Style transfer method based on lightweight attention mechanism

Abstract

The application provides a style transfer method based on a lightweight attention mechanism, and relates to the technical field of artificial intelligence, which comprises the following steps: constructing a neural network style transfer model based on a lightweight attention mechanism; in the model, a linear layer is used to project a content image and a style image into a plurality of image blocks, and the image blocks are reshaped into an image sequence; position encoding learned from the image blocks is added to the image sequence; an encoder network is used for encoding to obtain content features and style features; fusion features are generated based on the content features and the style features; a decoder network is used to convert the fusion features into a new image; a content image and a style image are input into the model, and an image that retains the original content and obtains artistic style is output. The application designs a lightweight attention network architecture, which can save time and avoid waste of device resources, and can efficiently generate a realistic and stylized image.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more particularly to a style transfer method based on a lightweight attention mechanism. Background Technology

[0002] Image style transfer is a highly anticipated research area in computer vision. With the rapid development of deep learning technology, image style transfer has achieved significant breakthroughs. Simply put, image style transfer combines the content of a real image with the style of another artwork (such as an oil painting) to create a new image that blends these artistic styles. In image style transfer, the input image typically includes a content image and a style image, and the output is a stylized result combining both.

[0003] Existing style transfer techniques can be divided into two types: optimization-based methods and neural network-based methods. Optimization-based methods transfer style by minimizing an objective function. The most classic algorithm maintains consistency between the content features of the image to be transferred and the target image, and also maintains consistency between the style features of the image to be transferred and the target image, ultimately resulting in an image that retains both the content of the image to be transferred and the style of the target image. This algorithm uses convolutional layers in a convolutional neural network to extract features and achieves style transfer by minimizing the differences between features.

[0004] Neural network-based methods refer to style transfer methods that use neural networks. The most classic algorithm uses a loss function called "style loss," which keeps the content features of the image to be transferred consistent with the content features of the target image, while keeping the style features of the image to be transferred inconsistent with the style features of the target image. The final result is an image that retains both the content of the image to be transferred and the style of the target image.

[0005] However, both of the above methods suffer from drawbacks such as high computational resource consumption and unrealistic generated images, resulting in some loss of content and style. Summary of the Invention

[0006] In view of this, the present invention provides a style transfer method based on a lightweight attention mechanism. The present invention designs a lightweight attention network architecture, which uses a shared weight layer to extract image features and uses an additional encoding layer in a high-dimensional space to further extract content features and style features. This saves time and avoids wasting device resources, and can efficiently generate realistic stylized images.

[0007] Therefore, the technical means adopted by the present invention are as follows:

[0008] This invention provides a style transfer method based on a lightweight attention mechanism, comprising:

[0009] A neural network style transfer model based on a lightweight attention mechanism is constructed. In this model, a linear layer projects the content image and style image into multiple image patches, reshaping these patches into a sequence of content images and a sequence of style images. Positional encodings learned from the image patches are added to the content image sequence and the style image sequence, respectively. An encoder network encodes the content image sequence and the style image sequence to obtain content features and style features. Mathematical operations or convolutional neural networks are performed on the content and style features to generate fused features. A decoder network converts the fused features into a new image.

[0010] Input a content image and a style image into a neural network style transfer model. The neural network style transfer model outputs an image that simultaneously preserves the original content and acquires the artistic style.

[0011] Furthermore, an encoder network is used to encode the content image sequence and the style image sequence to obtain content features and style features, including:

[0012] For content image sequences and style image sequences, features of the images are extracted using a shared weight coding layer;

[0013] High-dimensional image content features and style features are extracted using a content coding layer and a style coding layer, respectively.

[0014] Furthermore, a decoder network is used to transform the fused features into a new image, including:

[0015] Use the AdaIN module to obtain the sequence after feature fusion. Where σ(·) and μ(·) are used to calculate the characteristic variance and mean, respectively; F c F represents the content features obtained from feature extraction. s The style features obtained from feature extraction;

[0016] The sequence X, after feature fusion, is utilized by a decoding layer based on a cross-attention mechanism. cs The stylized features are obtained by combining stylized image sequences with added positional encoding;

[0017] The stylized image is generated through an upsampling decoder neural network.

[0018] Furthermore, the decoder neural network is a deconvolutional neural network or a variational autoencoder, used to transform the feature matrix into a new pixel-level image.

[0019] Furthermore, during the decoding process, regularization terms are added to ensure that the generated images maintain a certain level of visual appeal and content consistency.

[0020] Furthermore, content and style features are combined using mathematical operations or convolutional neural networks to generate fused features, including:

[0021] Calculate the Gram matrix in style images;

[0022] The fused features are generated by multiplying the Gram matrix with the feature matrix in the content image.

[0023] Furthermore, the backpropagation algorithm is used to optimize the model in order to minimize the difference between the generated image and the target image.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] Currently, traditional style transfer models require large datasets of artworks and advanced convolutional neural network architectures for training to produce realistic art images, a time-consuming and labor-intensive process. However, this invention designs a lightweight attention architecture to address the problem that CNN-based style transfer methods struggle to capture rich and long-range semantic concepts in artworks due to limitations in the spatial receptive domain. Compared to state-of-the-art style transfer attention models, this model reduces the parameter size by approximately 20% and improves iterative optimization speed by approximately 30%. Furthermore, the attention mechanism allows the content image to automatically and adaptively select detailed style elements to explore adaptive style transfer methods, thereby encouraging output images to be both attractive and plausible. This invention demonstrates excellent performance in content preservation and style rendering, while also reducing computational complexity, resulting in diverse and believable stylized images. This method has high practical value and application potential. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the basic framework for style transfer in an embodiment of the present invention;

[0028] Figure 2 This is a flowchart of the style transfer method based on a lightweight attention mechanism in an embodiment of the present invention;

[0029] Figure 3This is a schematic diagram comparing embodiments of the present invention with other stylized models;

[0030] Figure 4 This is a schematic diagram illustrating the generation of realistic stylized images according to an embodiment of the present invention;

[0031] Figure 5 This is an example of a stylized artwork based on an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] This invention is mainly used in style transfer in the artificial intelligence industry. It uses the style of artistic images to render real images so that they have the same artistic style and retain the original image content.

[0035] Style transfer is defined as follows: it involves using a convolutional neural network to extract content information from one image and style information from another, then combining this information to generate a new image. In this process, the content of the original image is preserved, while the style is transformed. This process is typically based on optimization algorithms, such as gradient descent, to minimize the differences in content and style between the generated image and the original image. Through this technique, it is possible to generate images with novel visual effects, such as transforming a photograph into an image resembling a Van Gogh painting.

[0036] Style transfer mainly consists of three parts: feature extraction, feature fusion, and feature decoding. In the feature extraction stage, an encoder network is used to encode the content image and the style image, obtaining their content features and style features. Next, in the feature fusion stage, mathematical operations or convolutional neural networks are performed on the content and style features to generate fused features. Finally, in the feature decoding stage, the fused features are mapped from the latent space to a higher-dimensional space to generate the final image with the new style. The specific process is as follows: Figure 1 As shown, it includes:

[0037] Part 1: Feature Extraction. Feature extraction is performed on both content and style images using a pre-trained convolutional neural network (typically a VGG network). Specifically, certain layers in the network can be selected as feature extractors to extract features at different scales and levels of abstraction. For content images, lower layers of the network, such as convolutional or pooling layers, can be selected to extract pixel-level details. For style images, higher layers of the network, such as convolutional or fully connected layers, can be selected to extract higher-level style features.

[0038] Part Two: Feature Fusion. After feature encoding, style features are fused into content features. This is achieved by calculating the correlations between different features in the style image. Specifically, the Gram matrix in the style image can be calculated and then multiplied by the feature matrix in the content image, thus fusing the style features into the content features. The fused feature matrix will contain both content and style information.

[0039] Part 3: Feature Decoding. After feature fusion, a decoder network is used to transform the fused feature matrix into a new image. The decoder network is typically a deconvolutional neural network or a variational autoencoder, which converts the feature matrix into a new pixel-level image. During this process, regularization terms can be added to ensure that the generated image maintains a certain level of visual appeal and content consistency.

[0040] Finally, by continuously adjusting and optimizing the parameters, the model can be gradually improved to generate more realistic images while retaining the characteristics of stylized images. The entire process can be optimized using the backpropagation algorithm to minimize the differences between the generated and target images.

[0041] The style transfer method in this invention is an attention-based approach that leverages long-term correlation and the ability to capture local semantics to extract content and style features from images. Specifically, the image sequence is treated as input obtained by projecting image patches through a block embedding module. Before encoding, positional codes learned from the image patches are added to the image sequence to achieve better global interaction. Unlike traditional style transfer methods that use multiple encoders for different feature extractions, the differences between content and style features should essentially exist in a high-dimensional space, which can be captured at the top level of the encoder. Therefore, this invention designs a first encoder layer with shared parameters, extracting content and style features separately at the high-dimensional layers of the encoder. Compared to current large-scale style transfer models, this invention reduces computational parameters by approximately 20% and improves model optimization speed by 30%. Lower computational consumption and memory requirements further enhance the application value of the attention mechanism.

[0042] like Figure 2 As shown in the embodiment of the present invention, a neural network model based on an attention mechanism for achieving realistic image stylization includes:

[0043] A. Image patch embedding

[0044] Attention mechanisms are a commonly used technique in deep learning, which can enhance the neural network's focus on relevant information, thereby improving the model's accuracy and robustness. Taking advantage of the long-term relevance of attention mechanisms, this invention proposes a novel image transfer algorithm. This algorithm defines the image transfer task as an image serialization task, mapping and reconstructing image patches to enable the attention mechanism to better capture information over long time spans. Specifically, a linear layer is used to transfer content images I... c ∈R H×W×C and style image I s ∈R H×W×C The projection is divided into N image patches, and then these image patches are reconstructed into an image sequence, i.e., the content image sequence X. c ∈R N×L and style image sequence X s ∈R N×L In this way, the original two-dimensional image is successfully transformed into a one-dimensional image sequence, effectively utilizing the long correlation advantage of the attention mechanism.

[0045] B. Location coding

[0046] Deep learning networks are a data-driven machine learning approach that uses multi-layered neural networks for feature learning and pattern recognition. Locality and spatial invariance are key to deep networks learning latent information. For sequence transfer tasks, positional encoding is crucial. Traditional attention networks use cosine functions to represent positional information, but this is limited by the fixed sequence dimension. Therefore, introducing positional encoding into image sequences helps the model learn more complete spatial information and enhances the interactions between each pair of sequences. This method learns the corresponding positional information P from content image patches and style image patches, respectively. c P s And it was fused into the image sequence, i.e., A c =X c +P c A s =X s +P s By incorporating positional encoding, each sequence possesses a positional code related to the source information. This results in better spatial awareness and helps the model capture more latent information. When processing image sequences, positional encoding can provide richer spatial information, enabling the model to have stronger visual perception and generalization abilities. Simultaneously, positional encoding can also be used to process other sequence data, such as sequence data in natural language processing, providing better positional information.

[0047] C. Feature Extraction

[0048] Building upon traditional attention networks, this method utilizes attention mechanisms to consider long-range dependencies and combines residual blocks to minimize information loss, demonstrating a powerful ability to learn sequence representations and effectively extract corresponding feature information from content and style sequences. However, constructing a corresponding encoder for each feature extraction consumes significant computational resources. In reality, the difference between content and style features lies in high-dimensional space. In this method, two image sequences X... c X s The image is input into the encoder E network for feature extraction, resulting in content features F. c and style characteristics F s First, using N enc The layers share a coding layer to extract features from the image, and then utilize an additional coding layer E. c E s Extract high-dimensional image content features and style features respectively, i.e. By setting shared weights, the model in this invention significantly reduces the number of parameters, further improves computational efficiency, and expands the application scenarios of the model.

[0049] D. Feature Decoding

[0050] Existing style transfer methods primarily focus on mitigating style loss during feature fusion. This method utilizes a multi-layer cross-attention mechanism combined with the AdaIN module (Adaptive Instance Normalization, which aligns the mean and variance of content image features to the mean and variance of style image features) to address both content and style loss issues. First, it obtains the sequence after feature fusion. Where σ(·) and μ(·) are used to calculate the characteristic variance and mean, respectively. Then, this method utilizes N dec The decoding layer R, based on a cross-attention mechanism, will stylize the feature sequence X. cs and style sequence A s By combining these elements and minimizing style loss, we can ultimately obtain the stylized feature F. cs ,Right now Finally, an upsampling decoder neural network D generates the stylized image I. cs =D(F cs The output image captures the style of the artistic image while retaining the content of the input image.

[0051] like Figure 3 As shown, it compares the model in the above embodiment with other existing stylization models, including: StyTr 2 The models used include AdaIN, SANet, Linear, Swap, MST, Avatar, MCC, and AAMS. Compared to these stylization models, the style transfer images obtained by the models in the above embodiments of this invention are more realistic (third column).

[0052] Compared to state-of-the-art style transfer attention models, this model reduces the parameter size by approximately 20% and improves the iteration optimization speed by approximately 30%.

[0053] like Figure 4 and Figure 5 As shown, it illustrates the generated realistic stylized image. The model, based on an attention mechanism as described above, takes a content image and a style image as input and outputs an image that simultaneously preserves the original content and captures the artistic style. The style transfer method of this invention is applicable to various styles and can generate realistic works of art.

[0054] This invention can be applied to multiple aspects:

[0055] 1. Style transfer technology in art creation is an image processing technique based on deep learning. It applies the style of one image to another, achieving a fusion of the two images. In the field of art creation, style transfer technology can create many stunning works of art. For example, it can merge Van Gogh's Starry Night style with Picasso's Cubist style, generating new artworks full of fantasy, mystery, and geometric forms. Furthermore, style transfer technology can also merge artworks of different styles with various media such as photographs and videos, creating more unique and interesting works of art.

[0056] 2. Image editing style transfer technology can also be used in image editing. In image editing, style transfer technology can transform an ordinary photo into a cartoon-style image, or a black and white photo into a color image. By using style transfer technology, image editors can easily apply different styles to the same image, thereby achieving a variety of interesting effects.

[0057] 3. Similar to image editing, style transfer technology can also be used in video processing. In video processing, style transfer can transform the style of a person in different scenes, making it appear as if the video was filmed in different environments. Furthermore, style transfer can also merge videos of different styles to generate entirely new video works. This has enormous potential for film, animation, and other video creation fields.

[0058] 4. In advertising design, style transfer technology can transfer the style of a landscape image into an advertisement, making it more attractive and artistic. Furthermore, style transfer technology can also blend different advertising design styles to create more unique and interesting advertisements. This is a very useful tool for advertising designers.

[0059] 5. Game Development: In game development, style transfer technology can be used to merge different game scenes to generate entirely new game scenes. For example, game developers can merge the style of a city scene with the style of an ancient scene to create a scene full of fantasy and history.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A style transfer method based on a lightweight attention mechanism, characterized in that, The application relates to a neural network style transfer model based on a lightweight attention mechanism. The neural network style transfer model projects a content image and a style image into multiple image blocks through a linear layer, and reshapes the image blocks into a content image sequence and a style image sequence; position encodings learned from the image blocks are added to the content image sequence and the style image sequence respectively; an encoder network is used to encode the content image sequence and the style image sequence to obtain content features and style features; The content features and the style features are combined through mathematical operations or a convolutional neural network to generate fusion features; a decoder network is used to convert the fusion features into a new image; A content image and a style image are input into the neural network style transfer model, and the neural network style transfer model outputs an image that retains the original content and acquires artistic style; The content image sequence and the style image sequence are encoded by an encoder network to obtain content features and style features, including: For the content image sequence and the style image sequence, a shared weight encoding layer is used to extract the features of the images; A content encoding layer and a style encoding layer are used to extract high-dimensional image content features and style features respectively; The decoder network used to convert the fusion features into a new image includes: The AdaIN module is used for acquiring the sequence after feature fusion wherein and are respectively used for calculating feature variance and mean value; is a content feature obtained through feature extraction, is a style feature obtained through feature extraction; Fusing sequences of features using a decoding layer based on a cross-attention mechanism and style image sequences with added positional encodings to get stylized features An up-sampling decoder neural network is used to generate a stylized image; The content features and the style features are combined through mathematical operations or a convolutional neural network to generate fusion features, including: A Gram matrix in the style image is calculated; The Gram matrix is multiplied with a feature matrix in the content image to generate fusion features.

2. The style transfer method based on lightweight attention mechanism according to claim 1, characterized in that, The decoder neural network is a deconvolutional neural network or a variational autoencoder, which is used to convert the feature matrix into a new pixel-level image.

3. The style transfer method based on lightweight attention mechanism according to claim 2, characterized in that, During the decoding process, a regularization term is added to ensure that the generated image maintains certain visual perception and content consistency.

4. The style transfer method based on lightweight attention mechanism according to claim 1, characterized in that, A backpropagation algorithm is used for model optimization to minimize the difference between the generated image and the target image.

Images