An image art style transfer algorithm, storage medium and device based on deep learning

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to an image artistic style transfer algorithm, storage medium, and device based on deep learning. The algorithm includes: step S1, constructing a TFEST network and performing training optimization; the TFEST network includes a dual-stream feature extraction module, a style feature supplementer, and a style-aware decoder, and the dual-stream feature extraction module includes a content feature encoder and a style feature encoder; step S2, extracting a content sequence and a style sequence from the content image and the style image respectively through independent encoding paths of the dual-stream feature extraction module; step S3, introducing a style feature supplementer to process the style image and extract style parameters; step S4, inputting the content sequence, style sequence, and style parameters into a style-aware decoder and outputting a corresponding style-transferred image. The present invention solves the technical problem in the prior art that style information excessively interferes with content information, resulting in semantic ambiguity and style inconsistency.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to an image art style transfer algorithm, storage medium and device based on deep learning. Background Art

[0002] Image style transfer, a key research direction in computer vision, aims to transform the visual expression of an image into a target style while preserving its semantic content. This technology uses deep learning algorithms to analyze and extract the content features of the source image and the artistic features of the style reference image, achieving an organic fusion of the two, thereby creating visual works that retain the recognizable features of the original image while presenting a new artistic style. In terms of implementation mechanism, mainstream methods rely on multi-level feature representations extracted by convolutional neural networks, finding the optimal output by minimizing the weighted sum of content loss and style loss. Professional artists and content creators can use style transfer technology to quickly implement stylization attempts and explore innovative expressions in traditional art forms such as painting and photography. In the process of digitizing cultural heritage, style transfer provides new technical means for the restoration and reproduction of historical relics and artworks, contributing to the protection and inheritance of traditional culture.

[0003] Existing methods for image style transfer using transformers exist. However, these methods suffer from the following issues: Most lack effective feature decoupling mechanisms, making it difficult to separate and control content and style features, resulting in uneven style application in complex scenes. When processing images with rich details and complex textures, existing algorithms are prone to artifacts and blocking effects, which affect the visual quality and artistic expression of the generated images. Traditional convolutional neural network (CNN)-based style transfer methods have limitations in feature expression and insufficiently capture global context, resulting in blurred semantic structure and inconsistent style in the generated images. Existing methods struggle to strike a balance between content preservation and style transfer: overemphasizing style can lead to loss of content semantics, while overpreserving content can lead to insufficient style expression. Lightweight models, while offering advantages such as low computational complexity and memory usage, also have the disadvantage of a small number of model parameters, making it even more difficult to balance content preservation and style transfer. Furthermore, the quality and consistency of style transfer can be compromised or inconsistent due to the lightweight nature of the model. Summary of the Invention

[0004] The purpose of the present invention is to provide an image art style transfer algorithm based on deep learning, which is used to solve the technical problems in the existing technology that lightweight style transfer models have difficulty in striking a balance between content preservation and style transfer, and the quality and consistency of style transfer are difficult to take into account with the requirements of model lightweighting.

[0005] The image art style transfer algorithm based on deep learning includes the following steps.

[0006] Step S1, construct the TFEST network and perform training optimization;

[0007] The TFEST network consists of a dual-stream feature extraction module, a style feature complementor, and a style-aware decoder. The dual-stream feature extraction module includes two independent encoding paths: a content feature encoder and a style feature encoder.

[0008] Step S2: extracting a content sequence and a style sequence from the content image and the style image respectively through independent encoding paths of the dual-stream feature extraction module;

[0009] Step S3: introducing a style feature supplementer to process the style image and extract style parameters;

[0010] Step S4: input the content sequence, style sequence, and style parameters into a style-aware decoder, and output a corresponding style-transferred image.

[0011] Preferably, in step S3, the style parameter supplementer extracts and quantizes style features from the style image through a multi-layer structure, and each layer of the multi-layer structure includes a scaled dot product attention mechanism and a residual block; after the style image is input into the style parameter supplementer, it is divided into equal-sized and non-overlapping patch units at the image blocking layer, and the features of each patch are spliced ​​by the original pixel RGB values, and are projected by a linear embedding layer and then input into the multi-layer structure of the style parameter supplementer, and the style parameters are output after processing by the multi-layer structure.

[0012] Preferably, in the multi-layer structure of the style parameter supplementer, the calculation formula of the scaled dot product attention mechanism is as follows:

[0013]

[0014] X=LN(MLP(X')+X')=LN(W2σ(W1X'+b1)+b2+X'),

[0015]

[0016] Among them, F s represents the input of the multi-layer structure, H represents the number of heads of the scaled dot product attention mechanism, Q, K and V represent the query vector, key vector and value vector respectively, and is the parameter matrix of the hth head of each scaled dot product attention mechanism head, d kis the dimension of the key vector, W1 and W2 are the weight matrices of the first and second layers in the MLP layer, b1 and b2 are the bias vectors of the first and second layers in the MLP layer, σ represents the activation function, LN represents the layer normalization operation, X', X' and X are the outputs of the first, second and third layers of the style parameter supplementer respectively, and the extracted style parameter X is subjected to dimensionality reduction to obtain the final style parameter

[0017] Preferably, in step S4, the style-aware decoder receives the content sequence Φ c , style sequence Φ s and style parameters For three types of input, in the feature fusion stage, the scaled dot product attention mechanism is first calculated for each input feature. The operation process is expressed as the following formula:

[0018]

[0019] Q=Φ c ,K=V=Φ s , where α and γ are learnable weight parameters, Φ″ cs Represents the content sequence Φ c With style parameters The intermediate features calculated by the scaled dot product attention mechanism, Φ' cs Represents the intermediate feature Φ″ cs With style sequence Φ s The result calculated by the scaled dot product attention mechanism, H is the number of scaled dot product attention mechanism heads, and is the parameter matrix of the hth head of each scaled dot product attention mechanism head, d k is the dimension of the key vector. The processed feature Φ' cs After further processing by a multi-layer perceptron, the calculation formula is as follows:

[0020]

[0021] Among them, δ is a learnable weight parameter, LN represents layer normalization operation, MLP represents multi-layer perceptron, Φ cs It represents the features output by the style-aware decoder after fusing the features of the content sequence, style sequence and style parameters.

[0022] Preferably, in step S2, the input content image and style image in RGB format are divided into equal-sized and non-overlapping patch units at the image segmentation layer. The features of each patch are composed of the original pixel RGB values ​​and are input to the content feature encoder and the style feature encoder for processing respectively. In the independent encoding path, the features obtained by the linear embedding layer projection are sequentially processed by several AFEM modules, which include AdaRF module, CAM module and AdaCW module. The calculation formula of the AFEM module is: Φ = z + F AdaCW (F CA (F AdaRF (z))), where Φ represents the output of the AFEM module, z represents the input feature, and F AdaRF Indicates the application of AdaRF module processing, F CA Indicates the application of CAM module processing, F AdaCW Indicates that the AdaCW module is applied for processing. The output of the previous AFEM module is downsampled and input to the next AFEM module, and finally the corresponding feature sequence is output. The feature sequence includes the content sequence Φ c and style sequence Φ s .

[0023] Preferably, the AdaRF module can dynamically adjust the receptive field size of the network while capturing both detail texture and global structure. The calculation formula of the AdaRF module is:

[0024] F AdaRF (z) = z + α Conv(z, Ψ(r(z))), where z represents the input feature, α is a learnable scaling parameter, Conv represents the convolution operation, Ψ is the kernel generating function, and r(z) is the adaptively calculated receptive field parameter. r(z) is calculated as follows:

[0025] r(z)=σ(MLP(AvgPool(z))),

[0026] Where: σ is the sigmoid activation function, MLP represents multi-layer perceptron, and AvgPool represents global average pooling.

[0027] Preferably, the calculation formula of the CAM module is:

[0028] F CA (z)=z⊙σ(FC2(ReLU(FC1(AvgPool(z))))),

[0029] Where: ⊙ represents element-wise multiplication, AvgPool represents global average pooling, FC1 and FC2 are learnable weight matrices, and σ is the sigmoid activation function.

[0030] Preferably, the calculation formula of the AdaCW module is:

[0031] F AdaCW (z)=z⊙σ(W2·ReLU(W1·AvgPool(z))),

[0032] Where: W1 is the weight parameter of the first layer transformation, AvgPool represents global average pooling, ReLU represents the ReLU activation function, and W2 is the weight parameter of the second layer transformation.

[0033] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned deep learning-based image art style transfer algorithm.

[0034] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the steps of the above-mentioned deep learning-based image art style transfer algorithm are implemented.

[0035] The present invention has the following advantages:

[0036] 1. The algorithm proposed in this invention innovatively designs a three-source feature processing architecture. In terms of the automatic balancing mechanism between semantic preservation and style transfer, the dual-stream feature extraction module of the present invention extracts content sequences and style sequences respectively; the style parameter supplementer quantifies style information and extracts style parameters to provide a more abstract style representation. When the style-aware decoder simultaneously receives two types of information, it can dynamically adjust the application weights of style parameters based on the semantic importance of content features, thereby forming an automatic balancing mechanism. This mechanism relies on the information complementarity and dynamic weight adjustment between the outputs of the dual-stream feature extraction module and the style parameter supplementer. Furthermore, this three-source feature fusion architecture can simultaneously process information at multiple scales, fusing the extracted content sequence, style sequence, and style parameters to ensure that both large-scale features (such as overall composition) and small-scale features (such as local texture) maintain appropriate style transfer. This solves the technical problems of excessive interference of style information on content information in the prior art and overcomes the technical problems of semantic ambiguity and style inconsistency.

[0037] 2. In terms of the construction of the style parameter supplementer, the present invention adopts a combination of a scaled dot product attention mechanism and a residual block, and introduces a style parameter supplementer of this structure into a lightweight network model. The increased computational complexity and the new consumption of computing resources are relatively small, and the overall network after construction still meets the characteristics of a lightweight network. At the same time, the multi-layer scaled dot product attention mechanism can extract and quantify more abstract and comprehensive style features, that is, summarize complex visual styles into key feature descriptors, thus achieving a balance between maintaining the lightweight of the network model and improving the quality and consistency of style transfer, and having a better style transfer effect. In terms of nonlinear optimization of computational complexity, the style parameter supplementer compresses the high-dimensional style sequence into a low-dimensional parameter vector, and the style-aware decoder uses compressed expression to fuse the three-source features, forming a chain for efficient utilization of computing resources.

[0038] 3. The style-aware decoder in the present invention implements a multi-level, progressive feature fusion strategy, accurately integrating the three-source feature information through the style-aware decoder and the multi-head scaling dot product attention mechanism. The style-aware decoder performs corresponding fusion processing at different levels, progressively at multiple levels of the network, from shallow texture fusion to deep semantic fusion, ensuring that style elements are fully expressed at different granularities while maintaining the integrity of the content structure. This fusion method also has the ability to dynamically adjust the application method of style features based on the semantic information of content features, achieving differentiated processing of different areas. For example, it can maintain more content information for important semantic areas such as the face of a person, and allow stronger style expression for areas such as the background.

[0039] 4. Regarding enhanced style generalization capabilities, the AFEM module's variable receptive field design, combined with the abstract expression of the style parameter supplementer, enables the system to handle unseen styles. AFEM provides a flexible feature expression foundation, the style parameter supplementer abstracts complex styles into parameter vectors, and the style-aware decoder flexibly applies these parameters. Together, these three create generalization capabilities that surpass their individual capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a basic flow chart of an image art style transfer algorithm based on deep learning in the present invention.

[0041] Figure 2 This is a structural diagram of the image style transfer network TFEST in the present invention.

[0042] Figure 3 This is a structural diagram of the dual-stream feature extraction module in the present invention.

[0043] Figure 4 The structural diagram of the AFEM module in this invention

[0044] Figure 5This is a structural diagram of the style parameter supplementer in the present invention.

[0045] Figure 6 This is a structural diagram of the style-aware decoder in the present invention.

[0046] Figure 7 This is the result diagram of the style arbitrariness of the present invention.

[0047] Figure 8 The figure shows the comparison between the results generated by the present invention and the existing baseline method.

[0048] Figure 9 A comparison chart of the content loss and style loss between the style transfer method of the present invention and the prior art.

[0049] Figure 10 A comparison chart of the stylization speed of the style transfer method of the present invention and the prior art at different pixels.

[0050] Figure 11 This figure compares the style transfer of the present invention and the prior art in terms of time and space complexity. DETAILED DESCRIPTION

[0051] The following is a further detailed description of the specific implementation methods of the present invention through the description of the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0052] Example 1.

[0053] like Figures 1-6 As shown, the present invention provides an image art style transfer algorithm based on deep learning, which includes the following steps.

[0054] Step S1: construct the TFEST network (three-source feature extraction image style transfer network) and perform training optimization.

[0055] The TFEST network includes a dual-stream feature extraction module, a style feature complementor, and a style-aware decoder. The dual-stream feature extraction module includes two independent encoding paths, namely, a content feature encoder and a style feature encoder. Figure 2 shown.

[0056] In step S2, a content sequence and a style sequence are extracted from the content image and the style image respectively through independent encoding paths of the dual-stream feature extraction module.

[0057] The structure of the dual-stream feature extraction module is as follows Figure 3As shown in the figure, after the image is input into the independent encoding path, the image patch embedding technology is used to split the RGB format input content image and style image into equal-sized and non-overlapping patch units at the image block layer. Each patch unit is regarded as a "token", and its features are composed of the original pixel RGB values ​​(3 channels) in a size of 4×4. Therefore, the feature dimension of each patch is 4×4×3=48. The features of the patch units generated by the content image and style image are then input into the content feature encoder and style feature encoder respectively for processing.

[0058] In the two feature encoders mentioned above, a linear embedding layer is applied to the original input features (the original features of the patch unit) to project them into an arbitrary dimension (denoted as C), and the corresponding features (content information and style information) are obtained for subsequent processing.

[0059] Both independent encoding paths use the AFEM module (Adaptive Feature Extraction Module), which fully utilizes the advantages of the AFEM module in modeling long-range dependencies and effectively extracts global context information. The projected features are processed by several AFEM modules in sequence. The output of the previous AFEM module is downsampled and input to the next AFEM module, and finally the corresponding feature sequence is output, such as content sequence or style sequence. The AFEM module is one of the core components of the TFEST network and consists of three key submodules working together. The structure is as follows: Figure 4 The three key submodules are AdaRF module (Adaptive Receptive Field Module, Adaptive Receptive Field Module, Figure 4 (a)), CAM module (channel attention module, such as Figure 4 (b)) and AdaCW module (Adaptive Channel Weighting Module, such as Figure 4(c)). The feature information input to the AFEM module is processed by the AdaRF module, the CAM module, and the AdaCW module. The AdaRF module can dynamically adjust the size of the network's receptive field, solving the limitation of the traditional convolutional network's fixed receptive field that cannot adapt to features of different scales, enabling the network to capture detailed textures and global structures simultaneously. The CAM module learns the dependencies and relative importance between channels through global pooling, FC1+ReLU, and FC2+Sigmoid components, and learns a universal inter-channel feature map through the FC layer, which is suitable for capturing global correlations between channels and enhancing the effectiveness of feature expression. The AdaCW module can further optimize the adaptive allocation strategy of channel weights, highlighting key features and suppressing secondary features, while keeping the number of tokens unchanged (H / 4, W / 4), achieving more refined adjustment of features, which is conducive to solving the fusion and matching problem of content features and style features in specific style transfer tasks.

[0060] During the feature extraction process, the calculation formula of the AdaRF module is:

[0061] F AdaRF (z) = z + α·Conv(z,Ψ(r(z))), where z represents the input feature, α is a learnable scaling parameter, Conv represents the convolution operation, Ψ is the kernel generating function, and r(z) is the adaptively calculated receptive field parameter. r(z) is calculated as follows:

[0062] r(z)=σ(MLP(AvgPool(z))),

[0063] Where: σ is the sigmoid activation function, MLP represents multi-layer perceptron, and AvgPool represents global average pooling.

[0064] The calculation formula of the CAM module is:

[0065] F CA (z)=z⊙σ(FC2(ReLU(FC1(AvgPool(z))))),

[0066] Where: ⊙ represents element-wise multiplication, AvgPool represents global average pooling, FC1 and FC2 are learnable weight matrices, and σ is the sigmoid activation function.

[0067] The calculation formula of the AdaCW module is:

[0068] F AdaCW (z)=z⊙σ(W2·ReLU(W1·AvgPool(z))),

[0069] Where: W1 is the weight parameter of the first layer transformation, which maps the pooled features to an intermediate feature space, and then performs a nonlinear transformation through the ReLU activation function, and then performs the second layer transformation. W2 is the weight parameter of the second layer transformation, and AvgPool represents global average pooling.

[0070] The calculation formula of the AFEM module is: Φ=z+F AdaCW (F CA (F AdaRF (z))), where Φ represents the output of the AFEM module, z represents the input feature, and F AdaRF Indicates the application of AdaRF module processing, F CA Indicates the application of CAM module processing, F AdaCW Indicates that the AdaCW module is used for processing.

[0071] In the overall style transfer network, the AFEM module plays the dual role of feature extraction and enhancement. As the network depth increases, through multi-stage processing implemented by connecting several AFEM modules in series (four in the embodiment), the network forms a multi-layered structure (from "Stage 1" to "Stage 4"), gradually reducing the spatial resolution (H / 8, W / 8 to H / 32, W / 32) and increasing the feature dimension (from C to 2C), achieving the transformation from low-level visual features to high-level semantic features. This hierarchical design enables the method to fuse style and content at different levels of abstraction, producing more natural and higher-quality style transfer results.

[0072] After being processed by several AFEM modules, the output content sequence Φ of the content feature encoder is c , and the output style sequence Φ of the style feature encoder s .

[0073] Step S3: introduce a style feature supplementer to process the style image and extract style parameters.

[0074] The style parameter supplementer extracts and quantifies style features from the style image through a multi-layer structure, each layer of which contains a scaled dot product attention mechanism and a residual block. The structure is as follows Figure 5 As shown in Figure 2. Similar to step S2, after the style image is input to the style parameter complementor, the image segmentation layer now segments the RGB input content image into equal-sized, non-overlapping patches. Each patch is considered a "token," whose features are a 4×4 concatenation of the original pixel RGB values ​​(3 channels). A linear embedding layer is then applied to the original features of the patch unit, projecting them into an arbitrary dimension (denoted as C). The style parameters are then processed sequentially through the aforementioned multi-layer structure to obtain the style parameters.

[0075] In the multi-layer structure of the style parameter supplementer, the calculation formula of the scaled dot product attention mechanism is as follows:

[0076] X=LN(MLP(X')+X')=LN(W2σ(W1X'+b1)+b2+X'),

[0077]

[0078] Among them, F s It represents the result of the style image processed by the image block layer and the linear embedding layer, that is, the input of the multi-layer structure. H represents the number of heads of the scaled dot product attention mechanism. This parameter determines the number of feature spaces that the model can focus on at the same time. The more heads there are, the stronger the model's ability to capture multi-dimensional features. Q, K, and V represent the query vector, key vector, and value vector, respectively. and is the parameter matrix of the hth head of each scaled dot product attention mechanism head, d k is the dimension of the key vector, W1 and W2 are the weight matrices of the first and second layers in the MLP (Multi-layer Perceptron) layer, b1 and b2 are the bias vectors of the first and second layers in the MLP layer, σ represents the activation function, LN represents the layer normalization operation, X', X' and X are the outputs of the first, second and third layers of the style parameter supplementer respectively. This operation stabilizes the training process and accelerates convergence by standardizing the output features of each layer. To improve the interpretability and efficiency of the model, the extracted style parameters X are subjected to dimensionality reduction to obtain the final style parameters Here U=100, which represents the dimension of the style parameter vector after dimensionality reduction.

[0079] The scaled dot product attention mechanism calculates the importance weights of each channel in the feature map, highlighting the feature channels that are most important for the current style transfer task while suppressing irrelevant channels, thereby achieving adaptive adjustment for different content-style combinations. In the spatial dimension, the scaled dot product attention mechanism can automatically identify key and secondary areas in the image and apply different intensities of style transfer to different areas. Through this multi-level and multi-dimensional application of the scaled dot product attention mechanism, the network can establish a precise feature mapping relationship between the content domain and the style domain, such as mapping specific stroke patterns in the style image to the corresponding texture areas of the content image, or appropriately applying the color scheme of the style image to different parts of the content image, thereby producing a more natural, harmonious, and artistically expressive stylized image.

[0080] Step S4: input the content sequence, style sequence, and style parameters into a style-aware decoder, and output a corresponding style-transferred image.

[0081] The feature reconstruction of the style-aware decoder uses an autoregressive prediction strategy, which first receives the content sequence Φ c , style sequence Φ s and style parameters Three types of input, through multiple levels of content sequence Φ c , style sequence Φ s and style parameters Fusion, the structure of the style-aware decoder is as follows Figure 6 This step generates decoupled features in sequence through a specific computational mechanism, ensuring that the semantic integrity of the original content is maintained while incorporating the target style features. This design enables the system to effectively capture and transform style features at different scales, and ultimately outputs the fused target style feature Φ. cs ∈R L×C .

[0082] Specifically, the features extracted at different levels are fused to form the input F of the style-aware decoder. fus , the corresponding formula is expressed as: Concat stands for cascade. In the feature fusion stage, the scaled dot product attention mechanism is first calculated for each input feature. The corresponding operation process is expressed as the following formula:

[0083]

[0084] Q=Φ c ,K=V=Φ s ,

[0085] Among them, α and γ are learnable weight parameters, Φ″ cs Represents the content sequence Φ c With style parameters The intermediate features calculated by the scaled dot product attention mechanism, Φ' cs Represents the intermediate feature Φ″ cs With style sequence Φ s The result is calculated by the scaled dot product attention mechanism, H is the number of scaled dot product attention mechanism heads in the style-aware decoder, and is the parameter matrix of the hth head of each scaled dot product attention mechanism head, d k is the dimension of the key vector. The processed feature Φ' cs Through further processing by the multi-layer perceptron, the final fusion feature is formed, and the calculation formula is as follows:

[0086]

[0087] Among them, δ is a learnable weight parameter, LN represents layer normalization operation, MLP represents multi-layer perceptron, Φ csIt represents the features output by the style-aware decoder after fusing the features of the content sequence, style sequence and style parameters.

[0088] Fusion target style features Φ cs After being processed by the linear projection layer and the VGG-19 decoder in sequence, the final output style migration image can be obtained to achieve the conversion of artistic style.

[0089] In the above steps, step S2 and step S3 can be performed in any order and can be performed simultaneously.

[0090] The following is an explanation of the specific technical effects of the above-mentioned deep learning-based image art style transfer algorithm through specific experiments.

[0091] like Figure 7 As shown in the embodiment, through sufficient network training and iterative optimization, the style parameter supplementer integrated into the network demonstrates excellent feature extraction capabilities. This extractor can accurately identify and extract key style features from any input style encoding and convert them into quantifiable parameter vectors. To test the generalization performance of the present invention for unknown styles, the present invention randomly selected six works from the WikiArt dataset and five photos from the MSCOCO dataset for style transfer experiments.

[0092] like Figure 8 As shown in Figure 2, these baseline methods fall into two categories: one is classic convolutional neural network style transfer methods, represented by AdaIN; the other is methods using the visual Transformer architecture, including StyTr2, WCT, STTR, and S2WAT. These methods cover both traditional convolutional and modern Transformer architectures, providing a comprehensive performance comparison baseline.

[0093] like Figure 9 As shown, in order to evaluate the stylization quality of the method proposed in the present invention, the present invention uses content loss and style loss as core evaluation indicators. These two indicators measure the degree of difference between the generated image and the original input image in terms of content and style, respectively. The lower the value, the better the migration effect. In the experimental evaluation framework, the present invention constructs a large-scale test dataset: 30 content images are randomly selected from the MSCOCO test set, and 30 artworks representing different genres and with different stylistic characteristics are carefully selected from the WikiArt test set. This combination produces 900 pairs of content-style image pairs, providing sufficient test samples for the present invention. Based on the method in the literature, the present invention calculates the average content loss and average style loss of each comparison method on all image pairs. The method proposed in the present invention achieves the smallest average loss value in both evaluation indicators. This experimental result strongly verifies the superiority of the method of the present invention over existing representative methods.

[0094] like Figure 10 As shown in the figure, the average time for stylization processing of various arbitrary style transfer methods on 256x256 pixel and 512x512 pixel images, all method models are run in the same experimental environment. Although the method of the present invention is slower than the AdaIN method, it is faster than the StyTr2, WCT, STTR and S2WAT methods. Therefore, the stylization rate of the network of the present invention is generally within an acceptable range. When the image size is increased to 512x512 pixels, the stylization speed is basically the same as the stylization speed of 256x256 pixels, and it also has advantages when compared with other benchmark methods. At the same time, the results also show the stability of the network of the present invention in style transfer.

[0095] like Figure 11 As shown in the figure, through comparative analysis, it is found that although the iterative model based on CNN has a smaller number of parameters, its number of floating-point operations is larger. When the visual Transformer is used for style transfer, under the condition of ensuring the same and sufficient amount of data, the pure visual Transformer model shows better computing performance, but its number of parameters is increased compared to the CNN method. The image decoder and method based on the fusion of three-source features of the present invention have achieved significant advantages in both indicators: the number of floating-point operations is reduced, and the number of parameters is significantly improved compared to the ordinary visual Transformer model. This result fully demonstrates the superiority of the method of the present invention in terms of computational efficiency.

[0096] Example 2.

[0097] Corresponding to the first embodiment of the present invention, the second embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the following steps are implemented according to the method of the first embodiment.

[0098] Step S1: construct the TFEST network and perform training optimization.

[0099] In step S2, a content sequence and a style sequence are extracted from the content image and the style image respectively through independent encoding paths of the dual-stream feature extraction module.

[0100] Step S3: introduce a style feature supplementer to process the style image and extract style parameters.

[0101] Step S4: input the content sequence, style sequence, and style parameters into a style-aware decoder, and output a corresponding style-transferred image.

[0102] In the above steps, step S2 and step S3 can be performed in any order and can be performed simultaneously.

[0103] The above-mentioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), optical disk and other media that can store program codes.

[0104] The specific limitations on the steps implemented after the program in the computer-readable storage medium is executed can be found in Example 1, and will not be described in detail here.

[0105] Example 3.

[0106] Corresponding to the first embodiment of the present invention, the third embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the following steps are implemented according to the method of the first embodiment.

[0107] Step S1: construct the TFEST network and perform training optimization.

[0108] In step S2, a content sequence and a style sequence are extracted from the content image and the style image respectively through independent encoding paths of the dual-stream feature extraction module.

[0109] Step S3: introduce a style feature supplementer to process the style image and extract style parameters.

[0110] Step S4: input the content sequence, style sequence, and style parameters into a style-aware decoder, and output a corresponding style-transferred image.

[0111] In the above steps, step S2 and step S3 can be performed in any order and can be performed simultaneously.

[0112] The specific limitations on the above steps of implementing the computer device can be found in Example 1 and will not be described in detail here.

[0113] It should be noted that each box in the block diagrams and / or flow charts in the accompanying drawings of the present invention, as well as the combination of boxes in the block diagrams and / or flow charts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and machine instructions.

[0114] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the scope of protection of the present invention.

Claims

1. A deep learning-based image style transfer algorithm, characterized by: The following steps are involved: Step S1, construct the TFEST network and perform training optimization; The TFEST network consists of a dual-stream feature extraction module, a style feature complementor, and a style-aware decoder. The dual-stream feature extraction module includes two independent encoding paths: a content feature encoder and a style feature encoder. Step S2: extracting a content sequence and a style sequence from the content image and the style image respectively through independent encoding paths of the dual-stream feature extraction module; Step S3: introducing a style feature supplementer to process the style image and extract style parameters; Step S4: input the content sequence, style sequence, and style parameters into a style-aware decoder, and output a corresponding style-transferred image; In step S2, the RGB input content image and style image are segmented into equal-sized, non-overlapping patches at the image segmentation layer. The features of each patch are concatenated from the original pixel RGB values ​​and fed into the content feature encoder and style feature encoder for processing, respectively. In the independent encoding paths, the features projected from the linear embedding layer are sequentially processed by several AFEM modules, including the AdaRF module, the CAM module, and the AdaCW module. In step S3, the style parameter supplementer extracts and quantizes style features from the style image through a multi-layer structure, each layer of which includes a scaled dot product attention mechanism and a residual block. After the style image is input to the style parameter supplementer, it is segmented into equal-sized and non-overlapping patch units at the image blocking layer. The features of each patch are spliced ​​together by the original pixel RGB values, projected by a linear embedding layer, and input into the multi-layer structure of the style parameter supplementer. After processing by the multi-layer structure, the style parameters are output. In step S4, the style-aware decoder receives the content sequence Φ c , style sequence Φ s and style parameters For three types of input, in the feature fusion stage, the scaled dot product attention mechanism calculation operation process of each input feature is first expressed as the following formula: Q=Φ c ,K=V=Φ s , Among them, α and γ are learnable weight parameters, Φ" cs Represents the content sequence Φ c With style parameters The intermediate features calculated by the scaled dot product attention mechanism, Φ' cs Represents the intermediate feature Φ” cs With style sequence Φ s The result calculated by the scaled dot product attention mechanism, H is the number of scaled dot product attention mechanism heads, and is the parameter matrix of the hth head of each scaled dot product attention mechanism head, d k is the dimension of the key vector; the processed feature Φ' cs After further processing by a multi-layer perceptron, the calculation formula is as follows: Among them, δ is a learnable weight parameter, LN represents layer normalization operation, MLP represents multi-layer perceptron, Φ cs It represents the features output by the style-aware decoder after fusing the features of the content sequence, style sequence and style parameters.

2. The deep learning-based image style transfer algorithm according to claim 1, characterized in that: In the multi-layer structure of the style parameter supplementer, the calculation formula of the scaled dot product attention mechanism is as follows: X=LN(MLP(X')+X')=LN(W2σ(W1X'+b1)+b2+X'), K=V=F s , Among them, F s represents the input of the multi-layer structure, H represents the number of heads of the scaled dot product attention mechanism, Q, K and V represent the query vector, key vector and value vector respectively, and is the parameter matrix of the hth head of each scaled dot product attention mechanism head, d k is the dimension of the key vector, W1 and W2 are the weight matrices of the first and second layers in the MLP layer, b1 and b2 are the bias vectors of the first and second layers in the MLP layer, σ represents the activation function, LN represents the layer normalization operation, X', X' and X are the outputs of the first, second and third layers of the style parameter supplementer respectively, and the extracted style parameter X is subjected to dimensionality reduction to obtain the final style parameter 3. The deep learning-based image style transfer algorithm according to claim 1, characterized in that: In step S2, the RGB input content image and style image are divided into equal-sized and non-overlapping patch units at the image segmentation layer. The features of each patch are composed of the original pixel RGB values ​​and input into the content feature encoder and style feature encoder for processing respectively. In the independent encoding path, the features obtained by the linear embedding layer projection are processed by several AFEM modules in sequence. The AFEM modules include the AdaRF module, the CAM module and the AdaCW module. The calculation formula of the AFEM module is: Φ = z + F AdaCW (F CA (F AdaRF (z))), where Φ represents the output of the AFEM module, z represents the input feature, and F AdaRF Indicates the application of AdaRF module processing, F CA Indicates the application of CAM module processing, F AdaCW Indicates that the AdaCW module is applied for processing. The output of the previous AFEM module is downsampled and input to the next AFEM module, and finally the corresponding feature sequence is output. The feature sequence includes the content sequence Φ c and style sequence Φ s .

4. The deep learning-based image style transfer algorithm according to claim 3, characterized in that: The AdaRF module can dynamically adjust the receptive field size of the network while capturing both detailed textures and global structures. The calculation formula of the AdaRF module is: F AdaRF (z)=z+α·Conv(z,Ψ(r(z))), Where: z represents the input feature, α is a learnable scaling parameter, Conv represents the convolution operation, Ψ is the kernel generating function, and r(z) is the adaptively calculated receptive field parameter; r(z) is calculated as follows: r(z)=σ(MLP(AvgPool(z))), Where: σ is the sigmoid activation function, MLP represents multi-layer perceptron, and AvgPool represents global average pooling.

5. The deep learning-based image style transfer algorithm according to claim 3, characterized in that: The calculation formula of the CAM module is: F CA (z)=z⊙σ(FC2(ReLU(FC1(AvgPool(z)))))), Where: ⊙ represents element-wise multiplication, AvgPool represents global average pooling, FC1 and FC2 are learnable weight matrices, and σ is the sigmoid activation function.

6. The deep learning-based image style transfer algorithm according to claim 3, characterized in that: The calculation formula of the AdaCW module is: F AdaCW (z)=z⊙σ(W2 ReLU(W1 AvgPool(z))), Where: W1 is the weight parameter of the first layer transformation, AvgPool represents global average pooling, ReLU represents the ReLU activation function, and W2 is the weight parameter of the second layer transformation.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the image art style transfer algorithm based on deep learning are implemented.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the image art style transfer algorithm based on deep learning are implemented.

Images