Lightweight image style migration method based on color and texture dual channels

Abstract

The invention discloses a lightweight image style migration method based on color and texture dual channels. The method is characterized in that firstly, a style image and a content image data set are prepared and normalized, and an image style migration model is constructed and trained; adjusting the to-be-migrated image to be 512 * 512 pixels, and carrying out normalization on the to-be-migrated image; then, the image is input into a color and texture style encoder, and color and texture features are extracted respectively; the features are then sent to corresponding decoders, and color and texture migration features are generated; and finally, fusing the two features, and generating a high-quality image with a target style through a fusion decoder. According to the invention, a color and texture dual-channel design is introduced, and an encoder-decoder structure is adopted, so that efficient and high-quality image style migration is realized. Meanwhile, the method provided by the invention has the characteristic of light weight, can generate a stylized image with an excellent visual effect while keeping relatively low calculation overhead, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a lightweight image style migration method based on color and texture dual channels. Background Art

[0002] Style transfer is a computer vision technique that aims to synthesize the content of one image with the style of another. By combining the content information of an image with the artistic style, style transfer technology can create unique and artistic synthetic images. The application of style transfer is wide-ranging and covers many fields, including image processing, filmmaking, game development, and virtual reality. Traditional style transfer methods are usually based on rules and hand-designed feature representations, such as texture descriptors and filter combinations. However, the effectiveness of these methods is limited by the hand-designed features and cannot capture the high-level semantic information of the image. With the rise of deep learning, especially the successful application of convolutional neural networks (CNN), style transfer methods based on deep learning have achieved significant breakthroughs.

[0003] Deep learning algorithms mainly achieve style transfer by training a neural network model. These models usually achieve stylization by two losses: content loss and style loss. Content loss measures the difference between the generated image and the target content image, while style loss measures the difference between the generated image and the target style image. By minimizing the content loss and style loss, the required stylized image can be generated, which retains the content characteristics of the target image and has the style characteristics of the target image.

[0004] Neural style transfer is a popular deep learning style transfer method that uses a pre-trained convolutional neural network as a feature extractor and synthesizes images with a target style by optimizing an objective function. The core idea of ​​this method is to achieve the fusion of content and style by matching the features of the content image with the features of the style image. Summary of the invention

[0005] The technical problem to be solved by the present invention is that traditional style transfer methods usually require large computing resources and time, and unnecessary texture information is often introduced in the style transfer process, especially in the secondary areas of the image. Existing methods are often unable to accurately and vividly reproduce the color characteristics of the style image.

[0006] In order to solve the above problems, the present invention proposes a lightweight image style transfer method based on color and texture dual channels, which is characterized by comprising the following steps:

[0007] Step 1: Prepare a style image and a content image dataset;

[0008] Step 2: The style image and content image datasets are preprocessed by scaling, center cropping, and normalization. The resulting image size is 512×512 to fit the input format of the model.

[0009] Step 3: Construct a color and texture dual-channel lightweight image style transfer method, including a color style encoder module, a texture style encoder module, a fusion decoder module, a color attention module, and a texture attention module;

[0010] Step 4: Construct a color style encoder module, including a downsampling convolution with doubled channels, 2 encoding blocks, and a downsampling convolution with an output channel of 3; construct a texture style encoder, which consists of 6 modules, including two downsampling convolutions with doubled channels, 2 encoding blocks, and 2 upsampling convolutions, and the final output channel is 3;

[0011] Step 5: Construct a texture style encoder module, including multiple depth-separable convolutions, instance regularization, and ReLU activation function stacking. The convolution layer applies reflection padding, the padding size is half the size of the convolution kernel, and depth-separable convolution is applied;

[0012] Step 6: The output of the color style encoder module is called the color style encoding feature matrix Constructing the color attention module Two statistical feature maps are generated using average pooling and maximum pooling respectively; then these two feature maps are processed by a multi-layer perceptron network with the same structure, and the processing results are added and a sigmoid activation function is used to generate a color channel attention map. The specific calculation formula is as follows:

[0013]

[0014] Step 7: Color channel attention map Multiply a matrix element-wise Get the color channel weighting matrix The calculation formula is as follows:

[0015]

[0016] Step 8: Right The matrix uses average pooling and maximum pooling operations respectively, and the two feature maps are integrated according to the average and maximum values, and then processed through a standard convolution layer, and finally a sigmoid activation function is used to generate a color space attention map. The specific calculation formula is as follows:

[0017]

[0018] Step 9: Color space attention map Element-wise multiplication Matrix to get the color space weighting matrix The specific calculation formula is as follows:

[0019]

[0020] Step 10: The output of the texture style encoder module is called the texture style encoding feature matrix Constructing texture attention module Two statistical feature maps are generated using average pooling and maximum pooling respectively; then these two feature maps are processed by a multi-layer perceptron network with the same structure, and the processing results are added and a texture channel attention map is generated through a sigmoid activation function. The specific calculation formula is as follows:

[0021]

[0022] Step 11: Texture channel attention map Multiply a matrix element-wise Get the texture channel weighting matrix The specific calculation formula is as follows:

[0023]

[0024] Step 12: Right The matrix uses average pooling and maximum pooling operations respectively, and the two feature maps are integrated according to the average and maximum values, and then processed through a standard convolution layer, and finally a sigmoid activation function is used to generate a texture space attention map. The specific calculation formula is as follows:

[0025]

[0026] Step 13: Texture Space Attention Map Element-wise multiplication Matrix gets the texture space weighting matrix The specific calculation formula is as follows:

[0027]

[0028] Step 14: Construct a fusion decoder module, including 1 color style decoding block, 1 texture style decoding block, and 1 downsampling module with 3 output channels; the fusion decoder can simultaneously realize the color migration of color style features and the texture migration of texture style features;

[0029] Step 15: Apply the Sobel operator to calculate the edge information of the content image image and obtain the edge graph Edge. The specific calculation formula is as follows:

[0030] Edge = Sobel(image);

[0031] Step 16: Set the edge threshold δ and extract the EdgeMask mask through the threshold function. The specific calculation formula is as follows:

[0032] EdgeMask=threshold(Edge,δ);

[0033] Step 17: Use cs to represent the image obtained after the content image is stylized by the model, and represent the edge mask value at position (x, y+1) as EdgeMask x,y+1 , multiply it by the intensity difference between the horizontally adjacent pixels of cs to obtain the variational mask L in the horizontal direction h , the specific calculation formula is as follows:

[0034]

[0035] Step 18: Use cs to represent the image obtained after the content image is stylized by the model, and represent the edge mask value at position (x, y+1) as EdgeMask x,y+1 , multiply it by the intensity difference between the vertically adjacent pixels of cs to obtain the variational mask L in the vertical direction v , the specific calculation formula is as follows:

[0036]

[0037] Step 19: Place L h and L v Add together to get the total variational loss function L mtv , the specific calculation formula is as follows:

[0038] L mtv =L h +L v ;

[0039] Step 20: Training the model for color and texture dual channel migration includes the following steps:

[0040] Step 20.1: Input the preprocessed content image into the color style encoder module and the texture style encoder module respectively to obtain corresponding color style encoding features and texture style encoding features;

[0041] Step 20.2: Process the preprocessed style image through the vgg network trained on the ImageNet dataset to obtain a style feature map containing C channels. Then, according to the height H and W of the style feature map, extract the style matrix Gram of size C×C = (G ij ),in;

[0042]

[0043] F ik is the value of the kth position in the style feature map of the i-th channel, F jk is the value of the kth position in the style feature map of the jth channel;

[0044] Step 20.3: Input the color style encoding feature into the color style decoder module to obtain the color migration feature, and input the texture style encoding feature into the texture style decoder module to obtain the texture migration feature;

[0045] Step 20.4: Input the content image into the color and texture dual-channel lightweight image style transfer method, and compare the content image features output by the model with the result F processed by vgg and the result F directly processed by vgg * Perform mean square error evaluation to obtain content loss L content (F,F * );

[0046] Step 20.5: Input the style image into the color and texture dual-channel lightweight image style transfer method, and process the style image features output by the model through vgg and calculate the result of the Gram matrix G, and the style image directly processed by vgg and calculate the result of the Gram matrix G * Perform mean square error estimation to obtain style loss L style (G,G * );

[0047] Step 20.6: Add the weighted sum of style loss, content loss and total variational loss to get the total model loss L. The specific calculation formula is as follows:

[0048] L=αL content (F,F * )+βL style (G,G * )+λL mtv ;

[0049] Step 20.7: According to the total loss L, select the Adam optimizer, set α, β, and λ to 1 for training, stop training when the preset number of iterations reaches 80,000, and select the model with the best verification result as the final model;

[0050] Step 21: Add the color style coding features and the texture style coding features and input them into the fusion decoder to obtain a stylized image with dual channel migration of color and texture.

[0051] Beneficial effects of the present invention:

[0052] The image style transfer method proposed in the present invention can provide users with interesting image filters and special effects, allowing users to easily convert their photographs or recorded videos into images or videos with different artistic styles.

[0053] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is an overall flow chart of the image style transfer method in the method of the present invention;

[0055] Figure 2 It is an overall diagram of the model of the image style transfer method in the method of the present invention;

[0056] Figure 3 is a structural diagram of a codec in the image style transfer method of the present invention;

[0057] Figure 4 is a structural diagram of the color attention module in the method of the present invention;

[0058] Figure 5 is a structural diagram of the texture attention module in the method of the present invention;

[0059] Figure 6 It is a stylized image obtained by the image style transfer method in the method of the present invention. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention, so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0061] The overall flow chart of the image style transfer method proposed in the present invention is as follows: Figure 1 The preferred embodiment of the present invention discloses a lightweight image style transfer method based on color and texture dual channels, which specifically includes the following steps:

[0062] Step 1: Prepare a style image and a content image dataset. Choose a famous work of art as the style image, such as Van Gogh's "Starry Night". This image will be the source of the artistic style we want to transfer. Use Microsoft's COCO dataset as the source of the content image; the COCO dataset contains more than 330,000 annotated images covering 80 object categories.

[0063] Step 2: reset the minimum side of the style image and the content image to 512, clip an image size of 512×512 resolution from the middle of the image, and normalize the image using the mean and standard deviation of imageNet for different channels to have a predefined mean and standard deviation;

[0064] Step 3: Construct a color and texture dual-channel lightweight image style transfer method, including a color style encoder module, a texture style encoder module, a fusion decoder module, a color attention module, and a texture attention module;

[0065] See also Figure 2 , the overall model diagram of the image style transfer method, in this embodiment, the color style features and texture style features of the image are first extracted by the color style encoder and the texture style encoder respectively, and then the color style features are input into the color style decoder to obtain the color transfer result. The texture style features are input into the texture style decoder to obtain the texture transfer result. The color style features and the texture style features are added and input into the fusion decoder to obtain a high-quality color and texture dual-channel transferred image. The image style transfer method proposed in the present invention can provide users with interesting image filters and special effects, so that users can easily convert their photos or recorded videos into images or videos with different artistic styles.

[0066] Step 4: Input the normalized image into the color style encoder to obtain the encoding features of the color style;

[0067] See also Figure 3 In a specific embodiment, step 4 includes:

[0068] Step 4.1: Input the image into the first convolution module to obtain the primary convolution feature after convolution, perform instance regularization and activation operations on the primary convolution feature, and obtain the primary content feature 1;

[0069] Step 4.2: Input the primary content feature 1 into the second convolution module, use depthwise separable convolution to reduce parameters, perform instance regularization and activation operations on the output features, and obtain the primary content feature 2;

[0070] Step 4.3: Input the primary content feature 2 into the third convolution module, and perform instance regularization and activation operations to obtain the intermediate convolution feature after convolution;

[0071] Step 4.4: Upsample the obtained intermediate convolutional features and pass them through the color attention module to obtain the feature output of the color style encoder;

[0072] In this embodiment, the content image is uniformly compressed into a (3, 512, 512) format; at the same time, the number of first convolutional layers is (3, 16, 3, 1), where (3, 16, 3, 1) indicates that the number of input channels of the first convolutional module is 3, the number of output channels is 16, the kernel size is 3×3, and the number of groups is 1; the number of second convolutional modules is (16, 32, 3, 16)(32, 32, 1, 1); the number of third convolutional modules is (32, 16, 3, 16)(16, 16, 1, 1); and the number of upsampling modules is (16, 3, 3, 1).

[0073] Among them, in this embodiment, the convolution modules all use the ReLU function to activate the intermediate features, and all use instance regularization to regularize the intermediate features, which helps to improve the stability and performance of the model.

[0074] The color style encoder of the present application can extract more color information from the input image and requires less calculation.

[0075] Step 5: Input the normalized image into the texture style encoder to obtain the encoding features of the texture style;

[0076] See also Figure 3 In a specific embodiment, step 5 includes:

[0077] Step 5.1: Input the image into the first convolution module to obtain the primary convolution feature after convolution, perform instance regularization and activation operations on the primary convolution feature, and obtain the primary content feature 1;

[0078] Step 5.2: Input the primary content feature 1 into the second convolution module, use depthwise separable convolution to reduce parameters, perform instance regularization and activation operations on the output features, and obtain a new primary content feature 2;

[0079] Step 5.3: Input the primary content feature 2 into the third convolution module, and perform instance regularization and activation operations to obtain the intermediate convolution content feature after convolution;

[0080] Step 5.4: Input the intermediate convolution content features into the fourth convolution module, and perform instance regularization and activation operations to obtain high-level convolution features after convolution;

[0081] Step 5.5: up-sample the high-level convolutional features, input the sampled features into the fifth convolutional module, and perform instance regularization and activation operations to obtain high-level up-sampled features;

[0082] Step 5.6: upsampling the advanced upsampled features and inputting them into the sixth convolution module to obtain the feature output of the convolutional texture style encoder;

[0083] In this embodiment, the content image is uniformly compressed into a (3, 512, 512) format; at the same time, the number of the first convolutional layers is (3, 16, 3, 1) (16, 32, 3, 16) (32, 32, 1, 1), where (3, 16, 3, 1) means that the number of input channels of the first convolutional module is 3, the number of output channels is 16, the kernel size is 3×3, and the number of groups is 1; the number of the second convolutional modules is (32, 64, 3, 32) (6 4, 64, 1, 1); the number of the third convolution modules is (64, 64, 3, 64)(64, 64, 1, 1); the number of the fourth convolution modules is (64, 64, 3, 64)(64, 64, 1, 1); the number of the fifth convolution modules is (64, 32, 3, 32)(32, 32, 1, 1); the number of the sixth convolution modules is (32, 16, 1, 16)(16, 16, 1, 1)(16, 3, 1, 1).

[0084] Among them, in this embodiment, the convolution modules all use the ReLU function to activate the intermediate features, and all use instance regularization to regularize the intermediate features, which helps to improve the stability and performance of the model.

[0085] The texture style encoder of the present application can extract more texture information from the input image and requires less calculation.

[0086] Step 6: The output of the color style encoder module is called the color style encoding feature matrix Constructing the color attention module Two statistical feature maps are generated using average pooling and maximum pooling respectively. These two feature maps are then processed by a multi-layer perceptron network with the same structure. The processed results are then added together and a sigmoid activation function is used to generate a color channel attention map. The specific calculation formula is as follows:

[0087]

[0088] Step 7: Color channel attention map Multiply a matrix element-wise Get the color channel weighting matrix The calculation formula is as follows:

[0089]

[0090] Step 8: Right The matrix uses average pooling and maximum pooling operations respectively, and the two feature maps are integrated according to the average and maximum values, and then processed through a standard convolution layer, and finally a sigmoid activation function is used to generate a color space attention map. The specific calculation formula is as follows:

[0091]

[0092] Step 9: Color space attention map Element-wise multiplication Matrix to get the color space weighting matrix The specific calculation formula is as follows:

[0093]

[0094] See also Figure 4 , the color attention module diagram, wherein the spatial attention mechanism adjusts the feature importance by generating a weight map of the same size as the input feature map, enhances the features of important regions, weakens the features of secondary regions, and makes the model pay more attention to color-rich regions. The channel attention mechanism adjusts the importance of color channels by generating weight vectors, emphasizing channels that can reflect color style. The combination of these two mechanisms enables the color style encoder module to adaptively adjust color features from both spatial and channel dimensions, improve the accuracy and efficiency of feature extraction, and better reflect the color style of the style image. This helps to improve the quality of style transfer, generate realistic color effects, and enhance the model's adaptability to diverse style images.

[0095] Step 10: The output of the texture style encoder module is called the texture style encoding feature matrix Constructing texture attention module Two statistical feature maps are generated using average pooling and maximum pooling respectively; then these two feature maps are processed by a multi-layer perceptron network with the same structure, and the processing results are added and a texture channel attention map is generated through a sigmoid activation function. The specific calculation formula is as follows:

[0096]

[0097] Step 11: Texture channel attention map Multiply a matrix element-wise Get the texture channel weighting matrix The specific calculation formula is as follows:

[0098]

[0099] Step 12: Right The matrix uses average pooling and maximum pooling operations respectively, and the two feature maps are integrated according to the average and maximum values, and then processed through a standard convolution layer, and finally a sigmoid activation function is used to generate a texture space attention map. The specific calculation formula is as follows:

[0100]

[0101] Step 13: Texture Space Attention Map Element-wise multiplication Matrix gets the texture space weighting matrix The specific calculation formula is as follows:

[0102]

[0103] See also Figure 5 , the texture attention module diagram, in the color and texture dual-channel lightweight image style transfer method of the present invention, in addition to introducing the space and channel attention mechanism in the color style encoder, we also introduce a similar attention mechanism in the texture style encoder module.

[0104] The spatial attention mechanism adjusts the spatial importance of texture features by generating a weight map, enhancing the features of key areas such as surface texture and edge contours of objects, and weakening the features of secondary areas such as the background. The role of the channel attention mechanism in the texture style encoder is to adaptively adjust the importance of different texture channels. In actual style images, the style information carried by different texture channels may be unbalanced. In order to adapt to this phenomenon of unbalanced texture channels, we introduced the channel attention mechanism. The channel attention mechanism adjusts the importance of different texture channels by generating a weight vector, highlighting the key channels that reflect the texture style. The combination of these two attention mechanisms enables the texture style encoder to adaptively adjust the texture features, better reflect the texture style of the style image, improve the quality of style transfer, and generate clearer and more realistic texture effects. In addition, the introduction of these mechanisms also keeps the model lightweight, without significantly increasing the model complexity and computational overhead, which is in line with the design concept of lightweight image style transfer.

[0105] Step 14: Construct a fusion decoder module, including 1 color style decoding block, 1 texture style decoding block, and 1 downsampling module with 3 output channels; the fusion decoder can simultaneously realize the color migration of color style features and the texture migration of texture style features;

[0106] See also Figure 3 In a specific embodiment, step 14 includes:

[0107] Step 14.1: Encode the above color style features Input the color style decoder, which consists of two depth-wise separable convolutions, instance regularization, and ReLU activation function stacks, and obtains the output of the color decoder;

[0108] Step 14.2: Encode the texture style features above Input an upsampling convolution, which is also composed of multiple convolutions, instance regularization, and ReLU activation function stacking, and get the output of the texture style decoder; encode the above color style features Texture style encoding features Add the input to the same upsampling convolution to get the output of the fused encoder.

[0109] In this embodiment, we implement the color style decoder and texture style decoder in a fusion decoder, and realize the reuse of the method. In addition, in this embodiment, we implement a lightweight image style transfer method based on color and texture dual channels. The method we proposed can achieve a high-quality color and texture style transfer, and our method can achieve a higher quality fusion style transfer.

[0110] Step 15: Apply the Sobel operator to calculate the edge information of the content image image and obtain the edge graph Edge. The specific calculation formula is as follows:

[0111] Edge = Sobel(image);

[0112] Step 16: Set the edge threshold δ and extract the EdgeMask mask through the threshold function. The specific calculation formula is as follows:

[0113] EdgeMask=threshold(Edge,δ);

[0114] Step 17: Use cs to represent the image obtained after the content image is stylized by the model, and represent the edge mask value at position (x, y+1) as EdgeMask x,y+1 , multiply it by the intensity difference between the horizontally adjacent pixels of cs to obtain the variational mask L in the horizontal direction h , the specific calculation formula is as follows:

[0115]

[0116] Step 18: Use cs to represent the image obtained after the content image is stylized by the model, and represent the edge mask value at position (x, y+1) as EdgeMask x,y+1 , multiply it by the intensity difference between the vertically adjacent pixels of cs to obtain the variational mask L in the vertical direction v , the specific calculation formula is as follows:

[0117]

[0118] Step 19: Place L h and L v Add together to get the total variational loss function L mtv , the specific calculation formula is as follows:

[0119] L mtv =L h +L v ;

[0120] In this example, we construct a total variational loss function L mtv The total variation loss function plays an important role in the color and texture dual-channel lightweight image style transfer method of the present invention. It effectively suppresses various visual defects in the style transfer process by encouraging the generated image to maintain local smoothness in space, and improves the clarity, coherence and beauty of the generated image.

[0121] Step 20: Training the color and texture dual-channel lightweight image style transfer method, specifically including the following steps:

[0122] Step 20.1: Input the preprocessed content image into the color style encoder module and the texture style encoder module respectively to obtain corresponding color style encoding features and texture style encoding features;

[0123] Step 20.2: Process the preprocessed style image through the vgg network trained on the ImageNet dataset to obtain a style feature map containing C channels. Then, according to the height H and W of the style feature map, extract the style matrix Gram of size C×C = (G ij ),in;

[0124]

[0125] F ik is the value of the kth position in the style feature map of the i-th channel, F jk is the value of the kth position in the style feature map of the jth channel;

[0126] Step 20.3: Input the color style encoding feature into the color style decoder module to obtain the color migration feature, and input the texture style encoding feature into the texture style decoder module to obtain the texture migration feature;

[0127] Step 20.4: Input the content image into the color and texture dual-channel lightweight image style transfer method, and compare the content image features output by the model with the result F processed by vgg and the result F directly processed by vgg * Perform mean square error evaluation to obtain content loss L content (F,F * );

[0128] Step 20.5: Input the style image into the color and texture dual-channel lightweight image style transfer method, and process the style image features output by the model through vgg and calculate the result of the Gram matrix G, and the style image directly processed by vgg and calculate the result of the Gram matrix G * Perform mean square error estimation to obtain style loss L style (G,G * );

[0129] Step 20.6: Add the weighted sum of style loss, content loss and total variational loss to get the total model loss L. The specific calculation formula is as follows:

[0130] L=αL content (F,F * )+βL style (G,G * )+λL mtv ;

[0131] Step 20.7: According to the total loss L, select the Adam optimizer, set α, β, and λ to 1 for training, stop training when the preset number of iterations reaches 80,000, and select the model with the best verification result as the final model;

[0132] Step 21: Add the color style coding features and the texture style coding features and input them into the fusion decoder to obtain a stylized image with dual channel migration of color and texture.

[0133] Through the above embodiments, we have implemented a lightweight image style transfer method based on color and texture dual channels.

[0134] See also Figure 6 In order to prove the actual effect of the method of the embodiment of the present application, the upper left 1 is the content image of the style transfer, and the upper left 2 is the style image of the style transfer. Figure 1 , the top left 3 is the style of style transfer Figure 2. Then a lightweight image style transfer method based on color and texture dual channels is used to produce the target style transfer. The lower left images 4 and 6 are the results of style image texture transfer. It is obvious that the results have rich texture information and more details. The lower left images 5 and 7 are the results of style image color transfer. The obtained image colors are more accurate and rich. Therefore, the method provided in this application has good practicality and applicability.

[0135] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0136] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A lightweight image style transfer method based on color and texture dual channels, characterized by The specific steps include: Step 1: Prepare a style image and a content image dataset; Step 2: The style image and content image datasets are preprocessed by scaling, center cropping, and normalization. The resulting image size is 512×512 to fit the input format of the model. Step 3: Construct a color and texture dual-channel lightweight image style transfer method, including a color style encoder module, a texture style encoder module, a fusion decoder module, a color attention module, and a texture attention module; Step 4: Construct a color style encoder module, including a downsampling convolution with doubled channels, 2 encoding blocks, and a downsampling convolution with an output channel of 3; construct a texture style encoder, which consists of 6 modules, including two downsampling convolutions with doubled channels, 2 encoding blocks, and 2 upsampling convolutions, and the final output channel is 3; Step 5: Construct a texture style encoder module, including multiple depth-separable convolutions, instance regularization, and ReLU activation function stacking. The convolution layer applies reflection padding, the padding size is half the size of the convolution kernel, and depth-separable convolution is applied; Step 6: The output of the color style encoder module is called the color style encoding feature matrix Constructing the color attention module Two statistical feature maps are generated using average pooling and maximum pooling respectively; then these two feature maps are processed by a multi-layer perceptron network with the same structure, and the processing results are added and a sigmoid activation function is used to generate a color channel attention map. The specific calculation formula is as follows: Step 7: Color channel attention map Multiply a matrix element-wise Get the color channel weighting matrix The calculation formula is as follows: Step 8: Right The matrix uses average pooling and maximum pooling operations respectively, and the two feature maps are integrated according to the average and maximum values, and then processed through a standard convolution layer, and finally a sigmoid activation function is used to generate a color space attention map. The specific calculation formula is as follows: Step 9: Color space attention map Element-wise multiplication Matrix to get the color space weighting matrix The specific calculation formula is as follows: Step 10: The output of the texture style encoder module is called the texture style encoding feature matrix Constructing texture attention module Two statistical feature maps are generated using average pooling and maximum pooling respectively; then these two feature maps are processed by a multi-layer perceptron network with the same structure, and the processing results are added and a texture channel attention map is generated through a sigmoid activation function. The specific calculation formula is as follows: Step 11: Texture channel attention map Multiply a matrix element-wise Get the texture channel weighting matrix The specific calculation formula is as follows: Step 12: Right The matrix uses average pooling and maximum pooling operations respectively, and the two feature maps are integrated according to the average and maximum values, and then processed through a standard convolution layer, and finally a sigmoid activation function is used to generate a texture space attention map. The specific calculation formula is as follows: Step 13: Texture Space Attention Map Element-wise multiplication Matrix gets the texture space weighting matrix The specific calculation formula is as follows: Step 14: Construct a fusion decoder module, including 1 color style decoding block, 1 texture style decoding block, and 1 downsampling module with 3 output channels; the fusion decoder can simultaneously realize the color migration of color style features and the texture migration of texture style features; Step 15: Apply the Sobel operator to calculate the edge information of the content image image and obtain the edge graph Edge. The specific calculation formula is as follows: Edge = Sobel(image); Step 16: Set the edge threshold δ and extract the EdgeMask mask through the threshold function. The specific calculation formula is as follows: EdgeMask=threshold(Edge,δ); Step 17: Use cs to represent the image obtained after the content image is stylized by the model, and represent the edge mask value at position (x, y+1) as EdgeMask x,y+1 , multiply it by the intensity difference between the horizontally adjacent pixels of cs to obtain the variational mask L in the horizontal direction h , the specific calculation formula is as follows: Step 18: Use cs to represent the image obtained after the content image is stylized by the model, and represent the edge mask value at position (x, y+1) as EdgeMask x,y+1 , multiply it by the intensity difference between the vertically adjacent pixels of cs to obtain the variational mask L in the vertical direction v , the specific calculation formula is as follows: Step 19: Place L h and L v Add together to get the total variational loss function L mtv , the specific calculation formula is as follows: L mtv =L h +L v ; Step 20: Training the model for color and texture dual channel migration includes the following steps: Step 20.1: Input the preprocessed content image into the color style encoder module and the texture style encoder module respectively to obtain corresponding color style encoding features and texture style encoding features; Step 20.2: Process the preprocessed style image through the vgg network trained on the ImageNet dataset to obtain a style feature map containing C channels. Then, according to the height H and W of the style feature map, extract the style matrix Gram of size C×C = (G ij ),in; F ik is the value of the kth position in the style feature map of the i-th channel, F jk is the value of the kth position in the style feature map of the jth channel; Step 20.3: Input the color style encoding feature into the color style decoder module to obtain the color migration feature, and input the texture style encoding feature into the texture style decoder module to obtain the texture migration feature; Step 20.4: Input the content image into the color and texture dual-channel lightweight image style transfer method, and compare the content image features output by the model with the result F processed by vgg and the result F directly processed by vgg * Perform mean square error evaluation to obtain content loss L content (F,F * ); Step 20.5: Input the style image into the color and texture dual-channel lightweight image style transfer method, and process the style image features output by the model through vgg and calculate the result of the Gram matrix G, and the style image directly processed by vgg and calculate the result of the Gram matrix G * Perform mean square error estimation to obtain style loss L style (G,G * ); Step 20.6: Add the weighted sum of style loss, content loss and total variational loss to get the total model loss L. The specific calculation formula is as follows: L=αL content (F,F * )+βL style (G,G * )+λL mtv ; Step 20.7: According to the total loss L, select the Adam optimizer, set α, β, and λ to 1 for training, stop training when the preset number of iterations reaches 80,000, and select the model with the best verification result as the final model; Step 21: Add the color style coding features and the texture style coding features and input them into the fusion decoder to obtain a stylized image with dual channel migration of color and texture.

Images