A multi-scale network device based on SECA and a training method thereof

Abstract

The application discloses a multi-scale network based on SECA, wherein a first layer is an encoder layer, an input of the first layer is a 3*H*W matrix, and finally, a hidden vector with a size of 512*H / 16*W / 16, content features and style features is output; a second layer is a conversion layer, the conversion layer comprises n SECA modules, an input of the second layer is the hidden vector with the content features and the style features output by the first layer, each SECA module accepts style features with different sizes and content features with the same size, outputs features with different heights and widths, then scales the features to the same height and width through an operation of scaling the features upwards, and then fuses the features, and finally, the style features of different layers are combined through a fusion layer after being converted respectively, and a style feature vector with a size of 512*H / 16*W / 16 is finally output; and a third layer is a decoder layer, a network structure of the third layer is a mirror image of the second layer, an input of the third layer is the style feature vector with the size of 512*H / 16*W / 16 output by the second layer, and finally, a 3*H*W matrix is output.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning and computer vision in artificial intelligence, and more particularly relates to a multi-scale network device based on style enhanced cross attention (SECA) and a training method thereof. BACKGROUND

[0002] The goal of style transfer is to render a picture in a given style while preserving its original content structure, which has great significance in the art field.

[0003] The research results on arbitrary style transfer can be classified into several categories. The first category of methods is based on global transformation, which aims to globally modify features to achieve the desired style transfer effect. One representative and breakthrough work is AdaIN, which rescales the content features by adaptively applying the mean and variance of the style features. Other methods such as WCT, which uses two transformation steps of whitening and coloring; Linear, which generates linear transformations according to content and style features to achieve global style transfer; the second category of methods is based on local transformation, such as style-swap, which uses the similarity between content and style patches to achieve stylization;

[0004] However, both of the above methods have some defects: the global transformation-based method has made great progress in the field of arbitrary style transfer, but it is usually difficult to preserve local information; the local transformation-based method effectively preserves local information, but may cause visual artifacts.

[0005] In order to solve the problems of the above two methods, people have developed methods based on attention mechanism, which can better model the fine-grained correspondence between local features of style and content images compared to the previous two works. Park et al. proposed a style-attentional network to match content and style features. Deng et al. proposed to adaptively separate content and style representations, and then calculate the similarity between content and style representations through attention mechanism to rearrange the distribution of style representations. Liu et al. proposed a new module AdaAttN, which performs adaptive attention normalization on each pixel point and simultaneously fuses statistical information from different network levels, achieving good results. AesUST can generate more aesthetic and realistic arbitrary style transfer results by introducing an aesthetic discriminator and a novel aesthetic style attention (Aesthetic-aware Style-Attention) module, further improving the performance of style transfer.

[0006] However, the above attention mechanism-based method still has some defects that cannot be ignored: the degree of stylization is not enough, many styles in the style picture do not appear in the result picture, resulting in less change in the picture, the picture looks monotonous, and the brush touch feeling is not enough. SUMMARY

[0007] In view of the above defects or improvement needs of the prior art, the present application provides a multi-scale network device based on SECA and a training method thereof. The purpose is to solve the technical problems that the existing deep learning-based style transfer method has insufficient degree of stylization, many styles in the style picture do not appear in the result picture, resulting in less change in the picture, the picture looks monotonous, and the brush touch feeling is not enough.

[0008] To achieve the above purpose, according to one aspect of the present application, a multi-scale network device based on SECA is provided, and the network structure is as follows:

[0009] The first layer is an encoder layer module, which adopts the part from the beginning to ReLU5_1 in the pre-trained VGG19 model, and uses the strategy of different activation layer features of the VGG model, that is, the features output by ReLU4_1 and ReLU5_1 of VGG19 are respectively fused after Transform, the input is a 3×H×W matrix, and the final output is a hidden vector of 512×H / 16×W / 16, content features and style features, wherein H represents the height of the input image, and W represents the width of the input image.

[0010] The second layer is a conversion layer module, which includes n SECA modules, wherein n represents the number of different size style pictures, the input is the hidden vector of the content features and style features output by the first layer, each SECA module accepts different size style features and the same size content features, outputs features of different height and width, and then scales these features to the same height and width through the operation of Upscale, and then fuses them. In combination with the strategy of Multi_level, the feature vectors of different layers are converted, and the stylized features of different layers are combined through a fusion layer, and finally a stylized feature vector of 512×H / 16×W / 16 is output.

[0011] The third layer is a decoder layer module, and the network structure thereof is a mirror image of the second layer, the input is the stylized feature vector of 512×H / 16×W / 16 output by the second layer, and the final output is a 3×H×W matrix.

[0012] Preferably, the network structure of the SECA module is as follows:

[0013] The first layer inputs a feature matrix of a content picture and a feature matrix of a style picture, and outputs three feature matrices Q, K and V.

[0014] The second layer inputs the feature matrices Q and K output by the first layer, performs matrix multiplication thereon, and outputs an attention score map.

[0015] The third layer inputs the attention score map output by the second layer, performs a normalization exponential function operation thereon to obtain an attention weight map, and outputs the same. The fourth layer inputs the attention weight map obtained by the third layer and the feature matrix obtained by the first layer, performs weighted average on the two in space to obtain a style feature, and outputs the same.

[0016] The fifth layer inputs the attention score map output by the second layer, selects a content feature most relevant to each style feature, and outputs the same. The sixth layer inputs the attention score map output by the second layer, performs Mask processing on the attention score map output by the second layer, retains only the content feature output by the fifth layer, that is, the maximum value of each column in the attention score map, and sets other values in the attention score map except the maximum value to a very small negative number, and outputs the same. The seventh layer inputs the attention score map output by the sixth layer, performs a Norm operation on the attention score map to obtain a style-enhanced attention map, and outputs the same. The eighth layer inputs the style-enhanced attention map output by the seventh layer and the attention weight map output by the third layer, mixes the two to obtain a new attention weight map, and outputs the same.

[0017] The ninth layer inputs the new attention weight map output by the eighth layer, performs a weighted average on the new attention weight map and the feature matrix of the content picture to obtain a new feature matrix of the content picture, and outputs the same. The tenth layer inputs the new feature matrix of the content picture output by the ninth layer and the feature matrix of the style picture, and outputs a new style feature.

[0018] The eleventh layer inputs the new style feature output by the tenth layer and the feature matrix of the content picture, and outputs a new content picture. The twelfth layer inputs the new content picture output by the eleventh layer and the feature matrix of the style picture, and outputs a new style picture. The thirteenth layer inputs the new style picture output by the twelfth layer and the feature matrix of the content picture, and outputs a new content picture. The fourteenth layer inputs the new content picture output by the thirteenth layer and the feature matrix of the style picture, and outputs a new style picture.

[0019] The fifteenth layer inputs the new style picture output by the fourteenth layer and the feature matrix of the content picture, and outputs a new content picture. The sixteenth layer inputs the new content picture output by the fifteenth layer and the feature matrix of the style picture, and outputs a new style picture.

[0020] The seventeenth layer inputs the new style picture output by the sixteenth layer and the feature matrix of the content picture, and outputs a new content picture. The eighteenth layer inputs the new content picture output by the seventeenth layer and the feature matrix of the style picture, and outputs a new style picture. The nineteenth layer inputs the new style picture output by the eighteenth layer and the feature matrix of the content picture, and outputs a new content picture. The twentieth layer inputs the new content picture output by the nineteenth layer and the feature matrix of the style picture, and outputs a new style picture.

[0021] The twenty-first layer inputs the new style picture output by the twentieth layer and the feature matrix of the content picture, and outputs a new content picture. ​​​​​​​​​​​​​The weighted average style features to obtain the final output.

[0022] Preferably, the first layer is using the following formula:

[0023]

[0024] K

[0025] V

[0026] where f(.), g(.), h(.) represent learnable 1x1 convolutions respectively, Norm represents channel-wise mean-variance normalization, represents the feature matrix of the content picture after the encoder layer, represents the feature matrix of the style picture after the encoder layer.

[0027] The fourth layer is using the following formula:

[0028]

[0029] where o(.) represents a 1x1 convolution.

[0030] The fifth layer is using the following formula:

[0031]

[0032] where dim represents the dimension of the attention score map A feature.

[0033] The sixth layer is using the following formula:

[0034]

[0035] where Mask operation represents keeping part of the original attention score map and setting the rest to a very small negative number.

[0036] The seventh layer is using the following formula:

[0037]

[0038] where Norm operation represents a normalization operation.

[0039] The eighth layer is using the following formula:

[0040]

[0041] where represent the mixing ratio of the attention weight map and the style enhanced attention map respectively, representing the intensity of the adjustment of the overall, wherein > 0.

[0042] According to another aspect of the present application, there is provided a method for training a SECA-based multi-scale network device, comprising the following steps:

[0043] (1) Obtain MS-COCO as a content picture dataset, obtain WikiArt as a style picture dataset, and divide the content picture dataset into a content picture training set and a content picture test set according to a ratio of 8:2, and divide the style picture dataset into a style picture training set and a style picture test set according to a ratio of 8:2;

[0044] (2) input the content picture training set and the style picture training set obtained in step (1) into the SECA-based multi-scale network device, and use a back propagation algorithm to update and optimize the weight parameters and the bias parameters of each layer in the SECA-based multi-scale network device, to obtain a trained SECA-based multi-scale network device.

[0045] Preferably, step (2) comprises the following sub-steps:

[0046] (2-1) input the content picture training set and the style picture training set obtained in step (1) into the SECA-based multi-scale network device, to obtain content pictures , style pictures and style pictures of different scales ; wherein n represents the number of style pictures of different scales of each style picture in the style picture training set;

[0047] (2-2) VGG encode the content pictures and the style pictures of different scales output in (2-1), to obtain content features and style features of different scales ;

[0048] (2-3) input the content features and the style features of different scales obtained in (2-2) into the SECA module respectively, to obtain n style features of different scales , ,..., ;

[0049] (2-4) input the style features of different scales , ,..., obtained in (2-3) into the conversion layer in the SECA-based multi-scale network device for mixing, to obtain final stylized features ;

[0050] (2-5) style the final features obtained in (2-4) Input the decoder layer of the SECA-based multi-scale network device to obtain the picture after style conversion ;

[0051] (2-6) based on the content picture obtained in (2-1) and the style picture , and the picture after style conversion in step (2-5) , the total loss function in the training process is back propagated to optimize the parameters of the SECA-based multi-scale network device, and a trained SECA-based multi-scale network device is obtained.

[0052] Preferably, the total loss function is:

[0053]

[0054] wherein , , , are weight coefficients greater than 0; represents the global content loss, represents the global style loss, represents the adversarial loss, represents the identity loss.

[0055] Preferably, the global content loss is equal to:

[0056]

[0057] The global style loss is equal to:

[0058]

[0059] wherein Φ i (.) represents the i-th activation layer output of the VGG Encoder, Relu4 1, Relu5 1 layer is used when calculating the global content loss, Relu1 1, Relu2 1, Relu3 1, Relu4 1, Relu5 1 layer is used when calculating the global content loss, μ() represents taking the mean of the features, σ() represents taking the variance of the features, and represent the number of pictures in the content picture training set and the style picture training set, respectively.

[0060] Preferably, the adversarial loss The calculation method is as follows:

[0061]

[0062] Wherein D represents the discriminator network, G represents the SECA-based multi-scale network device, Refers to the picture of the content picture data set, Refers to the picture of n different scales in the style picture data set, E() represents the expectation of the content.

[0063] Preferably, the identity loss The calculation formula is as follows:

[0064]

[0065] Wherein And Respectively represent the picture obtained by directly passing the decoder layer in the SECA-based multi-scale network device.

[0066] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0067] (1) The present application adopts the SECA module, which further processes the output features obtained by the cross-attention module and performs additional feature post-processing operations, thereby increasing the stylization degree of the result picture.

[0068] (2) The SECA module provided by the present application has strong versatility. Many models based on cross-attention modules have been trained before, and the cross-attention module can be directly replaced with the SECA module, and the style conversion effect can be improved. The SECA module does not use additional training parameters and does not require additional training after replacement.

[0069] (3) The present application adopts step (2-4), which obtains different scale style pictures by scaling, and then obtains different scale style features by respectively passing through the encoder. Then, these style features and normal scale content features are mixed in the SECA module to obtain different scale stylized results, and finally these stylized features are fused, so that the picture output by the SECA-based multi-scale network device is more varied and has a stronger brush touch. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 Is the structure diagram of the SECA module of the present application;

[0071] Figure 2 ​​is a structural diagram of a multi-scale network device based on SECA of the present application.

[0072] Figure 3 is a contrast test diagram used in the present application. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0074] The present application provides a SECA module, which only makes some modifications to the cross attention module, but greatly enhances the stylization effect of the generated image. The present application describes the differences in model structure between the SECA module and the general cross attention module in detail. A multi-scale network structure based on SECA is also proposed, which takes a content picture and style pictures of different scales as input, and then outputs the migrated picture.

[0075] As shown in Figure 2 The present application provides a multi-scale network device based on SECA, and the network structure is as follows:

[0076] The first layer is an encoder (Encoder) layer module, which adopts the part from the beginning to ReLU5_1 in the pre-trained VGG19 model, and uses the strategy of different activation layer features (Multi-level) of the VGG model, that is, the features output by ReLU4_1 and ReLU5_1 of VGG19 are respectively fused after Transform, the input is a 3xHxW matrix, and finally the hidden vector of the content feature and the style feature with the size of 512xH / 16xW / 16 is output, wherein H represents the height of the input image, and W represents the width of the input image.

[0077] The second layer is a transform layer module, which includes n SECA modules, where n represents the number of different sizes of picture styles, the input is the hidden vector of the content feature and the style feature output by the first layer, each SECA module accepts different size style features and the same size content features, outputs features of different height and width, and then scales these features to the same height and width through the operation of upscaling features, and then fuses them. In combination with the strategy of Multi_level, the feature vectors of different layers are respectively transformed, and then the stylized features of different layers are combined through a fusion layer to finally output a stylized feature vector of 512xH / 16xW / 16.

[0078] The third layer is a decoder layer module, and its network structure is the mirror image of the encoder. The input is the stylized feature vector of 512xH / 16xW / 16 output by the second layer, and the final output is a 3xHxW matrix.

[0079] As shown in Figure 1 , the network structure of the SECA module is as follows:

[0080] The first layer, the input is the feature matrix of the content picture and the feature matrix of the style picture, and the output is Q, K, and V three feature matrices;

[0081] This layer uses the following formula:

[0082]

[0083] K

[0084] V

[0085] Where f(.), g(.), and h(.) represent learnable 1x1 convolution, Norm represents channel-wise mean-variance normalization, represents the feature matrix of the content picture after the encoder layer, represents the feature matrix of the style picture after the encoder layer.

[0086] The second layer, the input is the feature matrix Q and K output by the first layer, and the matrix multiplication is performed on it to output the attention score map .

[0087] This layer uses the following formula:

[0088]

[0089] The third layer, input is the attention score map output by the second layer , and a Softmax operation is performed on it to obtain an attention weight map and output;

[0090] This layer uses the following formula:

[0091]

[0092] The fourth layer, input is the attention weight map obtained by the third layer and the feature matrix obtained by the first layer , and and are weighted and averaged in space to obtain style features and output;

[0093] This layer uses the following formula:

[0094]

[0095] where o(.) represents a 1x1 convolution.

[0096] The fifth layer, input is the attention score map output by the second layer , and the most relevant content features for each style feature are selected and output;

[0097] This layer uses the following formula:

[0098]

[0099] where dim represents the dimension of the attention score map A feature.

[0100] The sixth layer, input is the attention score map output by the second layer , and the attention score map output by the second layer is processed by Mask, only retaining the content features output by the fifth layer , that is, the maximum value of each column in the attention score map , and setting other values in except the maximum value to a very small negative number, and output ;

[0101] This layer uses the following formula:

[0102]

[0103] where the Mask operation means retaining the original attention score map the remaining values are set to be very small negative numbers.

[0104] the seventh layer, input is the output of the sixth layer , the output of the fourth layer is subjected to a Norm operation to obtain a style enhanced attention map and output;

[0105] This layer uses the following formula:

[0106]

[0107] wherein the Norm operation represents a normalization operation.

[0108] the eighth layer, input is the style enhanced attention map output by the seventh layer and the attention weight map output by the third layer , the output of the fourth layer and are mixed to obtain a new attention weight map and output.

[0109] This layer uses the following formula:

[0110]

[0111] wherein represent the respective mixing ratios of the attention weight map and the style enhanced attention map , and denotes the adjustment of the overall intensity, wherein >0.

[0112] the ninth layer, input is the new attention weight map output by the eighth layer , the new attention weight map is weighted and averaged with the style feature to obtain the final output.

[0113] This layer uses the following formula:

[0114]

[0115] wherein denotes a 1x1 convolution.

[0116] the output of the fourth layer is used in the training stage, and the output of the ninth layer is used in the inference stage.

[0117] The application also provides a training method of a multi-scale network device based on SECA, comprising:

[0118] (1) Obtain MS-COCO as a content picture dataset, obtain WikiArt as a style picture dataset, and divide the content picture dataset into a content picture training set and a content picture test set according to a ratio of 8:2, and divide the style picture dataset into a style picture training set and a style picture test set according to a ratio of 8:2;

[0119] (2) input the content picture training set and the style picture training set obtained in step (1) into the SECA-based multi-scale network device, and use a back propagation algorithm to update and optimize the weight parameters and the bias parameters of each layer in the SECA-based multi-scale network device, to obtain a trained SECA-based multi-scale network device;

[0120] This step includes the following sub-steps:

[0121] (2-1) input the content picture training set and the style picture training set obtained in step (1) into the SECA-based multi-scale network device, to obtain content pictures , style pictures and style pictures of different scales ; wherein n represents the number of style pictures of different scales of each style picture in the style picture training set;

[0122] (2-2) perform VGG encoding on the content pictures and the style pictures of different scales output in (2-1), to obtain content features and style features of different scales ;

[0123] (2-3) input the content features and the style features of different scales obtained in (2-2) into the SECA module respectively, to obtain n style features of different scales , ,..., ;

[0124] The above steps (2-1) to (2-3) have the advantage that the output features obtained by the cross-attention module are further subjected to additional feature post-processing operations, so that the degree of stylization of the result picture is increased.

[0125] (2-4) input the style features of different scales , ,..., obtained in (2-3) into the conversion layer in the SECA-based multi-scale network device for mixing, to obtain final stylized features ;

[0126] The step has the advantages that it fuses style information of different scales, and makes the picture output by the network more diverse and the brush strokes more powerful.

[0127] (2-5) The final stylized feature obtained in (2-4) Input the decoder layer of the SECA-based multi-scale network device to obtain the picture after style conversion ;

[0128] (2-6) Based on the content picture obtained in (2-1) and the style picture , and the picture after style conversion in step (2-5) , the total loss function in the training process is back propagated to optimize the parameters of the SECA-based multi-scale network device, and a trained SECA-based multi-scale network device is obtained.

[0129] Specifically, the total loss function in the application is:

[0130]

[0131] wherein , , , are weight coefficients greater than 0; represents the global content loss, represents the global style loss, represents the adversarial loss, represents the identity loss, and are calculated as follows:

[0132] The application uses a pre-trained VGG-19 network to calculate the global content loss and the global style loss.

[0133] The global content loss is to make the structure of the stylized picture as close as possible to that of the content picture :

[0134]

[0135] The global style loss is to make the overall style of the stylized picture closer to that of the style picture :

[0136]

[0137] wherein Φ i(.) represents the i-th activation layer output of VGG Encoder, Relu4 1, Relu5 1 layer is used when calculating the global content loss, Relu1 1, Relu2 1, Relu3 1, Relu4 1, Relu5 1 layer is used when calculating the global content loss, and μ () represents taking the mean of the features, and σ () represents taking the variance of the features, and respectively represent the number of pictures in the content picture training set and the style picture training set.

[0138] wherein L adv is calculated as follows:

[0139] An adversarial training method is used to further enhance the stylization result, that is, an additional discriminator network is used to perform a min-max game, and the network is further learned through the loss so that the output picture is as close as possible to domain:

[0140]

[0141] wherein D represents the discriminator network, G represents the SECA-based multi-scale network device, refers to the picture of the content picture data set, refers to the picture of the style picture data set, and E () represents the expectation of the content.

[0142] wherein L id is calculated as follows:

[0143] The present application directly passes the style features and the content features through the decoder layer in the SECA-based multi-scale network device without passing through the conversion layer in the SECA-based multi-scale network device:

[0144]

[0145] wherein and respectively represent the picture obtained by directly passing the content features and the unscaled style features through the decoder layer in the SECA-based multi-scale network device.

[0146] Experimental results

[0147] The experimental environment of the present application: using Adam optimizer, learning rate is 1e-4, the whole training process uses 4 content style image pairs of mini-batch size for training, and trains 80,000 iterations. During training, the present application first scales the smaller dimension of the two pictures to 512 while keeping the aspect ratio unchanged, and then randomly crops to 256x256 pixels. The present application performs all experiments and network training on an NVIDIA RTX 3060 12GB GPU. The weight coefficients of each in the total loss are set to: , , , . The hyperparameters for mixing two maps are set to , , .

[0148] Performance comparison

[0149] As Figure 3 shown, the present application and 5 arbitrary style transfer methods are compared, which are AesUST, AdaIN, MANet, SANet, and AdaAttN. AdaIN realizes the conversion of stylized features by globally readjusting the second-order statistics of content features, but since it only considers global information and ignores local information, it will introduce some unsightly flaws (1st, 4th, 5th, 7th, 8th rows) and distortions (2nd row) to the picture. MANet and SANet use attention mechanisms to match and fuse style features similar to content features to achieve style conversion, and the results they produce often have very obvious dirty textures (4th, 5th, 7th, 8th rows) and introduce some not-so-good distortions to the picture (2nd, 4th rows). Inspired by AdaIN and SANet, AdaAttn can better preserve the original content information, but the resulting results are not enough in terms of style—there is less content from the style picture in the results, and the changes in the picture in the results are too few and single (1st, 2nd, 3rd, 4th, 5th, 7th rows), and the brush strokes are weak—there is basically no brush stroke effect from the style picture in the results (2nd, 3rd, 4th, 5th, 7th, 8th rows). AesUST uses a two-stage training strategy, which adds aesthetic quality to the picture (Aesthetic quality), and it can be seen that the pictures it generates have fewer flaws and are more aesthetically pleasing, but it still faces the problem of not being stylized enough (1st, 2nd, 5th, 7th rows), and the brush strokes are single—no corresponding brush strokes are learned from the style picture (3rd, 5th, 6th, 8th rows).

[0150] The method can exhibit relatively delicate effects, has a stronger stylization degree, and can learn specific brush stroke information in a style picture, such as oil painting brush strokes in the style picture in the 5th row and pointillism brush strokes in the style picture in the 8th row.

[0151] The application proposes a multi-scale SECA module based style transfer scheme. Firstly, the SECA module can be regarded as a cross attention module plus a feature post-processing module. Replacing the cross attention module in the trained style transfer method with the SECA module of the application can obviously enhance the stylization effect. This replacement does not require additional training operations and can be directly applied to some existing networks. Due to the universality of the module, it can help to solve the problem of insufficient stylization degree of some existing style transfer methods to some extent. Secondly, a multi-scale structure is proposed, which can enable the network to learn more rich style information and obtain more delicate results. Experiments show that the method of the application achieves good stylization results.

[0152] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the application and is not intended to limit the application. Any modification, equivalent replacement and improvement made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A multi-scale network device based on SECA, characterized in that, Its network structure is as follows: The first layer is the encoder layer module, which adopts the part from the beginning to ReLU5_1 of the pre-trained VGG19 model. It uses the strategy of different activation layer features of the VGG model, that is, it uses the features output by ReLU4_1 and ReLU5_1 of VGG19 respectively, transforms them and then fuses them. The input is a 3×H×W matrix, and the final output is a hidden vector of size 512×H / 16×W / 16 containing content features and style features, where H represents the height of the input image and W represents the width of the input image. The second layer is the transformation layer module, which contains n SECA modules, where n represents the number of style images of different sizes. The input is the hidden vectors of the content features and style features output from the first layer. Each SECA module accepts style features of different sizes and content features of the same size, outputting features of different heights and widths. Then, these features are scaled up to the same height and width and then fused together. Combined with the Multi-level strategy, the feature vectors of different layers are transformed separately and then passed through a fusion layer to combine the stylized features of different layers, finally outputting a stylized feature vector of 512×H / 16×W / 16. The network structure of the SECA module is as follows: The first layer takes the feature matrix of the content image and the feature matrix of the style image as input, and outputs three feature matrices: Q, K, and V. The second layer takes the feature matrices Q and K from the first layer as input, performs matrix multiplication on them, and outputs an attention score map. ; The third layer takes the attention score map output from the second layer as input. The attention weight map is obtained by performing a normalized exponential function operation on it. And output; The fourth layer takes the attention weight map obtained from the third layer as input. and the feature matrix obtained from the first layer ,right and Spatially weighted averages are used to obtain stylistic features. And output; The fifth layer takes the attention score map output from the second layer as input. Select the content features most relevant to each style feature. And output; The sixth layer takes the attention score map output from the second layer as input. Attention score map of the second layer output Perform masking, retaining only the content features of the fifth layer output. Attention score map Find the maximum value in each column and put it into... All values ​​except the maximum value are set to extremely small negative numbers, and the output is: ; The seventh layer, whose input is the output of the sixth layer. ,right Perform a Norm operation to obtain a style-enhanced attention map. And output; The eighth layer takes the style-enhanced attention map as input, which is the output of the seventh layer. Attention weight map of the third layer output ,right and The data is then blended to obtain a new attention weight map. And output; The ninth layer takes as input a new attention weight map output from the eighth layer. For the new attention weight map Weighted average of style features is used to obtain the final output; The third layer is the decoder layer module. Its network structure is a mirror image of the second layer. The input is the stylized feature vector of 512×H / 16×W / 16 output from the second layer, and the final output is a 3×H×W matrix.

2. The SECA-based multi-scale network device according to claim 1, characterized in that, The first layer uses the following formula: ， Where f(.), g(.), and h(.) represent learnable 1x1 convolutions, and Norm represents channel-wise mean-variance standardization. This represents the feature matrix of the content image after passing through the encoder layer. This represents the feature matrix of the style image after passing through the encoder layer; The fourth layer uses the following formula: ， Where o(.) represents a 1x1 convolution; The fifth layer uses the following formula: ， Where dim represents the dimension of the attention score map A feature; The sixth layer uses the following formula: ， The Mask operation indicates that the original attention score map is preserved. The remaining values ​​are set to extremely small negative numbers, while only a portion of the values ​​are selected. The seventh layer uses the following formula: ， The Norm operation represents a normalization operation; The eighth layer uses the following formula: ， in Representative attention weight map and style-enhanced attention map Their respective mixing ratios, This indicates an adjustment to the overall intensity, where >

0.

3. A training method for a multi-scale network device based on SECA according to claim 1, characterized in that, Includes the following steps: (1) Obtain MS-COCO as the content image dataset and WikiArt as the style image dataset. Divide the content image dataset into a content image training set and a content image test set in an 8:2 ratio, and divide the style image dataset into a style image training set and a style image test set in an 8:2 ratio. (2) Input the content image training set and style image training set obtained in step (1) into the SECA-based multi-scale network device, and use the backpropagation algorithm to update and optimize the weight parameters and bias parameters of each layer in the SECA-based multi-scale network device to obtain the trained SECA-based multi-scale network device.

4. The training method for a multi-scale network device based on SECA according to claim 3, characterized in that, Step (2) includes the following sub-steps: (2-1) Input the content image training set and style image training set obtained in step (1) into the SECA-based multi-scale network device to obtain the content images. Style pictures Images of different scales and styles Where n represents the number of style images of different scales for each style image in the style image training set; (2-2) The content image output by (2-1) Images of different scales and styles VGG encoding is performed to obtain content features. and stylistic features at different scales ; (2-3) The content features obtained from (2-2) and stylistic features at different scales Input the data into the SECA module to obtain n style features at different scales. , ,..., ; (2-4) The style features obtained in (2-3) at different scales , ,..., The input is blended in the transformation layer of a SECA-based multi-scale network device to obtain the final stylized features. ; (2-5) The final stylization features obtained in (2-4) Input the decoder layer of a SECA-based multi-scale network device to obtain style-transferred images. ; (2-6) Content images obtained based on (2-1) and style pictures and the images style-transformed in steps (2-5). Backpropagation is performed on the total loss function during training to optimize the parameters of the SECA-based multi-scale network device and obtain the trained SECA-based multi-scale network device.

5. The training method for a SECA-based multi-scale network device according to claim 4, characterized in that, The total loss function is: ， in , , , All are weighting coefficients greater than 0; This represents the overall content loss. This represents the overall style loss. Indicating resistance to loss, This indicates a loss of identity.

6. The training method for a SECA-based multi-scale network device according to claim 5, characterized in that, Global content loss equal: ， Global style loss equal: ， Where Φ i (.) represents the output of the i-th activation layer of the VGG Encoder. When calculating the global content loss, Relu4 1, Relu5 1 layers are used; when calculating the global content loss, Relu1 1, Relu2 1, Relu3 1, Relu4 1, Relu5 1 layers are used. μ() represents taking the mean of the features, and σ() represents taking the variance of the features. and These represent the number of images in the content image training set and the style image training set, respectively.

7. The training method for a multi-scale network device based on SECA according to claim 6, characterized in that, Combat losses The calculation method is as follows: ， Where D represents the discriminator network, and G represents the SECA-based multi-scale network device. This refers to the images in the content image dataset. This refers to n images of different scales in a style image dataset, where E() represents the expectation of the content.

8. The training method for a multi-scale network device based on SECA according to claim 7, characterized in that, Loss of identity The calculation formula is as follows: ， in and These respectively represent the content features and unscaled style features Images obtained directly through the decoder layer in a SECA-based multi-scale network device.

Images