A method and system for transferring the style of any image based on deep reinforcement learning
Through deep reinforcement learning methods, a network of actors, critics and executors is established, and the content, style and multi-level comparison loss functions are used to solve the problems of complexity of style transfer models and difficult user needs in the existing technology, achieving lightweight Level and efficient style transfer effect.
Patent Information
- Application Number
- CN202411247402.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-09-06
AI Technical Summary
In any style migration method, the network structure is complex and the parameters are large, making it difficult to meet the different style needs of users. The user evaluation of the style migration results is subjective, making it difficult to achieve flexible style control.
Using a deep reinforcement learning method, a model including actor network, critic network and executor network is established. It is trained through the training set and total loss function, and uses content loss, style loss and multi-level stylized contrast loss function to gradually carry out the Style transfer, implementing a non-pre-trained unified strategy to model and coding content and styles.
It realizes a lightweight style migration model, with few parameters, high computing efficiency, fast inference speed, flexible control of the degree of stylization, meet the expectations of different users, and achieve good performance under any content and style.
Smart Images

Figure CN118761902B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a method and system for transferring the style of any image based on deep reinforcement learning. Background Art
[0002] Style transfer is the process of combining the semantic information of the content image with the style pattern of the style image through a neural network to form a stylized result. It is an interesting research topic in the field of computer vision and is also one of the applications that people often come into contact with in daily life. Style transfer provides new possibilities for artistic creation, and artists can explore different artistic styles through technical means. Style transfer has also shown great potential and opportunities in fields such as image editing and video processing.
[0003] Existing style transfer methods are divided into three categories: single style, multiple styles, and arbitrary style. Single style transfer is to transfer the style of any content image after selecting a style. However, in the arbitrary style transfer task, the content and style involved are arbitrarily selected, which means that the model needs to be able to accurately express the feature information of different content and styles, and perform correct feature fusion and reconstruction. These requirements greatly increase the challenge of controlling the degree of stylization of the results. In arbitrary style transfer, the common method of extracting content and style features is to use pre-trained neural networks to extract their feature information or to train two encoders to extract content features and style features respectively, but both methods will make the network structure complicated and the number of parameters increase significantly. Secondly, the existing deep learning style transfer method usually directly generates the final style transfer result through one reasoning after determining the content and style. However, the user evaluation of the style transfer result is very subjective. The existing method determines the result by one reasoning, which is difficult to meet the different stylization needs of users.
[0004] Therefore, it is necessary to provide a method and system for style transfer of any image based on deep reinforcement learning to improve the efficiency of style transfer of any image. Summary of the invention
[0005] The present invention provides a method for style transfer of any image based on deep reinforcement learning, comprising: establishing a deep reinforcement learning model, wherein the deep reinforcement learning model at least includes an actor network, a critic network and an executor network; establishing a training set; training the deep reinforcement learning model through the training set and a total loss function, wherein the total loss function at least includes a content loss function, a style loss function and a multi-level stylized contrast loss function; obtaining a content image to be transferred and a style image corresponding to the content image to be transferred; and generating a style-transferred image corresponding to the content image to be transferred based on the content image to be transferred and the style image corresponding to the content image to be transferred through the trained deep reinforcement learning model.
[0006] Furthermore, the actor network includes three feature extraction blocks, two depth-separable blocks, two residual blocks, three style signal modulation blocks and a distribution calculation block; the feature extraction block includes a reflection filling layer, a convolution layer and an activation layer; the depth-separable block includes a reflection filling layer, a grouped convolution layer, a 1x1 convolution layer and two activation layers, wherein the grouped convolution layer is used to extract spatial features, and the 1x1 convolution layer is used to extract channel features; the residual block includes two reflection filling layers, two convolution layers and an activation layer; the style signal modulation block includes a weight branch and a bias branch, wherein the weight branch is used to calculate the weight expressed by the current style, and the bias branch is used to calculate the bias expressed by the current style; the distribution calculation block includes a reflection filling layer, a grouped convolution layer and two 1x1 convolution layers, wherein one of the 1x1 convolution layers is used to calculate the mean, and the other 1x1 convolution layer is used to calculate the variance.
[0007] Furthermore, the performer network consists of two modulation residual blocks, two modulated upsampling blocks, a feature extraction block and two feature fusion blocks; the modulation residual block includes three reflection filling layers, three convolution layers and two activation layers; the modulated upsampling block includes one upsampling layer, two reflection filling layers, one grouped convolution layer, one 1x1 convolution layer and two activation layers; the feature extraction block includes one reflection filling layer, one convolution layer and one activation layer.
[0008] Furthermore, the critic network consists of a feature extraction block, two depth-separable blocks and an action scoring block; the feature extraction block includes a reflection filling layer, a convolution layer and an activation layer; the depth-separable block includes a reflection filling layer, a grouped convolution layer, a 1x1 convolution layer and two activation layers; the action scoring block includes a feature extraction block and a 1x1 convolution layer.
[0009] Furthermore, the deep reinforcement learning model is trained through the training set and the total loss function, including: S1, inputting the current state into the actor network, the actor network observes the current state, obtains style information, establishes a stylization strategy for the moving picture in the current state, and samples to obtain the optimal decision action corresponding to the current state, wherein the current state includes a moving picture and a style picture, and the moving picture is generated by initializing the content picture; S2, inputting the optimal decision action into the executor network, and synchronously inputting the style information generated in the actor network into the executor network through a jump connection; S3, the executor network uses the style information provided by the jump connection as a reference, executes the optimal decision action, obtains a decision execution result, and uses the decision execution result to update the moving picture to obtain a new state; S4, put the moving picture and style picture of the new state into the pre-trained neural network VGG19, obtain the multi-layer feature map of the moving picture of the new state and the multi-layer feature map of the style picture, and calculate the feedback reward obtained by executing the action; S5, input the moving picture of the current state, the optimal decision action and the feedback reward into the critic network, analyze the state features, and splice them with the action in the channel dimension to obtain the fusion features, and analyze the fusion features to obtain the maximum expected reward; S6, the actor network adjusts the update strategy according to the maximum expected reward; S7, calculates the loss through the content loss function, the style loss function and the multi-level stylization contrast loss function; S8, according to the loss, judge whether the preset training conditions are met, if so, complete the training, if not, optimize through gradient back propagation and execute S1-S7.
[0010] Furthermore, the total loss function is: ,in, is the total loss, are the model parameters of the actor network, are the model parameters of the actor network, , , are all learnable parameters, For content loss, is the style loss, is a multi-level stylization contrast loss.
[0011] Furthermore, the content loss function is: ,in, For the Executor Network The output of the step, For content pictures, It is the feature map of the input image output by the first activation function layer in the fourth convolutional block of the pre-trained neural network VGG19. , and They respectively represent the number of channels, height, and width of the feature map of the input image output by the first activation function layer in the fourth convolutional block of the pre-trained neural network VGG19.
[0012] Furthermore, the style loss function is: ,in, For the Executor Network The output of the step, For style pictures, The input image extracted by the pre-trained neural network VGG19 is The feature map of the layer, is the maximum number of layers selected, represents the mean value, Indicates variance.
[0013] Furthermore, the multi-level stylization contrast loss function is: ,in, is the number of batch samples, is the number of layers in the actor network, For the The moving pictures in the samples are in the actor network The output of the layer, For the The moving pictures in the samples are in the actor network The output of the layer, For the The style images in samples are output at the kth layer of the actor network.
[0014] The present invention provides a system for style transfer of any image based on deep reinforcement learning, comprising: a model building module, used to build a deep reinforcement learning model, wherein the deep reinforcement learning model at least includes an actor network, a critic network and an executor network; a sample building module, used to build a training set; a model training module, used to train the deep reinforcement learning model through the training set and a total loss function, wherein the total loss function at least includes a content loss function, a style loss function and a multi-level stylized contrast loss function; an image acquisition module, used to acquire a content image to be transferred and a style image corresponding to the content image to be transferred; a style transfer module, used to generate a style-transferred image corresponding to the content image to be transferred based on the content image to be transferred and the style image corresponding to the content image to be transferred through the trained deep reinforcement learning model.
[0015] Compared with the prior art, the method and system for style transfer of any image based on deep reinforcement learning provided by the present invention have at least the following beneficial effects:
[0016] 1. The network module used is lighter, which is a lightweight style transfer model with fewer parameters, higher calculation efficiency and faster reasoning speed.
[0017] 2. Use reinforcement learning to perform step-by-step style transfer, breaking down the one-step style transfer in the existing technology into step-by-step transfer, which can more flexibly control the degree of stylization. The degree of style gradually increases with the number of steps, which can meet the expectations of different users.
[0018] 3. For the first time, the reinforcement learning-based style transfer is extended to the field of arbitrary style transfer, and a non-pre-trained unified strategy is adopted to model and encode content and style. At the same time, a multi-level contrast loss is proposed to enhance the expressiveness of the model. Even with arbitrary content and style, good performance can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:
[0020] Figure 1 It is a flowchart of a method for transferring the style of any image based on deep reinforcement learning according to some embodiments of this specification;
[0021] Figure 2 is a schematic diagram of the structure of an actor network according to some embodiments of this specification;
[0022] Figure 3 is a schematic diagram of the structure of a feature extraction block according to some embodiments of this specification;
[0023] Figure 4 is a schematic diagram of a structure of a deeply separable block according to some embodiments of this specification;
[0024] Figure 5 is a schematic diagram of the structure of a residual block according to some embodiments of this specification;
[0025] Figure 6 is a schematic diagram of the structure of a style signal modulation block according to some embodiments of this specification;
[0026] Figure 7 is a schematic diagram of the structure of a distributed computing block according to some embodiments of this specification;
[0027] Figure 8 is a schematic diagram of the structure of an executor network according to some embodiments of this specification;
[0028] Fig. 9is a schematic diagram of the structure of a modulation residual block according to some embodiments of this specification;
[0029] Fig.10 is a schematic diagram of the structure of a modulatable upsampling block according to some embodiments of this specification;
[0030] Fig.11 is a schematic diagram of the structure of a critic network according to some embodiments of this specification;
[0031] Fig.12 is a schematic diagram of a process for training a deep reinforcement learning model according to some embodiments of this specification;
[0032] Fig.13 is a schematic diagram of a reward calculation process according to some embodiments of this specification;
[0033] Fig.14 This is a module diagram of a system for transferring the style of any image based on deep reinforcement learning according to some embodiments of this specification. DETAILED DESCRIPTION
[0034] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.
[0035] Figure 1 is a flowchart of a method for transferring the style of any image based on deep reinforcement learning according to some embodiments of this specification, such as Figure 1 As shown, a method for transferring the style of any image based on deep reinforcement learning may include the following process.
[0036] Step 110, establishing a deep reinforcement learning model.
[0037] Among them, the deep reinforcement learning model includes at least an actor network, a critic network and an executor network.
[0038] Figure 2 is a schematic diagram of the structure of an actor network according to some embodiments of this specification, such as Figure 2 As shown, in some embodiments, the actor network includes three feature extraction blocks, two depth-separable blocks, two residual blocks, three style signal modulation blocks, and one distribution calculation block. Figure 2 The black arrows in the figure represent the content path, and the white arrows represent the style path.
[0039] Figure 3 is a schematic diagram of the structure of a feature extraction block according to some embodiments of this specification, such as Figure 3 As shown, in some embodiments, the feature extraction block includes a reflection filling layer, a convolution layer and an activation layer, wherein the activation layer may be a ReLU (Rectified Linear Unit) activation layer.
[0040] Figure 4 is a schematic diagram of a structure of a depth-separable block according to some embodiments of this specification, such as Figure 4 As shown, in some embodiments, the depth separable block includes a reflection padding layer, a grouped convolution layer, a 1×1 convolution layer, and two activation layers, wherein the grouped convolution layer is used to extract spatial features, and the 1×1 convolution layer is used to extract channel features, wherein the activation layer can be a ReLU activation layer. Compared with fixed value padding, the reflection padding layer is more suitable for style transfer tasks.
[0041] Figure 5 is a schematic diagram of the structure of the residual block according to some embodiments of this specification, such as Figure 5 As shown, in some embodiments, the residual block includes two reflection filling layers, two convolutional layers, an activation layer and a residual connection, wherein the activation layer can be a ReLU activation layer.
[0042] Figure 6 is a schematic diagram of the structure of a style signal modulation block according to some embodiments of this specification, such as Figure 6 As shown, in some embodiments, the style signal modulation block includes a weight branch and a bias branch, wherein the weight branch is used to calculate the weight expressed by the current style, and the bias branch is used to calculate the bias expressed by the current style. The weight branch and the bias branch have the same structure, and the weight branch includes a reflection filling layer, a standard convolution layer, a ReLU activation layer, and an adaptive average pooling layer.
[0043] Figure 7 is a schematic diagram of the structure of the distributed computing block shown in some embodiments of this specification, such as Figure 7 As shown, in some embodiments, the distribution calculation block includes a reflection filling layer, a grouped convolution layer, and two 1×1 convolution layers, wherein one 1×1 convolution layer is used to calculate the mean and the other 1×1 convolution layer is used to calculate the variance.
[0044] Figure 8 is a schematic diagram of the structure of an executor network according to some embodiments of this specification, such as Figure 8 As shown, in some embodiments, the performer network consists of two modulation residual blocks, two modulated upsampling blocks, a feature extraction block and two feature fusion blocks.
[0045] Fig. 9 is a schematic diagram of the structure of the modulation residual block according to some embodiments of this specification, such as Fig. 9 As shown, in some embodiments, the modulated residual block includes three reflection filling layers, three convolution layers and two activation layers, wherein the activation layer may be a ReLU activation layer.
[0046] Fig.10 is a schematic diagram of the structure of a modulated upsampling block according to some embodiments of this specification, such as Fig.10 As shown, in some embodiments, the modulated upsampling block includes an upsampling layer, two reflection padding layers, a grouped convolution layer, a 1x1 convolution layer, and two activation layers. The activation layer may be a ReLU activation layer, and the upsampling layer uses bilinear interpolation.
[0047] The feature extraction block includes a reflection padding layer, a convolution layer, and an activation layer, wherein the activation layer may be a ReLU activation layer.
[0048] The feature fusion block adopts the design of AdaIN (Adaptive Instance Normalization), which has no learnable parameters and is simpler to calculate. This method uses the pre-trained VGG (Visual Geometry Group) network to extract deep feature maps of content images and style images, and then fuses the deep content features of the content image with the deep style features of the style image through adaptive instance normalization. The core idea of this fusion is to replace the mean and variance of the content features with the corresponding values of the style features, thereby realizing the transfer of any style. The calculation method is:
[0049] ,
[0050] in, is the result after feature fusion. , They are the feature maps after encoding the content image and the style image. is the mean of the feature map after encoding the content image, is the variance of the feature map after encoding the content image, is the mean of the feature map after encoding the style image, The variance of the feature map after encoding the style image.
[0051] Fig.11 is a schematic diagram of the structure of a critic network according to some embodiments of this specification, such as Fig.11As shown, in some embodiments, the critic network consists of a feature extraction block, two depthwise separable blocks, and an action scoring block.
[0052] The feature extraction block includes a reflection filling layer, a convolution layer and an activation layer, wherein the activation layer can be a ReLU activation layer.
[0053] The depth-wise separable block includes a reflection padding layer, a grouped convolution layer, a convolution layer, and two activation layers, wherein the activation layer may be a ReLU activation layer.
[0054] Step 120, establishing a training set.
[0055] The training set may include multiple training samples, which may include content images and style images. Content images refer to images that contain the main visual content or information. Content images usually have clear themes, objects, and scenes, and are the parts that are retained during the style transfer process. Content images provide the structural and semantic information required to generate images, such as the shape, layout, and color of objects. In the style transfer algorithm, the content features of the content image will be extracted and used to guide the construction of the generated image to ensure that the generated image can visually identify objects and structures similar to the content image. Style images refer to images with unique visual styles or expressions. Style images may come from different art schools, painters, or design styles, and have distinctive features such as color, texture, and brushstrokes. Style images provide information about the target style during the style transfer process, which will be integrated into the content image to generate images with a new style.
[0056] Step 130: training the deep reinforcement learning model using the training set and the total loss function.
[0057] Among them, the total loss function includes at least a content loss function, a style loss function and a multi-level stylized contrast loss function.
[0058] Fig.12 is a schematic diagram of a process for training a deep reinforcement learning model according to some embodiments of this specification, such as Fig.12 As shown, training a deep reinforcement learning model may include:
[0059] S1. Input the current state into the actor network. The actor network observes the current state, obtains style information, establishes a stylized strategy for the moving picture in the current state, and samples the optimal decision action corresponding to the current state. The current state includes the moving picture and the style picture. The moving picture is generated by initializing the content picture.
[0060] S2, input the optimal decision action into the performer network, and synchronously input the style information generated in the actor network into the performer network through skip connections;
[0061] S3, the executor network uses the style information provided by the jump connection as a reference, executes the optimal decision action, obtains the decision execution result, and uses the decision execution result to update the moving picture to obtain a new state;
[0062] S4. Put the moving picture and style picture of the new state into the pre-trained neural network VGG19, obtain the multi-layer feature map of the moving picture of the new state and the multi-layer feature map of the style picture, and calculate the feedback reward obtained by executing the action. Specifically, obtain the respective multi-layer feature maps of the pre-trained neural network, calculate the Euclidean distance of the Gram matrix of the corresponding feature map and sum them, and the inverse of the sum is the feedback reward obtained by executing the action. The pre-trained VGG19 network is used for feature extraction to capture the high-level features of the image, and the Gram matrix is used to capture the correlation between feature maps;
[0063] S5, input the current state moving picture, the optimal decision action and the feedback reward into the critic network, analyze the state features through the feature extraction block and the depth separable block, and splice them with the action in the channel dimension to obtain the fusion feature, and the action scoring block analyzes the fusion feature to obtain the maximum expected reward;
[0064] S6. The actor network adjusts the update strategy according to the maximum expected reward so that it can select the optimal stylized action at each step, thereby obtaining the maximum cumulative reward in this round of style transfer;
[0065] S7, calculating the loss through the content loss function, the style loss function and the multi-level stylized contrast loss function;
[0066] S8. Determine whether the preset training conditions are met based on the loss. If so, complete the training. If not, perform S1-S7 through gradient back propagation optimization. The preset training conditions may be loss convergence or the number of iterations reaching the maximum number of iterations.
[0067] Fig.13 is a flowchart of reward calculation according to some embodiments of this specification, such as Fig.13As shown, in some embodiments, the reward function used for feedback reward calculation is specifically: negative style loss. Using VGG19 as the feature extraction network, the outputs of the first activation function layer (ReLU1_1) in the first convolution block of VGG19, the first activation function layer (ReLU2_1) in the second convolution block, the first activation function layer (ReLU3_1) in the third convolution block, and the first activation function layer (ReLU4_1) in the fourth convolution block are selected as extracted features. The reward function is:
[0068] ,
[0069] in, For reward.
[0070] In some embodiments, the total loss function is:
[0071] ,
[0072] in, is the total loss, are the model parameters of the actor network, are the model parameters of the actor network, , , are all learnable parameters, For content loss, is the style loss, is a multi-level stylization contrast loss. Parameters , and The relative importance of content loss, style loss and multi-level stylization contrast loss in the total loss function to the overall goal is defined. These loss weights are obtained based on automatic learning. The loss weights are automatically adjusted using the homogeneous uncertainty principle, which can avoid relying solely on experience to set these hyperparameters, so that they can better adapt to different stages of training.
[0073] In some embodiments, the content loss is used to calculate the semantic similarity between the moving picture output by the performer network and the content picture, and the content loss function is:
[0074] ,
[0075] in, For the Executor Network The output of the step, For content pictures, It is the feature map of the input image output by the first activation function layer in the fourth convolutional block of the pre-trained neural network VGG19. , and They respectively represent the number of channels, height, and width of the feature map of the input image output by the first activation function layer in the fourth convolutional block of the pre-trained neural network.
[0076] In some embodiments, the style loss is used to measure the style similarity between the moving picture output by the performer network and the style picture. The style loss is defined as the squared Frobenius norm of the difference between the Gram matrix of the moving picture output by the performer network and the style picture, which is obtained by matching the mean and variance statistics. The style loss function is:
[0077] ,
[0078] in, For the Executor Network The output of the step, For style pictures, The input image extracted by the pre-trained neural network VGG19 is The feature map of the layer, is the maximum number of layers selected, represents the mean value, Indicates variance.
[0079] In some embodiments, in order to further improve the expressiveness of the model style, a multi-level stylized contrast loss is designed, and the style contrast loss is calculated for both deep features and shallow features at the same time, so as to further enhance the overall style expression ability. Specifically, the core idea of contrast loss is to bring the target sample and similar samples closer to each other (also called positive samples), and to push the target sample away from other samples (also called negative samples). Therefore, in the multi-level stylized contrast loss, the stylized result should be close to the style of its style image, and should be farther away from other styles. By strengthening the style signal in both shallow features and deep features, the stylized quality is further enhanced. Assume that Content samples and N style samples One-to-one correspondence Stylized output , the multi-level stylization contrast loss function is:
[0080] ,
[0081] in, is the number of batch samples, is the number of layers in the actor network, For the The moving pictures in the samples are in the actor network The output of the layer, For the The moving pictures in the samples are in the actor network The output of the layer, For the The style images in samples are output at the kth layer of the actor network.
[0082] Step 140: Obtain the content image to be migrated and the style image corresponding to the content image to be migrated.
[0083] Step 150 , using the trained deep reinforcement learning model, based on the content image to be transferred and the style image corresponding to the content image to be transferred, generates a style-transferred image corresponding to the content image to be transferred.
[0084] In order to further illustrate the beneficial effects of a method for style transfer of arbitrary images based on deep reinforcement learning, existing style transfer methods are selected as baseline methods for comparison, including MicroAST and AesFA. Among them, MicroAST adopts a structure of two encoders (content encoder and style encoder) and a decoder, and designs a style modulator to improve the stylization effect; while AesFA better distinguishes the aesthetic style from the reference image by frequency decomposing the image, thereby enhancing the stylization effect. The experiment includes qualitative and quantitative comparisons, where the quantitative comparison includes indicators such as content loss, style loss, structural similarity (SSIM) and parameter amount. The smaller the content loss, the better the semantic information is preserved; the higher the SSIM, the clearer the structure of the style transfer result; the smaller the style loss, the richer the style pattern; the fewer the number of parameters, the lighter the model. This experiment uses style dataset A and content dataset B for training. During testing, 70 content images are randomly selected as the content test set and 50 style images are randomly selected as the style test set. Among them, the size of the training image and the test image is 256×256. Table 1 shows the quantitative comparison results of the effects of a method for style transfer of arbitrary images based on deep reinforcement learning and all baseline methods. From the comparison results in Table 1, it can be seen that AesFA achieves the lowest style loss, but this is at the expense of more semantic information to obtain a higher stylization effect. In terms of other indicators, a method for style transfer of arbitrary images based on deep reinforcement learning is better than the baseline method. In addition, in all evaluation indicators, a method for style transfer of arbitrary images based on deep reinforcement learning shows better results than the baseline method MicroAST at the 9th step. Table 2 shows the quantitative comparison of performance. A method for style transfer of arbitrary images based on deep reinforcement learning has fewer parameters and realizes the control of the degree of stylization.
[0085] Table 1
[0086]
[0087] Table 2
[0088]
[0089] Fig.14 is a module diagram of a system for transferring the style of any image based on deep reinforcement learning according to some embodiments of this specification, such as Fig.14 As shown, a system for transferring the style of any image based on deep reinforcement learning may include a model building module, a sample building module, a model training module, an image acquisition module and a style transfer module.
[0090] The model building module can be used to build a deep reinforcement learning model, wherein the deep reinforcement learning model at least includes an actor network, a critic network, and an executor network;
[0091] The sample building module can be used to build a training set;
[0092] The model training module can be used to train the deep reinforcement learning model through the training set and the total loss function, wherein the total loss function at least includes a content loss function, a style loss function and a multi-level stylized contrast loss function;
[0093] The image acquisition module can be used to obtain the content image to be migrated and the style image corresponding to the content image to be migrated;
[0094] The style transfer module can be used to generate a style-transferred image corresponding to the content image to be transferred based on the content image to be transferred and the style image corresponding to the content image to be transferred through a trained deep reinforcement learning model.
[0095] A system for transferring the style of any image based on deep reinforcement learning can be used to execute a method for transferring the style of any image based on deep reinforcement learning. For more description of the system for transferring the style of any image based on deep reinforcement learning, please refer to the relevant description of the method for transferring the style of any image based on deep reinforcement learning, which will not be repeated here.
[0096] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.
Claims
1. A method for transferring the style of any image based on deep reinforcement learning, characterized in that: include: Establishing a deep reinforcement learning model, wherein the deep reinforcement learning model at least includes an actor network, a critic network, and an executor network; Create a training set; Training the deep reinforcement learning model through a training set and a total loss function, wherein the total loss function includes at least a content loss function, a style loss function, and a multi-level stylized contrast loss function; Obtaining a content image to be migrated and a style image corresponding to the content image to be migrated; Generate a style-transferred picture corresponding to the content picture to be transferred based on the content picture to be transferred and the style picture corresponding to the content picture to be transferred by the trained deep reinforcement learning model; The deep reinforcement learning model is trained by using a training set and a total loss function, including: S1. Input the current state into the actor network, the actor network observes the current state, obtains style information, establishes a stylization strategy for the moving picture in the current state, and samples to obtain the optimal decision action corresponding to the current state, wherein the current state includes the moving picture and the style picture, and the moving picture is generated by initialization of the content picture; S2, inputting the optimal decision action into the performer network, and synchronously inputting the style information generated in the actor network into the performer network through a skip connection; S3, the executor network uses the style information provided by the jump connection as a reference, executes the optimal decision action, obtains the decision execution result, and uses the decision execution result to update the moving picture to obtain a new state; S4, put the moving picture and style picture of the new state into the pre-trained neural network VGG19, obtain the multi-layer feature map of the moving picture of the new state and the multi-layer feature map of the style picture, and calculate the feedback reward obtained by executing the action; S5, input the current state moving picture, the optimal decision action and the feedback reward into the critic network, analyze the state features, and concatenate them with the action in the channel dimension to obtain the fusion features, and analyze the fusion features to obtain the maximum expected reward; S6, the actor network adjusts the update strategy according to the maximum expected reward; S7, calculating the loss through the content loss function, the style loss function and the multi-level stylized contrast loss function; S8, judging whether the preset training conditions are met according to the loss, if so, completing the training, if not, performing S1-S7 through gradient back propagation optimization; The total loss function is: , in, is the total loss, are the model parameters of the actor network, are the model parameters of the actor network, , , are all learnable parameters, For content loss, is the style loss, It is a multi-level stylized contrast loss; The content loss function is: , in, For the executor network The output of the step, For content pictures, It is the feature map of the input image output by the first activation function layer in the fourth convolutional block of the pre-trained neural network VGG19. , and They respectively represent the number of channels, height, and width of the feature map of the input image output by the first activation function layer in the fourth convolutional block of the pre-trained neural network VGG19; The style loss function is: , in, For the executor network The output of the step, For style pictures, The input image extracted by the pre-trained neural network VGG19 is The feature map of the layer, is the maximum number of layers selected, represents the mean value, Indicates the variance; The multi-level stylization contrast loss function is: , in, is the number of batch samples, is the number of layers in the actor network, For the The moving pictures in the samples are in the actor network The output of the layer, For the The moving pictures in the samples are in the actor network The output of the layer, For the The style images from samples are output at the kth layer of the actor network.
2. The method for transferring the style of any image based on deep reinforcement learning according to claim 1, characterized in that: The actor network includes three feature extraction blocks, two depth-separable blocks, two residual blocks, three style signal modulation blocks, and a distribution calculation block; The feature extraction block includes a reflection filling layer, a convolution layer and an activation layer; The depth separable block includes a reflection filling layer, a grouped convolution layer, a 1x1 convolution layer and two activation layers, wherein the grouped convolution layer is used to extract spatial features, and the 1x1 convolution layer is used to extract channel features; The residual block includes two reflection padding layers, two convolutional layers and an activation layer; The style signal modulation block includes a weight branch and a bias branch, wherein the weight branch is used to calculate the weight expressed by the current style, and the bias branch is used to calculate the bias expressed by the current style; The distribution calculation block includes a reflection filling layer, a group convolution layer and two 1x1 convolution layers, wherein one of the 1x1 convolution layers is used to calculate the mean, and the other 1x1 convolution layer is used to calculate the variance.
3. The method for transferring the style of any image based on deep reinforcement learning according to claim 1, characterized in that: The performer network consists of two modulated residual blocks, two modulated upsampling blocks, a feature extraction block and two feature fusion blocks; The modulated residual block includes three reflection padding layers, three convolutional layers and two activation layers; The modulatable upsampling block includes an upsampling layer, two reflection padding layers, a grouped convolution layer, a 1x1 convolution layer and two activation layers; The feature extraction block includes a reflection filling layer, a convolution layer and an activation layer.
4. The method for transferring the style of any image based on deep reinforcement learning according to claim 1, characterized in that: The critic network consists of a feature extraction block, two depth-wise separable blocks, and an action scoring block; The feature extraction block includes a reflection filling layer, a convolution layer and an activation layer; The depth-wise separable block includes a reflection padding layer, a grouped convolutional layer, a 1x1 convolutional layer, and two activation layers; The action scoring block includes a feature extraction block and a 1x1 convolutional layer.
5. A system for transferring the style of any image based on deep reinforcement learning, characterized in that: include: A model building module, used to build a deep reinforcement learning model, wherein the deep reinforcement learning model at least includes an actor network, a critic network and an executor network; Sample building module, used to build training sets; A model training module, used to train the deep reinforcement learning model through a training set and a total loss function, wherein the total loss function at least includes a content loss function, a style loss function, and a multi-level stylized contrast loss function; An image acquisition module, used to acquire a content image to be migrated and a style image corresponding to the content image to be migrated; A style transfer module, configured to generate a style-transferred image corresponding to the content image to be transferred based on the content image to be transferred and the style image corresponding to the content image to be transferred, through the trained deep reinforcement learning model; The deep reinforcement learning model is trained by using a training set and a total loss function, including: S1. Input the current state into the actor network, the actor network observes the current state, obtains style information, establishes a stylization strategy for the moving picture in the current state, and samples to obtain the optimal decision action corresponding to the current state, wherein the current state includes the moving picture and the style picture, and the moving picture is generated by initialization of the content picture; S2, inputting the optimal decision action into the performer network, and synchronously inputting the style information generated in the actor network into the performer network through a skip connection; S3, the executor network uses the style information provided by the jump connection as a reference, executes the optimal decision action, obtains the decision execution result, and uses the decision execution result to update the moving picture to obtain a new state; S4, put the moving picture and style picture of the new state into the pre-trained neural network VGG19, obtain the multi-layer feature map of the moving picture of the new state and the multi-layer feature map of the style picture, and calculate the feedback reward obtained by executing the action; S5, input the current state moving picture, the optimal decision action and the feedback reward into the critic network, analyze the state features, and concatenate them with the action in the channel dimension to obtain the fusion features, and analyze the fusion features to obtain the maximum expected reward; S6, the actor network adjusts the update strategy according to the maximum expected reward; S7, calculating the loss through the content loss function, the style loss function and the multi-level stylized contrast loss function; S8, judging whether the preset training conditions are met according to the loss, if so, completing the training, if not, performing S1-S7 through gradient back propagation optimization; The total loss function is: , in, is the total loss, are the model parameters of the actor network, are the model parameters of the actor network, , , are all learnable parameters, For content loss, is the style loss, It is a multi-level stylized contrast loss; The content loss function is: , in, For the executor network The output of the step, For content pictures, It is the feature map of the input image output by the first activation function layer in the fourth convolutional block of the pre-trained neural network VGG19. , and They respectively represent the number of channels, height, and width of the feature map of the input image output by the first activation function layer in the fourth convolutional block of the pre-trained neural network VGG19; The style loss function is: , in, For the executor network The output of the step, For style pictures, The input image extracted by the pre-trained neural network VGG19 is The feature map of the layer, is the maximum number of layers selected, represents the mean value, Indicates the variance; The multi-level stylization contrast loss function is: , in, is the number of batch samples, is the number of layers in the actor network, For the The moving pictures in the samples are in the actor network The output of the layer, For the The moving pictures in the samples are in the actor network The output of the layer, For the The style images in samples are output at the kth layer of the actor network.
