Image style gradual transfer method based on deep reinforcement learning

Abstract

The present invention relates to a method for gradual image style transfer based on deep reinforcement learning. The method splits the one-step style transfer process into a step-by-step style transfer process. A prepared content image and style image are placed in an environment. The initial state of the environment is set to the content image. The style image is used to calculate a feedback reward. The network used to calculate the feedback reward in the environment uses a pre-trained VGG model. The feedback reward is the inverse of the Gram matrix distance between the state and the multi-layer feature map extracted by the VGG model. Deep reinforcement learning is used to control the increase in the degree of stylization, retaining more details and structure of the content image in the early steps of style transfer, and synthesizing more style patterns in subsequent steps. In addition, the model adopted by the present invention is lightweight, has few parameters, and has low computational complexity. It is a style transfer method that allows users to easily control the degree of stylization, avoiding the tedious manual setting of hyperparameters to adjust the degree of stylization.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for gradual image style transfer based on deep reinforcement learning. Background Art

[0002] Style transfer aims to preserve the original content while presenting an image with artistic characteristics guided by a style reference. Gatys et al. discovered that different layers in a CNN can be used to extract image content structure and style texture information. They proposed an optimization-based iterative method for generating stylized images, using a pretrained VGG network to extract features. While this optimization-based method can achieve relatively satisfactory results, the significant time overhead of each style transfer iteration has prevented its industrial application. Johnson et al. employed an end-to-end model to achieve real-time style transfer of a specific style. The model reconstructs the stylized image by inputting the content image into a convolutional network for feature reconstruction. This end-to-end model speeds up style transfer by 1000 times, significantly increasing its practical application. However, the model must be retrained for each new style, significantly increasing the cost of practical application. Chen et al. proposed a more efficient style transfer model that combines multiple style models into a single model and achieves excellent stylized results. The convolutional network is divided into two branches: a content learning branch without a style convolutional block and a style learning branch with a style convolutional block. This method improves the expressive power of the network model to a certain extent, but it still has the limitation of limited style types.

[0003] Furthermore, arbitrary style transfer has garnered increasing attention in recent years. Huang et al. proposed Adaptive Instance Normalization (AdaIN) to achieve arbitrary style transfer. However, because AdaIN only uses first-order statistics, many details are not reflected in the stylized results. Liu et al. proposed a new attention normalization module, called Adaptive Attention Normalization (AdaAttN), which adaptively normalizes each point. An et al. proposed a method to prevent content leakage in general style transfer, consisting of a reversible neural flow and an unbiased feature transfer module (AdaIN / WCT). It supports both forward and backward inference and operates in a mapping-transfer-regression scheme. Forward inference projects the input image into deep features, while backward inference maps the deep features back to the input image in a lossless and unbiased manner. While this method addresses the content leakage issue in style transfer, it also results in a single stylized result, lacking stylistic diversity. Deng et al. introduced Transformer into style transfer, taking the long-term dependencies of the input images into account in unbiased style transfer, and to some extent alleviated the problem of content leakage, but it also brought expensive computational costs.

[0004] Current neural style transfer methods focus on achieving transfer to multiple styles. Despite this, more flexible control over the degree of stylization remains a challenge. Iteratively generating stylized images is an intuitive solution for enhancing stylization, but it can lead to content leakage, where image content is not preserved after several rounds of stylization. In contrast, our stepwise style transfer method can produce varying degrees of stylization without adjusting hyperparameters. Summary of the Invention

[0005] To address the shortcomings of existing technologies, a reinforcement-based image style distribution transfer method is proposed. The method includes: using deep reinforcement learning to solve the neural network style transfer problem, decomposing the one-step style transfer problem into a step-by-step style transfer problem, thereby enabling more flexible control of the stylization degree and reducing the number of network model parameters. The step-by-step transfer method proposes a step-by-step style transfer method, adopts a deep reinforcement learning framework trained by the actor-actor-critic method, uses reinforcement learning to control feature extraction, and adopts a lightweight network design to improve inference speed. Specifically, the method includes:

[0006] Step 1: Prepare the required content images and style images;

[0007] Step 2: Place the content image and the style image into the established environment. The initial state is set to the content image. A multi-layer network is used to approximate the policy function. The actor network samples the action of the time step according to the current state, and the actor network obtains the corresponding feedback reward by executing this action.

[0008] The neural network used mainly consists of three parts: actor network, performer network and critic network. The actor network consists of three convolutional modules and one residual module. Each convolutional module consists of a base module and an activation layer. The residual module consists of two base modules and an activation layer. The base module consists of a padding layer, a convolutional layer and a regularization layer.

[0009] The executor network consists of three upsampling modules. Each upsampling module consists of an upsampling layer, a basic module, and an activation layer. The input of each upsampling module is the output of the previous module and the feature map provided by the skip connection, which is concatenated channel by channel.

[0010] The critic network consists of one basic module 2, five basic modules 3, and one basic module 4. Basic module 2 consists of one convolutional layer and one activation layer, basic module 3 consists of one convolutional layer, one regularization layer, and one activation layer, and basic module 4 consists of one convolutional layer and one linear layer. The output of the convolutional layer is deformed by tensor expansion and then input into the linear layer.

[0011] Step 21: Input the current state into the actor network. The actor network observes the current state, establishes the distribution of stylized strategies corresponding to the current state, and samples the optimal decision action corresponding to the current state.

[0012] Step 22: Input the optimal decision action obtained in step 21 into the actor network, and input the output feature map of each convolutional module in the actor network into the actor network through a skip connection. The skip connection fuses the bottom feature map with the deep feature map by splicing, so that the model can simultaneously utilize the bottom position information and the deep semantic information;

[0013] The actor network uses the current state information provided by the jump connection as a reference, executes the current optimal decision action provided by the actor network to obtain the execution result of the decision, and uses the execution result of the decision to update the state in the environment to obtain a new state;

[0014] Step 23: Put the new state and style images into the pre-trained VGG network to obtain their respective multi-layer feature maps. Calculate the Euclidean distance of the Gram matrix of the corresponding feature maps and sum them. The inverse of the sum is the feedback reward obtained by executing the action.

[0015] Using a pre-trained network for feature extraction can capture the high-level features of the image, and the Gram matrix can capture the correlation between feature maps;

[0016] Step 24: The current state, the optimal decision action obtained based on the state, and the feedback reward are input into the critic network. The critic network uses the first two modules to analyze and extract features of the current state, and then concatenates the features with the action taken in the channel dimension to obtain a fusion feature of the state and action. The subsequent modules then analyze the fusion feature to obtain the maximum expected reward for executing the decision action in the current state.

[0017] Step 25: The actor network adjusts its update strategy based on the maximum expected reward, so that it can select the optimal stylized action at each step, thereby obtaining the maximum cumulative reward in this round of style transfer;

[0018] Step 3: Repeat the above steps until the maximum number of steps is reached, ending this round of style transfer.

[0019] Step 4: Finally, stylized result images with different degrees of stylization are obtained.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. This invention uses deep reinforcement learning to perform step-by-step style transfer, breaking down the existing one-step style transfer method into a step-by-step transfer. This allows for more flexible control of the degree of stylization, and a more refined balance between content and style at each step. This ensures that the degree of stylization increases gradually with the number of steps while preserving content information to the greatest extent possible. Different levels of stylization can be directly obtained through the steps.

[0022] 2. Since the one-step style transfer method is decomposed into multiple steps, the network module used in the present invention is more lightweight, which is a more lightweight style transfer model. Compared with the existing one-step style transfer network, the network of the present invention has smaller parameters, higher computational efficiency, and faster inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the overall flow chart of the step-by-step style transfer of the present invention;

[0024] Figure 2 It is a structural diagram of a neural network used in the present invention;

[0025] Figure 3 It is a gradual migration effect diagram of the present invention in two different styles. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0027] The following is a detailed description with reference to the accompanying drawings.

[0028] To address the traditional content leakage problem and the problem of manually setting hyperparameters to adjust the degree of stylization, deep reinforcement learning is introduced to automatically control stylization, splitting the one-step style transfer problem into a step-by-step style transfer problem. The trade-off between content and style in each step is more refined, ensuring that the degree of stylization gradually increases with the number of steps while retaining content information to the greatest extent. Results of different degrees of stylization can be obtained directly through the steps, and the model is lightweight.

[0029] At the same time, the degree of stylization can be controlled more flexibly, and the number of network model parameters is small. The step-by-step transfer method proposes a step-by-step style transfer method, which adopts a deep reinforcement learning framework trained by the actor-executor-critic method, uses reinforcement learning to control feature extraction, and adopts a lightweight network design to improve the inference speed. Figure 1 This is the overall flow chart of the gradual style transfer of the present invention; Figure 1 As shown, the method of the present invention specifically includes:

[0030] Step 1: Prepare the required content images and style images.

[0031] Step 2: Place the content image and the style image into the established environment. The initial state is set to the content image. A multi-layer network is used to approximate the policy function. The actor network samples the action of the time step according to the current state, and the actor network obtains the corresponding feedback reward by executing this action.

[0032] Figure 2 This is a structural diagram of a neural network used in the present invention; Figure 2 As shown in the figure, the neural network used mainly includes three parts: actor network, executor network and critic network, among which the actor network consists of three convolution modules and one residual module, each convolution module consists of a basic module and an activation layer, the residual module consists of two basic modules and an activation layer, and the basic module consists of a padding layer, a convolution layer and a regularization layer.

[0033] The executor network consists of three upsampling modules, each of which consists of an upsampling layer, a basic module and an activation layer. The input of each upsampling module is the output of the previous module and the feature map provided by the skip connection, which is spliced ​​channel by channel.

[0034] The critic network consists of one basic module two, five basic modules three and one basic module four. Basic module two consists of one convolutional layer and one activation layer, basic module three consists of one convolutional layer, one regularization layer and one activation layer, and basic module four consists of one convolutional layer and one linear layer. The output of the convolutional layer is deformed by tensor expansion and then input into the linear layer.

[0035] Step 21: Input the current state into the actor network. The actor network observes the current state, establishes the distribution of stylized strategies corresponding to the current state, and samples the optimal decision action corresponding to the current state.

[0036] A stylized distribution is established to calculate the entropy of the stylized strategy, thereby introducing an entropy regularization term during the training process, so that the stylized strategy can maximize its performance while ensuring its exploratory nature.

[0037] Step 22: The optimal decision action obtained in step 21 is input into the actor network, and the output feature map of each convolutional module in the actor network is also input into the actor network through a jump connection. The jump connection fuses the bottom feature map with the deep feature map by splicing, so that the model can simultaneously utilize the bottom position information and the deep semantic information.

[0038] It can help the model better capture the texture and other details of the target object.

[0039] The executor network uses the current state information provided by the jump connection as a reference, executes the current optimal decision action provided by the actor network to obtain the execution result of the decision, and uses the execution result of the decision to update the state in the environment to obtain a new state.

[0040] Step 23: Put the new state and style images into the pre-trained network VGG to obtain their respective multi-layer feature maps. Calculate the Euclidean distance of the Gram matrix of the corresponding feature map and sum them. The inverse of the sum is the feedback reward obtained by executing the action.

[0041] Using a pre-trained network for feature extraction can capture the high-level features of the image, and the Gram matrix can capture the correlation between feature maps.

[0042] Step 24: The current state, the optimal decision action obtained based on the state, and the feedback reward are input into the critic network. The critic network uses the first two modules to analyze the current state and extract its features. The features are then concatenated with the actions taken in the channel dimension to obtain a fusion feature of the state and action. The subsequent modules are then used to analyze the fusion feature to obtain the maximum expected reward for executing the decision action in the current state.

[0043] Step 25: The actor network adjusts its update strategy based on the maximum expected reward so that it can select the optimal stylized action at each step, thereby obtaining the maximum cumulative reward in this round of style transfer.

[0044] In other words, at each step, the actor produces the optimal stylized action by observing the state control, obtaining the optimal step-by-step stylization result.

[0045] Step 3: Repeat steps 21 to 25 until the maximum number of steps is reached, ending this round of style transfer.

[0046] Step 4: Finally, stylized results with different degrees of stylization are obtained. A maximum of twenty images can be obtained.

[0047] In a preferred embodiment, the prepared content image and style image are placed into the environment, the initial state of the environment is set to the content image, the style image is used to calculate the feedback reward, and the network for calculating the feedback reward in the environment uses a pre-trained VGG model. The feedback reward is the inverse of the sum of the Euclidean distances of the Gram matrices of the multi-layer feature maps extracted by the VGG model for the state and style images.

[0048] The present invention includes three main neural networks: the actor network, the executor network, and the critic network. Each neural network contains or partially contains multiple modules. Specifically, the actor network contains three convolutional modules and one residual module. The convolutional module consists of a basic module and a ReLU activation layer. The residual module consists of two basic modules and a ReLU activation layer. The basic module consists of a mirror padding layer, a convolutional layer, and an instance normalization layer. The input of the actor network is the state in the current environment, and the output of the actor network is the decision action obtained from the state.

[0049] The actor network consists of three upsampling modules. Each upsampling module consists of a 2x nearest neighbor interpolation upsampling layer, a base module, and a ReLU activation function. Each upsampling module receives as input the output of the previous module and a channel-wise fusion of feature maps obtained via skip connections. The actor network outputs the decision execution result. The critic module consists of two parts. The first part, consisting of a base module 2 and a base module 3, extracts state features. The second part, consisting of four base modules 3 and a given base module 4, estimates the maximum expected reward for the state and action. The first part takes the current state as input, while the second part takes the state feature map output by the first part and the channel-wise fusion of the optimal decision action. In one step of a style transfer cycle, the actor observes the state of the current environment, calculates the Gaussian distribution of decisions based on the state, and samples the optimal decision action. The actor then executes the decision action based on the state information obtained via skip connections, obtains the decision execution result, and uses the decision execution result to update the state of the environment.

[0050] The critic evaluates the decision-making actions taken with reference to the current state and obtains the maximum expected reward for executing the decision-making actions in the current state; the actor adjusts its own strategy based on the maximum expected reward calculated by the critic so that it can obtain the maximum cumulative reward in one round; the critic optimizes and updates its own parameters based on the difference between the maximum expected reward for executing the current optimal decision-making action in the current state and the maximum expected reward for executing the optimal decision-making action in the next state plus the current feedback reward; the parameters of the executor are updated through deep learning, and the learning of the actor is assisted and supervised. Specifically, the perceptual loss between the execution result of the decision and the content image and style image is calculated to update the executor and supervise the actor's learning.

[0051] Figure 3 This is a diagram showing the gradual migration effect of the network of the present invention in two different styles. Figure 3 (a) is the content image, Figure 3 (b) is the style picture, Figure 3 (c) is the first step of migration effect diagram, Figure 3 (d) is the effect diagram of the third step migration. Figure 3 (e) is the fifth step migration effect diagram, Figure 3 (f) is the seventh step migration effect diagram, Figure 3 (g) is the effect diagram of the tenth step migration. Figure 3 (h) is the twentieth step migration effect diagram. Figure 3 As can be seen from the figure, as the number of steps increases, the style of the content image transfer is closer to the style of the style image.

[0052] It should be noted that the above-described specific embodiments are illustrative only. Those skilled in the art may devise various solutions based on the disclosure of the present invention, and such solutions fall within the scope of the present invention and are intended to be protected by the present invention. Those skilled in the art should understand that the present description and its accompanying drawings are intended to be illustrative only and are not intended to limit the scope of the claims. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. A method for gradual image style transfer based on deep reinforcement learning, characterized by: Deep reinforcement learning is used to solve the problem of neural network style transfer. The one-step style transfer problem is decomposed into a step-by-step style transfer problem, which allows for more flexible control of the degree of stylization and reduces the number of network model parameters. The step-by-step transfer method proposes a step-by-step style transfer method. The deep reinforcement learning framework used is trained using the actor-actor-critic method. Reinforcement learning is used to control feature extraction. A lightweight network design is used to improve inference speed. Specifically, the following features are included: Step 1: Prepare the required content images and style images; Step 2: Place the content image and the style image into the established environment. The initial state is set to the content image. A multi-layer network is used to approximate the policy function. The actor network samples the action of the time step according to the current state, and the actor network obtains the corresponding feedback reward by executing this action. The neural network used mainly consists of three parts: actor network, performer network and critic network. The actor network consists of three convolutional modules and one residual module. Each convolutional module consists of a base module and an activation layer. The residual module consists of two base modules and an activation layer. The base module consists of a padding layer, a convolutional layer and a regularization layer. The executor network consists of three upsampling modules. Each upsampling module consists of an upsampling layer, a basic module, and an activation layer. The input of each upsampling module is the output of the previous module and the feature map provided by the skip connection, which is concatenated channel by channel. The critic network consists of one basic module 2, five basic modules 3, and one basic module 4. Basic module 2 consists of one convolutional layer and one activation layer, basic module 3 consists of one convolutional layer, one regularization layer, and one activation layer, and basic module 4 consists of one convolutional layer and one linear layer. The output of the convolutional layer is deformed by tensor expansion and then input into the linear layer. Step 21: Input the current state into the actor network. The actor network observes the current state, establishes the distribution of stylized strategies corresponding to the current state, and samples the optimal decision action corresponding to the current state. Step 22: Input the optimal decision action obtained in step 21 into the actor network, and input the output feature map of each convolutional module in the actor network into the actor network through a skip connection. The skip connection fuses the bottom feature map with the deep feature map by splicing, so that the model can simultaneously utilize the bottom position information and the deep semantic information; The actor network uses the current state information provided by the jump connection as a reference, executes the current optimal decision action provided by the actor network to obtain the execution result of the decision, and uses the execution result of the decision to update the state in the environment to obtain a new state; Step 23: Put the new state and style images into the pre-trained VGG network to obtain their respective multi-layer feature maps. Calculate the Euclidean distance of the Gram matrix of the corresponding feature maps and sum them. The inverse of the sum is the feedback reward obtained by executing the action. Using a pre-trained network for feature extraction can capture the high-level features of the image, and the Gram matrix can capture the correlation between feature maps; Step 24: The current state, the optimal decision action obtained based on the state, and the feedback reward are input into the critic network. The critic network uses the first two modules to analyze and extract features of the current state, and then concatenates the features with the action taken in the channel dimension to obtain a fusion feature of the state and action. The subsequent modules then analyze the fusion feature to obtain the maximum expected reward for executing the decision action in the current state. Step 25: The actor network adjusts its update strategy based on the maximum expected reward, so that it can select the optimal stylized action at each step, thereby obtaining the maximum cumulative reward in this round of style transfer; Step 3: Repeat steps 21 to 25 until the maximum number of steps is reached, ending this round of style transfer. Step 4: Finally, stylized result images with different degrees of stylization are obtained.

Images