Double-branch aesthetic image processing method based on active reinforcement learning

Abstract

The invention discloses a double-branch aesthetic image processing method based on active reinforcement learning. Firstly, a unified Markov decision process framework is constructed, two subtasks of color enhancement and composition optimization are processed in parallel, and a reinforcement learning training data set is constructed based on images. Thirdly, initializing a double-branch strategy network, a value network and a pixel-level agent system, providing pixel-level and channel-level instant feedback rewards through a pre-trained aesthetic model, and updating network parameters in combination with human subjective preference constraints; and after training is completed, a high-quality image sequence after progressive optimization is output. According to the invention, the pixels are defined as intelligent agents, so that the motion space and fineness of adjustment are expanded; and meanwhile, an aesthetic evaluation model is taken as a reward core, so that the result is consistent with expert modification in objective indexes and subjective scores, and personalized and progressive aesthetic modification requirements are met.

Description

Technical Field

[0001] This invention relates to the field of automatic aesthetic image retouching, and mainly to a two-branch aesthetic image processing method based on active reinforcement learning. Background Technology

[0002] In the field of digital image processing, improving image visual quality is a fundamental and crucial task. Limited by shooting hardware, environmental conditions, and user skill levels, raw images often suffer from various quality issues such as insufficient brightness and saturation, flat color and contrast, and poor composition. These problems are particularly pronounced in outdoor scenes and environments with rich colors. Although existing technologies have made progress in areas such as brightness enhancement, user expectations for images have shifted from basic clarity to a comprehensive visual experience that incorporates higher aesthetic standards. Currently, learning-based image retouching methods primarily focus on improving image quality through end-to-end mapping between the input and target images. Some methods utilize generative adversarial networks to treat image enhancement as an image-to-image transformation problem. However, these methods often produce images with loss of detail and the introduction of artifacts, limiting their practical application. Other methods employ convolutional neural networks, directly mapping the input to the enhanced output. While these models optimize visual quality through complex architectures and carefully designed loss functions, they often lack process interpretability and fail to adequately incorporate and optimize the crucial subjective factor of image aesthetics.

[0003] The aesthetic quality of an image, including color, contrast, saturation, and composition, is central to its visual appeal. Professional image retouching is often a gradual, iterative optimization process based on aesthetic principles. However, existing automated methods often overlook this gradual nature and human subjective aesthetic preferences, resulting in suboptimal visual effects. In color enhancement, existing methods often perform global adjustments, making it difficult to precisely handle contrast, color, and brightness issues in local areas. For example, while the ALL-E method significantly improves overall brightness, its enhancement of color saturation and contrast is limited, leading to unnatural and less vibrant results. Other methods, such as the PixelRL method, lack effective aesthetic guidance, and their color adjustments may produce artifacts or low-quality visual results. Regarding composition optimization, existing techniques often rephrase the problem as image cropping, aiming to find well-composed subviews within the original image. However, obtaining high-quality cropping results typically requires extensive expertise in photographic composition. Some methods attempt to train cropping networks by jointly modeling explicit photographic rules (such as the rule of thirds and centrality), but the abstract nature of compositional rules and the subjectivity of aesthetics make such models inadequate in practice. Other methods generate candidate regions through dense sliding windows and use aesthetic evaluation networks for sorting and filtering, but this method is computationally expensive and has limited aspect ratios to explore, making it difficult to locate the truly optimal composition region. Summary of the Invention

[0004] Objective: To address the problems existing in the aforementioned background technology, this invention provides a two-branch aesthetic image processing method based on active reinforcement learning. Due to the characteristics of color enhancement and composition optimization tasks, this invention constructs a two-branch aesthetic image retouching network based on active reinforcement learning, processing the two sub-tasks of color enhancement and composition optimization in parallel. In the color enhancement branch, this invention expands the action space, including not only image brightness adjustment actions but also contrast adjustment, saturation adjustment, and RGB three-channel adjustment actions. The enhancement operation is performed iteratively, treating the task process as a Markov decision process, learning an optimal strategy to maximize the aesthetic reward of all pixels and obtain the optimal result. In the composition optimization branch, this invention designs a set of candidate box shrinking adjustment action spaces. Initially, the candidate boxes are set to the same size as the input image. In subsequent iterative optimization processes, the agent can perform shrinking actions on the four sides of the candidate boxes. Each action receives a corresponding aesthetic reward, and ultimately, an optimal action space is learned to maximize the aesthetic reward, obtaining a candidate view with optimal composition.

[0005] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0006] A two-branch aesthetic image processing method based on active reinforcement learning includes the following steps:

[0007] Step S1: Collect images with different color defects and cropping imbalances, and construct a training dataset for the reinforcement learning system based on the images;

[0008] Step S2: Initialize the training dataset, policy network, and value network in the dual-branch reinforcement learning system;

[0009] Step S3: Update the policy network and value network of the color enhancement and composition optimization branches based on two aesthetic evaluation networks with different preferences;

[0010] Step S4: The model training is complete when all samples have been trained and all training iterations have been completed.

[0011] Step S5: Output the result after aesthetic enhancement of the image with color defects and cropping imbalance.

[0012] Furthermore, the specific method for initializing the policy network and value network in the dual-branch reinforcement learning system in step S2 includes:

[0013] For the color enhancement branch, use the current state. As input to the color enhancement policy network and the color enhancement value network, Represents time step The state at that time; the output of the color enhancement strategy network is Time step action set strategy set The output value of the color-enhanced value network is , representing from the current state The expected total reward. Similarly, for the graph optimization branch, the output of the graph optimization policy network is Time step to take action strategy The output value of the graph optimization value network is , representing the expected total reward for the current state.

[0014] The color enhancement action space and the composition optimization action space are independent. The color enhancement action space includes: contrast adjustment, saturation adjustment, brightness adjustment, and RGB three-channel adjustment. In addition, a NULL action is set to avoid over-adjustment. The composition optimization action space is set to a combination of indentation of the four boundaries of the candidate box plus the NULL action, with a total of 16 actions, ensuring that all positions in the image can be traversed without over-adjustment.

[0015] Furthermore, the specific steps for updating the policy network and value network in step S3 include:

[0016] Step S3.1: Train the training dataset based on historical stage images to obtain the following environmental reward values:

[0017]

[0018] in Discount factor The Power of 1 represent Environmental reward value at any given moment;

[0019] Step S3.2: Train the training dataset based on historical stage images to obtain the value network output value;

[0020] Step S3.3: Update the value network based on the environmental reward value and the value network output value:

[0021]

[0022] in Representative value network parameters;

[0023] Step S3.4: Update the policy network based on the environmental reward value and the predicted value:

[0024]

[0025] in Representative policy network parameters.

[0026] Furthermore, the environmental reward value in step S3.1 considers the following influencing factors:

[0027] (1) For the color enhancement branch, define the image in The instant reward that can be obtained in a time step is .in Rewards based on aesthetic perception It is composed of image feature rewards.

[0028]

[0029] in and These are hyperparameters and aesthetic perception rewards, respectively. The probability distribution is calculated from the output of the color aesthetics evaluation network.

[0030]

[0031] in Indicates the range of aesthetic scores. This represents the probability of each rating level. To preserve image features during the enhancement process pixel-by-pixel, an image feature reward is defined. The calculation process is as follows.

[0032]

[0033] in and This represents the width and height of the input image. It indicates the position of each pixel in the image.

[0034] (2) For the composition optimization branch, define the image at time step The instant rewards that can be obtained are .in It consists of an aesthetic score difference reward and an aspect ratio penalty reward, which are used to quantify the performance of cropping actions in terms of visual composition quality and geometric rationality, respectively.

[0035] The aesthetic score difference reward is implemented based on a pre-set composition aesthetic evaluation network, which quantifies the composition aesthetic quality of the input image and outputs a composition aesthetic score value between 0 and 1, denoted as .

[0036] Specifically, in the current state, the agent performs a cropping action, generates the corresponding cropping box, and obtains the cropped image accordingly. The system calculates the state and its corresponding compositional aesthetic score. and The difference between the two is used as a reward for the aesthetic score difference, that is:

[0037]

[0038] With the above settings, the agent receives a positive reward when the aesthetic score of the cropped image is higher than the previous state; conversely, it receives a negative reward if the score decreases. Thus, the agent is incentivized during training to adjust its cropping strategy in a way that improves the aesthetic quality of the composition.

[0039] To further constrain the geometric rationality of the cropping frame, this embodiment also introduces an aspect ratio penalty mechanism. When the aspect ratio of the cropping area deviates from a reasonable range, even if the composition itself has a high aesthetic score, it may lead to a sense of disharmony in the final image. Therefore, this system sets a preset aspect ratio threshold range to determine whether the geometric shape of the cropping frame is acceptable.

[0040] Specifically, let the aspect ratio of the cropped image window be... (defined as the ratio of width to height), if it is less than the first threshold (e.g., 0.5) or greater than the second threshold (e.g., 2.0), the clipping box is determined to be in a geometrically abnormal state, and the system will apply a negative penalty reward to the agent, denoted as ; otherwise, no additional penalty is applied.

[0041] The comprehensive aesthetic score difference reward and aspect ratio penalty reward mechanism can be described as follows:

[0042]

[0043] In this embodiment, the penalty value is a preset hyperparameter, preferably set to −1, so as to effectively suppress abnormal aspect ratio behavior during training.

[0044] Beneficial effects:

[0045] (1) This invention creatively proposes a parallel dual-branch reinforcement learning framework, which for the first time incorporates "composition optimization" as a core task parallel to "color enhancement" into the automatic image retouching process. The two branches make independent decisions based on sharing shallow features, and can simultaneously output the color-adjusted image and its optimal cropping region, realizing the synergistic optimization of color quality and composition structure, and effectively improving the overall aesthetic performance of the image.

[0046] (2) This invention designs a pixel-level multi-dimensional action space for color enhancement tasks, covering multiple adjustment dimensions such as brightness, contrast, saturation, and color cast correction, giving each pixel the ability to autonomously select adjustment actions based on local features; for composition optimization tasks, an intelligent action space based on a boundary shrinking mechanism is designed. In the initialization stage, the cropping box is set to the same size as the input image, and then independent shrinking actions are performed through the four boundaries of left, right, top, and bottom, realizing flexible search and positioning of any region of the image. On this basis, an "empty action" is specially introduced into the action space to allow the agent to maintain the current state under appropriate circumstances, effectively avoiding over-adjustment and invalid cropping, and enhancing the robustness of the strategy while ensuring the degree of freedom of search.

[0047] (3) This invention configures dedicated aesthetic evaluation networks for the color enhancement and composition optimization branches respectively: the color branch introduces a local aesthetic evaluation model that focuses on pixel-level color naturalness and saturation, while the composition branch adopts a composition quality evaluation model that focuses on overall layout and photographic principles. By generating targeted reward signals for each branch, the strategy network is accurately guided, significantly improving the quality of the retouching results in terms of subjective visual experience. Attached Figure Description

[0048] Figure 1 This is a flowchart of the dual-branch aesthetic image processing method based on active reinforcement learning provided by the present invention;

[0049] Figure 2 This is an algorithm framework diagram of the dual-branch aesthetic image processing method based on active reinforcement learning provided by the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] This invention provides a two-branch aesthetic image processing method based on active reinforcement learning. For example... Figure 2 As shown, it illustrates the core algorithm framework of the dual-branch aesthetic image processing system of this invention. This framework comprises two parallel branches:

[0052] Color enhancement branch: The color enhancement agent uses a color enhancement strategy based on the current image state. Select a color adjustment action After performing this action, the Color Aesthetic Evaluation Network (TANet) calculates the color reward. .

[0053] Composition optimization branch: The composition optimization agent, based on the current image state, uses a composition optimization strategy. Select a composition cropping action After this action is performed, the Composition Aesthetics Evaluation Network (VILA) calculates the composition reward. .

[0054] Actions of two branches and Together they act on the current state Update it to the next state. The process iterates continuously until the final output is an aesthetically enhanced image. The two agents share underlying image features, but have independent policies and value networks for decision-making.

[0055] Based on the above framework, this invention provides a two-branch aesthetic image processing method based on active reinforcement learning, the specific principle of which is as follows: Figure 1 As shown, it includes the following steps:

[0056] Step S1: Collect images with different color defects and cropping imbalances, and construct a training dataset for the reinforcement learning system based on the images.

[0057] This step constructs the training dataset used by the reinforcement learning system. First, 4500 original images are taken from the MIT-Adobe FiveK dataset, and the color ground truth is used with the Expert C retouching results to form color defect-correction image pairs. Then, 9000 images are randomly selected from the AVA aesthetic dataset in a 1:1:1 ratio of low, medium, and high ratings as composition imbalance samples, and finally merged into a color-composition joint training set. 500 images from FiveK and 900 images from AVA are reserved as corresponding test sets.

[0058] Step S2: Initialize the dual-branch reinforcement learning system.

[0059] This step initializes Figure 2 The system shown has trainable components. First, the training dataset constructed in step S1 is loaded. Next, the network parameters of the system are initialized:

[0060] Construct a shared feature encoding network to extract common features from the input image.

[0061] Initialize the color enhancement branch: Construct the color enhancement policy network And color enhances value network Both networks take features from the output of a shared feature encoding network as input. (Policy network) The output is the time step. Take action The probability distribution; the output value of the value network. Represents from state Estimated total expected reward for departure.

[0062] Initialize the graph optimization branch: Construct the graph optimization policy network And the value network of composition optimization Both networks also take the aforementioned shared features as input. Policy Network The output is the time step. Take action probability distribution Value network output value This represents the corresponding estimated total reward.

[0063] Among them, the color enhancement action space and the composition optimization action space are independent.

[0064] The color enhancement action space includes: contrast adjustment, saturation adjustment, brightness adjustment, and RGB three-channel adjustment. In addition, a NULL action is set to avoid over-adjustment.

[0065] The composition optimization action space is set to a combination of shrinking the four boundaries of the candidate bounding box plus a NULL action. By independently selecting "shrink" or "keep" on the top, bottom, left, and right boundaries, there are 16 possible actions, ensuring that the effective area in the image can be traversed without over-adjustment. During initialization, the candidate bounding box of the composition optimization branch is set to the same size as the input image.

[0066] Step S3: Based on the reward signal provided by the aesthetic evaluation network, iteratively update the policy network and value network of the two branches respectively.

[0067] This step is the core loop of model training, and it specifically includes the following sub-steps:

[0068] Step S3.1: Based on the current state The two agents select actions based on the policy network. and And execute, to obtain the new state. The system calculates immediate rewards based on status changes. The reward is calculated separately for each of the two branches:

[0069] (1) For the color enhancement branch, define the image in The immediate reward obtainable at each time step is comprised of aesthetic perception reward and image feature reward.

[0070]

[0071] in and These are hyperparameters.

[0072] The aesthetic perception reward is calculated from the probability distribution output by the color aesthetic evaluation network:

[0073]

[0074] in This indicates the range of aesthetic ratings, and represents the probability of each rating level.

[0075] To preserve image features pixel-by-pixel during the enhancement process, an image feature reward is defined and calculated as follows:

[0076]

[0077] in and This represents the width and height of the input image, and the position of each pixel in the image; this is the original input image.

[0078] (2) For the composition optimization branch, define the image at time step The immediate rewards available are comprised of an aesthetic score difference reward and an aspect ratio penalty reward, which are used to quantify the cropping action's performance in terms of visual composition quality and geometric rationality, respectively.

[0079] The aesthetic score difference reward is implemented based on a pre-set composition aesthetic evaluation network, which quantifies the composition aesthetic quality of the input image and outputs a composition aesthetic score value between 0 and 1, denoted as .

[0080] Specifically, in the current state, the agent performs a cropping action, generates the corresponding cropping box, and obtains the cropped image accordingly. The system calculates the state and its corresponding compositional aesthetic score. and The difference between the two is used as a reward for the aesthetic score difference, that is:

[0081]

[0082] With the above settings, the agent receives a positive reward when the aesthetic score of the cropped image is higher than the previous state; conversely, it receives a negative reward if the score decreases. Thus, the agent is incentivized during training to adjust its cropping strategy in a way that improves the aesthetic quality of the composition.

[0083] To further constrain the geometric rationality of the cropping frame, this embodiment also introduces an aspect ratio penalty mechanism. When the aspect ratio of the cropping area deviates from a reasonable range, even if the composition itself has a high aesthetic score, it may lead to a sense of disharmony in the final image. Therefore, this system sets a preset aspect ratio threshold range to determine whether the geometric shape of the cropping frame is acceptable.

[0084] Specifically, let the aspect ratio of the cropped image window be... (Defined as the ratio of width to height), if If the value is less than a first threshold (e.g., 0.5) or greater than a second threshold (e.g., 2.0), the clipping box is determined to be in a geometrically abnormal state, and the system will apply a negative penalty reward to the agent, denoted as . Otherwise, no additional penalty will be imposed.

[0085] The comprehensive aesthetic score difference reward and aspect ratio penalty reward mechanism can be described as follows:

[0086]

[0087] In this embodiment, the penalty value This is a preset hyperparameter, preferably set to −1, to effectively suppress abnormal aspect ratio behavior during training.

[0088] Step S3.2: Based on the immediate reward sequence, calculate the reward from the current state. Departure Discount Cumulative Return Estimate :

[0089]

[0090] in Discount factor The Power of 1 represent The environmental reward value at any given moment. This is an estimate of the future state from the value network. Simultaneously, it obtains the current state from the two branches of the value network. The output of .

[0091] Step S3.3: Update the value network based on the environmental reward value and the value network output value:

[0092]

[0093] in Representative value network parameters.

[0094] Step S3.4: Update the policy network based on the environmental reward value and the predicted value:

[0095]

[0096] in Representative policy network parameters.

[0097] The update process described above is performed through backpropagation. The gradients of the two branches work together on the shared feature encoding network, enabling it to learn feature representations useful for both color and composition tasks, thus achieving collaborative optimization between the two branches.

[0098] Step S4: Repeat step S3 for iterative training. When all training samples have been processed and the preset maximum number of training iterations has been reached, or when the model's performance on the validation set converges, the dual-branch reinforcement learning system is considered to have completed training.

[0099] Step S5: Process the input image using the trained system. The system performs one forward propagation: the two branches of the policy network output the action with the highest probability (i.e., the greedy policy). The system automatically performs these optimal color adjustments and cropping operations, ultimately outputting an aesthetically enhanced image. .

[0100] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A two-branch aesthetic image processing method based on active reinforcement learning, characterized in that, Includes the following steps: S1. Collect images with different color defects and cropping imbalances to construct the training dataset required to train the dual-branch reinforcement learning system; The dual-branch reinforcement learning system includes a color enhancement branch and a graph optimization branch that process in parallel and share underlying features. Each branch contains an independent policy network and a value network. S2. Initialize the dual-branch reinforcement learning system, load the training dataset, and initialize the color enhancement policy network and color enhancement value network for the color enhancement branch, and initialize the graph optimization policy network and graph optimization value network for the graph optimization branch. S3. Based on the reward signal provided by the aesthetic evaluation network, the policy network and value network of the two branches are iteratively updated respectively; wherein, the reward signal of the color enhancement branch is provided by the color aesthetic evaluation network, and the reward signal of the composition optimization branch is provided by the composition aesthetic evaluation network. S4. When all training samples have been processed and the preset number of iterations has been reached, the dual-branch reinforcement learning system is determined to have completed training. S5. Use the trained system to process the input image and output the aesthetically enhanced result image.

2. The method according to claim 1, characterized in that, In S2, the specific method for initializing the policy network and value network for each branch is as follows: For the color enhancement branch, the input image state at time step t... Input is fed into the color enhancement strategy network and the color enhancement value network; The color enhancement strategy network output is in the current state. Select action probabilistic strategies The color-enhanced value network output represents the current state. The estimated total expected reward that can be obtained from the departure. ; For the graph optimization branch, the state Inputs are fed into the graph optimization policy network and the graph optimization value network; the output of the graph optimization policy network is in the current state. Select clipping action probabilistic strategies The output of the graph-optimized value network represents the current state. The estimated total expected reward that can be obtained from the departure. .

3. The method according to claim 2, characterized in that, The color enhancement branch and the composition optimization branch have independently defined action spaces; The action space of the color enhancement branch includes the following action types: contrast adjustment, saturation adjustment, brightness adjustment, independent adjustment of RGB three channels, and a NULL action to avoid over-adjustment; The action space of the composition optimization branch includes the following action types: individual or combined shrinking actions on the upper, lower, left, and right boundaries of the candidate clipping box, and a NULL action to avoid over-adjustment; there are a total of 15 combinations of shrinking actions, plus the NULL action, forming a total of 16 selectable actions.

4. The method according to claim 3, characterized in that, During initialization, the composition optimization branch sets the candidate cropping box to be the same size as the input image. Subsequently, it iteratively executes the inward movement in the action space to gradually optimize the cropping area.

5. The method according to claim 1, characterized in that, In step S3, the specific process of updating the policy network and the value network includes the following sub-steps: S3.1 Calculate cumulative reward: based on the current time step From the Beginning to the Future The instant reward and status value of each step, calculated over time steps. Environmental reward estimates : ； in Discount factor The Power of 1 For the time step provided by the aesthetic evaluation network Instant rewards; S3.2 Obtain the value network output: Obtain the value network output for the current state. Value estimation output ,in For value network parameters; S3.3, Update the value network: Compute the environmental reward estimate With value network output The differences are analyzed, and the value network parameters are updated using gradient descent based on these differences. ; S3.4 Update the policy network: Based on the differences and the logarithmic probability of the selected action output by the policy network in the current state, update the policy network parameters using the gradient ascent method. This increases the probability of actions that yield higher rewards being selected.

6. The method according to claim 5, characterized in that, In S3.1, the immediate rewards for the two branches Calculate as follows: (1) For the color enhancement branch, its time step Instant rewards Rewards based on aesthetic perception Rewards for Image Features Linear weighted composition: ； in and These are hyperparameters; Aesthetic perception reward The state is evaluated by the color aesthetics evaluation network. and The evaluation probability distribution is calculated as follows: ； in Indicates the range of aesthetic scores. The image was rated as number one. The probability of a grade; Image feature reward The calculation formula is: ； in and This represents the width and height of the input image. This indicates the position of each pixel in the image. These are the pixel values ​​of the original input image; (2) For the graph optimization branch, its time step Instant rewards Rewards based on aesthetic rating differences Aspect Ratio Penalty constitute: The aesthetic score difference reward Scoring of the pre- and post-cropping states based on a compositional aesthetics evaluation network: ； in The network output is rated between 0 and 1 to assess the compositional aesthetics. The aspect ratio penalty item The trigger condition is: if the aspect ratio of the cropped image region is... If it is less than the first threshold or greater than the second threshold, then It is a preset negative value, otherwise Zero; The immediate reward of the graph optimization branch is represented as follows: .

7. The method according to claim 6, characterized in that, The first threshold is 0.5, the second threshold is 2.0, and the preset negative value penalty... It is -1.

8. The method according to claim 1, characterized in that, The color enhancement branch and the composition optimization branch share a basic feature encoding network, and the shared feature maps extracted by the feature encoding network serve as the inputs to the policy network and value network of the two branches, respectively.

Images