Aesthetics-guided automatic white balance method and device

Abstract

The embodiment of the invention provides an aesthetic-guided automatic white balance method and device, and the method is applied to computing equipment, and comprises the steps: obtaining a to-be-corrected image and corresponding instance segmentation information, inputting the to-be-corrected image into a reinforcement learning model to obtain a reward value and a correction strategy, and optimizing the reinforcement learning model based on the reward value to obtain a correction result. And based on a correction strategy and the instance segmentation information, carrying out correction processing on each pixel point of the to-be-corrected image to obtain a corrected image, and circulating the above steps until the number of times of circulating execution reaches a preset number of times to obtain a final image after final correction is completed. Based on the method provided by the invention, color balance can be realized, and the correction process is ensured to consider local and overall coordination, so that the effect of color fidelity is realized.

Description

Technical Field

[0001] The present application relates to the field of computer science and technology, and in particular to an aesthetically guided automatic white balance method and device. Background Art

[0002] Automatic white balance is a key image processing technology. Its main goal is to eliminate the impact of ambient light sources on the image and make the originally white area appear truly white.

[0003] Currently, automatic white balance processing of images mainly relies on data-driven methods. Data-driven methods usually extract lighting features in images through convolutional neural networks or other deep learning models to correct color casts.

[0004] However, the data-driven approach is highly dependent on the diversity and quality of the training data set. Therefore, it is difficult to achieve accurate illumination estimation when facing images of pure color scenes or complex multi-light source environments, which in turn greatly reduces the effect of white balance processing. Therefore, providing a method that can accurately correct the color cast of images in any environment is an urgent problem to be solved in this field. Summary of the invention

[0005] The embodiments of the present application provide an aesthetically guided automatic white balance method and device, which can improve the efficiency and accuracy of the automatic white balance solution.

[0006] In a first aspect, an embodiment of the present application provides an aesthetically guided automatic white balance method, the method being applied to a computing device, comprising:

[0007] Acquire the image to be corrected and instance segmentation information corresponding to the image to be corrected;

[0008] Inputting the image to be corrected into a reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the image to be corrected;

[0009] Optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model;

[0010] Based on the correction strategy for each pixel of the image to be corrected and the instance segmentation information, performing correction processing on each pixel of the image to be corrected to obtain a corrected image;

[0011] The corrected image is input into the reinforcement learning model in a loop to obtain a reward value and a correction strategy for each pixel of the corrected image, the reinforcement learning model is optimized based on the reward value to obtain an optimized reinforcement learning model, and each pixel of the corrected image is corrected based on the correction strategy and instance segmentation information, until the number of loop executions reaches a preset number, so as to obtain a final image after the final correction is completed.

[0012] In a possible implementation, the reinforcement learning model includes: a state module, an agent module and a reward function module, wherein the reward function module includes: a guidance reward function, an aesthetic quality reward function and an artifact loss function;

[0013] Accordingly, the step of inputting the image to be corrected into a reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the image to be corrected includes:

[0014] Inputting the image to be corrected into a reinforcement learning model, and performing illumination estimation on the image to be corrected based on the state module to obtain a preprocessing strategy;

[0015] The intelligent agent module obtains a correction strategy based on the preprocessing strategy and the historical correction strategy;

[0016] Based on the reward function module, a first reward value corresponding to the guidance reward function, a second reward value corresponding to the aesthetic quality reward function, and a third reward value corresponding to the artifact loss function are obtained.

[0017] In a possible implementation, the reinforcement learning model further includes: an environment module;

[0018] Accordingly, the step of performing illumination estimation on the image to be corrected based on the state module to obtain a preprocessing strategy includes:

[0019] Acquiring illumination information of the image to be corrected;

[0020] Based on the environment module, illumination information of the image to be corrected is estimated and globally corrected by using a diagonal matrix to obtain a corrected correction image;

[0021] The correction image is determined as the preprocessing result.

[0022] In a possible implementation, the optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model includes:

[0023] summing the first reward value, the second reward value and the third reward value to obtain a total reward value;

[0024] Based on a preset discount function, discounting the total reward value to obtain a discounted reward value;

[0025] Based on the discounted reward value, the parameters of the agent module in the reinforcement learning model are modified to obtain an optimized reinforcement learning model.

[0026] In a possible implementation, the performing correction processing on each pixel of the image to be corrected based on the correction strategy for each pixel of the image to be corrected and the instance segmentation information to obtain a corrected image includes:

[0027] Determine a plurality of target regions corresponding to the image to be corrected based on the instance segmentation information;

[0028] Determine a first correction strategy with the highest proportion among the correction strategies corresponding to the pixels included in each target area, and determine the first correction strategy as the target correction strategy corresponding to the target area;

[0029] A corresponding target correction strategy is executed on each pixel point in the target area to obtain a corrected image.

[0030] In a possible implementation, before inputting the image to be corrected into the reinforcement learning model, the method further includes:

[0031] Acquiring training data, wherein the training data includes multiple sets of training images and corresponding standard images;

[0032] Training is performed based on a preset pixel reinforcement learning framework of the multiple groups of training images and corresponding standard images to obtain the reinforcement learning model.

[0033] In a possible implementation, the training based on the pixel reinforcement learning framework preset by the multiple groups of training images and the corresponding standard images to obtain the reinforcement learning model includes:

[0034] For each set of training images and corresponding standard images, the training images and the corresponding standard images are input into a preset pixel reinforcement learning framework to obtain a reward value and a correction strategy for each pixel point of the training image;

[0035] Optimizing the pixel reinforcement learning framework based on the reward value to obtain an optimized pixel reinforcement learning framework;

[0036] Correcting each pixel of the training image based on the correction strategy to obtain a corrected image;

[0037] The corrected image is input into the pixel reinforcement learning framework in a loop to obtain a reward value and a correction strategy for each pixel of the corrected image, the pixel reinforcement learning framework is optimized based on the reward value to obtain an optimized pixel reinforcement learning framework, and each pixel of the corrected image is corrected based on the correction strategy, until the number of loop executions reaches a preset number;

[0038] The parameters of the pixel reinforcement learning framework are updated based on all reward values ​​corresponding to each group of training images and the corresponding standard images until the reward values ​​converge to obtain the reinforcement learning model.

[0039] In a second aspect, the present application provides an aesthetically guided automatic white balance device, the device being applied to a computing device, comprising:

[0040] An acquisition module, used for acquiring an image to be corrected and instance segmentation information corresponding to the image to be corrected;

[0041] A processing module, used for inputting the image to be corrected into a reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the image to be corrected;

[0042] The processing module is further used to optimize the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model;

[0043] The processing module is further used to perform correction processing on each pixel of the image to be corrected based on the correction strategy for each pixel of the image to be corrected and the instance segmentation information, so as to obtain a corrected image;

[0044] The processing module is also used to cyclically execute the steps of inputting the corrected image into the reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the corrected image, optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model, and correcting each pixel of the corrected image based on the correction strategy and instance segmentation information, until the number of loop executions reaches a preset number of times, so as to obtain a final image after the final correction is completed.

[0045] In a third aspect, the present application provides a computer device, including: a memory, a processor;

[0046] The memory stores computer-executable instructions;

[0047] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the above-mentioned method.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the method as described above when executed by a processor.

[0049] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0050] The aesthetically guided automatic white balance method and device provided in the embodiment of the present application first obtains the image to be corrected and the instance segmentation information corresponding to the image to be corrected, inputs the image to be corrected into the reinforcement learning model, obtains the reward value and the correction strategy for each pixel of the image to be corrected, optimizes the reinforcement learning model based on the reward value to obtain the optimized reinforcement learning model, performs correction processing on each pixel of the image to be corrected based on the correction strategy and the instance segmentation information to obtain the corrected image, and then repeats the above steps until the number of loop executions reaches a preset number of times to obtain the final image after the final correction is completed. The present application uses a reinforcement learning model to perform multiple corrections on each pixel of the image to be corrected to achieve color balance, and performs correction with reference to the instance segmentation information, which can ensure that the correction process takes into account the coordination of the local and the whole to achieve the effect of color fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] Figure 1 Schematic diagram of the scenario provided for this application;

[0053] Figure 2 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 1 ;

[0054] Figure 3 Schematic diagram of instance segmentation for example;

[0055] Figure 4 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 2 ;

[0056] Figure 5 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 3 ;

[0057] Figure 6 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 4 ;

[0058] Figure 7 A schematic diagram of the structure of a reinforcement learning model based on an example;

[0059] Figure 8 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 5 ;

[0060] Fig. 9 A schematic diagram showing a comparison between the present application and the prior art in image correction processing as an example;

[0061] Fig.10 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 6 ;

[0062] Fig.11 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 7 ;

[0063] Fig.12 A schematic diagram of the structure of the aesthetically guided automatic white balance device provided in the present application;

[0064] Fig.13 A schematic diagram of the structure of the electronic device provided in this application.

[0065] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0066] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0067] First, the terms involved in this application are explained:

[0068] Automatic white balance: It is an image processing technology widely used in digital cameras, camcorders, mobile phones and other devices. It estimates the scene lighting color during the shooting process and automatically adjusts the color temperature of the photo or video to ensure that all colors in the image are restored as close as possible to the real colors seen by the human eye under the corresponding lighting conditions.

[0069] Instance segmentation: Identifying the exact pixel boundaries of each individual object instance in an image.

[0070] GT image: Ground Truth (GT for short), in computer vision or aesthetics-guided automatic white balance tasks, a real annotated image or data used as a benchmark to test algorithm performance. In this invention, it specifically refers to images after expert color adjustment.

[0071] Pure color scene: mainly composed of a single or a small number of simple color areas, lacking complex textures and detail changes, such as blue sky, grass, walls, etc.

[0072] Color aesthetics: a discipline that studies the effects of color combination, contrast, and coordination on visual perception and emotional transmission, involving the application principles and aesthetic value of color in art, design, and daily life.

[0073] Pixel-based reinforcement learning: A multi-agent reinforcement learning method in which each pixel of an image is treated as an independent agent, which learns decision-making strategies from local information through interaction, collaboration, or competition to jointly complete the overall task.

[0074] Action space: A set of actions or decisions that define an agent in reinforcement learning. It describes all possible actions that the agent can choose at each time step.

[0075] Automatic white balance is a key image processing technology, and its main goal is to eliminate the influence of ambient light sources on the image, so that the originally white area presents a true white effect. At present, the automatic white balance processing of images mainly relies on data-driven methods. Data-driven methods usually extract the lighting features in the image through convolutional neural networks or other deep learning models to correct the color cast. However, the data-driven method is highly dependent on the diversity and quality of the training data set. Therefore, when faced with images of pure colors or complex multi-light source environments, it is difficult to achieve accurate lighting estimation, which in turn affects the white balance processing effect. Therefore, providing a method that can accurately correct the color cast of images in any environment is an urgent problem to be solved in this field.

[0076] Figure 1 The scene diagram provided for the present application can input the image to be corrected and the corresponding instance segmentation information into the reinforcement learning model, so that the reinforcement learning module discounts the obtained correction strategy according to the obtained reward value to obtain the correction strategy for each pixel point of the image to be corrected, and based on the correction strategy and the instance segmentation information, each pixel point of the image to be corrected is corrected to obtain the corrected image, and then the above steps are repeated until the number of loop executions reaches a preset number to obtain the final image after the final correction is completed.

[0077] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0078] Figure 2 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 1 ,The method is applied to computing devices, such as Figure 2 As shown, including:

[0079] S201: Obtain an image to be corrected and instance segmentation information corresponding to the image to be corrected.

[0080] The execution subject of the present application is a computing device, which can be a computer or server that provides services.

[0081] Combined with the scene example, instance segmentation (SEG) information refers to multiple regions in the image to be corrected. Figure 3 A schematic diagram of instance segmentation is shown in Figure 2. Figure 3 As shown, the image can be divided into four areas, namely the area where people are located, the area where pets are located, the sky area and the lawn area.

[0082] S202: Input the image to be corrected into a reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the image to be corrected.

[0083] Combined with the scene example, the initial value correction strategy includes a correction strategy for each pixel in the corrected image, the correction strategy includes color temperature adjustment processing, etc., and the reward value represents the optimization degree of the correction strategy for each pixel in the corrected image.

[0084] S203: Optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model.

[0085] In combination with the scenario example, the parameters in the reinforcement learning model can be discounted based on the preset discount reward coefficient, that is, the parameters in the reinforcement learning model can be corrected to optimize the reinforcement learning model.

[0086] S204 , based on the correction strategy for each pixel of the image to be corrected and the instance segmentation information, perform correction processing on each pixel of the image to be corrected to obtain a corrected image.

[0087] Combined with the scene example, according to the correction strategy for each pixel and the instance segmentation information obtained above, the correction strategy corresponding to each area is obtained, and the correction strategy corresponding to each pixel in each area is the same.

[0088] S205, looping and inputting the corrected image into the reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the corrected image, optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model, and correcting each pixel of the corrected image based on the correction strategy and instance segmentation information, until the number of loop executions reaches a preset number of times, to obtain a final image after the final correction is completed.

[0089] Combined with the scenario example, after obtaining the corrected image as mentioned above, the corrected image can be input into the reinforcement learning model again. Since the reinput reinforcement learning model has been optimized by the reward value obtained last time, a new correction strategy corresponding to this time can be obtained, and the corrected image can be corrected again based on the correction strategy corresponding to this time. The above steps are repeated until the number of cycles reaches the preset number, such as after 10 cycles of correction, the final image after the final correction is obtained.

[0090] Based on the method provided in this example, each pixel of the image to be corrected can be corrected multiple times through a reinforcement learning model to achieve color balance, and correction can be performed with reference to instance segmentation information to ensure that the correction process takes into account the coordination of the local and the whole to achieve a color fidelity effect.

[0091] Optional, Figure 4 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 2 , the reinforcement learning model includes: a state module, an agent module and a reward function module, wherein the reward function module includes: a guidance reward function, an aesthetic quality reward function and an artifact loss function;

[0092] Accordingly, if Figure 4 As shown, S202 includes:

[0093] S401: Input the image to be corrected into a reinforcement learning model, and perform illumination estimation on the image to be corrected based on the state module to obtain a preprocessing strategy.

[0094] In combination with the scenario example, the reinforcement learning model includes a state module, an agent module and a reward function module. After the image to be corrected is input into the reinforcement learning model, the state module is used to estimate the illumination of the image to be corrected and obtain a preprocessing strategy for the image to be corrected.

[0095] S402, the intelligent agent module obtains the correction strategy based on the preprocessing strategy and the historical correction strategy.

[0096] Combined with the scenario example, if this correction is the first correction, the policy network of the agent module will give a correction strategy based on the pre-processing strategy given by the state module. If this correction is not the first correction, the correction strategy given by each previous correction is used as the historical correction strategy, and the policy network of the agent module will give this correction strategy based on the pre-processing strategy given by the state module and the history given each time.

[0097] S403. Based on the reward function module, obtain a first reward value corresponding to the guidance reward function, a second reward value corresponding to the aesthetic quality reward function, and a third reward value corresponding to the artifact loss function.

[0098] Combined with the scene example, the preprocessing strategy is mainly used to correct the color cast of the image to be corrected. After that, it is necessary to further introduce the aesthetic guidance model to simulate people manually adjusting the color temperature to optimize the color of the image, and assist with other appropriate color adjustment actions to enhance the color aesthetic performance of the correction image. The main goal of the subsequent adjustment is to improve the aesthetic score of the image while minimizing the color cast. The overall objective function to be achieved is as follows:

[0099]

[0100] where / / Lc-L / / 2 Represents the distance between the estimated illumination and the real illumination, A(Lc) represents the aesthetic score of the image, and α and β are used to adjust the emphasis ratio of color fidelity and color aesthetics. Specifically, the above process can be realized through the reward values ​​given by the guidance reward function, aesthetic quality reward function and artifact loss function of the reward function module. The guidance reward function, aesthetic quality reward function and artifact loss function of the reward function module are used to optimize the policy network of the guidance agent module. Specifically, the guidance reward function is defined as the change in the distance between the image to be corrected and the preset target GT map, and the expert aesthetic and color fidelity information in the preset expert color adjustment map are learned, which can be expressed by the following formula:

[0101] Guidance reward function:

[0102] Among them I target Represents the target GT map, s t represents the state before correction, s (t+1) Represents the corrected state.

[0103] The aesthetic quality reward function is defined as the aesthetic change of the current image before and after correction. It is used to optimize the aesthetics of the image before and after correction and correct the image in a direction with better aesthetic evaluation. It can be expressed by the following formula:

[0104] Aesthetic quality reward function:

[0105] Among them Aes score (s t ) represents the evaluation of the image by the aesthetic model.

[0106] The artifact loss function is used to calculate the difference in gradient performance between the corrected image and the GT target image. By suppressing this difference, the artifact generation of the image is reduced. It can be expressed by the following formula:

[0107] Artifact loss function:

[0108] Among them, grad c (V i,j ) is used to calculate the gradient information of the image.

[0109] Based on the above guidance reward function, the aesthetic quality reward function and the artifact loss function, the corresponding first reward value, second reward value and third reward value are obtained respectively.

[0110] Based on the method provided in this example, the purpose of obtaining preprocessing strategy, correction strategy and reward value can be achieved.

[0111] Optional, Figure 5 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 3 ,The reinforcement learning model also includes: an environment module;

[0112] Accordingly, if Figure 5 As shown, S401 includes:

[0113] S501: Acquire illumination information of the image to be corrected.

[0114] Combined with the scene example, for the image I to be corrected, the illumination information can be defined as {LR, LG, LB}.

[0115] S502: Based on the environment module, perform illumination estimation and global correction on the illumination information of the image to be corrected by using a diagonal matrix to obtain a corrected correction image.

[0116] Combined with the scene example, in the environment module, the color deviation of the image to be corrected can be globally corrected by the diagonal matrix. The corresponding formula is as follows:

[0117]

[0118] Among them, Iwb is the correction diagram.

[0119] S503: Determine the correction image as the preprocessing result.

[0120] In combination with the scenario example, Iwb is determined as the preprocessing result.

[0121] Based on the method provided in this example, preprocessing of the image to be corrected can be achieved.

[0122] Optional, Figure 6 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 4 ,like Figure 6 As shown, S203 includes:

[0123] S601: sum the first reward value, the second reward value, and the third reward value to obtain a total reward value.

[0124] In combination with the scenario example, the first reward value, the second reward value and the third reward value obtained above are summed. For example, the first reward value is 40, the second reward value is 30, and the third reward value is 30, then the total reward value obtained is 100.

[0125] S602: Based on a preset discount function, the total reward value is discounted to obtain a discounted reward value.

[0126] Combined with the scenario example, the following is the discount function of the example:

[0127]

[0128] Where R(t) represents the reward corrected at step t, π represents the policy corrected at step t, and γ represents the discounted reward coefficient.

[0129] If the reward value obtained is 100 and the discount reward coefficient is 0.9, the final discounted reward value is 90.

[0130] S603: Based on the discounted reward value, modify the parameters of the agent module in the reinforcement learning model to obtain an optimized reinforcement learning model.

[0131] Combined with the scenario example, based on the discounted reward value obtained, the reward value is superimposed on the correction strategy to optimize the parameters of the agent module in the reinforcement learning model. Specifically, the policy network and value network in the agent module can be optimized to obtain the correction strategy.

[0132] Specifically, based on the correction strategy, the correction strategy and the action in the action space are mapped, and the adjustment is applied to the image. The specific adjustment formula is as follows:

[0133]

[0134] Where I(t+1) represents the corrected image, and xi represents the pixel position corresponding to the adjustment of the strategy "Action".

[0135] Figure 7 The structural diagram of the reinforcement learning model is shown as an example. Figure 7 As shown, the image to be corrected is input into the reinforcement learning model, and the illumination estimation is performed based on the state module to obtain the preprocessing strategy. The correction strategy is obtained based on the agent module. The correction strategy includes the adjustment action corresponding to each pixel point, but each pixel point can only have one adjustment action in a correction process, and the adjustment action specifically includes: color temperature increase and color temperature decrease. The total reward value is obtained based on the reward function module. Specifically, the reward function module includes an artifact loss function, an aesthetic quality reward function and a guidance reward function. Then, the agent module of the reinforcement learning model is optimized after discounting the total reward value. Then, based on the environment module, the color deviation of the image to be corrected is corrected based on the diagonal matrix correction of the preprocessing strategy, and the adjustment action corresponding to each pixel point in the image to be corrected is adjusted based on the correction strategy. In this way, the preset number of cycles, such as 10 cycles, and the correction strategy corresponding to each cycle includes the adjustment action corresponding to each pixel point.

[0136] Based on the method provided in this example, a correction strategy for the image to be corrected can be obtained, and the correction processing of the image to be corrected can be completed based on the correction strategy.

[0137] Figure 8 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 5 ,like Figure 8 As shown, S204 includes:

[0138] S801. Determine a plurality of target regions corresponding to the image to be corrected based on the instance segmentation information.

[0139] Combined with scene examples, different areas in the instance segmentation information are determined as different target areas.

[0140] S802: Determine a first correction strategy with the highest proportion among the correction strategies corresponding to the pixels included in each target area, and determine the first correction strategy as the target correction strategy corresponding to the target area.

[0141] Combined with the scene example, based on the correction strategy, determine the adjustment actions corresponding to the pixels contained in each target area, and determine the proportion of each type of adjustment action. For example, taking the target area where a person is located as an example, if the target area where the person is located contains 100 pixels, 30% of the pixels are processed with increased color temperature, and 70% of the pixels are processed with decreased color temperature. Therefore, the color temperature reduction process with a high proportion can be determined as the first correction strategy, and the color temperature adjustment process can be determined as the target correction strategy corresponding to the target area where the person is located. Similarly, in this way, the target correction strategy corresponding to each target area is determined.

[0142] S803: Execute a corresponding target correction strategy on each pixel in the target area to obtain a corrected image.

[0143] Combined with the scene example, based on the target correction strategy corresponding to each target area, the corresponding target correction strategy is executed on each pixel in the target area. For example, if the target correction strategy corresponding to the target area where a person is located is color temperature adjustment processing, then color temperature adjustment processing is executed on all 100 pixels in the target area where the person is located.

[0144] Based on the method provided in this example, the image to be corrected is corrected with reference to the instance segmentation information, which can ensure that the local and overall coordination is taken into account during the correction process.

[0145] Fig. 9 The following is a schematic diagram showing a comparison between the present application and the prior art in image correction processing, as shown in FIG. Fig. 9 As shown, with the same image to be corrected as the input image, based on the mainstream white balance of the prior art, the obtained output image can achieve color fidelity. Based on the method provided in this embodiment, the color deviation of the image to be corrected is corrected by diagonal matrix correction, which can achieve color fidelity. By optimizing the reward value of the reward function module, the aesthetic feeling of the image to be corrected can be improved to take into account the goal of color aesthetics.

[0146] Optional, Fig.10 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 6 , before S202, such as Fig.10 As shown, the method also includes:

[0147] S1001. Acquire training data, wherein the training data includes multiple groups of training images and corresponding standard images.

[0148] Combined with the scene examples, pairs of images before and after artificial color adjustment are obtained to obtain the dataset {I1, I2, ,In}, generate paired labels {L1,L2, ,Ln}, use the instance segmentation algorithm to obtain the instance segmentation information corresponding to each element in the data set {M1,M2, ,Mn}, combined with paired data sets and label data, and then divided into training set and test set in a ratio of 8:2. The training set is used as training data, where each element includes a pair of images. The image before color adjustment can be determined as the training image, and the image after artificial color adjustment can be determined as the standard image. The image before artificial color adjustment in the test set can be used as the aforementioned image to be corrected.

[0149] S1002: Perform training based on a preset pixel reinforcement learning framework of the multiple groups of training images and corresponding standard images to obtain the reinforcement learning model.

[0150] Combined with the scenario example, combined with the above Figure 7 Based on the training data obtained above, the preset pixel reinforcement learning framework is trained until each reward function in the reward function module converges to complete the training of the pixel reinforcement learning framework. Based on the method provided in this example, a reinforcement learning model after training can be obtained.

[0151] Optional, Fig.11 Schematic diagram of the process of the aesthetically guided automatic white balance method provided in this application Figure 7 ,like Fig.11 As shown, S1002 includes:

[0152] S1101. For each set of training images and corresponding standard images, the training images and the corresponding standard images are input into a preset pixel reinforcement learning framework to obtain a reward value and a correction strategy for each pixel point of the training image.

[0153] Combined with the scene example, combined with Figure 7 , the training images and standard images in the training set are output to the preset pixel reinforcement learning framework, and the illumination of the training images is estimated based on the state module to obtain the preprocessing strategy for the training images. If this is the first time to process the training images, the intelligent agent module can obtain the adjustment strategy based on the preprocessing strategy for the training images. If this is not the first time to process the training images, the intelligent agent module can obtain the adjustment strategy based on the preprocessing strategy for the training images and the previous historical adjustment strategy.

[0154] S1102: Optimize the pixel reinforcement learning framework based on the reward value to obtain an optimized pixel reinforcement learning framework.

[0155] Combined with the scene example, the reward function module obtains the total reward value based on the artifact loss function, the aesthetic quality reward function and the guidance reward function, and discounts the total reward value based on the preset discount function to obtain the discounted reward value. The parameters in the pixel reinforcement learning framework are optimized based on the discounted reward value.

[0156] S1103: Correct each pixel of the training image based on the correction strategy to obtain a corrected image.

[0157] Combined with the scene example, in the environment module, based on the preprocessing strategy, the color deviation of the training image is corrected by diagonal matrix correction, and the adjustment action corresponding to each pixel is performed based on the correction strategy to obtain the corresponding corrected image.

[0158] S1104, looping and inputting the corrected image into the pixel reinforcement learning framework to obtain a reward value and a correction strategy for each pixel of the corrected image, optimizing the pixel reinforcement learning framework based on the reward value to obtain an optimized pixel reinforcement learning framework, and correcting each pixel of the corrected image based on the correction strategy, until the number of loop executions reaches a preset number.

[0159] In combination with the scenario example, based on the above process, the training image is repeated for a preset number of times, such as 10 times in a loop, and the discounted reward values ​​corresponding to the 10 loops can be obtained.

[0160] S1105. Update the parameters of the pixel reinforcement learning framework based on all reward values ​​corresponding to each group of training images and the corresponding standard images until the reward values ​​converge to obtain the reinforcement learning model.

[0161] Combined with the scenario example, the 10 discounted reward values ​​are superimposed to obtain the total discounted reward value. Based on the total discounted reward value, the parameters of the pixel reinforcement learning framework are updated to complete the training of the pixel reinforcement learning framework based on the training image.

[0162] For each training image in the training set, repeat the steps S1201-S1205 above to complete the training of the pixel reinforcement learning framework for each training image until the artifact loss function, the aesthetic quality reward function and the guidance reward function all reach convergence during the training process, so as to obtain the reinforcement learning model after the training is completed.

[0163] This application uses a reinforcement learning model to perform multiple corrections on each pixel of the image to be corrected to achieve color balance, and performs correction with reference to instance segmentation information, which can ensure that the correction process takes into account the coordination of the local and the whole to achieve the effect of color fidelity.

[0164] Fig.12 The structure diagram of the aesthetically guided automatic white balance device provided by the present application is applied to a computing device, such as Fig.12 As shown, including:

[0165] An acquisition module 121 is used to acquire the image to be corrected and instance segmentation information corresponding to the image to be corrected;

[0166] A processing module 122, configured to input the image to be corrected into a reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the image to be corrected;

[0167] The processing module 122 is further used to optimize the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model;

[0168] The processing module 122 is further configured to perform correction processing on each pixel of the image to be corrected based on the correction strategy for each pixel of the image to be corrected and the instance segmentation information, so as to obtain a corrected image;

[0169] The processing module 122 is also used to cyclically execute the steps of inputting the corrected image into the reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the corrected image, optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model, and correcting each pixel of the corrected image based on the correction strategy and instance segmentation information, until the number of loop executions reaches a preset number of times, so as to obtain a final image after the final correction is completed.

[0170] The aesthetically guided automatic white balance device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0171] Fig.13 This is a schematic diagram of the structure of the electronic device provided in this application. Fig.13 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.

[0172] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0173] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0174] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0175] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0176] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0177] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0178] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0179] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0180] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0181] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0184] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0185] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0186] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An aesthetically guided automatic white balance method, characterized in that The method is applied to a computing device, comprising: Acquire the image to be corrected and instance segmentation information corresponding to the image to be corrected; Inputting the image to be corrected into a reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the image to be corrected; Optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model; Based on the correction strategy for each pixel of the image to be corrected and the instance segmentation information, performing correction processing on each pixel of the image to be corrected to obtain a corrected image; The corrected image is input into the reinforcement learning model in a loop to obtain a reward value and a correction strategy for each pixel of the corrected image, the reinforcement learning model is optimized based on the reward value to obtain an optimized reinforcement learning model, and each pixel of the corrected image is corrected based on the correction strategy and instance segmentation information, until the number of loop executions reaches a preset number, so as to obtain a final image after the final correction is completed.

2. The method according to claim 1, characterized in that The reinforcement learning model includes: a state module, an agent module and a reward function module, wherein the reward function module includes: a guidance reward function, an aesthetic quality reward function and an artifact loss function; Accordingly, the step of inputting the image to be corrected into a reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the image to be corrected includes: Inputting the image to be corrected into a reinforcement learning model, and performing illumination estimation on the image to be corrected based on the state module to obtain a preprocessing strategy; The intelligent agent module obtains a correction strategy based on the preprocessing strategy and the historical correction strategy; Based on the reward function module, a first reward value corresponding to the guidance reward function, a second reward value corresponding to the aesthetic quality reward function, and a third reward value corresponding to the artifact loss function are obtained.

3. The method according to claim 2, characterized in that The reinforcement learning model also includes: environment module; Accordingly, the step of performing illumination estimation on the image to be corrected based on the state module to obtain a preprocessing strategy includes: Acquiring illumination information of the image to be corrected; Based on the environment module, illumination information of the image to be corrected is estimated and globally corrected by using a diagonal matrix to obtain a corrected correction image; The correction image is determined as the preprocessing result.

4. The method according to claim 2, characterized in that: The optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model includes: summing the first reward value, the second reward value and the third reward value to obtain a total reward value; Based on a preset discount function, discounting the total reward value to obtain a discounted reward value; Based on the discounted reward value, the parameters of the agent module in the reinforcement learning model are modified to obtain an optimized reinforcement learning model.

5. The method according to claim 1, characterized in that The step of performing correction processing on each pixel of the image to be corrected based on the correction strategy for each pixel of the image to be corrected and the instance segmentation information to obtain a corrected image includes: Determine a plurality of target regions corresponding to the image to be corrected based on the instance segmentation information; Determine a first correction strategy with the highest proportion among the correction strategies corresponding to the pixels included in each target area, and determine the first correction strategy as the target correction strategy corresponding to the target area; A corresponding target correction strategy is executed on each pixel point in the target area to obtain a corrected image.

6. The method according to any one of claims 1 to 5, characterized in that: Before inputting the image to be corrected into the reinforcement learning model, the method further includes: Acquiring training data, wherein the training data includes multiple sets of training images and corresponding standard images; Training is performed based on a preset pixel reinforcement learning framework of the multiple groups of training images and corresponding standard images to obtain the reinforcement learning model.

7. The method according to claim 6, characterized in that The pixel reinforcement learning framework preset based on the multiple sets of training images and corresponding standard images is trained to obtain the reinforcement learning model, including: For each set of training images and corresponding standard images, the training images and the corresponding standard images are input into a preset pixel reinforcement learning framework to obtain a reward value and a correction strategy for each pixel point of the training image; Optimizing the pixel reinforcement learning framework based on the reward value to obtain an optimized pixel reinforcement learning framework; Correcting each pixel of the training image based on the correction strategy to obtain a corrected image; The corrected image is input into the pixel reinforcement learning framework in a loop to obtain a reward value and a correction strategy for each pixel of the corrected image, the pixel reinforcement learning framework is optimized based on the reward value to obtain an optimized pixel reinforcement learning framework, and each pixel of the corrected image is corrected based on the correction strategy, until the number of loop executions reaches a preset number; The parameters of the pixel reinforcement learning framework are updated based on all reward values ​​corresponding to each group of training images and the corresponding standard images until the reward values ​​converge to obtain the reinforcement learning model.

8. An aesthetically guided automatic white balance device, characterized in that: The device is applied to a computing device, comprising: An acquisition module, used for acquiring an image to be corrected and instance segmentation information corresponding to the image to be corrected; A processing module, used for inputting the image to be corrected into a reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the image to be corrected; The processing module is further used to optimize the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model; The processing module is further used to perform correction processing on each pixel of the image to be corrected based on the correction strategy for each pixel of the image to be corrected and the instance segmentation information, so as to obtain a corrected image; The processing module is also used to cyclically execute the steps of inputting the corrected image into the reinforcement learning model to obtain a reward value and a correction strategy for each pixel of the corrected image, optimizing the reinforcement learning model based on the reward value to obtain an optimized reinforcement learning model, and correcting each pixel of the corrected image based on the correction strategy and instance segmentation information, until the number of loop executions reaches a preset number of times, so as to obtain a final image after the final correction is completed.

9. A computer device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images