Image processing model training method and device, white balance processing method and device
By performing color cast processing on the original image to generate a training set image and using a loss function to optimize the neural network, the dependence on real light source annotation in existing technologies is solved, and efficient image processing model training and white balance processing are achieved.
Patent Information
- Application Number
- CN202110317455.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-03-25
AI Technical Summary
In existing technologies, white balance methods based on deep neural networks require a large amount of training data annotated with real light sources, which makes it difficult to obtain the training set and reduces the accuracy of the model.
By color-casting the original image to generate a color-cast training set image, which is used as a supervisory label to train the neural network, the network is optimized using a loss function to achieve white balance processing, thus avoiding dependence on annotations from real light sources.
It achieves efficient training without relying on annotations from real light sources, reducing training difficulty and improving the accuracy and efficiency of image processing models.
Smart Images

Figure CN112884693B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and specifically relates to a method and device for training an image processing model, and a method and device for white balance processing. Background Art
[0002] In the related art, for image acquisition devices such as cameras, due to the limitations of hardware sensors, there will be certain differences between the image output by the sensor and the image viewed by the human eye, resulting in the color of the object in the captured image being different from the actual color of the object recognized by the human eye.
[0003] Generally speaking, it is necessary to perform white balance processing on the source photos taken. The current white balance method based on deep neural networks is based on supervised and semi-supervised methods. During the model training process, this method requires a large amount of training set data containing real light source annotations. The real light source annotations take a lot of time, making it difficult to obtain the training set, and ultimately reducing the accuracy of the trained model.
[0004] Therefore, there is an urgent need for a training method that does not rely on training set data with real light source annotations. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a method and device for training an image processing model, and a method and device for white balance processing of an image, which can efficiently train the image processing model without relying on training set data annotated with real light sources.
[0006] In order to solve the above technical problems, this application is implemented as follows:
[0007] In a first aspect, an embodiment of the present application provides a method for training an image processing model, comprising:
[0008] Obtain a first training set image;
[0009] Performing color cast processing on the first training set image to obtain a second training set image after color cast;
[0010] The neural network to be trained is trained by using the first training set images and the second training set images to obtain an image processing model for white balance processing.
[0011] In a second aspect, an embodiment of the present application provides a training device for an image processing model, comprising:
[0012] A first acquisition unit, used to acquire a first training set image;
[0013] A first processing unit is used to perform color cast processing on the first training set image to obtain a second training set image after color cast;
[0014] The training unit trains the neural network to be trained by using the first training set images and the second training set images to obtain an image processing model for white balance processing.
[0015] In a third aspect, an embodiment of the present application provides a method for white balance processing of an image, comprising:
[0016] Obtaining an image processing model trained according to the image processing model training method provided in the first aspect;
[0017] The image to be processed is subjected to white balance processing through the image processing model to obtain a target image after white balance processing.
[0018] In a fourth aspect, an embodiment of the present application provides a white balance processing device for an image, comprising:
[0019] A second acquisition unit, used to acquire an image processing model trained according to the image processing model training method provided in the first aspect;
[0020] The second processing unit is used to perform white balance processing on the image to be processed through the image processing model to obtain a target image after white balance processing.
[0021] In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the training method of the image processing model provided in the first aspect is implemented; and / or
[0022] A method for processing white balance of an image as provided in the third aspect.
[0023] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, a training method for an image processing model as provided in the first aspect is implemented; and / or
[0024] A method for processing white balance of an image as provided in the third aspect.
[0025] In an embodiment of the present application, during the process of training the image processing model, a first training set image is first obtained, wherein the first training set image is an original image. The original image does not require annotation information of the real light source. Therefore, most standard data sets in the field of computer vision meet the training requirements, such as CIFAR-10, PASCAL VOC 2007, PASCAL VOC 2012 data sets, etc.
[0026] Furthermore, the first training set images are subjected to color cast processing. In this step, random color cast may be artificially added to the original images in the first training set images, so that the original images are color cast, thereby generating the second training set images after color cast for training. Specifically, the first training set images may be subjected to color cast processing by superimposing random color cast information on the feature images obtained after encoding by the encoder, or by performing color transfer on the original images, and color cast images corresponding to each original image in the first training set images are output, and formed as the second training set images.
[0027] After obtaining the second training set image, the first training set image is used as the supervision label of the second training set image (that is, the color cast image) to form the final training data, and the training data of the first training set and the second training set are used to train the neural network to be trained. During the training process, the loss function is used to calculate the loss value between the output image restored by the neural network based on the color cast image in the second training set image and the original image in the first training set image. When the loss function converges and the loss value meets the requirements, the training of the neural network is completed, and finally a neural network model that meets the requirements and can accurately complete the white balance adjustment of the original photo is obtained, that is, the above-mentioned image processing model.
[0028] In the training method of the image processing model provided in the embodiment of the present application, the applied training data does not need to be annotated with real light sources, thereby saving a lot of annotation time. At the same time, since the training process does not require annotation information of the real light source, the existing massive data sets can be used to train the neural network, which greatly increases the optional data sets while reducing the difficulty of training and achieving efficient training that does not rely on real light source annotations. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 One of the flowcharts of the training method of the image processing model according to the embodiment of the present application is shown;
[0030] Figure 2 A second flowchart of a method for training an image processing model according to an embodiment of the present application is shown;
[0031] Figure 3 A schematic diagram of the structure of an image processing model according to an embodiment of the present application is shown;
[0032] Figure 4 A third flowchart of a method for training an image processing model according to an embodiment of the present application is shown;
[0033] Figure 5 A structural block diagram of a training device for an image processing model according to an embodiment of the present application is shown;
[0034] Figure 6 One of the flowcharts of the white balance processing method of an image according to an embodiment of the present application is shown;
[0035] Figure 7 A second flowchart of the method for white balance processing of an image according to an embodiment of the present application is shown;
[0036] Figure 8 A structural block diagram of a white balance processing device for an image according to an embodiment of the present application is shown;
[0037] Fig. 9 A schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0039] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.
[0040] In the following, in combination with the accompanying drawings, the image processing model training method and device, and the white balance processing method and device provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.
[0041] In some embodiments of the present application, a training method for an image processing model is provided. Figure 1 One of the flowcharts of the training method of the image processing model according to the embodiment of the present application is shown, as shown in FIG. Figure 1 As shown, the training method includes:
[0042] Step 102, obtaining a first training set image;
[0043] Step 104, performing color cast processing on the first training set image to obtain a second training set image after color cast;
[0044] Step 106: Train the neural network to be trained using the first training set images and the second training set images to obtain an image processing model for white balance processing.
[0045] In an embodiment of the present application, during the process of training the image processing model, a first training set image is first obtained, wherein the first training set image is an original image, and the original image does not require annotation information of the real light source. Therefore, most standard data sets in the field of computer vision meet the training requirements, such as the CIFAR-10 data set, the PASCAL VOC 2007 data set, the PASCAL VOC 2012 data set, etc.
[0046] Furthermore, the first training set images are subjected to color cast processing. In this step, random color cast may be artificially added to the original images in the first training set images, so that the original images are color cast, thereby generating the second training set images after color cast for training. Specifically, the first training set images may be subjected to color cast processing by superimposing random color cast information on the feature images obtained after encoding by the encoder, or by performing color transfer on the original images, and color cast images corresponding to each original image in the first training set images are output, and formed as the second training set images.
[0047] After obtaining the second training set image, the first training set image is used as the supervision label of the second training set image (that is, the color cast image) to form the final training data, and the training data of the first training set and the second training set are used to train the neural network to be trained. During the training process, the loss function is used to calculate the loss value between the output image restored by the neural network based on the color cast image in the second training set image and the original image in the first training set image. When the loss function converges and the loss value meets the requirements, the training of the neural network is completed, and finally a neural network model that meets the requirements and can accurately complete the white balance adjustment of the original photo is obtained, that is, the above-mentioned image processing model.
[0048] In the training method of the image processing model provided in the embodiment of the present application, the applied training data does not need to be annotated with real light sources, thereby saving a lot of annotation time. At the same time, since the training process does not require annotation information of the real light source, the existing massive data sets can be used to train the neural network, which greatly increases the optional data sets while reducing the difficulty of training and achieving efficient training that does not rely on real light source annotations.
[0049] In some embodiments of the present application, the neural network to be trained includes an encoder network, Figure 2 FIG. 2 shows a second flowchart of a method for training an image processing model according to an embodiment of the present application. Figure 2As shown, the first training set image is subjected to color cast processing to obtain the second training set image, including:
[0050] Step 202, performing convolution processing on the first training set image through the encoder network to obtain first feature map data;
[0051] Step 204, obtaining preset color cast data and activation function;
[0052] Step 206, adding the color cast data to the first feature map data through an activation function to obtain second feature map data with color cast;
[0053] Step 208: Generate a second training set image based on the second feature map data.
[0054] In the embodiment of the present application, the model to be trained and the target image processing model both include an encoder network, wherein the encoder network can continue to use a 5-layer AlexNet structure, and correspondingly, a 5-layer decoder network symmetrical thereto can be set. Figure 3 A schematic diagram of the structure of an image processing model according to an embodiment of the present application is shown. Figure 3 As shown in the figure, the autoencoder structure contains a total of 10 layers of networks.
[0055] Specifically, the original image in the first training set image is input into the encoder, where the original image can be a three-channel color image with a size of 227×227. After the original image is input into the encoder network, it is subjected to 5 layers of convolution of the encoder network for feature extraction, and the encoder network outputs the first feature map data with a size of 6×6×256.
[0056] Further, the preset color cast data is obtained, which can be randomly selected color cast data, specifically the color cast data of the 6×6 feature map, and the color cast data is added to the first feature map data through the activation function. Specifically, the 6×6 color cast data can be bitwise multiplied with the 6×6×256 first feature map data, and the corresponding output is the second feature map data of the color cast after the activation function. After the obtained second feature map data is merged, the final second training set image is generated.
[0057] In this process, the second training set images are obtained by processing the first training set images. Therefore, the first feature map data does not need to be annotated with the real light source, which can save a lot of annotation time. At the same time, since the training process does not require annotation information of the real light source, the existing massive data sets can be used to train the neural network, which greatly increases the optional data sets while reducing the difficulty of training.
[0058] In some embodiments of the present application, the neural network to be trained includes a decoder network, Figure 4FIG. 3 shows a flowchart of a method for training an image processing model according to an embodiment of the present application, such as Figure 4 As shown, the neural network to be trained is trained by using the first training set image and the second training set image, including:
[0059] Step 402, performing deconvolution processing on the second training set image through a decoder network to obtain an output image;
[0060] Step 404, obtaining a preset loss function, and inputting the output image and the first training set image into the loss function;
[0061] Step 406, obtain the current loss value output by the loss function, and continue to train the neural network to be trained with the target loss value range as the target until the current loss value falls within the target loss value range.
[0062] In the embodiment of the present application, during the training of the neural network to be trained, that is, the above-mentioned image processing model, the second training set image, that is, the color cast image obtained by adding noise to the original image in the first training set image, is input into the decoder network, wherein, when the encoder network continues to use the 5-layer AlexNet structure, the decoder network can select a 5-layer decoder network symmetrical thereto. Through the decoder network, the second training set image is deconvolved to output a "restored" output image after adjusting the white balance.
[0063] Furthermore, the original image in the first training set image and the output image after white balance restoration by the decoder network are input into the loss function, and the loss value of the output image compared with the original image is calculated by the loss function, and the loss value can reflect the difference between the output image and the original image. During the training process, the neural network to be trained is continuously trained with the target loss value range as the target. In this process, the neural network to be trained is continuously trained with a large number of first training set images and processed second training set images, so that the neural network to be trained is continuously iterated and evolved during the training process, and the output image output by the decoder network is constantly "close" to the original image. In this process, the loss value calculated by the loss function is constantly close to the target loss value range.
[0064] When the loss value falls within the target loss value range, the loss function is considered to have converged. At this time, the output image of the neural network to be trained is very close to the original image. It can be determined that the trained neural network, that is, the above-mentioned image processing model, can accurately perform automatic white balance adjustment on "color cast" photos. At this time, the training is considered to be completed. In this process, the training is carried out in a back-propagation manner, which can accelerate the convergence of the network and improve the training efficiency.
[0065] In some implementations, during the training of the neural network to be trained, the Adam optimizer may be used to assist in the training, which can prevent the training results from falling into a "local optimum" on the one hand, and further accelerate the network convergence and improve the training efficiency on the other hand.
[0066] It can be understood that the above-mentioned target loss value range can be adjusted according to the training requirements and application iteration requirements. After obtaining an "image processing model" that meets the current requirements, the current version of the "image processing model" can continue to be trained during the use process, so as to obtain an "image processing model V2.0" with higher precision and more accurate white balance adjustment.
[0067] In some embodiments of the present application, the activation function is a linear rectification function, and the loss function is a loss function based on Wasserstein distance.
[0068] In an embodiment of the present application, the activation function is specifically a linear rectification function (Rectified Linear Unit, ReLU), which is a nonlinear activation function. The nonlinear activation function ReLU is used to process the product of the first feature map data obtained by the encoder according to the first training set image and the randomly selected color cast data, which can effectively increase the network sparsity, avoid overfitting, and improve the accuracy and effect of the image processing model.
[0069] Furthermore, the loss function selects a loss function based on the Wasserstein distance, which can narrow the distribution between the output image and the original image during the training process, and further improve the training effect and training efficiency of the training network.
[0070] In some embodiments of the present application, the loss function is:
[0071]
[0072] Among them, X is the input variable, Z is the encoded data, and Obj WCAE (P X ,P Y ) is the current loss value, is the expected value of the distribution of X, is the expected value of the conditional probability distribution of Z with respect to X. The function c(·) represents any measurable cost function, so c(X,G(Z)) is used to measure the distance between the output data of X and Z after decoding by the decoder. λ is the coefficient of the regularization term and λ is a constant. is the L2 norm of the encoder weights, is the L2 norm of the decoder weight, β is the coefficient of the sparse term and β is a constant, and ||h||1 is the L1 norm of the sparse term.
[0073] In the embodiment of the present application, the loss function used in the training process is specifically as shown in the above formula, wherein, in the training process, the encoder does not need to add any constraints, so that a relatively certain input-output mapping function can be obtained, thereby improving the training efficiency. Specifically, in the above loss function, its first term Represents the expectation of the cost function c(X,G(Z)) given the input data X, hidden layer data Z and decoder G.
[0074] In some embodiments of the present application, after the neural network to be trained is trained using the first training set images and the second training set images, the training method further includes: fixing the color cast weight value corresponding to the second feature map data.
[0075] In the embodiment of the present application, since no constraints are added to the encoder during the training process, the input-output mapping function obtained is relatively certain. The embodiment of the present application further adds a weight constraint item, that is, the above-mentioned color cast weight value, which can avoid overfitting and increase the sparse items of the hidden layer corresponding to the color cast, thereby improving the discrimination of the encoding, which is beneficial to improving the training effect.
[0076] In some embodiments of the present application, Figure 5 A structural block diagram of a training device for an image processing model according to an embodiment of the present application is shown. Figure 5 As shown, the training device 500 of the image processing model includes:
[0077] A first acquisition unit 502, used to acquire a first training set image;
[0078] A first processing unit 504 is used to perform color cast processing on the first training set image to obtain a second training set image after color cast;
[0079] The training unit 506 trains the neural network to be trained by using the first training set images and the second training set images to obtain an image processing model for white balance processing.
[0080] In an embodiment of the present application, during the process of training the image processing model, a first training set image is first obtained, wherein the first training set image is an original image. The original image does not require annotation information of the real light source. Therefore, most standard data sets in the field of computer vision meet the training requirements, such as CIFAR-10, PASCAL VOC 2007, PASCAL VOC 2012 data sets, etc.
[0081] Furthermore, the first training set images are subjected to color cast processing. In this step, random color cast may be artificially added to the original images in the first training set images, so that the original images are color cast, thereby generating the second training set images after color cast for training. Specifically, the first training set images may be subjected to color cast processing by superimposing random color cast information on the feature images obtained after encoding by the encoder, or by performing color transfer on the original images, and color cast images corresponding to each original image in the first training set images are output, and formed as the second training set images.
[0082] After obtaining the second training set image, the first training set image is used as the supervision label of the second training set image (that is, the color cast image) to form the final training data, and the training data of the first training set and the second training set are used to train the neural network to be trained. During the training process, the loss function is used to calculate the loss value between the output image restored by the neural network based on the color cast image in the second training set image and the original image in the first training set image. When the loss function converges and the loss value meets the requirements, the training of the neural network is completed, and finally a neural network model that meets the requirements and can accurately complete the white balance adjustment of the original photo is obtained, that is, the above-mentioned image processing model.
[0083] In the training method of the image processing model provided in the embodiment of the present application, the applied training data does not need to be annotated with real light sources, thereby saving a lot of annotation time. At the same time, since the training process does not require annotation information of the real light source, the existing massive data sets can be used to train the neural network, which greatly increases the optional data sets while reducing the difficulty of training and achieving efficient training that does not rely on real light source annotations.
[0084] In some embodiments of the present application, the first processing unit 504 is further configured to perform convolution processing on the first training set image through an encoder network to obtain first feature map data;
[0085] The first acquisition unit 502 is also used to acquire preset color cast data and activation function;
[0086] The first processing unit 504 is further configured to add the color cast data to the first feature map data through an activation function to obtain the second feature map data with color cast; and generate a second training set image according to the second feature map data.
[0087] In the embodiment of the present application, the model to be trained and the target image processing model both include an encoder network, wherein the encoder network can continue to use a 5-layer AlexNet structure, and correspondingly, a 5-layer decoder network symmetrical thereto can be set. Figure 3 A schematic diagram of the structure of an image processing model according to an embodiment of the present application is shown. Figure 3As shown in the figure, the autoencoder structure contains a total of 10 layers of networks.
[0088] Specifically, the original image in the first training set image is input into the encoder, where the original image can be a three-channel color image with a size of 227×227. After the original image is input into the encoder network, it is subjected to 5 layers of convolution of the encoder network for feature extraction, and the encoder network outputs the first feature map data with a size of 6×6×256.
[0089] Further, the preset color cast data is obtained, which can be randomly selected color cast data, specifically the color cast data of the 6×6 feature map, and the color cast data is added to the first feature map data through the activation function. Specifically, the 6×6 color cast data can be bitwise multiplied with the 6×6×256 first feature map data, and the corresponding output is the second feature map data of the color cast after the activation function. After the obtained second feature map data is merged, the final second training set image is generated.
[0090] In this process, the second training set images are obtained by processing the first training set images. Therefore, the first feature map data does not need to be annotated with the real light source, which can save a lot of annotation time. At the same time, since the training process does not require annotation information of the real light source, the existing massive data sets can be used to train the neural network, which greatly increases the optional data sets while reducing the difficulty of training.
[0091] In some embodiments of the present application, the neural network to be trained includes a decoder network, and the first processing unit 504 is further configured to perform deconvolution processing on the second training set image through the decoder network to obtain an output image;
[0092] The first acquisition unit 502 is also used to acquire a preset loss function, and input the output image and the first training set image into the loss function;
[0093] The training unit 506 is also used to obtain the current loss value output by the loss function, and continuously train the neural network to be trained with the target loss value range as the target until the current loss value falls within the target loss value range.
[0094] In the embodiment of the present application, during the training of the neural network to be trained, that is, the above-mentioned image processing model, the second training set image, that is, the color cast image obtained by adding noise to the original image in the first training set image, is input into the decoder network, wherein, when the encoder network continues to use the 5-layer AlexNet structure, the decoder network can select a 5-layer decoder network symmetrical thereto. Through the decoder network, the second training set image is deconvolved to output a "restored" output image after adjusting the white balance.
[0095] Furthermore, the original image in the first training set image and the output image after white balance restoration by the decoder network are input into the loss function, and the loss value of the output image compared with the original image is calculated by the loss function, and the loss value can reflect the difference between the output image and the original image. During the training process, the neural network to be trained is continuously trained with the target loss value range as the target. In this process, the neural network to be trained is continuously trained with a large number of first training set images and processed second training set images, so that the neural network to be trained is continuously iterated and evolved during the training process, and the output image output by the decoder network is constantly "close" to the original image. In this process, the loss value calculated by the loss function is constantly close to the target loss value range.
[0096] When the loss value falls within the target loss value range, the loss function is considered to have converged. At this time, the output image of the neural network to be trained is very close to the original image. It can be determined that the trained neural network, that is, the above-mentioned image processing model, can accurately perform automatic white balance adjustment on "color cast" photos. At this time, the training is considered to be completed. In this process, the training is carried out in a back-propagation manner, which can accelerate the convergence of the network and improve the training efficiency.
[0097] In some implementations, during the training of the neural network to be trained, the Adam optimizer may be used to assist in the training, which can prevent the training results from falling into a "local optimum" on the one hand, and further accelerate the network convergence and improve the training efficiency on the other hand.
[0098] It can be understood that the above-mentioned target loss value range can be adjusted according to the training requirements and application iteration requirements. After obtaining an "image processing model" that meets the current requirements, the current version of the "image processing model" can continue to be trained during the use process, so as to obtain an "image processing model V2.0" with higher precision and more accurate white balance adjustment.
[0099] In some embodiments of the present application, the activation function is a linear rectification function, and the loss function is a loss function based on Wasserstein distance.
[0100] In an embodiment of the present application, the activation function is specifically a linear rectification function (Rectified Linear Unit, ReLU), which is a nonlinear activation function. The nonlinear activation function ReLU is used to process the product of the first feature map data obtained by the encoder according to the first training set image and the randomly selected color cast data, which can effectively increase the network sparsity, avoid overfitting, and improve the accuracy and effect of the image processing model.
[0101] Furthermore, the loss function selects a loss function based on the Wasserstein distance, which can narrow the distribution between the output image and the original image during the training process, and further improve the training effect and training efficiency of the training network.
[0102] In some embodiments of the present application, the loss function is:
[0103]
[0104] Among them, X is the input variable, Z is the encoded data, and Obj WCAE (P X ,P Y ) is the current loss value, is the expected value of the distribution of X, is the expected value of the conditional probability distribution of Z with respect to X. The function c(·) represents any measurable cost function, so c(X,G(Z)) is used to measure the distance between the output data of X and Z after decoding by the decoder. λ is the coefficient of the regularization term and λ is a constant. is the L2 norm of the encoder weights, is the L2 norm of the decoder weight, β is the coefficient of the sparse term and β is a constant, and ||h||1 is the L1 norm of the sparse term.
[0105] In the embodiment of the present application, the loss function used in the training process is specifically as shown in the above formula, wherein, in the training process, the encoder does not need to add any constraints, so that a relatively certain input-output mapping function can be obtained, thereby improving the training efficiency. Specifically, in the above loss function, its first term Represents the expectation of the cost function c(X,G(Z)) given the input data X, hidden layer data Z and decoder G.
[0106] In some embodiments of the present application, after the neural network to be trained is trained using the first training set images and the second training set images, the training method further includes: fixing the color cast weight value corresponding to the second feature map data.
[0107] In the embodiment of the present application, since no constraints are added to the encoder during the training process, the input-output mapping function obtained is relatively certain. The embodiment of the present application further adds a weight constraint item, that is, the above-mentioned color cast weight value, which can avoid overfitting and increase the sparse items of the hidden layer corresponding to the color cast, thereby improving the discrimination of the encoding, which is beneficial to improving the training effect.
[0108] In some embodiments of the present application, a method for processing white balance of an image is provided. Figure 6 FIG. 1 shows one of the flow charts of the white balance processing method of an image according to an embodiment of the present application. Figure 6As shown, the white balance processing method of the image includes:
[0109] Step 602, obtaining an image processing model;
[0110] In step 602, the image processing model is specifically an image processing model trained according to the image processing model training method provided in any of the above embodiments.
[0111] Step 604: perform white balance processing on the image to be processed through the image processing model to obtain a target image after white balance processing.
[0112] In an embodiment of the present application, an image to be processed is processed by an image processing model trained according to the image processing model training method provided in the embodiment of the above-mentioned image processing model training method, wherein the image to be processed is the original color cast photo. After the original color cast photo is input into the image processing model, the image processing model automatically adjusts the white balance of the color cast photo to obtain an adjusted picture whose color is closer to the actual color information observed by the human eye, thereby avoiding the situation where the color of the object in the captured image is different from the actual color of the object recognized by the human eye due to the difference between the image output by the sensor and the image viewed by the human eye, thereby improving the color accuracy of the captured photos and improving the user experience.
[0113] Among them, in the training method of the image processing model provided in the embodiment of the present application, the applied training data does not need to be annotated with real light sources, thereby saving a lot of annotation time. At the same time, since the training process does not require annotation information of the real light source, the existing massive data sets can be used to train the neural network, which greatly increases the optional connected data sets while reducing the difficulty of training and achieving efficient training that does not rely on real light source annotations.
[0114] In some embodiments of the present application, the image processing model includes an encoder network and a decoder network. Figure 7 FIG. 2 shows a second flowchart of a method for processing white balance of an image according to an embodiment of the present application. Figure 7 As shown, the image to be processed is processed by the image processing model, including:
[0115] Step 702, obtaining a color cast weight value;
[0116] Step 704, performing convolution processing on the image to be processed through the encoder network to obtain third feature map data corresponding to the image to be processed;
[0117] Step 706, performing a weighted operation on the third feature map data by using the color cast weight value to obtain fourth feature map data;
[0118] Step 708: Deconvolution is performed on the fourth feature map data through a decoder network to obtain a target image after white balance processing.
[0119] In an embodiment of the present application, during the training of the image processing model, since no constraints are added to the encoder, the input-output mapping function obtained is relatively certain. Therefore, after the training is completed, the weight of the input color cast data, that is, the above-mentioned color cast weight value, is fixed. During the white balance adjustment of the processed image through the image processing model, the processed image is convolved through the encoder network to obtain the third feature map data, and the third feature map data is weighted by the color cast weight value to obtain the fourth feature map data. The fourth feature map data is then deconvolved through the decoder network. The target image after adjusting the white balance can be more consistent with the actual color of the subject recognized by the human eye, thereby improving the color accuracy of the taken photos and improving the user experience.
[0120] In some embodiments of the present application, a white balance processing device for an image is provided. Figure 8 FIG. 4 shows a structural block diagram of a white balance processing device for an image according to an embodiment of the present application. Figure 8 As shown, the image white balance processing device 800 includes:
[0121] A second acquisition unit 802 is used to acquire an image processing model, wherein the image processing model is an image processing model;
[0122] The second processing unit 804 is used to perform white balance processing on the image to be processed through the image processing model to obtain a target image after white balance processing.
[0123] In an embodiment of the present application, an image to be processed is processed by an image processing model trained according to the image processing model training method provided in the embodiment of the above-mentioned image processing model training method, wherein the image to be processed is the original color cast photo. After the original color cast photo is input into the image processing model, the image processing model automatically adjusts the white balance of the color cast photo to obtain an adjusted picture whose color is closer to the actual color information observed by the human eye, thereby avoiding the situation where the color of the object in the captured image is different from the actual color of the object recognized by the human eye due to the difference between the image output by the sensor and the image viewed by the human eye, thereby improving the color accuracy of the captured photos and improving the user experience.
[0124] Among them, in the training method of the image processing model provided in the embodiment of the present application, the applied training data does not need to be annotated with real light sources, thereby saving a lot of annotation time. At the same time, since the training process does not require annotation information of the real light source, the existing massive data sets can be used to train the neural network, which greatly increases the optional connected data sets while reducing the difficulty of training and achieving efficient training that does not rely on real light source annotations.
[0125] In some embodiments of the present application, the image processing model includes an encoder network and a decoder network, and the second acquisition unit 802 is further used to obtain a color cast weight value;
[0126] The second processing unit 804 is also used to perform convolution processing on the image to be processed through the encoder network to obtain first feature map data corresponding to the image to be processed; perform weighted operation on the third feature map data through the color cast weight value to obtain fourth feature map data; and perform deconvolution processing on the fourth feature map data through the decoder network to obtain a target image.
[0127] In an embodiment of the present application, during the training of the image processing model, since no constraints are added to the encoder, the input-output mapping function obtained is relatively certain. Therefore, after the training is completed, the weight of the input color cast data, that is, the above-mentioned color cast weight value, is fixed. During the white balance adjustment of the processed image through the image processing model, the processed image is convolved through the encoder network to obtain the third feature map data, and the third feature map data is weighted by the color cast weight value to obtain the fourth feature map data. The fourth feature map data is then deconvolved through the decoder network. The target image after adjusting the white balance can be more consistent with the actual color of the subject recognized by the human eye, thereby improving the color accuracy of the taken photos and improving the user experience.
[0128] The training device of the image processing model and the image white balance processing device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0129] The image processing model training device and the image white balance processing device in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0130] Optionally, an embodiment of the present application also provides an electronic device 1900, including a processor 1910, a memory 1909, and a program or instruction stored in the memory 1909 and executable on the processor 1910. When the program or instruction is executed by the processor 1910, each process of the training method of the above-mentioned image processing model and the white balance processing method of the image is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0131] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0132] Fig. 9 A schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application is shown.
[0133] The electronic device 1900 includes but is not limited to components such as a radio frequency unit 1901, a network module 1902, an audio output unit 1903, an input unit 1904, a sensor 1905, a display unit 1906, a user input unit 1907, an interface unit 1908, a memory 1909, and a processor 1910.
[0134] Those skilled in the art will appreciate that the electronic device 1900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0135] It should be understood that in the embodiment of the present application, the radio frequency unit 1901 can be used to send and receive information or send and receive signals during a call, specifically, receive downlink data from a base station or send uplink data to a base station. The radio frequency unit 1901 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low color cast amplifier, a duplexer, etc.
[0136] The network module 1902 provides users with wireless broadband Internet access, such as helping users to send and receive emails, browse web pages, and access streaming media.
[0137] The audio output unit 1903 can convert the audio data received by the RF unit 1901 or the network module 1902 or stored in the memory 1909 into an audio signal and output it as sound. Moreover, the audio output unit 1903 can also provide audio output related to a specific function performed by the electronic device 1900 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 1903 includes a speaker, a buzzer, a receiver, etc.
[0138] The input unit 1904 is used to receive audio or video signals. The input unit 1904 may include a graphics processor (GPU) 5082 and a microphone 5084. The graphics processor 5082 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 1906, or stored in the memory 1909 (or other storage medium), or sent via the radio frequency unit 1901 or the network module 1902. The microphone 5084 can receive sound and can process the sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 1901 in the case of a telephone call mode.
[0139] The electronic device 1900 also includes at least one sensor 1905, such as a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, a motion sensor, and other sensors.
[0140] The display unit 1906 is used to display information input by the user or information provided to the user. The display unit 1906 may include a display panel 5122, which may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like.
[0141] The user input unit 1907 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 1907 includes a touch panel 5142 and other input devices 5144. The touch panel 5142 is also called a touch screen, which can collect the user's touch operations on or near it. The touch panel 5142 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, and converts it into contact coordinates, and then sends it to the processor 1910, receives the command sent by the processor 1910 and executes it. Other input devices 5144 may include but are not limited to physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here.
[0142] Further, the touch panel 5142 may be covered on the display panel 5122. When the touch panel 5142 detects a touch operation on or near it, it is transmitted to the processor 1910 to determine the type of the touch event, and then the processor 1910 provides a corresponding visual output on the display panel 5122 according to the type of the touch event. The touch panel 5142 and the display panel 5122 may be two independent components or integrated into one component.
[0143] The interface unit 1908 is an interface for connecting an external device to the electronic device 1900. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / 0) port, a video I / 0 port, a headphone port, etc. The interface unit 1908 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 1900 or may be used to transmit data between the electronic device 1900 and an external device.
[0144] The memory 1909 can be used to store software programs and various data. The memory 1909 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile terminal (such as audio data, a phone book, etc.), etc. In addition, the memory 1909 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0145] The processor 1910 performs various functions of the electronic device 1900 and processes data by running or executing software programs and / or modules stored in the memory 1909 and calling data stored in the memory 1909, thereby monitoring the electronic device 1900 as a whole. The processor 1910 may include one or more processing units; preferably, the processor 1910 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications.
[0146] The electronic device 1900 may also include a power supply 1911 for supplying power to various components. Preferably, the power supply 1911 may be logically connected to the processor 1910 via a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.
[0147] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the training method of the above-mentioned image processing model and the white balance processing method of the image are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0148] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0149] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing model training method and image white balance processing method embodiments, and can achieve the same technical effect. To avoid repetition, they are not described here.
[0150] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0151] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0153] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A training method for an image processing model, characterized in that: include: Obtain a first training set image; Performing color cast processing on the first training set images to obtain second training set images after color cast; Training the neural network to be trained by using the first training set images and the second training set images to obtain the image processing model for white balance processing; The neural network to be trained includes an encoder network, and the color cast processing is performed on the first training set image to obtain the second training set image, including: Performing convolution processing on the first training set image through the encoder network to obtain first feature map data; Get the preset color cast data and activation function; Adding the color cast data to the first feature map data through the activation function to obtain second feature map data with color cast; Generate the second training set image according to the second feature map data; The color cast data is added to the first feature map data through the activation function to obtain the second feature map data with color cast, including: bitwise multiplying the color cast data with the first feature map data, and obtaining the corresponding second feature map data through the activation function.
2. The training method of the image processing model according to claim 1, characterized in that: The neural network to be trained further includes a decoder network, and the training of the neural network to be trained using the first training set images and the second training set images includes: Performing deconvolution processing on the second training set image through the decoder network to obtain an output image; Obtaining a preset loss function, and inputting the output image and the first training set image into the loss function; Obtain a current loss value output by the loss function, and continue training the neural network to be trained with the target loss value range as the target until the current loss value falls within the target loss value range.
3. The training method of the image processing model according to claim 2, characterized in that: The activation function is a linear rectification function, and the loss function is a loss function based on Wasserstein distance.
4. The training method of the image processing model according to claim 2 or 3, characterized in that: The loss function is: Among them, X is the input variable, Z is the encoded data, and Obj WCAE (P X ,P Y ) is the current loss value, is the expected value of the distribution of X, E Q(Z|X) is the expected value of the conditional probability distribution of Z with respect to X, λ is the coefficient of the regularization term and λ is a constant, is the L2 norm of the encoder weights, is the L2 norm of the decoder weight, β is the coefficient of the sparse term and β is a constant, and ||h||1 is the L1 norm of the sparse term.
5. The method for training an image processing model according to any one of claims 1 to 3, characterized in that: After the neural network to be trained is trained using the first training set images and the second training set images, the training method further includes: The color cast weight value corresponding to the second feature map data is fixed.
6. A training device for an image processing model, characterized in that: include: A first acquisition unit, used to acquire a first training set image; A first processing unit, configured to perform color cast processing on the first training set image to obtain a second training set image after color cast; A training unit, which trains a neural network to be trained by using the first training set images and the second training set images to obtain the image processing model for white balance processing; The first processing unit is further used to perform convolution processing on the first training set image through an encoder network to obtain first feature map data; The first acquisition unit is also used to acquire preset color cast data and activation function; The first processing unit is further configured to add the color cast data to the first feature map data through the activation function to obtain the second feature map data with color cast; and generate the second training set image according to the second feature map data; The color cast data is added to the first feature map data through the activation function to obtain the second feature map data with color cast, including: bitwise multiplying the color cast data with the first feature map data, and obtaining the corresponding second feature map data through the activation function.
7. A method for processing white balance of an image, characterized in that: include: Obtain the image processing model trained according to the image processing model training method according to any one of claims 1 to 5; The image to be processed is subjected to white balance processing by the image processing model to obtain a target image after white balance processing.
8. The image white balance processing method according to claim 7, characterized in that: The image processing model includes an encoder network and a decoder network, and performing white balance processing on the image to be processed by the image processing model includes: Get the color cast weight value; Performing convolution processing on the image to be processed through the encoder network to obtain third feature map data corresponding to the image to be processed; Performing a weighted operation on the third feature map data by using the color cast weight value to obtain fourth feature map data; The fourth feature map data is subjected to deconvolution processing through the decoder network to obtain the target image after white balance processing.
9. A white balance processing device for an image, characterized in that: include: A second acquisition unit, used to acquire the image processing model trained by the image processing model training method according to any one of claims 1 to 5; The second processing unit is used to perform white balance processing on the image to be processed through the image processing model to obtain a target image after white balance processing.
10. An electronic device, characterized in that: It comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the training method of the image processing model as described in any one of claims 1 to 5; and / or the white balance processing method of the image as described in claim 7 or 8.
11. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, they implement the training method of the image processing model as described in any one of claims 1 to 5; and / or the white balance processing method of the image as described in claim 7 or 8.
