White balance correction model training method, white balance correction method, device and medium

Abstract

The present application provides a white balance correction model training method, a white balance correction method, a device and a medium, and relates to the field of image processing technology. The method comprises: obtaining an image data set, the image data set comprises a plurality of groups of sample images, each group of sample images comprises a first single light source sample image and a first multi-light source sample image of a target object acquired by an image acquisition device having a three-color light source, determining a first standard light source sample image according to the first single light source sample image and the first multi-light source sample image; inputting the first multi-light source sample image into a pre-trained neural network model for prediction to obtain a first standard light source prediction image; optimizing the pre-trained neural network model according to the first standard light source sample image and the first standard light source prediction image to obtain a white balance correction model, so as to white balance the multi-light source image and output a standard light source image. The present application can realize automatic white balance correction of images taken under complex lighting conditions.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a white balance correction model training method, a white balance correction method, a device and a medium. Background Art

[0002] When using an image acquisition device to take images, it is important to consider that the captured image of the subject is affected by the device's own components and white balance mechanism. Under the same shooting conditions, different devices may have different imaging effects, and complex ambient lighting may affect the final imaging visual effect of the subject.

[0003] The imaging mechanism of existing image acquisition devices is that the photosensitive chip receives light information, the red, green and blue primary color diodes sense the corresponding signals according to the brightness difference, interpolate to obtain the three primary color values ​​of each pixel, and automatically perform white balance correction according to the preset algorithm of the device.

[0004] However, images that have been automatically white-balanced are usually unable to meet the color requirements of specific scenes, and further artificial correction of the influence of ambient light is required to make the subject close to its original color. Summary of the invention

[0005] The purpose of the present application is to provide a white balance correction model training method, a white balance correction method, a device and a medium to address the deficiencies in the above-mentioned prior art, so as to realize automatic white balance correction of images taken under complex lighting environments.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for training a white balance correction model, the method comprising:

[0008] Acquire an image data set, the image data set at least comprising: a plurality of groups of sample images, each group of sample images comprising: a first single-light source sample image and a first multi-light source sample image of a target object acquired by an image acquisition device having a three-color light source, wherein the first single-light source sample image is an image generated based on the three-color light source, and the first multi-light source sample image is an image generated based on the three-color light source and an ambient light source;

[0009] Determine a first standard light source sample image according to the first single light source sample image and the first multi-light source sample image;

[0010] Inputting the first multi-light source sample image into a pre-trained neural network model for prediction to obtain a first standard light source prediction image;

[0011] The pre-trained neural network model is optimized according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model, and the white balance correction model is used to perform white balance on a multi-light source image and output a standard light source image.

[0012] Optionally, acquiring the image data set includes:

[0013] The image acquisition device is used to acquire an image of the target object, and the initial image of the target object obtained includes: a red mixed light source image and a corresponding first ambient light source image, a green mixed light source image and a corresponding second ambient light source image, and a blue mixed light source image and a corresponding third ambient light source image, wherein the mixed light source images of corresponding colors are images captured during the process of the three-color fill light being turned on and off respectively within three preset exposure times, and the corresponding ambient light source images are images captured during the process of the three-color fill light being not turned on within three target exposure times, and each target exposure time is the off time of the corresponding fill light within the preset exposure time;

[0014] generating the first multi-light source sample image according to the red mixed light source image, the green mixed light source image and the blue mixed light source image;

[0015] The first single light source sample image is generated according to the red mixed light source image, the first ambient light source image, the green mixed light source image, the second ambient light source image, the blue mixed light source image and the third ambient light source image.

[0016] Optionally, before using the image acquisition device to acquire an image of the target object, the method further includes:

[0017] The image acquisition device is used to shoot a preset calibration color card when the three-color fill lights are turned on respectively, to obtain three grayscale images;

[0018] Fusing the three grayscale images to obtain a fused image;

[0019] According to the R value, G value and B value of the grayscale color block in the fused image, the lighting time ratio of the three-color fill light is adjusted to calibrate the image acquisition device. After the calibration, the R value, G value and B value of the grayscale color block in the fused image corresponding to the image acquisition device are equal.

[0020] Optionally, the sample image captured by the image acquisition device includes a preset calibration color chart in addition to the target object, and determining the first standard light source sample image according to the first single light source sample image and the first multi-light source sample image includes:

[0021] Determine an ambient light source sample image according to the first single light source sample image and the first multi-light source sample image;

[0022] Determining a first chromaticity vector of the three-color light source according to a pixel value of a grayscale color block of a preset calibration color card in the first single-light source sample image;

[0023] Determining a second chromaticity vector of the ambient light source according to a pixel value of a grayscale color block of the preset correction color card in the ambient light source sample image;

[0024] Determining a light source coefficient according to pixel intensities of the first single light source sample image and the ambient light source sample image;

[0025] The first multi-light source sample image is calibrated according to the first chromaticity vector, the second chromaticity vector and the light source coefficient by using a preset white balance mathematical model to determine the first standard light source sample image.

[0026] Optionally, before inputting the first multi-light source sample image into a pre-trained neural network model for prediction to obtain a first standard light source prediction image, the method further includes:

[0027] Determine a second standard light source sample image according to a second single light source sample image and a second multi-light source sample image, wherein the second single light source sample image and the second multi-light source sample image are sample images in a public data set;

[0028] Inputting the second multi-light source sample image into the initial neural network model to obtain a second standard light source prediction image;

[0029] The initial neural network model is optimized according to the second standard light source sample image and the second standard light source predicted image to obtain the pre-trained neural network model.

[0030] Optionally, the pre-trained neural network model includes: an encoder and a decoder, the encoder includes: a convolution layer and a multi-layer visual conversion layer, and the pre-trained neural network model is optimized according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model, including:

[0031] According to the first standard light source sample image and the first standard light source predicted image, only the parameters of the convolution layer and the decoder are optimized to obtain the white balance correction model.

[0032] Optionally, optimizing the pre-trained neural network model according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model includes:

[0033] Calculating object edge loss and global color loss of the first standard light source sample image and the first standard light source prediction image;

[0034] The pre-trained neural network model is optimized according to the object edge loss and the global color loss to obtain the white balance correction model.

[0035] In a second aspect, an embodiment of the present application further provides a white balance correction method, the method comprising:

[0036] An image acquisition device with a three-color light source is used to acquire multi-light source images of a preset application scenario;

[0037] A pre-trained white balance correction model is used to perform white balance correction on the multi-light source image to output a standard light source image, wherein the white balance correction model is trained using the training method for the white balance correction model as described in any one of the first aspects.

[0038] In a third aspect, an embodiment of the present application provides a training device for a white balance correction model, the device comprising:

[0039] An image acquisition module is used to acquire an image data set, wherein the image data set at least includes: a plurality of groups of sample images, each group of sample images includes: a first single-light source sample image and a first multi-light source sample image of a target object acquired by an image acquisition device having a three-color light source, wherein the first single-light source sample image is an image generated based on the three-color light source, and the first multi-light source sample image is an image generated based on the three-color light source and an ambient light source;

[0040] A standard image generating module, configured to determine a first standard light source sample image according to the first single light source sample image and the first multi-light source sample image;

[0041] A model prediction module, used for inputting the first multi-light source sample image into a pre-trained neural network model for prediction to obtain a first standard light source prediction image;

[0042] A model correction module is used to optimize the pre-trained neural network model according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model, wherein the white balance correction model is used to white balance the multi-light source image and output a standard light source image.

[0043] Optionally, the image acquisition module is specifically used to use the image acquisition device to perform image acquisition on the target object, and the initial image of the target object obtained includes: a red mixed light source image and a corresponding first ambient light source image, a green mixed light source image and a corresponding second ambient light source image, a blue mixed light source image and a corresponding third ambient light source image, the mixed light source images of corresponding colors are images captured during the process of the three-color fill light being turned on and off respectively within three preset exposure times, and the corresponding ambient light source images are images captured during the three target exposure times when the three-color fill light is not turned on, and each target exposure time is the off time of the corresponding fill light within the preset exposure time; the first multi-light source sample image is generated according to the red mixed light source image, the green mixed light source image and the blue mixed light source image; the first single light source sample image is generated according to the red mixed light source image, the first ambient light source image, the green mixed light source image, the second ambient light source image, the blue mixed light source image and the third ambient light source image.

[0044] Optionally, the device further comprises:

[0045] The device calibration module is used to use the image acquisition device to shoot a preset correction color card when the three-color fill light is turned on respectively to obtain three grayscale images; fuse the three grayscale images to obtain a fused image; adjust the lighting time ratio of the three-color fill light according to the R value, G value and B value of the grayscale color block in the fused image to calibrate the image acquisition device, so that the R value, G value and B value of the grayscale color block in the fused image corresponding to the image acquisition device after the calibration is completed are equal.

[0046] Optionally, in addition to the target object, the sample image captured by the image acquisition device also includes a preset correction color card, and the standard image generation module is specifically used to determine the ambient light source sample image based on the first single light source sample image and the first multi-light source sample image; determine the first chromaticity vector of the three-color light source based on the pixel values ​​of the grayscale color blocks of the preset correction color card in the first single light source sample image; determine the second chromaticity vector of the ambient light source based on the pixel values ​​of the grayscale color blocks of the preset correction color card in the ambient light source sample image; determine the light source coefficient based on the pixel intensities of the first single light source sample image and the ambient light source sample image; and calibrate the first multi-light source sample image using a preset white balance mathematical model based on the first chromaticity vector, the second chromaticity vector and the light source coefficient to determine the first standard light source sample image.

[0047] Optionally, the standard image generation module is further used to determine a second standard light source sample image according to a second single light source sample image and a second multi-light source sample image, wherein the second single light source sample image and the second multi-light source sample image are sample images in a public data set;

[0048] The model prediction module is further used to input the second multi-light source sample image into the initial neural network model to obtain a second standard light source prediction image;

[0049] The model correction module is further used to optimize the initial neural network model according to the second standard light source sample image and the second standard light source predicted image to obtain the pre-trained neural network model.

[0050] Optionally, the pre-trained neural network model includes: an encoder and a decoder, the encoder includes: a convolutional layer and a multi-layer visual conversion layer, and the model correction module is specifically used to optimize only the parameters of the convolutional layer and the decoder according to the first standard light source sample image and the first standard light source predicted image to obtain the white balance correction model.

[0051] Optionally, the model correction module is specifically used to calculate the object edge loss and global color loss of the first standard light source sample image and the first standard light source predicted image; according to the object edge loss and the global color loss, the pre-trained neural network model is optimized to obtain the white balance correction model.

[0052] In a fourth aspect, an embodiment of the present application further provides a white balance correction device, the device comprising:

[0053] An image acquisition module, used to acquire multi-light source images of a preset application scene using an image acquisition device with a three-color light source;

[0054] An image correction module is used to perform white balance correction on the multi-light source image using a pre-trained white balance correction model to output a standard light source image, wherein the white balance correction model is trained using the training method for the white balance correction model as described in any one of the first aspects.

[0055] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to execute the steps of the training method of the white balance correction model as described in any one of the first aspect, or executes the steps of the white balance correction method as described in the second aspect.

[0056] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program executes the steps of the training method of the white balance correction model as described in any one of the first aspects, or executes the steps of the white balance correction method as described in the second aspect.

[0057] The beneficial effects of this application are:

[0058] The white balance correction model training method, white balance correction method, device and medium provided in the present application adopt a pre-trained neural network model for training, which can reduce the requirement for the number of samples in the preset application scenario and improve the training efficiency; the white balance correction module obtained after the training is completed can automatically perform white balance correction on images taken under complex lighting conditions in the preset application scenario, and there is no need to place a correction color card when taking pictures of objects in the preset application scenario. It can generate a real color image that conforms to the visual perception of the human eye and highlights the fine texture structure of the photographed object, saving manpower and time, and can be applied to scenes with high requirements on the lighting effect of the object image. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of the structure of an image acquisition device provided in an embodiment of the present application;

[0061] Figure 2 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 1 ;

[0062] Figure 3 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0063] Figure 4 This is the architecture diagram of Mobile-ViT Block;

[0064] Figure 5 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 2 ;

[0065] Figure 6 A schematic diagram of the exposure time and the on and off time of the fill light provided in an embodiment of the present application;

[0066] Figure 7 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 3 ;

[0067] Figure 8 A schematic diagram of a correction color block;

[0068] Fig. 9 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 4 ;

[0069] Fig.10 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 5 ;

[0070] Fig.11 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 6 ;

[0071] Fig.12 A schematic diagram of a flow chart of a white balance correction method provided in an embodiment of the present application;

[0072] Fig.13 A schematic diagram of the structure of a training device for a white balance correction model provided in an embodiment of the present application;

[0073] Fig.14 A schematic diagram of the structure of a white balance correction device provided in an embodiment of the present application;

[0074] Fig.15 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0076] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0077] In addition, the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0078] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0079] Traditional methods of white balance correction of images are mostly based on estimating the spectral sensitivity and device response of the image acquisition device based on statistical models, or using standard color cards for color correction. However, since it is difficult to build a shooting environment with a standard light source during the shooting process, and in some scenes such as outdoor shooting, it is even more impossible to guarantee a single lighting. For the problem of local color cast, existing methods mostly perform white balance correction on a single lighting in a color image, and there is little research on complex lighting scenes. In addition, in application scenarios with high real-time requirements, manually laying out standard color cards is time-consuming and labor-intensive, and at the same time, due to the limited field of view, the placement of multiple color cards is easily restricted.

[0080] Based on the problems existing in the above-mentioned prior art, the present application intends to provide a training method for a white balance correction model and a white balance correction method, and trains a neural network model based on single-light source sample images and multi-light source sample images, so that the trained white balance correction model can be used to perform white balance correction on multi-light source images and output standard light source images. In this way, when capturing images of the subject, there is no need to place a correction color card, and white balance correction can be automatically performed through the white balance correction model, saving manpower time and improving the efficiency of white balance correction.

[0081] The training method of the white balance correction model and the white balance correction method provided in the present application can be applied to general application scenarios as well as specific scenarios. For example, the specific scenarios may be cultural relic shooting scenes, etc., where it is necessary to restore the image of the fine texture structure of the subject and avoid the influence of the light source in the scene on the fine texture structure of the subject.

[0082] Before introducing the training method of the white balance correction model and the white balance correction method provided in the present application, the image acquisition device used in the present application is first introduced.

[0083] Specifically, Figure 1 A schematic diagram of the structure of the image acquisition device provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the image acquisition device may include: a lens 11 , a three-color fill light 12 and a trigger button 13 .

[0084] Unlike traditional color cameras that directly capture color images, this application uses a self-developed camera that can project stable red, green, and blue narrow-band light sources onto the subject respectively. The red fill light can project a light source with a band of 620-630nm, the green fill light can project a light source with a band of 520-535nm, and the blue fill light can project a light source with a band of 440-455nm.

[0085] The software corresponding to the image acquisition device can be used to set the camera exposure time of the image acquisition device and the lighting time of the three-color flash to perform image acquisition. When the image acquisition device performs image acquisition, it can be triggered once by the trigger button 13, and the image acquisition device performs three exposures, and sequentially captures images of the three-color flash being lit and extinguished respectively. It can also be triggered three times by the trigger button 13, and the image acquisition device performs three exposures, each time capturing an image of the flash being lit and extinguished. It can also be exposed three times, and the images of the three-color flash not being lit are captured respectively.

[0086] In some embodiments, there may be three trigger buttons 13 to respectively control the lighting of three-color flash lights.

[0087] It should be noted that the training method of the white balance correction model provided in the present application can be executed in a computer device, the white balance correction method can be directly executed in an image acquisition device, or can be executed in a computer device. When the white balance correction method is executed in an image acquisition device, the trained white balance correction model needs to be deployed in the image acquisition device, and this embodiment does not impose any restrictions on this.

[0088] The following describes the specific implementation of the white balance correction model training method, white balance correction method, white balance correction model training device, white balance correction device, electronic device and storage medium provided in the present application in combination with the embodiments.

[0089] Figure 2 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 1 ,like Figure 2 As shown, the method may include:

[0090] S101. Acquire an image data set, where the image data set at least includes: multiple groups of sample images, each group of sample images includes: a first single-light source sample image and a first multi-light source sample image of a target object acquired by an image acquisition device having a three-color light source, wherein the first single-light source sample image is an image generated based on the three-color light source, and the first multi-light source sample image is an image generated based on the three-color light source and an ambient light source.

[0091] In this embodiment, if Figure 1 As shown, the image acquisition device has a three-color fill light that can emit red, green, and blue light sources respectively. The image acquisition device is used to take pictures of objects in a preset application scene to construct an image data set for the preset application scene.

[0092] Specifically, image acquisition is performed on multiple objects in a preset application scene to obtain multiple groups of sample images for the multiple objects, wherein, for each target object in the preset application scene, an image acquisition device with a three-color light source is used to photograph each target object, and by controlling the three-color light source and the ambient light source in the scene where the target object is located to light up, a first multi-light source sample image including the three-color light source and the ambient light source is captured.

[0093] In some embodiments, the ambient light of the scene where the target object is located is complex, such as the light source reflected by the surface of other objects, and it is difficult to control the ambient light source to be extinguished. Therefore, the three-color light source can be controlled to be extinguished, and only the ambient light source can be retained to obtain an image containing only the ambient light source. The lighting information of the image containing only the ambient light source is removed from the lighting information of the first multi-light source sample image to obtain a first single-light source sample image containing only the three-color light source.

[0094] It should be noted that the single light source in the first single light source sample image refers to a mixed light source of three-color lights, and the mixed light source does not represent multiple light sources. The multiple light sources in this embodiment refer to the mixed light source corresponding to the three-color lights and the ambient light source.

[0095] S102: Determine a first standard light source sample image according to the first single light source sample image and the first multi-light source sample image.

[0096] In this embodiment, according to the first single light source sample image Pixel values, calculate the first single light source sample image The light source chromaticity l1 of the three-color light source and the illumination weight α1 of each pixel are calculated according to the first single light source sample image. and the first multi-light sample image The pixel value of the ambient light source is used to calculate the light source chromaticity l2 and the lighting weight α2 of each pixel.

[0097] Determine the white balance correction matrix of multiple light sources based on the light source chromaticity l1 and lighting weight α1 of the three-color light source, the light source chromaticity l2 and lighting weight α2 of the ambient light source The first multi-light source sample image is corrected according to the white balance correction matrix Perform calibration to determine the first standard light source sample image The first standard light source sample image As a reference image of the target object under a standard light source, it is used to participate in the calculation of the loss function during the training of the neural network model.

[0098] Example, first standard light source sample image The calculation formula can be expressed as:

[0099]

[0100] S103: Input the first multi-light source sample image into a pre-trained neural network model for prediction to obtain a first standard light source prediction image.

[0101] In this embodiment, the pre-trained neural network model is a neural network model that has been preliminarily trained in advance. For example, a neural network model can be obtained by training with a data set that does not limit the application scenario. If the number of sample images that can be collected in the preset application scenario is small and cannot meet the needs of model training, before using the sample images of the preset application scenario for model training, a data set of a general scenario can be used to train the model to obtain a pre-trained neural network model. Thereafter, the pre-trained neural network model can be refined by using the sample images of the preset application scenario.

[0102] Specifically, the first multi-light source sample image is input into the pre-trained neural network model, the pre-trained neural network model extracts features from the first multi-light source sample image, and determines standard light source information according to the extracted features to output a first standard light source predicted image.

[0103] For example, Figure 3 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the pre-trained neural network model is a U-Net network composed of an encoder and a decoder, wherein the encoder includes a convolution layer and a multi-layer visual conversion layer, the visual conversion layer can use Mobile-ViT Block, the convolution layer and the multi-layer Mobile-ViT Block are used to downsample the input first multi-light source sample image, gradually reduce the image resolution, and increase the number of channels to extract the abstract features of the image, and the decoder includes a pooling layer and a convolution layer, which gradually restores the image resolution through upsampling, and finally outputs the first standard light source prediction image.

[0104] For example, Figure 4 The architecture diagram of Mobile-ViT Block is shown in Figure 1. Figure 4 As shown, the principle of Mobile-ViT Block is: for the input tensor X∈R H×W×C The tensor is projected into a high-dimensional space (d>C) by learning the linear combination of input channels through n×n standard convolutional layers and 1×1 convolutional layers, generating X∈R H×W×d ; Further expand the high-dimensional tensor into N non-overlapping flat patches X∈R P×N×d , where P = w*h, N = H*W, P is the number of patches, h≤n and w≤n are the height and width of the patch respectively; for each p∈{1,····,P}, the relationship between patches is encoded by applying Transformer; fold X∈R P×N×d Get X∈R H×W×d ; Point-wise convolution projects X into C-dimensional space; concatenates with the input tensor and fuses the concatenated features using another n×n convolutional layer.

[0105] S104. Optimize the pre-trained neural network model according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model, where the white balance correction model is used to perform white balance on the multi-light source image and output a standard light source image.

[0106] In this embodiment, the image is predicted according to the first standard light source. and the first standard light source sample image The error between the two is used to adjust the model parameters of the pre-trained neural network model through the back propagation algorithm, so that the model can better adapt to the specific lighting environment of the preset application scenario. After multiple rounds of training of the pre-trained neural network model, when the training rounds reach the preset rounds, or the error value converges, the training of the pre-trained neural network model is completed, and a white balance correction model is obtained.

[0107] Since Mobile-ViT learns a global representation with spatial inductive bias, the U-Net network with Mobile-ViT Block added is applied to the white balance correction model, and the high-level features of the multi-light source image are extracted layer by layer. The corrected image in the multi-light source scene is generated through the decoder to eliminate the influence of multiple light sources. The white balance correction model can effectively encode the local and global information of the image tensor, perform white balance correction on the multi-light source color cast image, and improve the overall computational efficiency of the model, especially in application scenarios such as processing large-scale images or with high real-time requirements.

[0108] The training method of the white balance correction model provided in the above embodiment adopts a pre-trained neural network model for training, which can reduce the requirement for the number of samples in the preset application scenario and improve the training efficiency; the white balance correction module obtained after the training is completed can automatically perform white balance correction on the images taken under the complex lighting environment in the preset application scenario, and there is no need to place a correction color card when taking pictures of the objects in the preset application scenario. It can generate a real color image that conforms to the visual perception of the human eye and highlights the fine texture structure of the photographed object, saving manpower and time, and can be applied to scenes with high requirements on the lighting effect of the object image.

[0109] In one possible implementation, Figure 5 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 2 ,like Figure 5 As shown, the process of obtaining the image data set in S101 may include:

[0110] S201: Use an image acquisition device to acquire an image of a target object to obtain an initial image of the target object.

[0111] Among them, the initial image may include: a red mixed light source image and a corresponding first ambient light source image, a green mixed light source image and a corresponding second ambient light source image, a blue mixed light source image and a corresponding third ambient light source image, the mixed light source images of corresponding colors are images taken when the three-color fill light is turned on and off respectively within three preset exposure times, and the corresponding ambient light source images are images taken when the three-color fill light is not turned on within three target exposure times, and the target exposure time of each time is the off-time of the corresponding fill light within the preset exposure time.

[0112] In this embodiment, a target object is photographed by an image acquisition device. During the photographing process, within a preset exposure time of the image acquisition device, only one fill light is on, and the time the fill light is on is less than or equal to the preset exposure time. Within one exposure time, in the process of the fill light being turned on and off, the image acquired by the image acquisition device is a mixed light source image including the fill light source and the ambient light source. During the time period when the fill light is off, the image acquisition device is still acquiring images, but because the fill light is not on, only the lighting information brought by the ambient light is acquired. In order to collect the lighting information brought only by ambient light in each mixed light source image, the off-time of each fill light can be determined according to the preset exposure time and the on-time of each fill light within the preset exposure time. The corresponding target exposure time is determined according to the off-time of each fill light. The target object is photographed with the corresponding target exposure time. The collected image is an ambient light source image containing only ambient light. If the on-time of the fill light is equal to the exposure time, the ambient light source image is an image that does not contain any pixel information.

[0113] Specifically, Figure 6 The schematic diagram of the exposure time and the on and off time of the fill light provided in the embodiment of the present application is as follows: Figure 6 As shown, the sum of the on time and off time of each fill light is equal to the first preset exposure time of the image acquisition device. Within one preset exposure time, when the red fill light is turned on and off, the image acquired by the image acquisition device is a red mixed light source image including the red fill light source and the ambient light source. Within another preset exposure time, when the green fill light is turned on and off, the image acquired by the image acquisition device is a green mixed light source image including the green fill light source and the ambient light source. Within another preset exposure time, when the blue fill light is turned on and off, the image acquired by the image acquisition device is a blue mixed light source image including the blue fill light source and the ambient light source.

[0114] According to the preset exposure time and the on time of the red fill light, the off time of the red fill light is determined, and the first target exposure time is determined according to the off time of the red fill light. The target object is imaged within the first target exposure time, and the image acquired is a first ambient light image that only contains the ambient light source; according to the preset exposure time and the on time of the green fill light, the off time of the green fill light is determined, and the second target exposure time is determined according to the off time of the green fill light. The target object is imaged within the second target exposure time, and the image acquired is a second ambient light image that only contains the ambient light source; according to the preset exposure time and the on time of the blue fill light, the off time of the blue fill light is determined, and the third target exposure time is determined according to the off time of the blue fill light. The target object is imaged within the third target exposure time, and the image acquired is a third ambient light image that only contains the ambient light source.

[0115] S202: Generate a first multi-light source sample image according to the red mixed light source image, the green mixed light source image, and the blue mixed light source image.

[0116] In this embodiment, the red mixed light source image, the green mixed light source image and the blue mixed light source image are fused using the red, green and blue channels of the color image respectively to obtain a first multi-light source sample image. First multi-light source sample image It can be regarded as a color image taken by a color camera under multiple light sources (ambient light and mixed light of fill light that causes color shift).

[0117] It should be noted that the single image captured by the image capture device provided in this embodiment is a grayscale image, and the images corresponding to the three-color fill light need to be fused in the red, green and blue channels to obtain a color image.

[0118] S203 . Generate a first single light source sample image according to the red mixed light source image, the first ambient light source image, the green mixed light source image, the second ambient light source image, the blue mixed light source image, and the third ambient light source image.

[0119] In this embodiment, since the image captured when the three-color fill light is on will be affected by the ambient light source, it is necessary to remove the influence of the ambient light source from the red mixed light source image, the green mixed light source image and the blue mixed light source image, respectively, and fuse the images without the influence of the ambient light source in the red, green and blue channels to obtain a first single light source sample image that only contains the mixed light of the three-color fill light.

[0120] Specifically, a subtraction operation is performed pixel by pixel on the red mixed light source image and the first ambient light source image to obtain a red single light source image; a subtraction operation is performed pixel by pixel on the green mixed light source image and the second ambient light source image to obtain a green single light source image; a subtraction operation is performed pixel by pixel on the blue mixed light source image and the pixel value of the third ambient light source image to obtain a blue single light source image; the red single light source image, the green single light source image and the blue single light source image are fused in the red, green and blue channels to obtain a first single light source sample image First single light source sample image It can be regarded as a color image taken by a color camera under a single light source (mixed light of the fill light that causes color shift) environment.

[0121] For example, the calculation formula of the single light source sample image of each light source can be expressed as:

[0122]

[0123] Among them, P gr , P gg , P gb It represents the grayscale images captured during the preset exposure time of the image acquisition device when the three-color fill light is turned on and off respectively. gΔtr , P gΔtg , P gΔtb represents the target exposure time set according to the off time of the three-color fill light, the grayscale image captured by the image acquisition device, P gR , P gG , P gB Represents the single light source image corresponding to the three-color fill light.

[0124] Different objects in a preset application scene are photographed at different exposure times, and the above S201-S203 are used for processing to obtain a plurality of first single-light source sample images and first multi-light source sample images, so as to complete the self-construction of the image data set of the preset application scene.

[0125] It should be noted that the preset application scenario is an application scenario of a preset type, such as a cultural relic shooting scene, rather than a single scene.

[0126] It should be further explained that the preset correction color card part contained in the first multi-light source sample image can be deleted by taking a screenshot before inputting the pre-trained neural network model, or a mask can be set in the preset correction color card area so that the RGB value of the preset correction color card area is (0, 0, 0), and the preset correction color card area does not participate in the white balance prediction of the model.

[0127] In some embodiments, Figure 7 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 3 ,like Figure 7 As shown, before the above S201 uses the image acquisition device to acquire an image of the target object, the method may further include:

[0128] S301, using an image acquisition device to shoot a preset calibration color card when three-color fill lights are turned on respectively, to obtain three grayscale images.

[0129] S302: Fuse the three grayscale images to obtain a fused image.

[0130] S303. According to the R value, G value and B value of the grayscale color block in the fused image, adjust the lighting time ratio of the three-color fill light to calibrate the image acquisition device. After the calibration, the R value, G value and B value of the grayscale color block in the fused image corresponding to the image acquisition device are equal.

[0131] In this embodiment, the exposure time of the image acquisition device and the lighting time of each of the three-color fill lights within the exposure time are set. The preset correction color card includes multiple rows of correction color blocks. Figure 8 is a schematic diagram of a correction color block, such as Figure 8 As shown, one row of correction color blocks is a grayscale color block, which is composed of a white correction color block, a black correction color block and a gray correction color block. An image acquisition device is used to shoot a preset correction color card. When the three-color fill light is turned on respectively, the image acquisition device projects the light source of the three-color fill light onto the correction color card to acquire three grayscale images. The three grayscale images are fused in the red, green and blue channels to obtain a fused image, and the fused image is a color image.

[0132] In order to avoid significant color cast in the fused image, it is necessary to calibrate the on-time ratio of the three-color fill light based on the grayscale color block in the correction color card. Specifically, the on-time ratio of the three-color fill light within the exposure time is adjusted according to the R, G, B values ​​of the grayscale color block in the fused image, so that the R, G, B values ​​of the grayscale color block in the fused image are equal or approximately equal. Approximately equal means that the difference between the R, G, B values ​​is less than a preset threshold.

[0133] It should be noted that after setting a different exposure time and the lighting time of the three-color fill light each time, it is necessary to use the above steps S301-S303 to calibrate the lighting time ratio of the three-color fill light of the image acquisition device to avoid large color deviation caused by the three-color fill light.

[0134] The training method of the white balance correction model provided in the above embodiment generates a first multi-light source sample image and a first single-light source sample image based on the mixed light source image when the three-color fill light is on and the ambient light source image when the three-color fill light is off, which is captured by the image acquisition device, so as to self-build an image data set for a preset application scenario, so that the white balance correction model trained using the image data set is more matched with the preset application scenario, thereby improving the effect of the white balance correction model on white balance correction of images in the preset application scenario.

[0135] In one possible implementation, Fig. 9 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 4 ,like Fig. 9 As shown, the process of determining the first standard light source sample image according to the first single light source sample image and the first multi-light source sample image in S102 may include:

[0136] S401 : Determine an ambient light source sample image according to a first single-light source sample image and a first multi-light source sample image.

[0137] In this embodiment, the first single-light source sample image is a sample image corresponding to the three-color light source 1, and the first multi-light source sample image is a sample image corresponding to the mixed light source 12, i.e., the three-color light source 1 and the ambient light source 2. The first single-light source sample image is subtracted from the first multi-light source sample image to obtain the ambient light source sample image.

[0138] The image captured under the environment corresponding to the mixed light source can be modeled as follows:

[0139] P RGB (x,y)=r(x,y)⊙η(x,y)l

[0140] Where P is an RGB image, r is the surface reflectance of the RGB image, η is a scaling term that includes the intensity of illumination and shadow, l is the light source chromaticity vector of the RGB image, (x, y) is the pixel position, and ⊙ represents element-by-element multiplication.

[0141] According to the above formula, the calculation formula of the ambient light source sample image can be expressed as:

[0142]

[0143] S402: Determine a first chromaticity vector of the three-color light source according to the pixel value of the grayscale color block of the preset calibration color card in the first single-light source sample image.

[0144] In this embodiment, when an image acquisition device is used to photograph a target object, a preset correction color card is placed around the target object so that the captured image contains both the target object and the correction color card. The chromaticity vector of the light source can be calculated based on the pixel values ​​of the grayscale color blocks in the correction color card.

[0145] Specifically, an affine transformation is performed on the preset correction color card in the first single light source sample image to obtain a correction color card of a fixed size, and the R, G, and B three-channel intensity values ​​of each pixel point of the grayscale color block in the correction color card are accumulated respectively. For the accumulated R, G, and B three-channel intensity values, the green channel is normalized, and the normalized R, G, and B three-channel intensity values ​​of the green channel are used as the first chromaticity vector l1 of the three-color light source.

[0146] In some embodiments, the first chromaticity vector l1 of the three-color light source may be calculated by accumulating the intensity values ​​of the three channels R, G, and B of each pixel point of the grayscale color block with the highest brightness in the calibration color card.

[0147] It should be noted that the Bayer color filter array (Bayer pattern) commonly used in digital cameras is most sensitive to green light. Therefore, when performing white balance processing, using the data of the green channel provides more accurate and stable brightness information.

[0148] S403: Determine a second chromaticity vector of the ambient light source according to the pixel value of the grayscale color block of the preset calibration color card in the ambient light source sample image.

[0149] In this embodiment, the second chromaticity vector l2 of the ambient light source is calculated by the same method as that of calculating the first chromaticity vector of the first single light source sample image. The specific method is as described in S402 above and will not be described in detail here.

[0150] S404: Determine a light source coefficient according to pixel intensities of the first single light source sample image and the ambient light source sample image.

[0151] In this embodiment, the preset white balance mathematical model can be expressed as the following formula:

[0152]

[0153] in, α(x,y) and 1-α(x,y) are the illumination weights of the three-color light source and the ambient light source at the pixel (x,y), namely α1 and α2, respectively.

[0154] According to the above formula, P is proportional to η, and the pixel intensities of P1 and P2 can be used as approximate values ​​of η1 and η2, respectively. In the RGB three-color channels, since the sensitivity of the Bayer pattern sensor of the digital camera is the highest in the green channel, the green channel is used to calculate the illumination coefficient α. The green channel is normalized by the ratio of the green channel values ​​of the first single light source sample image and the ambient light source sample image. The illumination coefficient α is calculated based on the pixel intensities of the first single light source sample image and the ambient light source sample image after green channel normalization. The calculation formula of the illumination coefficient α can be converted to:

[0155]

[0156] S405 , calibrating the first multi-light source sample image according to the first chromaticity vector, the second chromaticity vector and the light source coefficient using a preset white balance mathematical model to determine a first standard light source sample image.

[0157] In this embodiment, after the first chromaticity vector l1, the second chromaticity vector l2 and the light source coefficient α are calculated, the white balance data model can be used to calculate the first multi-light source sample image. Perform calibration to determine the first standard light source sample image

[0158] The training method of the white balance correction model provided in the above embodiment adopts a preset white balance mathematical model to calculate the first standard light source sample image. During the model training process, the first standard light source sample image is used as a reference image, so that the training result of the model is consistent with the effect of the preset white balance mathematical model, thereby improving the correction effect of the model.

[0159] In one possible implementation, Fig.10 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 5 ,like Fig.10 As shown, before the first multi-light source sample image is input into the pre-trained neural network model for prediction in the above S103 to obtain the first standard light source prediction image, the method may further include:

[0160] S501. Determine a second standard light source sample image according to a second single-light source sample image and a second multi-light source sample image, wherein the second single-light source sample image and the second multi-light source sample image are sample images in a public data set.

[0161] S502: Input the second multi-light source sample image into the initial neural network model to obtain a second standard light source prediction image.

[0162] S503: Optimize the initial neural network model according to the second standard light source sample image and the second standard light source predicted image to obtain a pre-trained neural network model.

[0163] In this embodiment, in order to improve the training speed of the white balance correction model for the preset application scenario, a pre-trained neural network model for a general scenario can be pre-trained, and then the pre-trained neural network model is applied to the image data set of the preset application scenario for further training.

[0164] Specifically, a public data set is used to train the initial neural network model, and the public data set includes a one-to-one corresponding second single light source sample image and a second multi-light source sample image, and the second single light source sample image and the second multi-light source sample image also include a preset correction color card. The sample images included in the public data set involve various scenes, and the number of sample images is much larger than the image data set built by this embodiment. The process of training the initial neural network model using the public data set is the same as the process of training the pre-trained neural network model in S102-S104 above, and will not be repeated here.

[0165] The training method of the white balance correction model provided in the above embodiment uses a public data set for pre-training first, and then migrates the pre-trained neural network model to a preset application scenario for further training. It has stronger mobility and a faster training speed in the preset application scenario, ensuring better training results even when the number of samples in the preset application scenario is small.

[0166] In a possible implementation, the pre-trained neural network model includes: an encoder and a decoder, the encoder includes: a convolution layer and a multi-layer visual conversion layer, and the above S104 optimizes the pre-trained neural network model according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model, which may include:

[0167] According to the first standard light source sample image and the first standard light source predicted image, only the parameters of the convolution layer and the decoder are optimized to obtain a white balance correction model.

[0168] In this embodiment, when the pre-trained neural network model is subjected to targeted training for a preset application scenario, some model parameters of the pre-trained neural network model can be frozen, that is, the frozen model parameters are no longer adjusted during the targeted training process, and only the remaining model parameters are adjusted, wherein the frozen model parameters are model parameters that characterize common features in the image.

[0169] Specifically, Figure 3 As shown, the parameters of W / 2, W / 4, W / 8 and W / 16 of the Mobile-ViT Block layer in the pre-trained neural network model are frozen, the pre-trained general features are retained, and only the parameters of the unfrozen convolutional layers, decoders and regressors are adjusted, so that the model can better adapt to the performance of white balance correction in the characteristic lighting environment of the image dataset of the preset application scenario.

[0170] In one possible implementation, Fig.11 Schematic diagram of the process of training the white balance correction model provided in the embodiment of the present application Figure 6 ,like Fig.11 As shown, the process of optimizing the pre-trained neural network model according to the first standard light source sample image and the first standard light source predicted image to obtain the white balance correction model in S104 may include:

[0171] S601 : Calculate object edge loss and global color loss of a first standard light source sample image and a first standard light source prediction image.

[0172] S602: Optimize the pre-trained neural network model according to the object edge loss and the global color loss to obtain a white balance correction model.

[0173] In this embodiment, an edge extraction algorithm is used to identify the edge of an object on the first standard light source sample image and the first standard light source prediction image to determine the edge line of transition between the target object and the surrounding environment in the first standard light source sample image, and the edge line of transition between the target object and the surrounding environment in the first standard light source prediction image, and the object edge loss is calculated based on the two edge lines. For example, the edge extraction algorithm may be Sobel Filters.

[0174] The difference between the pixel values ​​of each pixel point of the first standard light source sample image and the first standard light source prediction image is calculated to determine the global color loss of the first standard light source sample image and the first standard light source prediction image.

[0175] According to the total loss value or weighted loss value of the object edge loss and the global color loss, the pre-trained neural network model is optimized to obtain a white balance correction model.

[0176] The training method of the white balance correction model provided in the above embodiment optimizes the model according to the object edge loss and the global color loss, which can not only ensure the clear edges of objects in the image after white balance correction, but also ensure the overall color of the image is accurate, thereby improving the robustness and adaptability of the white balance correction model and optimizing the image visual effect.

[0177] Based on the above-mentioned training method of the white balance correction model, the embodiment of the present application also provides a white balance correction method. Fig.12 A schematic diagram of a flow chart of a white balance correction method provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the method may include:

[0178] S701: Use an image acquisition device with a three-color light source to acquire a multi-light source image of a preset application scenario.

[0179] S702: Use a pre-trained white balance correction model to perform white balance correction on the multi-light source image and output a standard light source image.

[0180] The white balance correction model is trained using the above-mentioned white balance correction model training method.

[0181] In this embodiment, an image acquisition device with three-color light sources is used to photograph objects in a preset application scene, and the grayscale images corresponding to the three-color light sources are fused to obtain a colorful multi-light source image. The multi-light source image is input into a white balance correction model, and white balance correction is performed on the multi-light source image through the white balance correction model to obtain a corrected standard light source image. The standard light source image can be used in scenes such as texture mapping.

[0182] The white balance correction method provided in the above embodiment uses a pre-trained white balance correction model to realize automatic white balance correction. There is no need to place the correction color card during the image acquisition process, which saves manpower time. The standard light source image corrected by the white balance correction model conforms to the visual perception of the person, and can highlight the fine texture structure and color of the subject, and can be applied in a variety of scenarios.

[0183] Based on the above method embodiment, the embodiment of the present application provides a training device for a white balance correction model. Fig.13 A schematic diagram of the structure of a training device for a white balance correction model provided in an embodiment of the present application, such as Fig.13 As shown, the device may include:

[0184] The image acquisition module 801 is used to acquire an image data set, wherein the image data set at least includes: a plurality of groups of sample images, each group of sample images includes: a first single-light source sample image and a first multi-light source sample image of a target object acquired by an image acquisition device having a three-color light source, wherein the first single-light source sample image is an image generated based on the three-color light source, and the first multi-light source sample image is an image generated based on the three-color light source and an ambient light source;

[0185] The standard image generating module 802 is used to determine a first standard light source sample image according to the first single light source sample image and the first multi-light source sample image;

[0186] A model prediction module 803 is used to input the first multi-light source sample image into a pre-trained neural network model for prediction to obtain a first standard light source prediction image;

[0187] The model correction module 804 is used to optimize the pre-trained neural network model according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model. The white balance correction model is used to white balance the multi-light source image and output the standard light source image.

[0188] Optionally, the image acquisition module 801 is specifically used to use an image acquisition device to acquire an image of the target object, and the initial image of the target object obtained includes: a red mixed light source image and a corresponding first ambient light source image, a green mixed light source image and a corresponding second ambient light source image, a blue mixed light source image and a corresponding third ambient light source image, the mixed light source images of corresponding colors are images captured during the process of the three-color fill light being turned on and off respectively within three preset exposure times, and the corresponding ambient light source images are images captured during the three target exposure times when the three-color fill light is not turned on, and each target exposure time is the off time of the corresponding fill light within the preset exposure time; a first multi-light source sample image is generated according to the red mixed light source image, the green mixed light source image and the blue mixed light source image; a first single light source sample image is generated according to the red mixed light source image, the first ambient light source image, the green mixed light source image, the second ambient light source image, the blue mixed light source image and the third ambient light source image.

[0189] Optionally, the device may further include:

[0190] The device calibration module is used to use an image acquisition device to shoot a preset correction color card when the three-color fill light is turned on respectively to obtain three grayscale images; fuse the three grayscale images to obtain a fused image; adjust the lighting time ratio of the three-color fill light according to the R value, G value and B value of the grayscale color block in the fused image to calibrate the image acquisition device. After the calibration is completed, the R value, G value and B value of the grayscale color block in the fused image corresponding to the image acquisition device are equal.

[0191] Optionally, in addition to the target object, the sample image captured by the image acquisition device also includes a preset correction color card. The standard image generation module 802 is specifically used to determine the ambient light source sample image based on the first single light source sample image and the first multi-light source sample image; determine the first chromaticity vector of the three-color light source based on the pixel values ​​of the grayscale color blocks of the preset correction color card in the first single light source sample image; determine the second chromaticity vector of the ambient light source based on the pixel values ​​of the grayscale color blocks of the preset correction color card in the ambient light source sample image; determine the light source coefficient based on the pixel intensities of the first single light source sample image and the ambient light source sample image; and calibrate the first multi-light source sample image using a preset white balance mathematical model based on the first chromaticity vector, the second chromaticity vector and the light source coefficient to determine the first standard light source sample image.

[0192] Optionally, the standard image generating module 802 is further configured to determine a second standard light source sample image according to the second single light source sample image and the second multi-light source sample image, wherein the second single light source sample image and the second multi-light source sample image are sample images in a public data set;

[0193] The model prediction module 803 is further used to input the second multi-light source sample image into the initial neural network model to obtain a second standard light source prediction image;

[0194] The model correction module 804 is further used to optimize the initial neural network model according to the second standard light source sample image and the second standard light source predicted image to obtain a pre-trained neural network model.

[0195] Optionally, the pre-trained neural network model includes: an encoder and a decoder, the encoder includes: a convolution layer and a multi-layer visual conversion layer, and the model correction module 804 is specifically used to optimize only the parameters of the convolution layer and the decoder according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model.

[0196] Optionally, the model correction module 804 is specifically used to calculate the object edge loss and global color loss of the first standard light source sample image and the first standard light source prediction image; according to the object edge loss and the global color loss, the pre-trained neural network model is optimized to obtain a white balance correction model.

[0197] Based on the above method embodiment, the embodiment of the present application also provides a white balance correction device. Fig.14 A schematic diagram of the structure of a white balance correction device provided in an embodiment of the present application is shown in FIG. Fig.14 As shown, the device may include:

[0198] An image acquisition module 805 is used to acquire a multi-light source image of a preset application scene using an image acquisition device with a three-color light source;

[0199] The image correction module 806 is used to perform white balance correction on the multi-light source image using a pre-trained white balance correction model and output a standard light source image. The white balance correction model is trained using the training method of the white balance correction model of any one of the first aspects.

[0200] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0201] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0202] Fig.15 A schematic diagram of an electronic device provided in an embodiment of the present application, such as Fig.15 As shown, the electronic device 900 includes: a processor 901, a storage medium 902 and a bus, the storage medium 902 stores program instructions executable by the processor 901, when the electronic device 900 is running, the processor 901 and the storage medium 902 communicate through the bus, and the processor 901 executes the program instructions to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0203] Optionally, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method embodiment is executed.

[0204] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0207] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0208] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for training a white balance correction model, characterized in that: The method comprises: Acquire an image data set, the image data set at least comprising: a plurality of groups of sample images, each group of sample images comprising: a first single-light source sample image and a first multi-light source sample image of a target object acquired by an image acquisition device having a three-color light source, wherein the first single-light source sample image is an image generated based on the three-color light source, and the first multi-light source sample image is an image generated based on the three-color light source and an ambient light source; Determine a first standard light source sample image according to the first single light source sample image and the first multi-light source sample image; Inputting the first multi-light source sample image into a pre-trained neural network model for prediction to obtain a first standard light source prediction image; The pre-trained neural network model is optimized according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model, and the white balance correction model is used to perform white balance on a multi-light source image and output a standard light source image.

2. The method according to claim 1, characterized in that: The acquiring of the image data set comprises: The image acquisition device is used to acquire an image of the target object to obtain an initial image of the target object, wherein the initial image includes: a red mixed light source image and a corresponding first ambient light source image, a green mixed light source image and a corresponding second ambient light source image, and a blue mixed light source image and a corresponding third ambient light source image, wherein the mixed light source images of corresponding colors are images captured during the process of the three-color fill light being turned on and off respectively within three preset exposure times, and the corresponding ambient light source images are images captured during the process of the three-color fill light being not turned on within three target exposure times, and each target exposure time is the off time of the corresponding fill light within the preset exposure time; generating the first multi-light source sample image according to the red mixed light source image, the green mixed light source image and the blue mixed light source image; The first single light source sample image is generated according to the red mixed light source image, the first ambient light source image, the green mixed light source image, the second ambient light source image, the blue mixed light source image and the third ambient light source image.

3. The method according to claim 2, characterized in that Before using the image acquisition device to acquire an image of the target object, the method further includes: The image acquisition device is used to shoot a preset calibration color card when the three-color fill lights are turned on respectively, to obtain three grayscale images; Fusing the three grayscale images to obtain a fused image; According to the R value, G value and B value of the grayscale color block in the fused image, the lighting time ratio of the three-color fill light is adjusted to calibrate the image acquisition device. After the calibration, the R value, G value and B value of the grayscale color block in the fused image corresponding to the image acquisition device are equal.

4. The method according to claim 1, characterized in that: The sample image acquired by the image acquisition device includes, in addition to the target object, a preset calibration color card, and determining the first standard light source sample image according to the first single light source sample image and the first multi-light source sample image, including: Determine an ambient light source sample image according to the first single light source sample image and the first multi-light source sample image; Determining a first chromaticity vector of the three-color light source according to a pixel value of a grayscale color block of a preset calibration color card in the first single-light source sample image; Determining a second chromaticity vector of the ambient light source according to a pixel value of a grayscale color block of the preset correction color card in the ambient light source sample image; Determine a light source coefficient according to pixel intensities of the first single light source sample image and the ambient light source sample image; The first multi-light source sample image is calibrated according to the first chromaticity vector, the second chromaticity vector and the light source coefficient by using a preset white balance mathematical model to determine the first standard light source sample image.

5. The method according to claim 1, characterized in that Before inputting the first multi-light source sample image into a pre-trained neural network model for prediction to obtain a first standard light source prediction image, the method further includes: Determine a second standard light source sample image according to a second single light source sample image and a second multi-light source sample image, wherein the second single light source sample image and the second multi-light source sample image are sample images in a public data set; Inputting the second multi-light source sample image into the initial neural network model to obtain a second standard light source prediction image; The initial neural network model is optimized according to the second standard light source sample image and the second standard light source predicted image to obtain the pre-trained neural network model.

6. The method according to claim 5, characterized in that The pre-trained neural network model includes: an encoder and a decoder, the encoder includes: a convolution layer and a multi-layer visual conversion layer, and the pre-trained neural network model is optimized according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model, including: According to the first standard light source sample image and the first standard light source predicted image, only the parameters of the convolution layer and the decoder are optimized to obtain the white balance correction model.

7. The method according to claim 1, characterized in that The step of optimizing the first neural network model according to the first standard light source sample image and the first standard light source predicted image to obtain a white balance correction model includes: Calculating object edge loss and global color loss of the first standard light source sample image and the first standard light source prediction image; The pre-trained neural network model is optimized according to the object edge loss and the global color loss to obtain the white balance correction model.

8. A white balance correction method, characterized in that: The method comprises: An image acquisition device with a three-color light source is used to acquire multi-light source images of a preset application scenario; A pre-trained white balance correction model is used to perform white balance correction on the multi-light source image to output a standard light source image, wherein the white balance correction model is trained using the training method for the white balance correction model according to any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the training method of the white balance correction model as described in any one of claims 1 to 7, or the steps of the white balance correction method as described in claim 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the computer program executes the steps of the training method of the white balance correction model according to any one of claims 1 to 7, or executes the steps of the white balance correction method according to claim 8.

Images