Light source estimation model training, light source estimation method and apparatus

Abstract

The application discloses a light source estimation model training method, a light source estimation method and device, and belongs to the computer field. The light source estimation model training method comprises the following steps: determining a training data set comprising multiple groups of training data, each group of training data comprising a first image sequence, a second image sequence and a reference light source estimation result, the first image sequence comprising multiple frames of images continuously collected under a first light source scene, and the second image sequence being obtained by performing brightness transformation on the first image sequence; inputting the first image sequence in a group of training data into a light source estimation model to obtain a first light source estimation result, and inputting the second image sequence in the same group of training data into the light source estimation model to obtain a second light source estimation result; determining a loss value according to the first light source estimation result, the second light source estimation result, the reference light source estimation result and a loss function; and adjusting parameters of the light source estimation model according to the loss value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and particularly relates to a light source estimation model training method and device and a light source estimation method and device. BACKGROUND

[0002] The human visual system has the ability to perceive the color of an object itself independently of the lighting conditions, and this ability to stably perceive the color is referred to as color constancy (CC). White balance is an important correction method for realizing image color constancy, to eliminate the influence of color temperature under different light sources on the color of an image, and restore the true color of an object. When performing white balance processing on an image, the color of a light source when the image is captured needs to be estimated (hereinafter referred to as light source estimation).

[0003] At present, methods for light source estimation include two categories: 1) a method based on statistics, represented by the gray world algorithm, which links the statistical characteristics of an image and light source estimation based on certain prior assumptions; and 2) a method based on learning, which mainly relies on a specific database to learn the features of an input image through deep learning and then map them to light source estimation.

[0004] In these two methods, the former has certain limitations in the accuracy of light source estimation, and although the latter has improved accuracy, in actual applications, due to deviations in the shooting angle, changes in the scene and lighting, etc., the light source estimation result of a single frame of image is unstable, thereby causing color jumping in the image preview process, so that the color of the image after white balance processing does not match the true scene color. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a light source estimation model training method and device and a light source estimation method and device, to solve the problem of unstable light source estimation result of a single frame of image.

[0006] To solve the above technical problem, the present application is implemented as follows:

[0007] In a first aspect, the embodiments of the present application provide a light source estimation model training method, comprising:

[0008] determining a training data set, the training data set comprising a plurality of groups of training data, each group of training data comprising a first image sequence, a second image sequence and a reference light source estimation result corresponding to the first image sequence, the first image sequence comprising a plurality of frames of images continuously captured under a first light source scene, the second image sequence being obtained by performing brightness transformation on the first image sequence, and the reference light source estimation result being used to describe the color characteristics of a true light source under the first light source scene;

[0009] inputting the first image sequence in a group of the training data into a light source estimation model to obtain a first light source estimation result, and inputting the second image sequence in the same group of the training data into the light source estimation model to obtain a second light source estimation result, wherein the light source estimation model comprises at least two layers of light source prediction networks, each layer of the light source prediction network comprises a space-time feature extraction network and a light source estimation network, the space-time feature extraction network is configured to extract space-time features from the input image sequence, and the light source estimation network is configured to perform light source estimation according to the space-time features;

[0010] determining a loss value according to the first light source estimation result, the second light source estimation result, the reference light source estimation result in the same group of the training data, and a loss function;

[0011] adjusting parameters of the light source estimation model according to the loss value.

[0012] In a second aspect, an embodiment of the present application provides a light source estimation method, comprising:

[0013] determining a third image sequence to be processed;

[0014] inputting the third image sequence into a light source estimation model, wherein the light source estimation model is trained by the light source estimation model training method in the first aspect, and the number of images contained in the third image sequence is consistent with the number of images of the first image sequence input for training the light source estimation model;

[0015] taking an output of the light source estimation model as a light source estimation result of the third image sequence, and the light source estimation result is used to adjust a color of at least one image in the third image sequence.

[0016] In a third aspect, an embodiment of the present application provides a light source estimation model training device, comprising:

[0017] a data determination module configured to determine a training data set, the training data set comprising a plurality of groups of training data, each group of the training data comprising a first image sequence, a second image sequence, and a reference light source estimation result corresponding to the first image sequence, the first image sequence comprising a plurality of frames of images continuously collected under a first light source scene, the second image sequence being obtained by performing brightness transformation on the first image sequence, and the reference light source estimation result being used to describe color characteristics of a real light source under the first light source scene;

[0018] a data input module, configured to input a first image sequence in a group of training data into a light source estimation model to obtain a first light source estimation result, and input a second image sequence in the same group of training data into the light source estimation model to obtain a second light source estimation result, wherein the light source estimation model comprises at least two layers of light source prediction networks, each layer of the light source prediction networks comprises a space-time feature extraction network and a light source estimation network, the space-time feature extraction network is configured to extract space-time features from an input image sequence, and the light source estimation network is configured to perform light source estimation according to the space-time features;

[0019] a loss determination module, configured to determine a loss value according to the first light source estimation result, the second light source estimation result, a reference light source estimation result in the same group of training data, and a loss function;

[0020] a parameter adjustment module, configured to adjust parameters of the light source estimation model according to the loss value.

[0021] In a fourth aspect, an embodiment of the present application provides a light source estimation device, comprising:

[0022] an image determination module, configured to determine a third image sequence to be processed;

[0023] an image input module, configured to input the third image sequence into a light source estimation model, wherein the light source estimation model is trained by the light source estimation model training method in the first aspect, and the number of images contained in the third image sequence is consistent with the number of images of a first image sequence input for training the light source estimation model;

[0024] a result output module, configured to output an output of the light source estimation model as a light source estimation result of the third image sequence, and the light source estimation result is used to adjust a color of at least one image in the third image sequence.

[0025] In a fifth aspect, an embodiment of the present application provides an electronic device, which comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and the program or instruction is executed by the processor to implement the steps of the method in the first aspect or the second aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a readable storage medium, which stores a program or instruction, and the program or instruction is executed by a processor to implement the steps of the method in the first aspect or the second aspect.

[0027] In a seventh aspect, an embodiment of the present application provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, and the processor is configured to run programs or instructions to implement the method in the first aspect or the second aspect.

[0028] In the embodiments of the present application, the training of the light source estimation model and the estimation of the light source are performed by using image sequences (multiple frames of images), compared with the light source estimation performed by using a single frame of image in the related art, the use of multiple frames of images can better learn the information of adjacent frames of images, thereby reducing the errors of single frame estimation, adapting to the environment jump, improving the stability of the estimation result, preventing the color jump phenomenon in the image preview process, and making the image color after the white balance processing more consistent with the real scene color. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 FIG. 1 is a schematic diagram of a light source estimation scheme framework provided by an embodiment of the present application.

[0030] Figure 2 FIG. 2 is a schematic diagram of a light source estimation model training method provided by an embodiment of the present application.

[0031] Figure 3 FIG. 3 is a structural schematic diagram of a light source estimation model provided by an embodiment of the present application.

[0032] Figure 4 FIG. 4 is a structural schematic diagram of ConvLSTM provided by an embodiment of the present application.

[0033] Figure 5 FIG. 5 is a structural schematic diagram of a light source estimation network provided by an embodiment of the present application.

[0034] Figure 6 FIG. 6 is a schematic diagram of a light source estimation method provided by an embodiment of the present application.

[0035] Figure 7 FIG. 7 is a structural schematic diagram of a light source estimation model training device provided by an embodiment of the present application.

[0036] Figure 8 FIG. 8 is a structural schematic diagram of a light source estimation device provided by an embodiment of the present application.

[0037] Figure 9 FIG. 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0038] Figure 10 FIG. 10 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0040] The terms "first", "second", and the like in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the specification means at least one of the connected objects, and the character " / ", generally means that the front and rear associated objects are in a "or" relationship.

[0041] In order to solve the problem of unstable light source estimation result of single frame image, the present application provides a light source estimation model training method and a light source estimation method. Wherein, the light source estimation model training method can be executed by an electronic device, which can include but is not limited to a server or a terminal device; the light source estimation method can be executed by a terminal device.

[0042] Specifically, as shown in the figure, Figure 1 The light source estimation scheme provided by the present application includes two parts: offline training 1 and online prediction 2.

[0043] Wherein, the offline training 1 corresponds to the light source estimation model training method provided by the present application, and the offline training 1 generally includes: step 11, constructing a training data set containing image sequence in time sequence; step 12, training the light source estimation model; step 13, outputting the light source estimation result, and adjusting the model parameters according to the light source estimation result.

[0044] Wherein, the online prediction 2 corresponds to the light source estimation method provided by the present application, and the online prediction 2 generally includes: step 21, inputting the image sequence to be estimated to the light source estimation model obtained by the offline training 1; step 22, the light source estimation model estimates the light source corresponding to the input image sequence; step 23, the light source estimation model outputs the light source estimation result.

[0045] The light source estimation model training method and the light source estimation method provided by the present application will be described in detail below with reference to the drawings and specific embodiments.

[0046] First, a light source estimation model training method provided by an embodiment of the present application is described.

[0047] As shown in the figure, Figure 2As shown, the embodiment of the present application provides a light source estimation model training method, which can include:

[0048] Step 201, determining a training data set, the training data set including multiple groups of training data, each group of training data including a first image sequence, a second image sequence, and a reference light source estimation result corresponding to the first image sequence, the first image sequence including multiple frames of images continuously collected under a first light source scene, the second image sequence being obtained by performing brightness transformation on the first image sequence, and the reference light source estimation result being used to describe color characteristics of a real light source under the first light source scene.

[0049] Wherein, the training data set is pre-constructed, and the training data set contains multiple groups of training data under multiple light source scenes, and one light source scene corresponds to at least one group of training data. The light source colors of the multiple light source scenes are different from each other. The multiple light source scenes include the first light source scene, or in other words, the first light source scene can be any one of the multiple light source scenes.

[0050] In some embodiments, the determining of the training data set can include: taking each of the multiple light source scenes as the first light source scene respectively, and performing the following steps:

[0051] Step 1, continuously collecting n frames of images under the first light source scene, and shooting a reference image containing a color card within a first time length after the n frames of images are collected, wherein the first time length is a relatively short time length set in advance, for example, the first time length is 3s, aiming to shoot the reference image containing the color card immediately after the preview frame data is collected.

[0052] After collecting the images and the reference image for each light source scene, the data is pre-processed to obtain training data suitable for training the light source estimation model.

[0053] In some embodiments, the images shot by the camera are generally raw format images, which contain a lot of information. In order to speed up the calculation, the collected raw format images can be converted into stats format images before step 2 is performed.

[0054] Step 2, extracting t frames of images from every m frames of the n frames of images according to the collection time, to obtain the first image sequence under the first light source scene.

[0055] Wherein, the first image sequence can be represented as: I=(I1, I2, I3, …, It). t

[0056] Step 3, processing the reference image under the first light source scene to obtain the reference light source estimation result under the first light source scene. ​

[0057] Specifically, first, the colors of all pixels in the area where the color card is located in the reference image can be averaged by color channel to obtain a first vector containing the color average values under three color channels, for example, a first vector containing the color average values under RGB three color channels can be obtained, and it can be understood that the first vector contains 3 elements; then, the first vector is normalized by a first color channel to obtain a second vector, wherein the first color channel is any one of the three color channels, for example, the first color channel can be G, and it can be understood that the second vector obtained by normalization also includes 3 elements; then, the second vector is taken as the reference light source estimation result gtillof the light source scene. The reference light source estimation result of a light source scene can describe the color characteristics of the real light source in the light source scene, which is equivalent to the label of the first image sequence in the light source scene.

[0058] Up to now, the first image sequence in each light source scene and the corresponding reference light source estimation constitute a set of training data (I, gtillof), wherein I=(I1, I2, I3, …, I t ).

[0059] In some embodiments, training data of different brightness levels in the same light source scene can also be constructed by the following step 4.

[0060] Step 4: performing brightness transformation on the first image sequence in the first light source scene to obtain a second image sequence with different brightness in the first light source scene.

[0061] Since the way of human perception of brightness is approximately logarithmic relationship, this way is more in line with human perception, logarithmic transformation changes the brightness of the image, which can reduce the extreme brightness difference and enhance the dark details of the image, therefore, the first image sequence in the first light source scene can be transformed in brightness by logarithmic transformation.

[0062] Suppose the first image sequence is I=(I1, I2, I3, …, I t ), then the brightness of the image in the first image sequence can be changed by the following logarithmic transformation formula:

[0063] I t ′=c*log(1+I t ) (1)

[0064] Wherein, c=limit / log(1+f max (I t )), I t represents the original stats image before transformation, I t ′ is the stats image after brightness change, limit represents the maximum value under the corresponding radix of the storage stats image, and fmax (·) represents the maximum value function.

[0065] The second image sequence obtained after transformation can be represented as: I' = (I1', I2', I3', …, I t

[0066] Step 5, the first image sequence, the second image sequence and the reference light source estimation result under the first light source scene are saved correspondingly to obtain a set of training data.

[0067] For example, (I, gtill, I') is saved as a set of training data.

[0068] In some other embodiments, the determination of the training data set can further include:

[0069] Step 6, at least one first image sequence is selected from the multiple sets of training data.

[0070] For example, a certain ratio of first image sequences is randomly selected from the multiple sets of training data as the object of light source enhancement.

[0071] Step 7, the selected first image sequence is subjected to white balance processing.

[0072] Specifically, the selected first image sequence can be subjected to white balance processing by using an existing white balance processing method.

[0073] Step 8, the first image sequence subjected to white balance processing is subjected to light source enhancement using a light source under a second light source scene, wherein the color of the light source under the second light source scene is different from the color of the light source under the first light source scene.

[0074] Specifically, the implementation principle of the light source enhancement on the first image sequence subjected to white balance processing can be described as follows: a new light source, i.e., the first light source, is used to strike the image subjected to white balance, and the new light source is the real light source gtill new of the new image. new Thus, the data pair subjected to light source enhancement can be obtained. For one set of training data (I, gtill), a light source ill is randomly selected from other light sources in the training data set as the first light source, and the training data pair (I new , gtill new ) subjected to light source enhancement is calculated as follows:

[0075]

[0076] gtill new = ill (3) ​

[0077] In some embodiments, determining the training dataset may further include:

[0078] Step 9: Perform brightness transformation on the first image sequence after light source enhancement to obtain the second image sequence with different brightness levels under the second light source scene.

[0079] The brightness transformation method can also be a logarithmic transformation.

[0080] Step 10: Save the first image sequence enhanced with light source in the second light source scene, the second image sequence in the second light source scene, and the corresponding light source to obtain a new set of training data.

[0081] Assume that the first image sequence after light source enhancement is I. new First image sequence I new The corresponding reference light source estimation result is gtill new The second image sequence after brightness transformation is I′ new Then (I) new ,gtill new , I′ new () as a new set of training data.

[0082] It is understandable that generating new image sequences by enhancing the light source and adding them to the training dataset can compensate for the insufficient generalization ability of the model due to insufficient dataset and enrich the diversity of data.

[0083] Step 202: Input the first image sequence from the set of training data into the light source estimation model to obtain a first light source estimation result, and input the second image sequence from the same set of training data into the light source estimation model to obtain a second light source estimation result. The light source estimation model includes at least two layers of light source prediction network. Each layer of the light source prediction network includes a spatiotemporal feature extraction network and a light source estimation network. The spatiotemporal feature extraction network is used to extract spatiotemporal features from the input image sequence, and the light source estimation network is used to estimate the light source based on the spatiotemporal features.

[0084] Figure 3 A schematic diagram of the structure of a light source estimation model proposed in an embodiment of this application is shown. Figure 3 As shown, the light source estimation model proposed in this application embodiment includes a layer light source prediction network 31, wherein the layer light source prediction network 31 includes a spatiotemporal feature extraction network 311 and a light source estimation network 312.

[0085] Furthermore, such as Figure 3As shown, the spatio-temporal feature extraction network 311 includes k layers of network units connected in sequence, where k is a positive integer, k is consistent with the number of images contained in the input target image sequence (i.e., k = t), the input of the first layer network unit in the k layers of network units is the first image I1 in the target image sequence input into the spatio-temporal feature extraction network 3, the input of the i-th layer network unit in the k layers of network units includes the i-th image Ii in the target image sequence input into the spatio-temporal feature extraction network 311 and the output of the (i-1)-th layer network unit, 1 < i < k, and the output of the k-th layer network unit in the k layers of network units is the spatio-temporal feature of the target image sequence input into the spatio-temporal feature extraction network, the target image sequence including the first image sequence or the second image sequence. i As shown, the spatio-temporal feature extraction network 311 includes k layers of network units connected in sequence, where k is a positive integer, k is consistent with the number of images contained in the input target image sequence (i.e., k = t), the input of the first layer network unit in the k layers of network units is the first image I1 in the target image sequence input into the spatio-temporal feature extraction network 3, the input of the i-th layer network unit in the k layers of network units includes the i-th image Ii in the target image sequence input into the spatio-temporal feature extraction network 311 and the output of the (i-1)-th layer network unit, 1 < i < k, and the output of the k-th layer network unit in the k layers of network units is the spatio-temporal feature of the target image sequence input into the spatio-temporal feature extraction network, the target image sequence including the first image sequence or the second image sequence.

[0086] As an example, the network unit in the spatio-temporal feature extraction network 311 can be a Convolutional Long Short-Term Memory Network (ConvLSTM), Figure 4 which is a structural diagram of ConvLSTM. ConvLSTM not only learns the spatial feature information of images but also has certain memory capability, and can capture the long-term dependency relationship in the input image sequence. This way can reduce the error of single-frame estimation, adapt to the environment jump, and improve the accuracy, stability and algorithm adaptability of the white balance result. It should be noted that, Figure 4 the structure of the ConvLSTM shown is prior art, Figure 4 the meanings of the physical symbols shown in the above formula are known in the art.

[0087] Specifically, as shown in Figure 3 , the spatio-temporal feature of the image sequence is obtained by inputting the image sequence (I1, I2, …, I t-1 , I t ) into the spatio-temporal feature extraction network 311. Assuming that the image matrix of each frame of image is represented as (3, h, w), 3 represents that the stats image is RGB three-channel, and h and w represent the height and width of the input image. Each frame of image in the image sequence is sequentially input into each network unit 3111, and the first frame of image I1 is output h1 (1) ,c1 (1) after the processing of the first network unit 3111, which is input into the next network unit together with the next frame of image I2, until the last frame of image I t is input, and then the spatio-temporal feature output h t (1), the dimension of the spatio-temporal feature is (c, h, w), and the spatio-temporal features of each subsequent layer can be obtained in turn, and then the light source estimation value y of each layer is obtained through the light source estimation network 1 ,y 2 to y l .

[0088] As an example, as shown in Figure 5 , the light source estimation network 312 includes at least one convolutional layer, a global average pooling (GAP) layer, and an output data dimension adjustment layer, wherein:

[0089] The at least one convolutional layer is used for further feature extraction from the spatio-temporal features extracted by the spatio-temporal feature extraction network.

[0090] The GAP layer is used to determine the average value of the features extracted by the at least one convolutional layer by color channel to obtain a third vector formed by the average values of each color channel.

[0091] The output data dimension adjustment layer is used to perform a dimension adjustment (reshape) operation on the third vector output by the GAP layer to obtain the light source estimation result of the image sequence input to the spatio-temporal feature extraction network.

[0092] Specifically, as shown in Figure 5 , wherein the convolutional layer is used to receive the spatio-temporal features and further extract features therefrom, and the dimension of the output feature map is (3, h, w); the GAP layer is used to perform an average operation on the color channel of the feature map to obtain the average value of the feature map under each color channel, and the output dimension of this layer is (3, 1, 1); and finally, a dimension adjustment operation is performed to obtain the predicted light source estimation result y. Here, the GAP layer is used to directly obtain the average value output of each color channel, which can reduce the parameter quantity and calculation quantity of the model, simplify the network structure, and help to reduce overfitting and improve the generalization ability of the model compared with the fully connected layer.

[0093] Based on the above description, the first light source estimation result includes a plurality of first light source estimation sub-results output by the multi-layer light source prediction network, and the plurality of first light source estimation sub-results can include: to The second light source estimation result includes a plurality of second light source estimation sub-results output by the multi-layer light source prediction network, and the plurality of second light source estimation sub-results can include: to

[0094] In step 203, a loss value is determined according to the first light source estimation result, the second light source estimation result, the reference light source estimation result in the same set of training data, and a loss function.

[0095] wherein the loss function is weighted by the loss function of the multi-layer light source prediction network, and specifically, the loss function can be represented as:

[0096]

[0097] wherein, represents the loss function corresponding to the single-layer light source prediction network, and α layer represents the weight of the loss function of the single-layer light source prediction network.

[0098] In some embodiments, the loss function corresponding to the last layer of the multi-layer light source prediction network is the main loss function, and its weight can be set to 1. The loss functions of the remaining light source prediction networks are auxiliary losses, and their weights are set by hyperparameters, and the values do not exceed the weight of the main loss function.

[0099] The hyperparameters in the neural network refer to the parameters set before the training process that affect the model training. Here, the weight of the auxiliary loss function is one of the hyperparameters, and different values are set according to the network depth, experience and experimental debugging. Taking the default setting of 4 layers of the multi-layer light source prediction network as an example, the loss function corresponding to the last layer of the multi-layer light source prediction network is the main loss function, and the weight is 1. The initial values of the weights of the loss functions corresponding to the first layer to the third layer of the light source prediction network can be set to 0.01, 0.05 and 0.125, respectively.

[0100] In some embodiments, the loss function of the single-layer light source prediction network includes a color correction loss function and a light source color consistency loss function under different brightness, wherein:

[0101] The color correction loss function is used to calculate the loss value of the light source estimation result predicted by the light source estimation model relative to the reference light source estimation result. Specifically, in order to evaluate the accuracy of the light source estimation result, the angle error between the predicted light source and the true light source can be used as the color correction loss.

[0102] The light source color consistency loss function under different brightness is used to calculate the light source estimation loss value between the sequence images of the same light source scene under different brightness predicted by the light source estimation model.

[0103] In some embodiments, the loss function of the single-layer light source prediction network is adaptively weighted and fused by the color correction loss function and the light source color consistency loss function under different brightness.

[0104] Specifically, the loss function of the single-layer light source prediction network can be represented as:

[0105]

[0106] wherein,

[0107]

[0108]

[0109]

[0110]

[0111] wherein, represents a color correction loss, w color is a weight of the color correction loss, represents a light source color consistency loss under different brightness, w bright is a weight of the light source color consistency loss under different brightness; y1 and y2 respectively represent the light source estimation results of the first image sequence and the light source estimation results of the second image sequence in the same group of the training data, represents a reference light source estimation result, n represents a sample number, “·” represents a dot product of a vector, ‖·‖ represents a length of a vector, arccos represents an inverse cosine function for converting a cosine value into an angle, and lr is a learning rate, and represents a gradient of the loss function with respect to the weight.

[0112] As described above, the loss function designed by the embodiments of the present application not only considers the color correction loss, but also designs the light source color consistency loss under different brightness considering the consistency problem of color under different brightness, and performs adaptive fusion, which is helpful to improve the accuracy and stability of light source prediction. In addition, the embodiments of the present application also apply an additional auxiliary loss in the middle layer of the multi-layer light prediction network, which can help the network to learn more useful features, enhance the feature expression ability and improve the generalization.

[0113] Step 204, adjusting the parameters of the light source estimation model according to the loss value.

[0114] It can be understood that the two time series image data with different brightness in the same group of training data are respectively input into the light source estimation model, and the first light source estimation result y1 and the second light source estimation result y2 are output, and then the parameters and adaptive weight parameters of the light source estimation model are continuously optimized by the back propagation algorithm to minimize the loss function. After training a relatively small loss value through multiple iterations, the model training is considered successful, and the model structure and parameters are saved. Specifically, the reference light source estimation result corresponding to the first light source scene, the plurality of first light source estimation sub-results, and the plurality of second light source estimation sub-results are respectively input into the above formula (6) and formula (7), so that the color estimation loss value and the light source estimation loss value under different brightness corresponding to each single-layer light source prediction network in the plurality of light source prediction networks are obtained; then, the color estimation loss value and the light source estimation loss value under different brightness corresponding to each single-layer light source prediction network are input into the above formula (5), so that the loss value corresponding to each single-layer light source prediction network is obtained; and then the loss value corresponding to each single-layer light source prediction network is input into the above formula (4), so that the final loss value is obtained. Based on the final loss value and the loss value corresponding to each single-layer light source prediction network, the parameters of the light source estimation model are iteratively optimized until the preset convergence condition is met, and the trained light source estimation model is obtained.

[0115] The light source estimation model training method provided in the embodiments of the present application uses image sequences with spatial and temporal information to train the light source estimation model and estimate the light source. Compared with the related art of using a single frame image to estimate the light source, using multiple frame images can better learn the information of adjacent frame images, thereby reducing the error of single frame estimation, adapting to environmental jump conditions, improving the stability of the estimation result, preventing color jump phenomenon in the image preview process, and making the image color after white balance processing more consistent with the true scene color.

[0116] In addition, the light source estimation model training method provided in the embodiments of the present application uses ConvLSTM as the basic unit of the single-layer light source prediction network. Since ConvLSTM can not only learn the spatial feature information of the image but also has certain memory ability and can capture the long-term dependence relationship in the input image sequence, the error of single frame estimation can be further reduced, the environmental jump condition can be adapted, and the accuracy, stability, and algorithm adaptability of the white balance result can be improved.

[0117] Furthermore, the light source estimation model training method provided in the embodiments of the present application generates new training data by light source enhancement and adds the new training data to the training data set, which can compensate for the influence of insufficient data set on the generalization of the model and enrich the diversity of data.

[0118] Further, the light source estimation model training method provided by the embodiment of the present application considers not only the color correction loss when designing the loss function, but also designs the light source color consistency loss under different brightness and performs adaptive fusion, which helps to improve the accuracy and stability of light source prediction. In addition, an additional auxiliary loss is applied to the middle layer of the multi-layer light prediction network, which can help the network to learn more useful features, enhance the feature expression ability and improve the generalization.

[0119] The light source estimation model training method provided by the embodiment of the present application is introduced above, and on this basis, the embodiment of the present application also provides a light source estimation method, which is described below.

[0120] As shown in Figure 6 The light source estimation method provided by the embodiment of the present application can include:

[0121] Step 601, determine a third image sequence to be processed.

[0122] Step 602, input the third image sequence into a light source estimation model, wherein the light source estimation model is trained by the light source estimation model training method described above, and the number of images contained in the third image sequence is consistent with the number of images of the image sequence input for training the light source estimation model.

[0123] Step 603, take the output of the light source estimation model as the light source estimation result of the third image sequence, and the light source estimation result is used to adjust the color of at least one image in the third image sequence.

[0124] In some embodiments, Figure 6 The method shown can further include adjusting the color of at least one image in the third image sequence according to the light source estimation result of the third image sequence.

[0125] The light source estimation method provided by the embodiment of the present application uses the light source estimation model trained by the light source estimation model training method provided by the embodiment of the present application to perform light source estimation, so that the error of single-frame estimation can be reduced, the environmental jump condition can be adapted, the stability of the estimation result can be improved, the color jump phenomenon in the image preview process can be prevented, and the image color after white balance processing is more consistent with the real scene color.

[0126] It should be noted that the light source estimation model training method provided in this application embodiment can be executed by a light source estimation model training device, or by a control module within that device for executing the loading light source estimation model training method. This application embodiment uses the execution of the loading light source estimation model training method by a light source estimation model training device as an example to illustrate the light source estimation model training method provided in this application embodiment.

[0127] It should be noted that the light source estimation method provided in this application embodiment can be executed by a light source estimation device, or by a control module within the light source estimation device for executing the loading light source estimation method. This application embodiment uses the execution of the loading light source estimation method by a light source estimation device as an example to illustrate the light source estimation method provided in this application embodiment.

[0128] The light source estimation device provided in the embodiments of this application will be described below.

[0129] like Figure 7 As shown, an embodiment of this application provides a light source estimation model training device, which may include: a data determination module 701, a data input module 702, a loss determination module 703, and a parameter adjustment module 704.

[0130] The data determination module 701 is used to determine the training dataset, which includes multiple sets of training data. Each set of training data includes a first image sequence, a second image sequence, and a reference light source estimation result corresponding to the first image sequence. The first image sequence includes multiple frames of images continuously acquired under the first light source scene. The second image sequence is obtained by performing a brightness transformation on the first image sequence. The reference light source estimation result is used to describe the color characteristics of the real light source under the first light source scene.

[0131] The data input module 702 is used to input the first image sequence from a set of training data into the light source estimation model to obtain a first light source estimation result, and to input the second image sequence from the same set of training data into the light source estimation model to obtain a second light source estimation result. The light source estimation model includes at least two layers of light source prediction networks. Each layer of the light source prediction network includes a spatiotemporal feature extraction network and a light source estimation network. The spatiotemporal feature extraction network is used to extract spatiotemporal features from the input image sequence, and the light source estimation network is used to estimate the light source based on the spatiotemporal features.

[0132] In some embodiments, the data determination module 701 is specifically used to: take each of the above-mentioned multiple light source scenes as a first light source scene, and perform the following steps:

[0133] Step 1, under the first light source scene, continuously collect n frames of images, and shoot a reference image containing a color card within a first time length after collecting the n frames of images, wherein the first time length is a pre-set short time length, for example, the first time length is 3s, aiming to shoot the reference image containing the color card immediately after collecting the preview frame data.

[0134] After collecting a plurality of images and reference images for each light source scene, the data is pre-processed to obtain training data suitable for training a light source estimation model.

[0135] In some embodiments, the images obtained by the camera are generally raw format images, which contain a lot of information. In order to speed up the calculation speed, the collected raw format images can be converted into stats format images before step 2 is performed.

[0136] Step 2, extract t frames of images from every m frames of the n frames of images according to the collection time, to obtain the first image sequence under the first light source scene.

[0137] Step 3, process the reference image under the first light source scene to obtain the reference light source estimation result under the first light source scene.

[0138] Specifically, first, the average value of the color of all pixels in the area where the color card is located in the reference image is taken according to the color channel to obtain a first vector containing the average value of the color under three color channels, for example, a first vector containing the average value of the color under RGB three color channels is obtained. It can be understood that the first vector contains 3 elements at this time; then, the first vector is normalized according to the first color channel to obtain a second vector, wherein the first color channel is any one of the three color channels, for example, the first color channel can be G. It can be understood that the second vector obtained by normalization also includes 3 elements; then, the second vector is taken as the reference light source estimation result gtil of the light source scene. The reference light source estimation result of a light source scene can describe the color characteristics of the real light source under the light source scene, which is equivalent to the label of the first image sequence under the light source scene.

[0139] In some embodiments, training data under different brightness levels of the same light source scene can also be constructed by the following step 4.

[0140] Step 4, perform brightness transformation on the first image sequence under the first light source scene to obtain a second image sequence under the first light source scene with different brightness.

[0141] Step 5, save the first image sequence, the second image sequence and the reference light source estimation result under the first light source scene correspondingly to obtain a set of training data.

[0142] In some embodiments, the data determining module 701 can be further configured to perform the following steps:

[0143] Step 6, selecting at least one first image sequence from the plurality of training data sets.

[0144] For example, a certain ratio of first image sequences in the plurality of training data sets can be randomly selected as the light source enhanced objects.

[0145] Step 7, performing white balance processing on the selected first image sequence.

[0146] Specifically, the selected first image sequence can be subjected to white balance processing by using an existing white balance processing method.

[0147] Step 8, performing light source enhancement on the first image sequence subjected to white balance processing using a light source under a second light source scene, wherein the color of the light source under the second light source scene is different from the color of the light source under the first light source scene.

[0148] Specifically, the implementation principle of performing light source enhancement on the first image sequence subjected to white balance processing can be described as follows: using a new light source, i.e., the first light source, to strike the image subjected to white balance, and the new light source is the real light source of the new image. new The image subjected to white balance is restored using the new light source to obtain I new Thus, the data pair subjected to light source enhancement can be obtained. For one training data set (I, gtill), a light source ill is randomly selected from other light sources in the training data set as the first light source, and the training data pair (I new ,gtill new ) subjected to light source enhancement is calculated as follows:

[0149]

[0150] gtill new =ill (3)

[0151] In some embodiments, the determining training data set can further include:

[0152] Step 9, performing brightness transformation on the first image sequence subjected to light source enhancement to obtain the second image sequence under the second light source scene with different brightness.

[0153] The brightness transformation method can also be logarithmic transformation.

[0154] Step 10, the first image sequence enhanced by the light source in the second light source scene, the second image sequence in the second light source scene, and the light source corresponding to the storage are obtained, and a new set of training data is obtained.

[0155] Suppose the first image sequence enhanced by the light source is I new , the first image sequence I new , and the corresponding reference light source estimation result is gtill new , the second image sequence after brightness transformation is I' new , then (I new , gtill new , I' new ) can be used as a new set of training data.

[0156] It can be understood that the new image sequence generated by the light source enhancement is added to the training data set, which can make up for the influence of insufficient data set on the generalization of the model and enrich the data diversity.

[0157] The loss determination module 703 is configured to determine a loss value according to the first light source estimation result, the second light source estimation result, the reference light source estimation result in the same set of training data, and a loss function.

[0158] In some embodiments, the spatio-temporal feature extraction network includes k layer network units connected in sequence, where k is a positive integer, k is consistent with the number of images contained in the input target image sequence (i.e. k = t), the input of the first layer network unit in the k layer network unit is the first image I1 in the target image sequence input into the spatio-temporal feature extraction network 3, the input of the i-th layer network unit in the k layer network unit includes the i-th image I i in the target image sequence input into the spatio-temporal feature extraction network 311 and the output of the i-1-th layer network unit, 1 < i < k, and the output of the k-th layer network unit in the k layer network unit is the spatio-temporal feature of the target image sequence input into the spatio-temporal feature extraction network, and the target image sequence includes the first image sequence or the second image sequence.

[0159] The network unit is a convolutional long short-term memory network ConvLSTM.

[0160] The light source estimation network includes at least one convolutional layer, a global average pooling GAP layer, and an output data dimension adjustment layer, where:

[0161] The at least one convolutional layer is used for further feature extraction from the spatio-temporal features extracted by the spatio-temporal feature extraction network.

[0162] The GAP layer is configured to determine average values of features extracted by the at least one convolutional layer according to color channels, to obtain a third vector formed by average values in each color channel.

[0163] The output data dimension adjustment layer is configured to perform a dimension adjustment operation on the third vector output by the GAP layer, to obtain a light source estimation result of the image sequence input into the spatio-temporal feature extraction network.

[0164] In some embodiments, the loss function is obtained by weighting a loss function of the multi-layer light source prediction network, and the loss function of the single-layer light source prediction network includes a color correction loss function and a light source color consistency loss function under different brightness, wherein,

[0165] The color correction loss function is configured to calculate a loss value of the light source estimation result predicted by the light source estimation model relative to a reference light source estimation result.

[0166] The light source color consistency loss function under different brightness is configured to calculate a light source estimation loss value between sequence images of different brightness in the same light source scene predicted by the light source estimation model.

[0167] The loss function of the single-layer light source prediction network is detailed in the above formulas (5) to (9), and will not be described here.

[0168] The parameter adjustment module 704 is configured to adjust parameters of the light source estimation model according to the loss value.

[0169] It can be understood that the two time series image data with different brightness in the same set of training data are respectively input into the light source estimation model, and first light source estimation result y1 and second light source estimation result y2 are output, and then the parameters and adaptive weight parameters of the light source estimation model are continuously optimized by back propagation algorithm to minimize the loss function. After training a relatively small loss value through multiple iterations, the model training is considered successful, and the model structure and parameters are saved. Specifically, the reference light source estimation result corresponding to the first light source scene, the plurality of first light source estimation sub-results, and the plurality of second light source estimation sub-results are respectively input into the above formula (6) and formula (7), so that the color estimation loss value and the light source estimation loss value under different brightness corresponding to each single-layer light source prediction network in the plurality of light source prediction networks are obtained; then, the color estimation loss value and the light source estimation loss value under different brightness corresponding to each single-layer light source prediction network are input into the above formula (5), so that the loss value corresponding to each single-layer light source prediction network is obtained; and then the loss value corresponding to each single-layer light source prediction network is input into the above formula (4), so that the final loss value is obtained. Based on the final loss value and the loss value corresponding to each single-layer light source prediction network, the parameters of the light source estimation model are iteratively optimized until the preset convergence condition is met, and the trained light source estimation model is obtained.

[0170] The light source estimation model training device provided in the embodiments of the present application uses image sequences with spatial and temporal information to train the light source estimation model and estimate the light source. Compared with the related art of using a single frame image to estimate the light source, using multiple frame images can better learn the information of adjacent frame images, thereby reducing the error of single frame estimation, adapting to environmental jump conditions, improving the stability of the estimation result, preventing color jump phenomenon in the image preview process, and making the image color after white balance processing more consistent with the true scene color.

[0171] In addition, the light source estimation model training device provided in the embodiments of the present application uses ConvLSTM as the basic unit of the single-layer light source prediction network. Since ConvLSTM can not only learn the spatial feature information of the image but also has certain memory ability and can capture the long-term dependence relationship in the input image sequence, the error of single frame estimation can be further reduced, the environmental jump condition can be adapted, and the accuracy, stability, and algorithm adaptability of the white balance result can be improved.

[0172] In addition, the light source estimation model training device provided in the embodiments of the present application uses ConvLSTM as the basic unit of the single-layer light source prediction network. Since ConvLSTM can not only learn the spatial feature information of the image but also has certain memory ability and can capture the long-term dependence relationship in the input image sequence, the error of single frame estimation can be further reduced, the environmental jump condition can be adapted, and the accuracy, stability, and algorithm adaptability of the white balance result can be improved.

[0173] Further, the light source estimation model training device provided by the embodiment of the present application considers not only the color correction loss when designing the loss function, but also designs the light source color consistency loss under different brightness and performs adaptive fusion, which helps to improve the accuracy and stability of light source prediction. In addition, the additional auxiliary loss is applied to the middle layer of the multi-layer light prediction network, which can help the network to learn more useful features, enhance the feature expression ability and improve the generalization.

[0174] The light source estimation model training device provided by the embodiment of the present application is introduced above, and on this basis, the embodiment of the present application also provides a light source estimation device, which is described below.

[0175] As shown in Figure 8 The light source estimation device provided by the embodiment of the present application can include an image determination module 801, an image input module 802 and a result output module 803.

[0176] The image determination module 801 is configured to determine a third image sequence to be processed.

[0177] The image input module 802 is configured to input the third image sequence into a light source estimation model, wherein the light source estimation model is trained by the light source estimation model training method described above, and the number of images contained in the third image sequence is consistent with the number of images of the image sequence input for training the light source estimation model.

[0178] The result output module 803 is configured to take the output of the light source estimation model as the light source estimation result of the third image sequence, and the light source estimation result is used to adjust the color of at least one image in the third image sequence.

[0179] In some embodiments, Figure 8 The device shown in the figure can also include a color adjustment module configured to adjust the color of at least one image in the third image sequence according to the light source estimation result of the third image sequence.

[0180] The light source estimation device provided by the embodiment of the present application uses the light source estimation model trained by the light source estimation model training method provided by the embodiment of the present application to perform light source estimation, so that the error of single-frame estimation can be reduced, the environmental jump condition can be adapted, the stability of the estimation result can be improved, the color jump phenomenon in the image preview process can be prevented, and the image color after white balance processing is more consistent with the true scene color.

[0181] Optionally, as Figure 9As shown, the embodiments of the present application further provide an electronic device 900, comprising a processor 901 and a memory 902, wherein the memory 902 stores a program or instructions executable on the processor 901, the program or instructions are executed by the processor 901 to implement each step of the above-mentioned light source estimation model training method or light source estimation method embodiments, and achieve the same technical effects. To avoid repetition, details are not repeated here.

[0182] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0183] Figure 10 To implement the hardware structure of an electronic device in the embodiments of the present application.

[0184] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010, etc.

[0185] Those skilled in the art can understand that the electronic device 1000 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 1010 through a power management system, so as to realize the functions of power management, such as charging, discharging, and power consumption management, through the power management system. Figure 10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not repeated here.

[0186] The processor 1010 is configured to determine a training data set, the training data set including a plurality of groups of training data, each group of training data including a first image sequence, a second image sequence, and a reference light source estimation result corresponding to the first image sequence, the first image sequence including a plurality of frames of images successively captured in a first light source scene, the second image sequence being obtained by performing brightness transformation on the first image sequence, and the reference light source estimation result being used to describe color characteristics of a real light source in the first light source scene; inputting the first image sequence in one group of the training data into a light source estimation model to obtain a first light source estimation result, and inputting the second image sequence in the same group of the training data into the light source estimation model to obtain a second light source estimation result, wherein the light source estimation model includes at least two layers of light source prediction networks, each layer of the light source prediction network including a space-time feature extraction network and a light source estimation network, the space-time feature extraction network being configured to extract space-time features from an input image sequence, and the light source estimation network being configured to perform light source estimation according to the space-time features; determining a loss value according to the first light source estimation result, the second light source estimation result, the reference light source estimation result in the same group of the training data, and a loss function; and adjusting parameters of the light source estimation model according to the loss value.

[0187] Alternatively, the processor 1010 is configured to determine a third image sequence to be processed; input the third image sequence into a light source estimation model; and take an output of the light source estimation model as a light source estimation result of the third image sequence, the light source estimation result being used to adjust colors of at least one image in the third image sequence.

[0188] In the embodiments of the present application, the image sequence (a plurality of frames of images) is used to train the light source estimation model and estimate the light source. Compared with the related art in which a single frame of image is used to estimate the light source, the use of the plurality of frames of images can better learn the information of adjacent frames of images, thereby reducing errors of single frame estimation, adapting to environmental jump conditions, improving stability of the estimation result, preventing color jump phenomenon in the image preview process, and making the image color after the white balance processing more consistent with the real scene color.

[0189] It should be understood that in the embodiments of the present application, the input unit 1004 can include a graphics processor (GPU) 10041 and a microphone 10042. The graphics processor 10041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 can include a display panel 10061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1007 includes at least one of a touch panel 10071 and other input devices 10072. The touch panel 10071 is also referred to as a touch screen. The touch panel 10071 can include two parts of a touch detection device and a touch controller. The other input devices 10072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, a joystick, and the like, which will not be described here.

[0190] The memory 1009 can be used to store software programs and various data. The memory 1009 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), etc. In addition, the memory 1009 can include a volatile memory or a non-volatile memory, or the memory 1009 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 1009 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0191] The processor 1010 can include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1010.

[0192] The embodiments of the present application also provide a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize each process of the above-mentioned light source estimation model training method or light source estimation method embodiment, and can achieve the same technical effects, to avoid repetition, which will not be repeated here.

[0193] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0194] The chip provided in the embodiments of the present application includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used to run programs or instructions, realizes each process of the above light source estimation model training method or light source estimation method embodiments, and can achieve the same technical effects. To avoid repetition, it will not be described here.

[0195] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip.

[0196] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to the order of functions shown or discussed, but can also include functions performed in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0197] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0198] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the scope of the present application, and all belong to the protection of the present application.

Claims

1. A method for training a light source estimation model, characterized in that, The method includes: A training dataset is determined, which includes multiple sets of training data. Each set of training data includes a first image sequence, a second image sequence, and a reference light source estimation result corresponding to the first image sequence. The first image sequence includes multiple frames of images continuously acquired under the first light source scene. The second image sequence is obtained by performing a brightness transformation on the first image sequence. The reference light source estimation result is used to describe the color characteristics of the real light source under the first light source scene. The first image sequence from a set of training data is input into the light source estimation model to obtain a first light source estimation result, and the second image sequence from the same set of training data is input into the light source estimation model to obtain a second light source estimation result. The light source estimation model includes at least two layers of light source prediction network. Each layer of the light source prediction network includes a spatiotemporal feature extraction network and a light source estimation network. The spatiotemporal feature extraction network is used to extract spatiotemporal features from the input image sequence, and the light source estimation network is used to estimate the light source based on the spatiotemporal features. The loss value is determined based on the first light source estimation result, the second light source estimation result, the reference light source estimation result in the same set of training data, and the loss function. The parameters of the light source estimation model are adjusted based on the loss value.

2. The method according to claim 1, characterized in that, The determination of the training dataset includes: Under the first light source scenario, n1 frames of images are continuously acquired, and a reference image containing the color chart is captured within a first time period after the acquisition of the n1 frames of images. Based on the acquisition time, extract t frames of images every m frames from the n1 frames of images to obtain the first image sequence under the first light source scene; The reference image in the first light source scene is processed to obtain the reference light source estimation result in the first light source scene; The first image sequence is subjected to brightness transformation to obtain the second image sequence with different brightness under the first light source scene; wherein, n1, m, and t are all positive integers, and n1 > t; The first image sequence, the second image sequence, and the reference light source estimation result under the first light source scene are saved accordingly to obtain a set of training data.

3. The method according to claim 2, characterized in that, The determination of the training dataset also includes: Select at least one first image sequence from the plurality of training data sets; White balance processing is performed on the selected first image sequence; The first image sequence after white balance processing is enhanced using a light source in a second light source scene, wherein the color of the light source in the second light source scene is different from the color of the light source in the first light source scene. The brightness of the first image sequence after light source enhancement is transformed to obtain the second image sequence with different brightness under the second light source scene; The first image sequence enhanced with light source under the second light source scenario, the second image sequence under the second light source scenario, and the corresponding light source are saved to obtain a new set of training data.

4. The method according to claim 2 or 3, characterized in that, The step of processing the reference image under the first light source scene to obtain the reference light source estimation result under the first light source scene includes: Take the average value of all pixels in the region where the color card is located in the reference image under the first light source scene according to the color channel to obtain a first vector containing the average color value under the three color channels; The first vector is normalized according to the first color channel to obtain the second vector, wherein the first color channel is any one of the three color channels; The second vector is used as the reference light source estimation result for the first light source scenario.

5. The method according to claim 1, characterized in that, The spatiotemporal feature extraction network includes sequentially connected... k Layer network unit, where, k It is a positive integer. k The number of images contained in the input target image sequence is consistent with the number of images contained therein. k The input to the first layer of the network unit is the first image in the target image sequence. k The input of the i-th layer network unit in the layer network unit includes the i-th image in the target image sequence and the output of the (i-1)-th layer network unit, 1 < i < 1. k The k The first layer of network unit k The output of the layer network unit is the spatiotemporal features of the target image sequence, which includes either the first image sequence or the second image sequence.

6. The method according to claim 1, characterized in that, The light source estimation network includes at least one convolutional layer, a global average pooling (GAP) layer, and an output data dimension adjustment layer, wherein... The at least one convolutional layer is used for further feature extraction from the spatiotemporal features extracted by the spatiotemporal feature extraction network; The GAP layer is used to determine the average value of the features extracted by the at least one convolutional layer according to the color channel, and obtain a third vector formed by the average value of each color channel. The output data dimension adjustment layer is used to perform a dimension adjustment operation on the third vector output by the GAP layer to obtain the light source estimation result of the image sequence input to the spatiotemporal feature extraction network.

7. The method according to claim 1, characterized in that, The loss function is obtained by weighting the loss functions of the multi-layer light source prediction network. The loss function of the single-layer light source prediction network includes a color correction loss function and a light source color consistency loss function under different brightness levels. The color correction loss function is used to calculate the loss value of the light source estimation result predicted by the light source estimation model relative to the reference light source estimation result; The light source color consistency loss function under different brightness is used to calculate the light source estimation loss value between sequential images of different brightness under the same light source scene predicted by the light source estimation model.

8. The method according to claim 7, characterized in that, The loss function of a single-layer light source prediction network is: in, in, Indicates color correction loss. The weights for color correction loss, This indicates the loss of color consistency of the light source under different brightness levels. The weights for the loss of color consistency of the light source under different brightness levels; Indicates the sample size, " represents the dot product of vectors, Represents the magnitude of a vector. This represents the inverse cosine function, used to convert cosine values ​​into angles. It's the learning rate. and This represents the gradient of the loss function with respect to the weights.

9. A light source estimation method, characterized in that, The method includes: Determine the third image sequence to be processed; The third image sequence is input into the light source estimation model, wherein the light source estimation model is trained by the light source estimation model training method according to any one of claims 1-8, and the number of images contained in the third image sequence is the same as the number of images in the first image sequence input for training the light source estimation model; The output of the light source estimation model is used as the light source estimation result of the third image sequence, and the light source estimation result is used to adjust the color of at least one image in the third image sequence.

10. A light source estimation model training device, characterized in that, The device includes: The data determination module is used to determine the training dataset, which includes multiple sets of training data. Each set of training data includes a first image sequence, a second image sequence, and a reference light source estimation result corresponding to the first image sequence. The first image sequence includes multiple frames of images continuously acquired under the first light source scene. The second image sequence is obtained by performing a brightness transformation on the first image sequence. The reference light source estimation result is used to describe the color characteristics of the real light source under the first light source scene. The data input module is used to input the first image sequence from a set of training data into the light source estimation model to obtain a first light source estimation result, and to input the second image sequence from the same set of training data into the light source estimation model to obtain a second light source estimation result. The light source estimation model includes at least two layers of light source prediction networks. Each layer of the light source prediction network includes a spatiotemporal feature extraction network and a light source estimation network. The spatiotemporal feature extraction network is used to extract spatiotemporal features from the input image sequence, and the light source estimation network is used to estimate the light source based on the spatiotemporal features. The loss determination module is used to determine the loss value based on the first light source estimation result, the second light source estimation result, the reference light source estimation result in the same set of training data, and the loss function. The parameter adjustment module is used to adjust the parameters of the light source estimation model according to the loss value.

11. A light source estimation device, characterized in that, The device includes: Image determination module, used to determine the third image sequence to be processed; An image input module is used to input the third image sequence into a light source estimation model, wherein the light source estimation model is trained by the light source estimation model training method according to any one of claims 1-8, and the number of images contained in the third image sequence is the same as the number of images in the first image sequence input for training the light source estimation model; The result output module is used to take the output of the light source estimation model as the light source estimation result of the third image sequence, and the light source estimation result is used to adjust the color of at least one image in the third image sequence.

Images