Light source value calculation method and device, electronic equipment and medium

Abstract

The application discloses a light source value calculation method and device, electronic equipment and medium, and belongs to the technical field of artificial intelligence. The light source value calculation method comprises the following steps: generating a first image based on image data collected by a camera; extracting first image feature information corresponding to the first image based on N candidate light sources, wherein N is a positive integer; obtaining prediction information corresponding to the N candidate light sources based on the first image feature information, wherein the prediction information is used to represent the similarity between each candidate light source in the N candidate light sources and the real light source of the first image; and calculating a predicted light source value based on the prediction information and the light source value of each candidate light source, wherein the predicted light source value is used to correct the first image.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a method, apparatus, electronic device, and medium for calculating light source values. Background Technology

[0002] With the development of smart terminal technology, the camera function of electronic devices has also undergone tremendous changes. When taking photos with electronic devices, users pay more attention to the color accuracy of the captured images.

[0003] In related technologies, when an electronic device takes a picture of a target object, it needs to process the raw format image captured by the camera to generate the final image. For example, the gray areas in the raw format image that reflect the true light source value are processed to obtain a predicted value of the true light source value illuminating the target object. Then, based on the predicted light source value, the non-gray areas in the original image format are color corrected to generate the final image.

[0004] However, when the above scheme is used, if there is no gray area in the above raw format image, or if the gray area is incorrectly identified, the predicted light source value will deviate from the actual light source value illuminating the target object. As a result, the final image obtained by correcting the above raw format image with the predicted light source value cannot represent the true color of the target object. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, electronic device, and medium for calculating light source values, which can solve the problem of deviation between predicted light source values ​​and actual light source values.

[0006] In a first aspect, embodiments of this application provide a method for calculating light source values. The method includes: generating a first image based on image data acquired by a camera; extracting first image features corresponding to the first image based on N candidate light sources, where N is a positive integer; obtaining prediction information corresponding to the N candidate light sources based on the first image feature information, wherein the prediction information is used to characterize the similarity between each candidate light source among the N candidate light sources and the real light source of the first image; and calculating a predicted light source value based on the prediction information and the light source value of each candidate light source, wherein the predicted light source value is used to correct the first image.

[0007] Secondly, embodiments of this application provide a device for calculating light source values. This device includes a generation module, an extraction module, a prediction module, and a calculation module. The generation module generates a first image based on image data acquired by a camera. The extraction module extracts first image features corresponding to the first image generated by the generation module based on N candidate light sources, where N is a positive integer. The prediction module obtains prediction information corresponding to the N candidate light sources based on the first image feature information extracted by the extraction module. This prediction information characterizes the similarity between each candidate light source and the real light source of the first image. The calculation module calculates a predicted light source value based on the prediction information obtained by the processing module and the light source value of each candidate light source. This predicted light source value is used to correct the first image.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, the electronic device first generates a first image based on image data captured by a camera; then, based on N candidate light sources, it extracts first image features corresponding to the first image, where N is a positive integer; next, based on the first image feature information, it obtains prediction information corresponding to the N candidate light sources, which characterizes the similarity between each candidate light source and the real light source of the first image; finally, based on the prediction information and the light source value of each candidate light source, it calculates a predicted light source value, which is used to correct the first image. Thus, because the electronic device can predict the similarity between each candidate light source and the real light source, the predicted light source value calculated by the electronic device based on these multiple similarities is more accurate, thereby making the image corrected by the predicted light source value more accurately represent true colors. Attached Figure Description

[0013] Figure 1 This is one of the flowcharts illustrating a method for calculating a light source value provided in an embodiment of this application;

[0014] Figure 2 This is a light source value distribution map applied to a light source value calculation method provided in an embodiment of this application;

[0015] Figure 3 This is a linear graph of the function used in the method for calculating light source values ​​provided in the embodiments of this application;

[0016] Figure 4 This is a second schematic flowchart of a method for calculating a light source value provided in an embodiment of this application;

[0017] Figure 5 This is a schematic diagram of the structure of a light source value calculation device provided in an embodiment of this application;

[0018] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0019] Figure 7 This is a hardware schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0021] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0022] The following description, in conjunction with the accompanying drawings, details the calculation method, apparatus, electronic device, and medium for the light source value provided in this application through specific embodiments and application scenarios.

[0023] When taking photos, the camera automatically adjusts its white balance function to address the issue of varying colors in photos due to differences in the response of different camera image sensors to light sources, as well as the human eye's requirement for color constancy. Color constancy means that regardless of whether a white, blue, or yellow light source shines on a red strawberry, the human eye will perceive the strawberry as red, even though the light received by the human eye is not actually red (similarly, the light received by the camera is not actually red). To reproduce the true color of the object in a photograph, the camera needs to adjust the white balance to ensure that the colors in the photograph match the true colors of the object.

[0024] In related technologies, the methods used by electronic devices to predict light source values ​​can be broadly divided into two types:

[0025] One approach is to use an automatic white balance algorithm, which typically relies on the gray world assumption. The gray world assumption states that the average color of our world is gray (the average of the red, green, and blue channels represents the response of the light source to the camera's image sensor). Under this assumption, the electronic device directly averages each pixel block in the linear image to obtain the predicted light source values ​​R / G and B / G. However, this approach is problematic when the gray world assumption is not met (such as when the linear image is composed of a single color), resulting in a significant deviation between the predicted and actual light source values.

[0026] Another approach is based on a gray-block finding algorithm. This algorithm searches for gray blocks within the area covered by the camera's image sensor lens. Since gray blocks have the same response value to light, the average response of the three color channels (R (Red), G (Green), and B (Blue)) of the gray block is the response value of the real light source on the camera's image sensor. The electronic device can then calculate the predicted light source values ​​R / G and B / G based on this response value. However, this approach has limitations. When no gray block exists within the area covered by the image sensor lens, or when the electronic device misjudges a gray block (i.e., misleading color conditions occur—where the electronic device cannot accurately distinguish between a yellow light source illuminating a white object and a white light source illuminating a yellow object, potentially misclassifying it as a gray block), the algorithm cannot predict the accurate light source value.

[0027] In the light source value calculation method, apparatus, electronic device, and medium provided in this application embodiment, the electronic device can first generate a first image based on image data acquired by a camera; then, based on N candidate light sources, extract first image feature information corresponding to the first image, where N is a positive integer; next, based on the first image feature information, obtain prediction information corresponding to the N candidate light sources, which is used to characterize the similarity between each of the N candidate light sources and the real light source of the first image; finally, based on the prediction information and the light source value of each candidate light source, calculate a predicted light source value, which is used to correct the first image. Thus, since the electronic device can predict the similarity between each candidate light source and the real light source, the predicted light source value calculated by the electronic device based on these multiple similarities is more accurate, thereby making the image corrected by the predicted light source value more accurately represent true colors.

[0028] The execution entity of the light source value calculation method provided in this embodiment can be a light source value calculation device, which can be an electronic device, or a control module or processing module in the electronic device, etc. The following uses an electronic device as an example to illustrate the technical solution provided in this application embodiment.

[0029] This application provides a method for calculating light source values, such as... Figure 1 As shown, the method for calculating the light source value may include the following steps 201 to 204:

[0030] Step 201: The electronic device generates a first image based on the image data captured by the camera.

[0031] In the embodiments of this application, the above-mentioned image data can be the raw data of light source signals captured by the camera of an electronic device and converted into digital signals.

[0032] In this embodiment of the application, the first image may be an original image that has not been processed by an electronic device.

[0033] Step 202: The electronic device extracts the first image feature information corresponding to the first image based on N candidate light sources.

[0034] Where N is a positive integer.

[0035] In this embodiment of the application, the above-mentioned N candidate light sources can be N light sources preset by the user.

[0036] For example, an electronic device can collect the R / G and B / G ratios of all possible real light sources, obtaining the distribution of R / G and B / G. Then, the user can select a certain range with a high R / G and B / G distribution density and uniformly divide it into N points. Figure 2 As shown, each of the above N points represents a candidate light source, and the R / G and B / G values ​​corresponding to each point are the light source values ​​of the candidate light source.

[0037] In this embodiment of the application, the first image feature information may include the feature vector of each pixel block in the first image.

[0038] Step 203: The electronic device obtains prediction information corresponding to N candidate light sources based on the first image feature information.

[0039] In this embodiment of the application, the prediction information is used to characterize the similarity between each of the N candidate light sources and the real light source of the first image.

[0040] In this embodiment of the application, the aforementioned real light source refers to the light source that actually illuminates the aforementioned first image.

[0041] Step 204: The electronic device calculates the predicted light source value based on the prediction information and the light source value of each candidate light source.

[0042] In this embodiment of the application, the predicted light source value is used to correct the first image.

[0043] In this embodiment of the application, the electronic device can adjust the color of the original image captured by the camera based on the predicted light source value in order to restore the true color.

[0044] In this embodiment of the application, the electronic device can first use the above-mentioned predicted light source value to calculate the values ​​of correction parameters R, G, and B, and then perform color adjustment on all pixels in the first image based on the correction parameters so that the R, G, and B values ​​of each pixel in the adjusted image are consistent with the correction parameters.

[0045] In the light source value calculation method provided in this application embodiment, the electronic device can first generate a first image based on image data acquired by a camera; then, based on N candidate light sources, extract the first image feature information corresponding to the first image, where N is a positive integer; next, based on the first image feature information, obtain prediction information corresponding to the N candidate light sources, which is used to characterize the similarity between each of the N candidate light sources and the real light source of the first image; finally, based on the prediction information and the light source value of each candidate light source, calculate the predicted light source value, which is used to correct the first image. Thus, since the electronic device can predict the similarity between each candidate light source and the real light source, the predicted light source value calculated by the electronic device based on these multiple similarities is more accurate, thereby making the image corrected by the predicted light source value more accurately represent true colors.

[0046] Optionally, in this embodiment of the application, when the first image includes a first sub-image, step 202, "the electronic device extracts the first image feature information corresponding to the first image based on N candidate light sources," may include the following steps 202a and 202b:

[0047] Step 202a: After the electronic device inputs the first sub-image into the feature prediction model, it extracts features from the first sub-image based on the convolutional neural network in the feature prediction model to obtain the semantic feature information of the first sub-image under each candidate light source.

[0048] In this embodiment of the application, the first sub-image can be a linear graph (such as a graph in raw format).

[0049] For example, the linear graph described above can be a graph generated from raw, unprocessed data in an electronic device.

[0050] In this embodiment of the application, the semantic feature information mentioned above includes low-level information such as the color, texture, and shape of the first sub-image.

[0051] Step 202b: The electronic device performs feature transformation on the semantic feature information based on the global pooling layer in the feature prediction model to obtain the second image feature information of the first sub-image under each candidate light source.

[0052] In this embodiment of the application, the second image feature information is the same as the first image feature information.

[0053] In this embodiment of the application, the second image feature information may include a second image feature vector.

[0054] In this embodiment, the feature prediction model includes a convolutional neural network and a global pooling layer. An electronic device can adjust the first sub-image to a size matching the convolutional neural network, then input it into the convolutional neural network to extract semantic features of the first sub-image. These semantic features are then input into the global pooling layer to obtain a deep semantic feature vector of the first sub-image.

[0055] It should be noted that the aforementioned deep semantic feature vector is the same as the aforementioned second image feature vector.

[0056] Thus, by inputting the first sub-image into the feature prediction model, the second image feature information of the first sub-image under each candidate light source is extracted, so as to improve the efficiency of subsequent processing of the first sub-image.

[0057] Optionally, in this embodiment of the application, when the first image includes a first sub-image and a second sub-image, in combination with the above steps 202a and 202b, the step 202, "the electronic device extracts the first image feature information corresponding to the first image based on N candidate light sources," may include the following step 202c:

[0058] Step 202c: The electronic device concatenates the second image feature information with the color temperature feature information and grayscale world hypothesis features contained in the second sub-image to obtain the first image feature information.

[0059] In this embodiment of the application, the second sub-image may include an image in Joint Photographic Experts Group (jpg) format.

[0060] In the embodiments of this application, after acquiring image data, the electronic device can use the image data to save images in different formats in the electronic device.

[0061] In this embodiment of the application, the aforementioned color temperature feature information may include a color temperature feature vector.

[0062] In this embodiment of the application, the gray-scale world hypothesis features mentioned above may include gray-scale world hypothesis feature vectors.

[0063] It should be noted that the grayscale world hypothesis features mentioned above are used to set the average reflection of natural objects to a fixed value, and to force this fixed value to be applied to the second image feature information to eliminate the influence of ambient light on the first image, thereby obtaining a more realistic first image.

[0064] In this embodiment, the color temperature feature vector, the grayscale world hypothesis feature vector, and the second image feature vector have the same vector dimension.

[0065] In this embodiment of the application, the first image feature vector is the image feature vector obtained by vector summing the above-mentioned color temperature feature vector, the above-mentioned grayscale world hypothesis feature vector, and the above-mentioned second image feature vector.

[0066] Thus, by adding the color temperature feature vector, the grayscale world hypothesis feature vector, and the second image feature vector together to obtain the first image feature vector, the accuracy of the electronic device's subsequent prediction of light source values ​​is further improved.

[0067] Optionally, in the embodiments of this application, before step 202c above, the method for calculating the light source value provided in the embodiments of this application may further include the following steps 202c1 and 202c2:

[0068] Step 202c1: The electronic device inputs the color temperature data in the second sub-image into the first multilayer perceptron for encoding to obtain color temperature feature information.

[0069] In this embodiment of the application, the first multilayer sensor is used to convert the color temperature data into color temperature feature information.

[0070] In this embodiment, the electronic device can read the color temperature data stored in the exchangeable image file information that comes with the above-mentioned JPG format image, and then encode the color temperature data into a color temperature feature vector with the same dimension as the above-mentioned convolutional neural network through a multilayer perceptron.

[0071] Step 202c2: The electronic device inputs the grayscale world hypothesis value into the second multilayer perceptron for encoding to obtain the grayscale world hypothesis features.

[0072] In this embodiment of the application, the second multilayer perceptron is used to convert the gray-scale world hypothesis value into gray-scale world hypothesis features.

[0073] In this embodiment, the electronic device can, based on the grayscale world hypothesis, average the pixels in the unexposed areas of the raw format image according to the R, G, and B channels to obtain the grayscale world hypothesis value RGB1. Then, the grayscale world hypothesis value RGB1 is encoded into a grayscale world hypothesis feature vector with the same dimension as the convolutional neural network by passing it through a multilayer perceptron.

[0074] In this way, by using a multilayer perceptron to convert color temperature data and grayscale world assumptions into feature information, the efficiency of subsequent processing in electronic devices is further improved.

[0075] Optionally, in this embodiment of the application, step 203 above, "the electronic device obtains prediction information corresponding to N candidate light sources based on the first image feature information," may include step 203a:

[0076] Step 203a: The electronic device inputs the first image feature information into the feature information classification model, classifies the first image feature information, and obtains the prediction information corresponding to N candidate light sources.

[0077] In this embodiment of the application, the above-mentioned feature information classification model is used to distinguish the feature information corresponding to each candidate light source.

[0078] In this embodiment, the feature information classification model includes a fully connected layer and a normalized exponential function (SoftMax).

[0079] In this embodiment of the application, the prediction information is used to characterize the similarity between each of the N candidate light sources and the real light source of the first image.

[0080] For example, the similarity mentioned above can be a similarity score or a similarity probability.

[0081] In this embodiment, the electronic device can input the first image feature vector into the fully connected layer to obtain a similarity score between each candidate light source and the real light source of the first image. Then, the similarity score is used to calculate the similarity probability using the SoftMax function.

[0082] Thus, by inputting the first image feature information into the feature information classification model, specific prediction information for N candidate light sources is obtained, thereby improving the accuracy of the candidate predicted light source values ​​for electronic devices.

[0083] Optionally, in this embodiment of the application, after step 204 above, the method for calculating the light source value provided in this embodiment of the application may further include the following steps 301 to 304:

[0084] Step 301: The electronic device calculates the dispersion value of the N candidate light sources based on the predicted information and the light source values ​​of the N candidate light sources.

[0085] In this embodiment of the application, after obtaining the above similarity probability and the light source values ​​of the above N candidate light sources, the electronic device can substitute the above similarity probability and the light source values ​​of the above N candidate light sources into formula (1) and formula (2) to calculate the dispersion value of the N candidate light sources.

[0086] For example, after obtaining the above similarity probabilities, the electronic device can combine N candidate light sources to calculate the variance and covariance of R / G and B / G, and then calculate the Σ value using formula (1). Formula (1) is as follows:

[0087]

[0088] Among them, Var r / g Var represents the variance of R / G. b / g Let R represent the variance of B / G, and Cov(r / g,b / g) represent the covariance of R / G and B / G. ε is a constant that can take the value 1.

[0089] Then, the natural logarithm of the obtained Σ value is taken using formula (2) to obtain the dispersion value. Formula (2) is as follows:

[0090] dispersion=log(|Σ|) formula (2)

[0091] Step 302: The electronic device calculates the angular error between the predicted light source value and the actual light source value based on the predicted light source value and the actual light source value.

[0092] In this embodiment of the application, the aforementioned angle error value can be the cosine of the angle between the vectors corresponding to the predicted light source value and the actual light source value.

[0093] For example, the smaller the cosine value mentioned above, the closer the predicted light source value is to the actual light source value.

[0094] The following will provide an example of how to obtain the light source values ​​of the aforementioned real light source:

[0095] For example, taking the above-mentioned raw format image including a standard color chart as an example. After generating the raw format image, the electronic device can obtain the key points in the above-mentioned raw format image that match the above-mentioned standard color chart according to the image key point detection algorithm (such as SURF key point detection) and key point matching algorithm (such as KNNMatch). Then, using the matched key points, the transformation matrix is ​​calculated according to the homography matrix transformation principle, and the transformed standard color chart is obtained according to the transformation matrix. Finally, according to the preset 6 gray levels, the corresponding pixel blocks are extracted respectively, and the mean values ​​of R, G, and B are calculated for the pixel blocks of each gray level. The brightness value L is calculated according to the brightness calculation formula L = 0.3*R + 0.59*G + 0.11*B to ensure that the brightness value L is within a good linear range (such as (3000, 55000)). Next, the mean values ​​of the pixel blocks whose brightness values ​​L are within the linear range for the above 6 gray levels are calculated again to obtain the final R, G, and B, which are the light source values ​​of the real light source.

[0096] Step 303: The electronic device fits the discreteness value and the angle error value to obtain the fitting function.

[0097] In the embodiments of this application, the above-mentioned fitting function is used to characterize the correlation between the discreteness of the candidate light source and the angle error.

[0098] In this embodiment of the application, the electronic device can fit the dispersion value and angle error value between the predicted light source value obtained from multiple predictions and their respective corresponding real light source values ​​to obtain the corresponding fitting function.

[0099] Step 304: The electronic device inputs the dispersion value between the predicted light source value and the actual light source value into the fitting function to obtain the target angle error value.

[0100] In this embodiment of the application, the target angle error is used to indicate whether the predicted light source value is valid.

[0101] In this embodiment, the electronic device can establish a relationship between the predicted light source value and the actual light source value, and fit the relationship between the degree of dispersion and the angle error using a fitting function (such as the least squares method). Then, based on the degree of dispersion value calculated from the similarity probability output by the feature information classification model, the corresponding angle error value range can be estimated, providing an evaluation index for the prediction of the feature information classification model and guiding whether subsequent prediction results should be adopted.

[0102] For example, consider the above electronic device performing five light source value predictions. The electronic device can obtain the dispersion values ​​D1, D2, D3, D4, and D5 corresponding to the five predictions, and the angle error values ​​E1, E2, E3, E4, and E5 respectively. Then, assuming D = k * E + b, and substituting the dispersion values ​​and angle error values ​​corresponding to the five predictions into formulas (3) and (4), the corresponding k and b are calculated. Figure 3 The diagram illustrates the relationship between the dispersion (k = 0.002, b = 0.0005) and the angle error (i.e., the least squares method described above). Finally, the electronic device can substitute the currently predicted dispersion value into the fitting function calculated above to obtain the corresponding target angle error value.

[0103] Formulas (3) and (4) are as follows:

[0104]

[0105]

[0106] In this way, by calculating and statistically analyzing data from multiple test sets, the correlation between dispersion and angle error can be obtained, enabling electronic devices to determine the accuracy of the predicted light source value based on this correlation.

[0107] The following will provide an exemplary description of the method for calculating the light source value provided in the embodiments of this application:

[0108] For example, taking the first sub-image as a RAW format image and the second sub-image as a JPG format image, as follows: Figure 4 As shown, the method for calculating the light source value provided in this application may include the following steps P1 to P9:

[0109] Step P1: The electronic device acquires image data and generates images in RAW and JPG formats.

[0110] Step P2: When the electronic device is collecting image data, the actual light source value in the environment is displayed.

[0111] Step P3: The electronic device divides the distribution range of R / G and B / G into N grid points based on the light source values ​​of all possible real light sources, thus obtaining N candidate light sources.

[0112] Step P4: The electronic device parses the color temperature value from the above JPG format image.

[0113] Step P5: The electronic device adjusts the above raw format image to a suitable size and inputs it into the convolutional neural network to extract semantic features 1.

[0114] Step P6: The electronic device inputs the light source values ​​predicted in the gray-scale world hypothesis into the multilayer perceptron 1 and encodes them to obtain feature 2.

[0115] Step P7: The electronic device inputs the above color temperature data into the multilayer perceptron 2 and encodes it to obtain feature 3.

[0116] Step P8: The electronic device fuses the above features 1, 2 and 3 to obtain a new feature 4. Then, it inputs feature 4 into the fully connected layer and the SoftMax layer to obtain the probability distribution of N candidate light sources. Finally, it calculates the expectation of the probability distribution with the above N candidate light sources to obtain the predicted light source value.

[0117] Step P9: The electronic device calculates the correlation between the degree of dispersion and the angle error using the above formulas (1) and (2) based on the above probability distribution, and generates a fitting function to calculate the angle error of the predicted light source value each time.

[0118] The light source value calculation method provided in this application can be executed by a light source value calculation device. This application uses the example of a light source value calculation device executing the light source value calculation method to illustrate the light source value calculation device provided in this application.

[0119] This application provides a device for calculating light source values, such as... Figure 5 As shown, the light source value calculation device 400 includes: a generation module 401, an extraction module 402, a prediction module 403, and a calculation module 404, wherein: the generation module 401 is used to generate a first image based on image data acquired by a camera; the extraction module 402 is used to extract first image feature information corresponding to the first image generated by the generation module 401 based on N candidate light sources, where N is a positive integer; the prediction module 403 is used to obtain prediction information corresponding to the N candidate light sources based on the first image feature information extracted by the extraction module 402, and the prediction information is used to characterize the similarity between each of the N candidate light sources and the real light source of the first image; the calculation module 404 is used to calculate a predicted light source value based on the prediction information obtained by the prediction module 403 and the light source value of each candidate light source, and the predicted light source value is used to correct the first image.

[0120] Optionally, in this embodiment of the application, when the first image includes a first sub-image, the extraction module 402 is specifically configured to: input the first sub-image into the feature prediction model, and then extract features from the first sub-image based on the convolutional neural network in the feature prediction model to obtain semantic feature information of the first sub-image under each candidate light source; and perform feature transformation on the semantic feature information based on the global pooling layer in the feature prediction model to obtain second image feature information of the first sub-image under each candidate light source; wherein the second image feature information is the first image feature information.

[0121] Optionally, in this embodiment of the application, when the first image includes a first sub-image and a second sub-image, the extraction module 402 is specifically configured to: input the first sub-image into the feature prediction model, and then extract features from the first sub-image based on the convolutional neural network in the feature prediction model to obtain semantic feature information of the first sub-image under each candidate light source; perform feature transformation on the semantic feature information based on the global pooling layer in the feature prediction model to obtain second image feature information of the first sub-image under each candidate light source; and concatenate the second image feature information with the color temperature feature information and grayscale world hypothesis feature information contained in the second sub-image to obtain the first image feature information.

[0122] Optionally, in this embodiment of the application, the prediction module 403 is specifically used to input the first image feature information into the feature information classification model, classify the first image feature information, and obtain the prediction information corresponding to the N candidate light sources.

[0123] Optionally, in this embodiment, the calculation module 404 is further configured to: calculate the predicted light source value based on the predicted information and the light source value of each candidate light source; calculate the dispersion value of the N candidate light sources based on the predicted information and the light source values ​​of the N candidate light sources; calculate the angle error value between the predicted light source value and the light source value of the actual light source based on the predicted light source value and the light source value of the actual light source; fit the dispersion value and the angle error value to obtain a fitting function, which is used to characterize the correlation between the dispersion of the candidate light source and the angle error; input the dispersion value between the predicted light source value and the light source value of the actual light source into the fitting function to obtain a target angle error value; wherein the target angle error value is used to indicate whether the predicted light source value is valid.

[0124] In the light source value calculation device provided in this application embodiment, the light source value calculation device can first generate a first image based on image data captured by a camera; then, based on N candidate light sources, extract the first image features corresponding to the first image, where N is a positive integer; next, based on the first image feature information, obtain prediction information corresponding to the N candidate light sources, which is used to characterize the similarity between each of the N candidate light sources and the real light source of the first image; finally, based on the prediction information and the light source value of each candidate light source, calculate the predicted light source value, which is used to correct the first image. Thus, since the light source value calculation device can predict the similarity between each candidate light source and the real light source, the predicted light source value calculated by the light source value calculation device based on these multiple similarities is more accurate, thereby making the image corrected by the predicted light source value more accurately represent true colors.

[0125] The device for calculating the light source value in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0126] The device for calculating the light source value in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit the specific operating system used.

[0127] The light source value calculation device provided in this application embodiment can realize Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0128] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described light source value calculation method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0129] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0130] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0131] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0132] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0133] The processor 110 is configured to generate a first image based on image data acquired by a camera; extract first image feature information corresponding to the first image based on N candidate light sources, where N is a positive integer; obtain prediction information corresponding to the N candidate light sources based on the first image feature information, wherein the prediction information is used to characterize the similarity between each of the N candidate light sources and the real light source of the first image; and calculate a predicted light source value based on the prediction information and the light source value of each of the candidate light sources, wherein the predicted light source value is used to correct the first image.

[0134] Optionally, in this embodiment of the application, when the first image includes a first sub-image, the processor 110 is specifically configured to: input the first sub-image into a feature prediction model, and then extract features from the first sub-image based on the convolutional neural network in the feature prediction model to obtain semantic feature information of the first sub-image under each candidate light source; and perform feature transformation on the semantic feature information based on the global pooling layer in the feature prediction model to obtain second image feature information of the first sub-image under each candidate light source; wherein the second image feature information is the first image feature information.

[0135] Optionally, in this embodiment of the application, when the first image includes a first sub-image and a second sub-image, the processor 110 is specifically configured to: input the first sub-image into a feature prediction model, and then extract features from the first sub-image based on the convolutional neural network in the feature prediction model to obtain semantic feature information of the first sub-image under each candidate light source; perform feature transformation on the semantic feature information based on the global pooling layer in the feature prediction model to obtain second image feature information of the first sub-image under each candidate light source; and concatenate the second image feature information with the color temperature feature information and grayscale world hypothesis feature information contained in the second sub-image to obtain the first image feature information.

[0136] Optionally, in this embodiment of the application, the processor 110 is specifically used to input the first image feature information into a feature information classification model, classify the first image feature information, and obtain prediction information corresponding to the N candidate light sources.

[0137] Optionally, in this embodiment of the application, the processor 110 is further configured to: after calculating the predicted light source value based on the predicted information and the light source value of each candidate light source, calculate the dispersion value of the N candidate light sources based on the predicted information and the light source values ​​of the N candidate light sources; calculate the angle error value between the predicted light source value and the light source value of the actual light source based on the predicted light source value and the light source value of the actual light source; fit the dispersion value and the angle error value to obtain a fitting function, which is used to characterize the correlation between the dispersion of the candidate light source and the angle error; input the dispersion value between the predicted light source value and the light source value of the actual light source into the fitting function to obtain a target angle error value; wherein, the target angle error value is used to indicate whether the predicted light source value is valid.

[0138] In the electronic device provided in this application embodiment, the electronic device can first generate a first image based on image data captured by a camera; then, based on N candidate light sources, extract first image feature information corresponding to the first image, where N is a positive integer; next, based on the first image feature information, obtain prediction information corresponding to the N candidate light sources, which is used to characterize the similarity between each of the N candidate light sources and the real light source of the first image; finally, based on the prediction information and the light source value of each candidate light source, calculate a predicted light source value, which is used to correct the first image. Thus, since the electronic device can predict the similarity between each candidate light source and the real light source, the predicted light source value calculated by the electronic device based on these multiple similarities is more accurate, thereby making the image corrected by the predicted light source value more accurately represent true colors.

[0139] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0140] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0141] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0142] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described light source value calculation method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0143] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0144] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described light source value calculation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0145] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0146] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described light source value calculation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0149] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method of calculating a light source value, characterized by, The method comprises: generating a first image based on image data collected by a camera; extracting first image feature information corresponding to the first image based on N candidate light sources, N being a positive integer; obtaining prediction information corresponding to the N candidate light sources based on the first image feature information, the prediction information being used to represent the similarity between each candidate light source in the N candidate light sources and the real light source of the first image; calculating a predicted light source value based on the prediction information and the light source value of each candidate light source, the predicted light source value being used to correct the first image; calculating a correction parameter based on the predicted light source value, and performing color adjustment on all pixel points in the first image based on the correction parameter.

2. The method of claim 1, wherein, In the case where the first image comprises a first sub-image, the second image feature information is the first image feature information, and the extraction of the first image feature information corresponding to the first image based on the N candidate light sources comprises: after inputting the first sub-image into a feature prediction model, performing feature extraction on the first sub-image based on a convolutional neural network in the feature prediction model to obtain semantic feature information of the first sub-image under each candidate light source; performing feature conversion on the semantic feature information based on a global pooling layer in the feature prediction model to obtain second image feature information of the first sub-image under each candidate light source, wherein the second image feature information is the first image feature information.

3. The method of claim 1, wherein, In the case where the first image comprises a first sub-image and a second sub-image, the extraction of the first image feature information corresponding to the first image based on the N candidate light sources comprises: after inputting the first sub-image into a feature prediction model, performing feature extraction on the first sub-image based on a convolutional neural network in the feature prediction model to obtain semantic feature information of the first sub-image under each candidate light source; performing feature conversion on the semantic feature information based on a global pooling layer in the feature prediction model to obtain second image feature information of the first sub-image under each candidate light source; splicing the second image feature information with color temperature feature information and gray world assumption feature information contained in the second sub-image to obtain the first image feature information.

4. The method according to any one of claims 1 to 3, characterized in that, The obtaining of the prediction information corresponding to the N candidate light sources based on the first image feature information comprises: inputting the first image feature information into a feature information classification model to classify the first image feature information to obtain the prediction information corresponding to the N candidate light sources.

5. The method of claim 1, wherein, After the calculation of the predicted light source value based on the prediction information and the light source value of each candidate light source, the method further comprises: calculating a dispersion degree value of the N candidate light sources based on the prediction information and the light source value of the N candidate light sources; calculating an angle error value of the predicted light source value and the light source value of the real light source based on the predicted light source value and the light source value of the real light source; fitting the discrete degree value and the angle error value to obtain a fitting function, the fitting function being used to represent a correlation between the discrete degree and the angle error of the candidate light source; inputting the discrete degree value between the predicted light source value and the light source value of the real light source into the fitting function to obtain a target angle error value; wherein the target angle error value is used to indicate whether the predicted light source value is valid.

6. A light source value calculating apparatus characterized by comprising: The device comprises a generation module, an extraction module, a prediction module, a calculation module and a processing module, wherein: The generation module is configured to generate a first image based on image data collected by a camera. The extraction module is configured to extract first image feature information corresponding to the first image generated by the generation module based on N candidate light sources, N being a positive integer. The prediction module is configured to obtain prediction information corresponding to the N candidate light sources based on the first image feature information extracted by the extraction module, the prediction information being used to represent the similarity between each candidate light source in the N candidate light sources and the real light source of the first image. The calculation module is configured to calculate a predicted light source value based on the prediction information obtained by the prediction module and the light source value of each candidate light source, the predicted light source value being used to correct the first image, and calculate a correction parameter based on the predicted light source value. The processing module is configured to perform color adjustment on all pixel points in the first image based on the correction parameter.

7. The apparatus of claim 6, wherein, In the case where the first image comprises a first sub-image, The extraction module is specifically configured to: input the first sub-image into a feature prediction model, perform feature extraction on the first sub-image based on a convolutional neural network in the feature prediction model, and obtain semantic feature information of the first sub-image under each candidate light source; perform feature conversion on the semantic feature information based on a global pooling layer in the feature prediction model, and obtain second image feature information of the first sub-image under each candidate light source; wherein the second image feature information is the first image feature information.

8. The apparatus of claim 6, wherein, In the case where the first image comprises a first sub-image and a second sub-image, The extraction module is specifically configured to: input the first sub-image into a feature prediction model, perform feature extraction on the first sub-image based on a convolutional neural network in the feature prediction model, and obtain semantic feature information of the first sub-image under each candidate light source; perform feature conversion on the semantic feature information based on a global pooling layer in the feature prediction model, and obtain second image feature information of the first sub-image under each candidate light source; splice the second image feature information with color temperature feature information and gray world assumption feature information contained in the second sub-image to obtain the first image feature information.

9. The device of any one of claims 6 to 8, wherein: The prediction module is specifically configured to input the first image feature information into a feature information classification model, classify the first image feature information, and obtain the prediction information corresponding to the N candidate light sources.

10. The apparatus of claim 6, wherein, the computing module is further configured to: calculate a dispersion degree value of the N candidate light sources based on the prediction information and the light source values of the N candidate light sources after calculating a predicted light source value based on the prediction information and the light source value of each candidate light source; calculate an angle error value between the predicted light source value and the light source value of the real light source based on the predicted light source value and the light source value of the real light source; fit the dispersion degree value and the angle error value to obtain a fitting function, the fitting function being used to represent a correlation between the dispersion degree of a candidate light source and the angle error; input the dispersion degree value between the predicted light source value and the light source value of the real light source into the fitting function to obtain a target angle error value; wherein the target angle error value is used to indicate whether the predicted light source value is valid.

11. An electronic device, comprising: a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the light source value calculation method according to any one of claims 1 to 5.

12. A readable storage medium, characterized by, programs or instructions stored on the readable storage medium, the programs or instructions being executed by the processor to implement the steps of the light source value calculation method according to any one of claims 1 to 5.

Images