Methods, apparatus, storage media and electronic devices for illumination detection of images

Abstract

This disclosure provides a method, apparatus, computer-readable storage medium, and electronic device for image illumination detection, relating to the field of computer vision technology. The image illumination detection method includes: acquiring a target image collected by an image sensor, and spectral data of multiple detection regions collected by multiple spectral sensors; each spectral sensor corresponds to a detection region, and each detection region corresponds to a sub-region in the target image; parsing the spectral data of each detection region to obtain illumination information of each sub-region in the target image. This disclosure addresses the problems of low detection accuracy and long processing time in illumination information detection by using spectral sensors to perform illumination information detection on images, thereby improving the detection efficiency of illumination information detection in images to a certain extent.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a method for detecting illumination in images, a device for detecting illumination in images, a computer-readable storage medium, and an electronic device. Background Technology

[0002] The technology of image illumination detection has been applied in many scenarios. For example, the colors of an image can be adjusted based on the illumination detection results to make the image display on the screen consistent with the human eye.

[0003] In related technologies, clustering algorithms are typically used to detect illumination in images. However, this method is easily misled by large areas of confusing colors in the image, leading to inaccurate detection results. Summary of the Invention

[0004] This disclosure provides a method for detecting illumination in an image, an apparatus for detecting illumination in an image, a computer-readable storage medium, and an electronic device, thereby at least partially solving the problem of inaccurate illumination detection in related technologies.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to a first aspect of this disclosure, a method for detecting illumination in an image is provided, comprising: acquiring a target image collected by an image sensor, and spectral data of multiple detection regions collected by multiple spectral sensors; each spectral sensor corresponding to a detection region, and each detection region corresponding to a sub-region in the target image; and parsing the spectral data of each detection region to obtain illumination information of each sub-region in the target image.

[0007] According to a second aspect of this disclosure, an illumination detection apparatus for an image is provided, comprising: a data acquisition module configured to acquire a target image collected by an image sensor, and spectral data of a plurality of detection regions collected by a plurality of spectral sensors; each spectral sensor corresponds to a detection region, and each detection region corresponds to a sub-region in the target image; and an illumination information detection module configured to parse the spectral data of each detection region to obtain illumination information of each sub-region in the target image.

[0008] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the illumination detection method for an image described in the first aspect and its possible implementations.

[0009] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor. The processor is configured to execute the illumination detection method for an image and possible implementations thereof as described in the first aspect by executing the executable instructions.

[0010] The technical solution disclosed herein has the following beneficial effects:

[0011] This disclosure first acquires the target image to be detected for illumination, as well as the spectral data of each sub-region of the target image collected by multiple spectral sensors. Then, the illumination information of each sub-region is obtained by analyzing the spectral data of each sub-region. On one hand, each spectral sensor acquires spectral data from its corresponding sub-region of the target image, improving the accuracy of spectral data acquisition. Obtaining the illumination information of each sub-region based on its spectral data strongly suppresses interference caused by one or more large areas of color in the target image, thus improving the accuracy of illumination detection. On the other hand, multiple spectral sensors simultaneously act on the target image, acquiring spectral data in parallel, reducing the detection process time, increasing the detection speed, and thereby improving the efficiency of illumination detection for the image.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0014] Figure 1 This shows the distribution of illumination information in the target image obtained using a clustering algorithm;

[0015] Figure 2 This shows the clustering results of illumination detection using a clustering algorithm when multiple colors have a high proportion in the target image;

[0016] Figure 3 This illustrates the system architecture of the operating environment for this exemplary embodiment;

[0017] Figure 4 This illustration shows a flowchart of a method for detecting illumination in an image according to an exemplary embodiment of the present invention;

[0018] Figure 5 This diagram illustrates the detection area in this exemplary embodiment.

[0019] Figure 6 This diagram illustrates the process of acquiring illumination information for each sub-region in this exemplary embodiment.

[0020] Figure 7 This diagram illustrates the structure of a light source classification model in this exemplary embodiment.

[0021] Figure 8 This diagram illustrates the process of extracting feature data for each detection region in this exemplary embodiment.

[0022] Figure 9 This illustration shows a schematic diagram of the process of determining the light source of the target image based on illumination information in this exemplary embodiment;

[0023] Figure 10 This example shows a histogram of color temperature and a histogram of color deviation values ​​in this exemplary embodiment;

[0024] Figure 11 This exemplary embodiment illustrates a flowchart for target image illumination detection;

[0025] Figure 12 This diagram illustrates the structure of a light detection device for an image according to this exemplary embodiment.

[0026] Figure 13 This diagram illustrates the structure of a camera module according to an exemplary embodiment.

[0027] Figure 14A A schematic diagram of a traditional Bayer array arrangement is shown;

[0028] Figure 14B A schematic diagram showing the arrangement of a four-Bayer array is provided.

[0029] Figure 15 This diagram illustrates a set of spectral spectrometers and a set of spectral sensors in this exemplary embodiment;

[0030] Figure 16 This diagram illustrates the arrangement of the spectral sensors in this exemplary embodiment.

[0031] Figure 17 This diagram illustrates the light source distribution and window division in this exemplary embodiment.

[0032] Figure 18A The image shown is the result of global color correction to the reference image;

[0033] Figure 18B The image shown is the result of local color correction processing on the reference image;

[0034] Figure 19 This diagram illustrates a 3×3 extraction template in this exemplary embodiment;

[0035] Figure 20 This diagram illustrates the structure of another camera module in this exemplary embodiment.

[0036] Figure 21 A schematic diagram of an electronic device according to this exemplary embodiment is shown;

[0037] Figure 22 A schematic diagram of a mobile terminal is shown in this exemplary embodiment. Detailed Implementation

[0038] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0039] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0040] In related technologies, pixels in an image are typically clustered according to the RGB color space, and the distribution of illumination information in the image is determined based on the clustering results. Commonly used clustering algorithms include the k-means clustering algorithm. For example, the result of clustering illumination information in an image using the k-means clustering algorithm might look like this: Figure 1 As shown, Figure 1 The horizontal and vertical axes can represent the R / G and B / G color gamuts, respectively.

[0041] However, when one or more colors occupy a large area in an image, clustering algorithms for detecting illumination information can easily be misled by the large areas of color, leading to detection failure. For example... Figure 2 As shown, Figure 2 (a) can be an image containing only one type of lighting information, but including large areas of plants and large areas of ground of the same color. Figure 2 (b) shows the clustering results of the images, derived from... Figure 2 (b) It can be seen that the relevant technology uses clustering algorithms to identify plants and the ground as two types of light information. Therefore, Figure 2 In (a), the colors of the large areas of plants and the ground misled the clustering algorithm, causing the illumination information detection to fail and reducing the accuracy of illumination information detection.

[0042] Furthermore, when similar clustering algorithms are used to detect illumination in images, related technologies typically require at least two rounds of clustering to obtain relatively accurate clustering results. Therefore, this technology is time-consuming and has low detection efficiency.

[0043] In view of one or more of the above-mentioned problems, this disclosure first provides an exemplary embodiment of a method for detecting illumination in an image. The following is in conjunction with... Figure 3 The system architecture of the operating environment for this exemplary embodiment will be described.

[0044] refer to Figure 3 As shown, the system architecture 300 may include a terminal device 310 and a server 320. The terminal device 310 may be an electronic device with photographic capabilities, such as a smartphone, tablet, or camera. The terminal device 310 may include an image sensor and multiple spectral sensors, which may be integrated into the camera module of the terminal device 310 for acquiring target images and spectral data. The server 320 generally refers to the backend system providing illumination detection-related services for images in this exemplary embodiment, such as a server implementing an illumination detection method. The server 320 may be a single server or a cluster of multiple servers; this disclosure does not limit this. The terminal device 310 and the server 320 can be connected via a wired or wireless communication link for data interaction.

[0045] In one embodiment, the terminal device 310 can first acquire the target image collected by the image sensor, and the spectral data of multiple detection areas collected by multiple spectral sensors, wherein each spectral sensor acquires the spectral data of a detection area, and each detection area corresponds to a sub-region in the target image; then the spectral data of each detection area is analyzed to obtain the illumination information of each sub-region in the target image.

[0046] In one embodiment, the terminal device 310 acquires the target image collected by the image sensor and the spectral data of multiple detection areas collected by multiple spectral sensors, and then sends the spectral data of the multiple detection areas to the server 320. After receiving the spectral data of the multiple detection areas sent by the terminal device 310, the server 320 parses the spectral data of each detection area to obtain the illumination information of each sub-region in the target image.

[0047] As can be seen from the above, the illumination detection method for images in this exemplary embodiment can be executed by the terminal device 310 or the server 320 described above.

[0048] The following is combined Figure 4 The illumination detection method for images is described. Figure 4 An exemplary flow diagram for a method of light detection for an image is shown, including the following steps S410 to S420:

[0049] Step S410: Acquire the target image captured by the image sensor and the spectral data of multiple detection areas captured by multiple spectral sensors; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the target image;

[0050] Step S420: Analyze the spectral data of each detection area to obtain the illumination information of each sub-region in the target image.

[0051] Based on the above method, on the one hand, each spectral sensor acquires spectral data from the corresponding sub-region of the target image, improving the acquisition accuracy of spectral data; and by obtaining the illumination information of each sub-region based on the spectral data of each sub-region in the target image, it has a strong suppression effect on interference caused by one or more large-area colors in the target image, thus improving the illumination detection accuracy of the image. On the other hand, multiple spectral sensors act on the target image simultaneously, acquiring spectral data in parallel, reducing the detection process time, increasing the detection speed, and thus improving the efficiency of illumination detection of the image.

[0052] The following is about Figure 4 Each step in the process will be explained in detail.

[0053] refer to Figure 4 In step S410, the target image acquired by the image sensor and the spectral data of multiple detection areas acquired by multiple spectral sensors are obtained; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the target image.

[0054] The image sensor is used to acquire an image of the target to be detected. In one embodiment, the image sensor may be an imaging sensor with three color channels (R / G / B) used in a mobile phone or other imaging device. This type of sensor has a high resolution; for example, the resolution of such an image sensor may be 4000*3000 or 2000*1500, etc. This image sensor can cover an imaging area of ​​380–780 nm (close to the wavelength range perceptible to the human eye) in the visible light band. This disclosure does not specifically limit the resolution of the image sensor or other parameters such as the wavelength range it can cover.

[0055] The target image can be an image that needs to be detected by light, and may include one or more light sources. The target image can be an image already stored in the terminal device 310, or it can be an image re-captured by the imaging device. This disclosure does not specifically limit the source of the target image. For example, the target image can be an image stored in the mobile phone's photo album.

[0056] A spectral sensor can be a sensor used to acquire spectral data of a target image. The system structure of such a spectral sensor may include an optical section and a control / display section. The optical section may include imaging optical elements and a spectroscopic element that uses a beam splitter to divide the spectrum. When dividing the spectrum, detectors are typically required; therefore, a spectral sensor usually includes multiple detectors to detect multiple spectra in the image. Each spectrum in the image may correspond to a channel in the image. This disclosure does not specifically limit the spectral bands that the spectral sensor can detect. For example, the spectral sensor may include one pixel and seven detectors. The total wavelength range that the seven detectors can cover may be 350–1000 nm. These seven detectors can detect seven spectra with dominant wavelengths of 430 nm, 450 nm, 500 nm, 530 nm, 575 nm, 650 nm, and 850 nm, and a half-width of at least 20 nm. Therefore, the spectral sensor can perform spectroscopic processing based on a pixel in the target image to obtain the spectral data of the seven channels of that pixel.

[0057] In one embodiment, multiple spectral sensors can be combined to achieve illumination detection of the entire target image. This disclosure does not impose any particular limitation on the number or arrangement of the multiple spectral sensors; for example, multiple spectral sensors can be arranged in a six-row, eight-column configuration to perform illumination detection of the entire target image. In one embodiment, the combination of spectral sensors can be determined based on the maximum number of light sources to be detected; variations in the number of spectral sensors primarily affect the number of detectable light sources.

[0058] Spectral data can be spectral distribution data collected by a spectral sensor based on a sub-region of the corresponding target image. This disclosure does not specifically limit the content of the spectral data. In one embodiment, the spectral data may include monospectral data and multispectral data. When generating multispectral data, the spectral sensor may correspond to multiple detectors, each of which can detect the spectrum of different wavelengths in the detection area. These different wavelengths can constitute the multispectral data for that detection area.

[0059] In addition, spectral sensors can also be used in spectral cameras, which can cover a wider range of spectral bands and detect more types of spectra. For example, a spectral camera can cover a band of 300nm to 2000nm and can detect 6 to 40 types of spectra. Its resolution can be lower than that of the image sensors mentioned above, but higher than that of the main screen resolution (QuarterVideo Graphics Array, QVGA).

[0060] The aforementioned detection area is the image region detected by the spectral sensor. In one embodiment, each spectral sensor can acquire spectral data of its corresponding detection area. This disclosure does not impose any special limitations on the shape, size, arrangement, or number of detection areas.

[0061] The sub-regions of the target image mentioned above are a part of the target image. For example, the target image can be divided into 64*48 sub-regions for illumination detection. This disclosure does not specifically limit the shape, size, arrangement, or number of sub-regions. In one embodiment, each detection region corresponds to a sub-region in the target image. For example, 48 spectral sensors can be arranged in a 6*8 pattern, and these 48 6*8 distributed spectral sensors can be approximated as a spectral camera with 6*8 pixels and a 120-degree viewing angle. In this exemplary embodiment, each spectral sensor corresponds to one detection region, such as... Figure 5 As shown, the target image can be divided into 48 sub-regions based on the distribution of the detection area. Each detection area corresponds to a sub-region in the target image. Therefore, each spectral sensor can detect the spectral distribution of a sub-region.

[0062] After obtaining the spectral data for each detection area, the spectral data can be analyzed and further referenced. Figure 4 In step S420, the spectral data of each detection area is parsed to obtain the illumination information of each sub-region in the target image.

[0063] The illumination information can be the light source information of each sub-region obtained by analyzing the spectral data of each sub-region. This disclosure does not make any special limitation on the specific content of the illumination information. For example, the illumination information may include the light source type of each sub-region, and may also include the color temperature, color deviation value and other index values ​​of the light source of each sub-region.

[0064] In one implementation, the aforementioned illumination information includes light source classification results; such as... Figure 6 As shown, the above-described method of parsing the spectral data of each detection region to obtain the illumination information of each sub-region in the target image may include the following steps S610 to S620:

[0065] Step S610: Extract feature data for each detection area based on the spectral data of each detection area.

[0066] Step S620: The feature data of each detection region is processed using a pre-trained light source classification model to obtain the light source classification result of each sub-region in the target image.

[0067] The feature data for each detection region can be obtained from the spectral data of each detection region. This disclosure does not specifically limit the content of the feature data; in one embodiment, the feature data can be obtained based on the derivatives of the spectral data of adjacent bands.

[0068] The light source classification model can be a model that processes the feature data of each detection region to obtain the light source classification result for each sub-region in the target image. This disclosure does not specifically limit the specific structure of the light source classification model. In one embodiment, such as... Figure 7 As shown, the light source classification model may include multiple hidden layers, multiple batch normalization (BN) layers, multiple activation function layers, and a classification layer. In this exemplary embodiment, feature data can be input into multiple levels of hidden layers, batch normalization layers, and activation function layers for processing, and then the processed intermediate data can be sent to the classification layer to obtain the category of the light source in the current sub-region of the target image.

[0069] In one implementation, the hidden layer of the light source classification model can abstract the input feature data to another dimension space. Therefore, the number of nodes in the hidden layer can be adjusted according to the input feature data. The Batch Normalization (BN) layer can prevent overfitting during the light source classification process based on the feature data. Therefore, the BN layer can also be replaced by other neural networks with the same function, such as the Drop-out layer. The activation function layer can map the input of the light source classification model to the output. Therefore, other activation functions such as ReLU and ReLU6 can be used in the model to implement the mapping process.

[0070] In one implementation, indoor light sources with a color temperature range of 8000–2600K that can be collected, and outdoor light sources with a color temperature range of 6000–3800K collected at different times such as sunny days, cloudy days, noon, and sunset, can be selected to train the above light source classification model.

[0071] In one implementation, the light source classification model can be trained based on the light source information of each sub-region of the target image and the aforementioned indoor and outdoor light sources to obtain a light source classification model for each detection region. That is, the structure and feature data used in the light source classification model for each detection region can be different, thereby obtaining a more accurate light source classification result for each sub-region of the target image.

[0072] In one implementation, the light source classification model corresponding to each detection region can be executed in parallel, thereby improving the acquisition speed of light source classification results for each sub-region in the target image and thus improving the light detection efficiency.

[0073] In another implementation, a light source classification model can be trained based on the above-mentioned indoor and outdoor light sources, and then the light source classification model can be applied to each detection area. That is, the key classification model structure used in each detection area is the same, so as to obtain the light source classification result of each sub-region of the target image.

[0074] The aforementioned light source classification result can be the light source category information of each detection area obtained by the light source classification model of that sub-area based on the feature data of that sub-area. For example, the light source classification result can include one or more of the aforementioned indoor light sources or outdoor light sources. In one embodiment, the light source classification result can be a set of color index values ​​such as color temperature value and color deviation value corresponding to a certain light source. This disclosure does not specifically limit the specific content of the light source classification result.

[0075] In step S610, feature data for each detection region can be extracted based on the spectral data of each detection region. In one embodiment, such as... Figure 8 As shown, the above-described extraction of feature data for each detection region based on the spectral data of each detection region may include the following steps S810 to S820:

[0076] Step S810: Preprocess the spectral data and fit the reflectance spectrum to obtain the fitted data;

[0077] Step S820: Extract feature data for each detection region from the fitted data.

[0078] The fitted data can be obtained by fitting the reflectance spectrum of the preprocessed data. This disclosure does not specifically limit the method or content of obtaining the fitted data. In one embodiment, after obtaining the spectral data, the discrete spectral data can be expanded into smooth continuous spectral data to obtain the fitted data.

[0079] In step S810, the spectral data can be preprocessed and the reflectance spectrum can be fitted to obtain fitted data. In one embodiment, the spectral data may include multi-channel response data; the preprocessing of the spectral data may include channel pre-correction of the multi-channel response data.

[0080] The multi-channel response data can be the spectral information of different bands of the detection area output by the spectral sensor after receiving the image data of the detection area. This disclosure does not make any special limitations on the acquisition method and specific content of the multi-channel response data.

[0081] In one exemplary embodiment, multi-channel response data can be corrected using a spectral calibrator. For example, the spectral parameters of the multi-channel response data can be analyzed first to determine whether peak shifts occur between the multi-channel response data. Then, a peak shift correction coefficient can be obtained based on the peak shift. Finally, a parameter corrector can be used to correct the multi-channel response data based on the peak shift correction coefficient to ensure the consistency of the data output by the multi-channel sensor.

[0082] In one embodiment, after performing channel pre-correction on the multi-channel response data, reflectance spectrum fitting can be performed on the corrected data to expand the discrete multi-channel response data into continuous fitted data. This disclosure does not specifically limit the specific implementation method of reflectance spectrum fitting.

[0083] After obtaining the fitted data, feature data for each detection region can be extracted from the fitted data in step S820.

[0084] In one embodiment, the fitting data can be obtained by fitting the reflectance spectrum of the aforementioned multi-channel response data. Therefore, in this exemplary embodiment, the first reciprocal of the response data of adjacent channels can be extracted as the feature data of each detection region. This disclosure does not impose any special limitations on the method of feature data extraction.

[0085] After obtaining the feature data through the above steps, continue to refer to... Figure 6 In step S620, the feature data of each detection area can be processed using a pre-trained light source classification model to obtain the light source classification result of each sub-region in the target image.

[0086] In one implementation, feature data can be input into the aforementioned light source classification model. The light source classification model can perform data dimension mapping, data fitting, and data transformation using activation functions on the feature data. The transformed feature data is then input into the classification model for light source classification. In this exemplary embodiment, hidden layers can be used for data dimension mapping, Batch Normalization (BN) layers for data fitting, and ReLU activation functions for data transformation. Hidden layers, BN layers, and ReLU activation functions can be combined into a processing unit. The light source classification model can include multiple processing units. When the light source classification model receives feature data, it can input the feature data into the processing unit for processing. The output of the previous processing unit can be used as the input of the next processing unit. The data processed by multiple levels of processing units is then input into the classification model for classification to obtain the light source classification result for the current sub-region in the target image.

[0087] Based on the above method, the spectral data of each sub-region in the target image is analyzed in parallel by the light source classification model to obtain the light source classification result of each sub-region in the target image. Since the light source classification model corresponding to each detection region has been adaptively adjusted according to the feature data of each sub-region, the network structure is relatively simple, so the model running time is short. Moreover, the light source classification model of each detection region can be executed in parallel, thereby improving the accuracy of illumination information detection, increasing the detection speed, and thus improving the efficiency of illumination information detection.

[0088] In one embodiment, the illumination information includes light source classification results; the illumination detection method for images may further include the following steps S910 to S920:

[0089] Step S910: Convert the light source classification result of each sub-region into the color index value of each sub-region;

[0090] Step S920: Calculate the color index values ​​of each sub-region, and determine the light source in the target image based on the statistical results of the color index values.

[0091] The color index value can be one or more of the index values ​​corresponding to a certain light source. For example, the color index value can be the color temperature (CCT) and color deviation value (DUV) corresponding to a certain light source.

[0092] In step S910, the light source classification result for each sub-region can be converted into a color index value for each sub-region. In one embodiment, the conversion of the light source classification result into its corresponding color index value can be achieved through a light source information database. In this exemplary embodiment, a light source information database can be pre-established to store the classification result of each type of light source and its corresponding color index value. When converting the light source classification result into a color index value, the light source information database can be searched according to the light source classification result to obtain the color index value corresponding to that type of light source, thereby realizing the conversion of the light source classification result. This disclosure does not impose any special limitations on the conversion method of light source classification result into color index value. For example, the existing light source classification model can be improved so that the light source classification model directly outputs the color index value of each type of light source.

[0093] After obtaining the color index value of each sub-region in the target image, in step S920, the color index value of each sub-region can be statistically analyzed, and the light source in the target image can be determined based on the statistical results of the color index values ​​of all sub-regions.

[0094] In one implementation, the aforementioned color index values ​​may include color deviation values ​​and color temperature; the aforementioned statistical analysis of color index values ​​for each sub-region and determination of light sources in the target image based on the statistical results of the color index values ​​may include statistical analysis of histograms of color deviation values ​​and histograms of color temperature for each sub-region, and determination of n light sources in the target image based on the histograms of color deviation values ​​and histograms of color temperature; n is a positive integer not less than 2.

[0095] In one implementation, the histogram can be determined based on the proportion of color deviation values ​​and color temperature values ​​in all sub-regions of the target image. In this exemplary implementation, as... Figure 10 As shown, the horizontal axis of the histogram represents the range of color temperature and color deviation values, while the vertical axis can be the proportion of the number of sub-regions corresponding to a certain color temperature or color deviation value out of the total number of sub-regions. For example, if the target image has 48 sub-regions, and 6 of them have a color temperature of 2000K, the proportion corresponding to the color temperature value of 2000K is 6 / 48 = 0.125. This disclosure does not impose any special limitations on the horizontal and vertical axes of the histogram.

[0096] After obtaining the histograms of color deviation values ​​and color temperature, in one embodiment, the n color temperature values ​​and n color deviation values ​​with the highest proportions can be selected from each of the two histograms. These n color temperature values ​​and color deviation values ​​are then combined to determine n sets of color index values ​​corresponding to the light source of the target image. Based on these n sets of color index values, the corresponding n light sources are then obtained to determine the light source of the target image. In this exemplary embodiment, the n light sources can be determined based on the n sets of color index values ​​using the aforementioned light source information database.

[0097] In one implementation, after obtaining the color index value corresponding to the light source classification result of each sub-region of the target image, a mean filter can be used to process the color index value of each sub-region to determine the light source of the target image. In this exemplary embodiment, a mean filter can be used to filter the color index values ​​corresponding to the light source classification result. If the color index values ​​of adjacent sub-regions are very similar, they are determined to be the same color index value, corresponding to the same light source. By using a mean filter to process the light source classification result of each sub-region, it is not necessary to limit the maximum number of light sources detected in the target image. This disclosure does not impose a special limitation on the size of the mean filter described above. For example, the mean filter can be 3*3, 2*2, or other sizes.

[0098] In one embodiment, the above-described illumination detection method for images may further include performing color correction processing on the target image based on the illumination information of each sub-region in the target image. In this exemplary embodiment, color deviations in the target image caused by various illumination information can be corrected based on the illumination information of each sub-region, so that the display effect of the target image on the terminal is consistent with the visual effect of the human eye, thereby improving the user experience.

[0099] Based on the above method, spectral data of each sub-region in the target image is obtained by a spectral sensor. The spectral data of each sub-region is analyzed using a light source classification model for each detection region to obtain the light source classification result for each sub-region. Then, the color index values ​​corresponding to the light source classification results of all sub-regions in the target image are statistically analyzed to determine the light source in the target image. This avoids interference from large areas of confusing colors in the target image on the light source detection, improves the accuracy and speed of light source detection, and thus improves the efficiency of light source detection.

[0100] In one embodiment, an exemplary process of the illumination information detection method for images disclosed herein is as follows: Figure 11 As shown, illumination information detection of an image can be achieved according to steps S1101 to S1109.

[0101] Step S1101: Obtain the target image;

[0102] Step S1102: The detection areas of multiple spectral sensors are mapped one-to-one with the sub-regions of the target image;

[0103] Step S1103: The spectral sensor acquires spectral data of the corresponding detection area;

[0104] Step S1104: Preprocess the spectral data of each detection area and fit the reflectance spectrum;

[0105] Step S1105: Perform feature extraction on the fitted data to obtain feature data for each detection region;

[0106] Step S1106: Process the feature data using the light source classification model to obtain the light source classification result for each sub-region;

[0107] Step S1107: Convert the light source classification results of each sub-region into the color deviation value and color temperature of the sub-region;

[0108] Step S1108: Calculate the histogram of color deviation values ​​and the histogram of color temperature for each sub-region;

[0109] Step S1109: Determine the light source in the target image based on the color temperature and color deviation value that have the highest proportion in the histogram of color deviation values ​​and the histogram of color temperature.

[0110] Exemplary embodiments of this disclosure also provide an illumination detection apparatus for images. For example... Figure 12 As shown, the illumination detection device 1200 for images may include:

[0111] The data acquisition module 1210 is configured to acquire a target image collected by an image sensor and spectral data of multiple detection areas collected by multiple spectral sensors; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the target image;

[0112] The illumination information detection module 1220 is configured to analyze the spectral data of each detection area to obtain the illumination information of each sub-region in the target image.

[0113] In one embodiment, the illumination information includes light source classification results; the process of parsing the spectral data of each detection region to obtain the illumination information of each sub-region in the target image may include:

[0114] Based on the spectral data of each detection area, feature data of each detection area is extracted;

[0115] The feature data of each detection region are processed using a pre-trained light source classification model to obtain the light source classification results for each sub-region in the target image.

[0116] In one implementation, the above-described extraction of feature data for each detection region based on the spectral data of each detection region may include:

[0117] The spectral data are preprocessed and fitted with reflectance spectra to obtain fitted data;

[0118] Extract feature data for each detection region from the fitted data.

[0119] In one embodiment, the spectral data includes multi-channel response data; the preprocessing of the spectral data may include:

[0120] Perform channel pre-calibration on multi-channel response data.

[0121] In one embodiment, the illumination information includes light source classification results; the device may further include:

[0122] The light source classification results for each sub-region are converted into color index values ​​for each sub-region.

[0123] The color index values ​​of each sub-region are statistically analyzed, and the light source in the target image is determined based on the statistical results of the color index values.

[0124] In one embodiment, the aforementioned color index values ​​include color deviation values ​​and color temperature; the above-mentioned statistical analysis of color index values ​​for each sub-region, and determination of the light source in the target image based on the statistical results of the color index values, may include:

[0125] Calculate the histogram of color deviation values ​​and the histogram of color temperature for each sub-region, and determine n light sources in the target image based on the histogram of color deviation values ​​and the histogram of color temperature; n is a positive integer not less than 2.

[0126] In one embodiment, the above-mentioned device further includes:

[0127] Color correction is performed on the target image based on the illumination information of each sub-region in the target image.

[0128] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation, and therefore will not be repeated here.

[0129] An exemplary embodiment of this disclosure provides a camera module. (See reference...) Figure 13 As shown, the camera module 1300 may include: an image filter 1310, an image sensor 1320, a K-group spectral splitter 1330, and a K-group spectral sensor 1340. Each component is described below.

[0130] Image filter 1310 is a filter formed by arranging monochromatic filters in an array. It can be located in the incident light path of image sensor 1320, enabling image sensor 1320 to receive monochromatic light passing through image filter 1310. For example, image filter 1310 can be composed of RGB monochromatic filters, which can filter out light of three different spectral ranges (R, G, and B) (i.e., red, green, and blue monochromatic light). This disclosure does not limit the arrangement of image filter 1310.

[0131] In one implementation, image filter 1310 may be a Bayer filter, such as the reference filter. Figure 14A As shown, Bayer filters use the traditional Bayer array arrangement, or refer to... Figure 14B As shown, Bayer filters are arranged in a quad-bayer array.

[0132] Image sensor 1320 is a sensor that converts light signals into electrical signals, and achieves imaging by quantitatively characterizing the light signals. This disclosure does not limit the specific type of image sensor 1320, such as a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) sensor. In this exemplary embodiment, image sensor 1320 is located in the outgoing light path of image filter 1310, and is used to sense the light signals passing through image filter 1310 to generate a target image of the subject. The subject refers to the scene, object, or person being photographed located directly in front of camera module 1300. The target image is the image obtained by camera module 1300 of the subject, which can be a raw image, such as a RAW image, or an RGB or YUV image processed by ISP (Image Signal Processor).

[0133] Generally, the image sensor 1320 consists of a certain number of photosensitive elements arranged in an array, with each photosensitive element corresponding to one pixel of the target image. The number of photosensitive elements represents the resolution of the image sensor 1320. For example, if the photosensitive elements are arranged in an H×W array, where H represents the number of rows and W represents the number of columns, then the resolution of the image sensor 1320 can be H×W, and the size of the generated target image is also H×W, where H represents the image height and W represents the image width. For example, H is 3000 and W is 4000.

[0134] The spectral splitter 1330 is used to separate light of a specific wavelength from the incident light. It can be located in the incident light path of the spectral sensor 1340, enabling the spectral sensor 1340 to receive the light of the specific wavelength after passing through the spectral splitter 1330, thereby sensing spectral data. In contrast, the image filter 1310 typically only provides red, green, and blue monochromatic light within the visible light range, while the spectral splitter 1330 can provide a wider variety of different wavelengths of light over a larger spectral range (e.g., 350–1000 nm, covering the ultraviolet to infrared bands). The number of light types split by the spectral splitter 1330 is referred to as the number of channels of the spectral splitter 1330 or the spectral sensor 1340.

[0135] The spectral splitter 1330 can be an optical device such as a filter or a pyramid. Taking a filter as an example, in one embodiment, each set of spectral splitters 1330 can include L filters with different peak wavelengths (or center wavelengths), such that the incident light is separated into L different wavelength bands after passing through the spectral splitter 1330, and the number of channels of the spectral splitter 1330 is L. If L is 1, that is, the number of channels of the spectral splitter is 1, then the spectral splitter is a single-spectrum splitter; if L is a positive integer not less than 2, that is, the number of channels of the spectral splitter is greater than or equal to 2, then the spectral splitter is a multispectral splitter. For example, L can be 13.

[0136] In one implementation, the L filters in each group of spectral splitters can be arranged in a p×q array, where p represents the number of rows and q represents the number of columns, and L = p×q. Figure 15 A schematic diagram of a spectral splitter 1330 is shown, which has 3×4 channels. The spectral splitter includes 12 filters arranged in a 3×4 array, denoted as C1 to C12, indicating that they are used to filter the light in channels C1 to C12. The peak wavelength and full width at half maximum (FWHM) of the light in each channel can be referred to Table 1, which covers 12 important wavelength bands in the range of 350 to 1000 nm.

[0137] Table 1

[0138]

[0139]

[0140] The spectral sensor 1340 can be located in the output light path of the spectral beam splitter 1330 to sense the light signal passing through the spectral beam splitter 1330 and obtain spectral data of a specific detection area in the object being photographed. This disclosure does not limit the specific type of the spectral sensor 1340; it can be a CMOS or CCD sensor, and its type can be the same as or different from that of the image sensor 1320.

[0141] In one embodiment, each set of spectral sensors 1340 may include L photosensitive elements, each used to sense the light signal filtered by the corresponding L filters in the spectral splitter 1330, to obtain response data for L channels, i.e., spectral data for different L bands. If L is 1, i.e., the number of channels of the spectral sensor is 1, then the spectral sensor is a monospectral sensor; if L is a positive integer not less than 2, i.e., the number of channels of the spectral sensor is greater than or equal to 2, then the spectral sensor is a multispectral sensor.

[0142] In one implementation, the L photosensitive elements in each group of spectral sensors can be arranged in a p×q array, where p represents the number of rows and q represents the number of columns, and L = p×q. For example, refer to... Figure 15 As shown, the spectral sensor 1340 may include 3×4 photosensitive elements, denoted as Z1 to Z12, which correspond one-to-one with the filters C1 to C12 of the spectral spectrometer 1330, respectively receiving optical signals from 12 channels and obtaining response data from 12 channels.

[0143] It should be noted that the spectrometer 1330 and the spectrometer sensor 1340 can be considered as a single unit, referred to as the spectrometer sensor. For the sake of consistency, this article will still refer to the spectrometer 1330 and the spectrometer sensor 1340 as two separate components.

[0144] The camera module 1300 may also include a bracket 1350 for fixing one or more of the Bayer filter 1310, image sensor 1320, spectral spectrometer 1330, and spectral sensor 1340.

[0145] In this exemplary embodiment, K sets of spectral splitters 1330 and K sets of spectral sensors 1340 can be provided in the camera module 1300, where K is a positive integer not less than 2, that is, the camera module 1300 includes at least two sets of spectral splitters 1330 and at least two sets of spectral sensors 1340. For example, K can be 12 or 48.

[0146] Each set of spectral splitters 1330 corresponds to a set of spectral sensors 1340 and a detection area. The detection area refers to a localized region within the object being photographed. During the shooting process, light reflected from each detection area enters the camera module 1300, is split by its corresponding set of spectral splitters 1330, and finally enters its corresponding set of spectral sensors 1340. The spectral sensors 1340 sense the light signal reflected from the detection area after being split by the spectral splitters 1330, thus obtaining the spectral data of that detection area. The spectral splitters 1330 and spectral sensors 1340 can be configured correspondingly. For example, the positions of the spectral splitters 1330 and spectral sensors 1340 can be arranged one-to-one along the optical axis, so that each set of spectral sensors 1340 receives the light transmitted by its corresponding set of spectral splitters 1330 (in some cases, each set of spectral sensors 1340 may also receive a small amount of light transmitted by adjacent spectral splitters 1330, which can be eliminated through algorithms).

[0147] Each detection region can correspond to a sub-region in the target image. For example, by using K sets of spectral sensors 1340 to detect the spectral data of K detection regions, these K detection regions are equivalent to dividing the object into K blocks. Similarly, the image sensor 1320 captures the target image of the same object, and this target image can also be divided into K sub-regions, with each detection region corresponding to one sub-region.

[0148] Therefore, by detecting the spectral data of K detection areas, the spectral data of K sub-regions in the target image are actually obtained. On the one hand, the camera module in this exemplary embodiment can provide spectral data based on the conventional capture of the target image, which is beneficial to improving image quality or broadening the application range of the image. For example, based on the spectral data, the lighting conditions of the image shooting scene can be characterized more fully, thereby enabling accurate color correction and other optimization processing of the image; for another example, based on the RGB data and spectral data of the target image, more accurate target detection and industrial monitoring can be achieved, which can be applied to fields such as surveying and mapping and industrial inspection. On the other hand, the camera module is equipped with multiple sets of spectral splitters and multiple sets of spectral sensors to refine the localized spectral data to the detection sub-regions, which is beneficial to perform more refined optimization processing of the target image. For example, different lighting conditions in different sub-regions of the target image can be characterized separately, and different processing can be applied to different sub-regions, or more refined information can be provided. For example, based on the spectral data of different sub-regions in the target image, temperature information or defect information in different parts of the shooting scene can be monitored.

[0149] K can be regarded as the resolution of the spectral sensor 1340. Each group of spectral sensors 1340 outputs spectral data of a detection area, which can be regarded as a pixel of the spectral data. K groups of spectral sensors 1340 can output spectral data with a pixel of K. That is to say, a pixel of the spectral data corresponds to a sub-area of the target image. In this exemplary embodiment, K < H×W, that is, the resolution of the spectral sensor 1340 is lower than that of the image sensor 1320. Among them, the image sensor 1320 is used for imaging, and a high-resolution image sensor 1320 can generate a high-definition target image; while the spectral sensor 1340 is used for detecting spectral data. The detection of spectral data does not need to be refined to the extent of image pixels. By detecting the spectral data of K sub-areas in the target image by K groups of spectral sensors 1340, the differences between different local parts in the target image are characterized, which can already meet the usage requirements. Therefore, the camera module 1300 in this exemplary embodiment can combine the characteristics of high-definition imaging by the image sensor 1320 and spectral information detection by the spectral sensor 1340, and can obtain both a high-definition target image and relatively rich and detailed spectral information. Moreover, increasing the resolution of the spectral sensor 1340 will lead to an increase in the manufacturing cost of the camera module 1300. Controlling the resolution of the spectral sensor 1340 to be lower than (generally much lower than) the resolution level of the image sensor 1320 is beneficial to cost control.

[0150] In one embodiment, K groups of spectral sensors 1340 can be arranged in an m×n array, where m represents the number of rows and n represents the number of columns, then K = m×n. It should be understood that K groups of spectral beam splitters 1330 can also be arranged in an m×n array. Figure 16 The schematic diagram of the arrangement array of the spectral sensor 1340 is shown, where m = 6 and n = 8, that is, the camera module 1300 includes 48 groups of spectral beam splitters 1330 and 48 groups of spectral sensors 1340. Each group of spectral sensors 1340 can further include 3×4 photosensitive elements to output response data of 12 channels.

[0151] In one embodiment, H / W = m / n, that is, the ratio of the image sensor 1320 to the m×n groups of spectral sensors 1340 is the same. For example, the resolution of the image sensor 1320 is 3000*4000, and 48 groups of spectral sensors 1340 are arranged in a 6×8 array, and the aspect ratios of both are the same. This makes it easier for the m×n detection areas to correspond to the sub-areas in the target image. For example, each group of spectral sensors 1340 corresponds to a detection area, and this detection area corresponds to a sub-area of 500*500 in the target image, and 6×8 detection areas respectively correspond to 6×8 sub-areas in the target image.

[0152] In one implementation, if spectral data is used to detect illumination in the subject, it is generally rare for the subject to have more than five light sources. When capturing the target image, refer to... Figure 17 As shown, the most even distribution of the five light sources in the image is achieved when they are located at the four corners and the center of the image. This allows for the complete separation of areas with different lighting conditions using 3×3=9 windows. In other cases, where two or more of the five light sources are more concentrated, fewer than nine windows are needed to completely separate areas with different lighting conditions. Therefore, when setting up the spectral sensor 1340, considering the need to detect lighting conditions, m and n can be set to positive integers not less than 3. This allows the target image to be divided into at least nine sub-regions. By detecting the spectral data of each sub-region, the lighting conditions of each sub-region can be determined, thus fully detecting the different lighting conditions in different parts of the subject and meeting practical usage requirements.

[0153] Generally, the higher the resolution of the 1340 spectral sensor (i.e., the larger the values ​​of m and n), the more refined the detection of local spectral data. However, considering the impact on image processing, a higher resolution for the 1340 spectral sensor is not always better. Acquiring a reference image with large areas of confusing colors under a single light source, such as in an indoor scene where large areas of solid color on walls and ceilings can interfere with light source detection (considered confusing colors), is helpful. Dividing the reference image into image blocks, such as 6×8 blocks, is also beneficial. Each image block has a smaller FOV (Field of Vision), making it easier to focus on large areas of confusing colors. Figure 18A The image shown is the result of detecting global spectral data from a reference image and performing global color correction. Figure 18B The image shown is the result of detecting local spectral data (i.e., detecting spectral data for each image patch individually) in a reference image and then performing local color correction. By comparison, Figure 18B The image showed some color anomalies because the image patch included a large area of ​​yellow wall material, which was mistakenly identified as a light source, resulting in a noticeably cool color during local color correction. To reduce the risk of image processing errors caused by illumination analysis mistakes, the resolution of the spectral sensor 1340 can be controlled to not exceed 6×8.

[0154] In addition, refer to Figure 19As shown, when performing operations such as feature extraction on the target image, a 3×3 extraction template (i.e., 3×3 sub-regions) may be used. This facilitates operations such as filtering, averaging, and edge extraction on the RGB or spectral data within the 3×3 range, which is beneficial for achieving refined illumination analysis. To facilitate the use of the 3×3 extraction template, the resolution of the spectral sensor 1340 can be set to 6×8, meaning the camera module 1300 can include 48 sets of spectral sensors 1340 arranged in a 6×8 array.

[0155] In one embodiment, each set of spectral splitters 1330 includes L filters with different peak wavelengths; each set of spectral sensors 1340 includes L photosensitive elements, each used to sense the light signal filtered by the corresponding L filters; the spectral data of each detection region includes response data from L channels. Wherein, L is a positive integer not less than 2, meaning the number of channels in the spectral splitter 1330 or spectral sensor 1340 is at least 2. Referring to the above... Figure 15 As shown, L filters can be arranged in a p×q array, and L photosensitive elements can also be arranged in a p×q array, where p represents the number of rows and q represents the number of columns.

[0156] In one implementation, considering that the light intensity of a general light source gradually decreases in a circular diffusion manner, p = q can be set, that is, each group of spectral sensors 1340 arranges its photosensitive elements in a square array to conform to the circular diffusion law of light intensity, so as to more accurately detect the spectral data of each detection area (or sub-region).

[0157] In one implementation, the number of channels of the spectral sensor 1340 can be determined according to the application scenario of the camera module 1300. For example, in the field of remote sensing, the number of channels of the spectral sensor 1340 can be set to 40 (e.g., 4×5), covering a spectral range of 350 to 2000 nm.

[0158] In one implementation, reference Figure 20 As shown, the camera module 1300 may further include a first lens 1360 and a second lens 1370. The first lens 1360 is disposed in the incident light path of the Bayer filter 1310. Light passing through the first lens 1360 is incident on the Bayer filter 1310 and then on the image sensor 1320. The second lens 1370 is disposed in the incident light path of the spectral splitter 1330. Light passing through the second lens 1370 is incident on the spectral splitter 1330 and then on the spectral sensor 1340. Thus, by using two lenses, the total amount of light entering the camera is increased. Furthermore, the Bayer filter 1310 and the spectral splitter 1330 use different lenses, which can reduce interference between image imaging and spectral data detection, thereby improving image quality and the accuracy of spectral data.

[0159] In one embodiment, the camera module 1300 may include a first camera and a second camera. The first camera is for imaging and includes a first lens 1360, a Bayer filter 1310, and an image sensor 1320. The second camera is for detecting spectral data and includes a second lens 1370, a K-group spectral splitter 1330, and a K-group spectral sensor 1340. In practical use, one of the cameras can be activated independently according to specific needs, thereby further expanding the application scenarios of the camera module 1300.

[0160] In one implementation, the first and second cameras can be calibrated to obtain calibration parameters between them. These calibration parameters can be used to determine the correspondence between each set of spectral sensors 1340 and a sub-region of the image. When the camera module is actually shooting, the spectral data of each set of spectral sensors 1340 can be associated with a sub-region of the target image, which is beneficial for achieving faster and higher-quality image optimization processing or detection.

[0161] Exemplary embodiments of this disclosure also provide an electronic device. (See reference...) Figure 21 As shown, the electronic device 2100 may include: a housing 2110, a main circuit board 2120, and a camera module 2130. The main circuit board 2120 is located within the housing 2110. The camera module 2130 can be any type of camera module in this exemplary embodiment, such as the camera module 1300 described above. The camera module 2130 is electrically connected to the main circuit board 2120 to enable signal or data transmission between the camera module 2130 and the main circuit board 2120. For example, the camera module 2130 may be disposed on the main circuit board 2120, or connected to the main circuit board 2120 via a medium such as a flexible circuit board to form an electrical connection.

[0162] For example, electronic device 2100 can be a mobile phone, camera, tablet computer, wearable device, drone, or other device requiring shooting capabilities. Within camera module 2130, the area of ​​each spectral sensor can be within 1cm × 1cm, thus keeping the overall size of camera module 2130 compact and integrated into electronic device 2100 without occupying excessive space. Integrating camera module 2130 can improve the image processing performance of electronic device 2100 or enable it to perform more image and spectral detection-based functions, such as surveying and industrial monitoring.

[0163] In one embodiment, the electronic device 2100 may further include a main processor and an image signal processor (ISP) (not shown in the figure). Both the main processor and the ISP are electrically connected to the main circuit board 2120. Exemplarily, the main processor may be located on the main circuit board 2120 and may be a CPU (Central Processing Unit) or an AP (Application Processor), etc. The ISP may be located on the main circuit board 2120 as a device independent of the camera module 2130, or it may be located within the camera module 2130 as a device integrated within the camera module 2130. The target image generated by the image sensor of the camera module 2130 may be a RAW image, which is input to the ISP for processing. Spectral data detected by the spectral sensor of the camera module can be input to the main processor. For example, spectral sensors can be connected to the main processor via I2C (Inter-Integrated Circuit) or SPI (Serial Peripheral Interface). The amount of spectral data is typically much smaller than that of image data, making it suitable for transmission via I2C and SPI protocols. This allows the spectral data processing flow to bypass the ISP (Internet Service Provider Interface), eliminating the need to modify the ISP to handle spectral data. Furthermore, spectral data processing does not require the use of ISP interfaces (such as MIPI (Mobile Industry Processor Interface)), thereby reducing the manufacturing cost of the electronic device 2100.

[0164] The following is for reference. Figure 22 The following description uses a mobile terminal as an example to illustrate an electronic device. It should be understood that... Figure 22 The electronic device 2200 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0165] refer to Figure 22 As shown, the electronic device 2200 may include: a processor 2201, a memory 2202, a mobile communication module 2204, a wireless communication module 2205, a display screen 2206, a camera module 2207, an audio module 2208, a power supply module 2209, and a sensor module 2210.

[0166] Processor 2201 may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a display processing unit (DPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The illumination detection method for images in this exemplary embodiment can be executed by the CPU. In one embodiment, sensor module 2210 acquires a target image and collects spectral data from multiple detection regions based on multiple spectral sensors. Each spectral sensor corresponds to a detection region, and each detection region corresponds to a sub-region in the target image. After receiving the spectral data from each detection region, the CPU analyzes the spectral data of each detection region to obtain the illumination information of each sub-region in the target image. After obtaining the illumination information of each sub-region in the target image, the light source of the target image can be determined based on the statistical results of the illumination information of each sub-region.

[0167] The memory 2202 can be used to store computer executable program code, which includes instructions. The processor 2201 executes various functional applications and data processing of the electronic device 2200 by running the instructions stored in the memory 2202. The memory 2202 can also store application data and various intermediate data, such as images, videos, and the aforementioned spectral data.

[0168] The communication function of electronic device 2200 can be implemented through mobile communication module 2204, antenna 1, wireless communication module 2205, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 2204 can provide 3G, 4G, 5G and other mobile communication solutions for electronic device 2200. Wireless communication module 2205 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for electronic device 2200.

[0169] The display screen 2206 is used to implement display functions, such as displaying user interfaces, images, videos, etc. In one embodiment, the display screen 2206 can be used to display the display result after color correction of the target image based on the illumination detection result according to the present disclosure embodiment. The camera module 2207 is used to implement shooting functions, such as capturing images, videos, etc. For example, the camera module 2207 may include an image sensor and the above-mentioned spectral sensor. The image sensor can be used to acquire target images. Combining multiple spectral sensors can more accurately acquire the spectral data of the target image, thereby improving the illumination detection accuracy and detection efficiency of the image. The audio module 2208 is used to implement audio functions, such as playing audio and acquiring voice. The power module 2209 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring the battery status. The sensor module 2210 may include one or more sensors, which are used to acquire status assessments of various aspects of the electronic device 2200.

[0170] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0171] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0172] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A method for detecting illumination in an image, characterized in that, include: Acquire target images captured by image sensors, as well as spectral data of multiple detection areas captured by multiple spectral sensors; Each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the target image; The spectral data of each detection region is analyzed to obtain the illumination information of each sub-region in the target image; The illumination information includes light source classification results; the method further includes: The light source classification result of each sub-region is converted into a color index value for each sub-region; the color index value includes a color deviation value and a color temperature. The histograms of color deviation values ​​and color temperature for each sub-region are statistically analyzed. The histogram of color deviation values ​​represents the ratio of the number of sub-regions corresponding to different color deviation values ​​to the total number of sub-regions. The histogram of color temperature represents the ratio of the number of sub-regions corresponding to different color temperature values ​​to the total number of sub-regions. The n color deviation values ​​with the highest percentages are determined from the color deviation value histogram, and the n color temperature values ​​with the highest percentages are determined from the color temperature histogram. The n color deviation values ​​and the n color temperature values ​​are combined to obtain n sets of color index values. Based on the n sets of color index values, the n light sources in the target image are determined; n is a positive integer not less than 2.

2. The method according to claim 1, characterized in that, The illumination information includes light source classification results; the step of parsing the spectral data of each detection region to obtain the illumination information of each sub-region in the target image includes: Based on the spectral data of each detection region, feature data of each detection region is extracted; The feature data of each detection region are processed using a pre-trained light source classification model to obtain the light source classification result of each sub-region in the target image.

3. The method according to claim 2, characterized in that, The step of extracting feature data for each detection region based on the spectral data of each detection region includes: The spectral data is preprocessed and fitted with reflectance spectra to obtain fitted data; Feature data for each detection region is extracted from the fitted data.

4. The method according to claim 3, characterized in that, The spectral data includes multi-channel response data; the preprocessing of the spectral data includes: Channel pre-calibration is performed on the multi-channel response data.

5. The method according to claim 3, characterized in that, The preprocessing and reflectance spectrum fitting of the spectral data to obtain fitted data includes: Discrete spectral data is expanded into smooth, continuous spectral data to obtain fitted data.

6. The method according to claim 2, characterized in that, The process of processing the feature data of each detection region using a pre-trained light source classification model includes: The feature data of each detection area are processed using the light source classification model corresponding to each detection area.

7. The method according to claim 1, characterized in that, The method further includes: The target image is color-corrected based on the illumination information of each sub-region in the target image.

8. A light detection device for images, characterized in that, include: The data acquisition module is configured to acquire a target image collected by an image sensor, and spectral data of multiple detection areas collected by multiple spectral sensors; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the target image; The illumination information detection module is configured to analyze the spectral data of each detection region to obtain the illumination information of each sub-region in the target image; The illumination information includes light source classification results; the device is further configured to: The light source classification result of each sub-region is converted into a color index value for each sub-region; the color index value includes a color deviation value and a color temperature. The histograms of color deviation values ​​and color temperature for each sub-region are statistically analyzed. The histogram of color deviation values ​​represents the ratio of the number of sub-regions corresponding to different color deviation values ​​to the total number of sub-regions. The histogram of color temperature represents the ratio of the number of sub-regions corresponding to different color temperature values ​​to the total number of sub-regions. The n color deviation values ​​with the highest percentages are determined from the color deviation value histogram, and the n color temperature values ​​with the highest percentages are determined from the color temperature histogram. The n color deviation values ​​and the n color temperature values ​​are combined to obtain n sets of color index values. Based on the n sets of color index values, the n light sources in the target image are determined; n is a positive integer not less than 2.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 7 by executing the executable instructions.

Images