Image processing method, model training method and related equipment

Abstract

The embodiments of the present application provide an image processing method, a model training method and related equipment, which are applied in the field of electronic technology. The method obtains a first image captured by a camera and first reflection spectrum data captured by a multi-spectral sensor, the first reflection spectrum data and the first image correspond to the same shooting scene, the second reflection spectrum data is determined according to the first reflection spectrum data and the calibration matrix, the second reflection spectrum data is processed using a light source prediction model, at least one light source prediction type and the confidence corresponding to each light source prediction type are predicted to determine the white balance parameters corresponding to the first image, the first image is processed using the white balance parameters to obtain the second image. Therefore, the embodiments of the present application can realize white balance processing of the first image captured by the camera in certain pure color scenes to reduce the problem of image color cast, thereby improving the white balance processing effect of the image.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method, a model training method and related equipment. Background Art

[0002] With the continuous development of electronic technology, electronic devices such as mobile phones and tablet computers have become common tools in people's daily life and work. At present, some electronic devices are provided with cameras to provide users with photo taking or video recording functions based on the cameras.

[0003] When an electronic device uses a camera to capture images, it is affected by the color of the light source in the environment, which can cause a difference between the color of the object in the image captured by the camera and the actual color of the object. Currently, electronic devices can use automatic white balance algorithms to perform white balance processing on the images captured by the camera to reduce the impact of the light source color on the images captured by the camera.

[0004] However, in some pure color scenes, the automatic white balance algorithm cannot effectively perform white balance processing on the images captured by the camera, resulting in color cast problems in the images. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, a model training method and related equipment, which can perform white balance processing on images captured by a camera in certain pure color scenes to reduce the problem of image color cast.

[0006] In a first aspect, an embodiment of the present application proposes an image processing method, which is applied to an electronic device, wherein the electronic device includes a camera and a multispectral sensor, and the image processing method includes: the electronic device obtains a first image captured by the camera; the electronic device obtains first reflection spectrum data captured by the multispectral sensor, the first reflection spectrum data and the first image correspond to the same shooting scene; the electronic device determines second reflection spectrum data based on the first reflection spectrum data and a calibration matrix; the electronic device processes the second reflection spectrum data using a light source prediction model to predict at least one light source prediction type and a confidence level corresponding to each light source prediction type; the electronic device determines white balance parameters corresponding to the first image based on the at least one light source prediction type and the corresponding confidence level; the electronic device processes the first image using the white balance parameters to obtain a second image.

[0007] In this way, based on the first reflection spectrum data collected by the multispectral sensor and in combination with the light source prediction model, the light source prediction type of the ambient light source when the camera collects the first image can be predicted, and the white balance parameters can be determined based on the light source prediction type to perform white balance processing on the first image. Therefore, it can realize white balance processing on the first image collected by the camera in certain pure color scenes (pure color scenes other than black, white and gray), improve the consistency between the color of the subject in the second image obtained after processing and the actual color of the subject, that is, make the color restoration degree of the subject in the second image obtained after processing higher, so as to reduce the problem of image color cast, thereby improving the white balance processing effect of the image.

[0008] In a possible implementation, the electronic device determines the second reflection spectrum data according to the first reflection spectrum data and the calibration matrix, including: the electronic device multiplies the first reflection spectrum data by the calibration matrix to obtain the second reflection spectrum data. The calibration matrix is ​​used to expand the 1×n-dimensional first reflection spectrum data to 1×m-dimensional second reflection spectrum data; n is the number of channels of the multi-spectral sensor, m is the number of wavelengths within the preset spectral range, and m is greater than n. In this way, the second reflection spectrum data with richer spectral dimension information can be obtained, and when the second reflection spectrum data is input into the light source prediction model, the accuracy of the light source prediction type predicted by the light source prediction model can be improved.

[0009] In a possible implementation, the electronic device determines the white balance parameters corresponding to the first image according to at least one light source prediction type and the corresponding confidence level, including: the electronic device determines the first color temperature according to the at least one light source prediction type and the corresponding confidence level; the electronic device determines the white balance parameters according to the first color temperature. In this way, the white balance parameters corresponding to the first image can be determined based on the first color temperature.

[0010] In a possible implementation, the electronic device determines the first color temperature according to at least one light source prediction type and the corresponding confidence level, including: the electronic device obtains the second color temperature corresponding to each light source prediction type; the electronic device uses the confidence level of each light source prediction type to perform weighted summation on the second color temperature corresponding to the light source prediction type to obtain the first color temperature. In this way, the confidence level of each light source prediction type can be used to perform weighted summation on the second color temperature corresponding to the light source prediction type to calculate the first color temperature, so that the calculated first color temperature has higher accuracy.

[0011] In a possible implementation, the electronic device obtains the second color temperature corresponding to each light source prediction type, including: the electronic device queries the second color temperature corresponding to each light source prediction type from a first storage table. The first storage table stores second color temperatures corresponding to multiple light source types; the second color temperatures corresponding to the multiple light source types are obtained by using an illuminometer to collect light sources corresponding to the multiple light source types. In this way, the second color temperature corresponding to each light source prediction type can be directly queried from the first storage table, making the method of obtaining the second color temperature more simple and quick.

[0012] In a possible implementation, the electronic device obtains the second color temperature corresponding to each light source prediction type, including: the electronic device queries the first light source spectrum data corresponding to each light source prediction type from the second storage table; the electronic device determines the second light source spectrum data corresponding to each light source prediction type according to the first light source spectrum data corresponding to each light source prediction type and the calibration matrix; the electronic device determines the second color temperature corresponding to each light source prediction type according to the second light source spectrum data corresponding to each light source prediction type and the tristimulus value curve. The second storage table stores the first light source spectrum data corresponding to multiple light source types; the first light source spectrum data corresponding to the multiple light source types is obtained by using a multi-spectral sensor to collect light sources corresponding to the multiple light source types. In this way, the second color temperature can be calculated based on the first light source spectrum data corresponding to the multiple light source types stored in the electronic device, which provides a method for obtaining the second color temperature; and since the first light source spectrum data is obtained by collecting light sources by the multi-spectral sensor, the number of data in the first light source spectrum data is related to the number of channels of the multi-spectral sensor, so that the data amount of the first light source spectrum data is limited, so as to reduce the storage space occupied by the electronic device when storing the first light source spectrum data corresponding to multiple light source types.

[0013] In a possible implementation, the white balance parameter includes a first white balance parameter and a second white balance parameter. The electronic device processes the first image using the white balance parameter to obtain the second image, including: the electronic device multiplies the pixel value of the first color channel in the first image by the first white balance parameter, and multiplies the pixel value of the second color channel in the first image by the second white balance parameter to obtain the second image. In this way, the white balance processing of the first image can be implemented based on the first white balance parameter and the second white balance parameter.

[0014] In the second aspect, the embodiment of the present application proposes a model training method, including: obtaining the first light source spectrum data corresponding to the multiple light sources collected by the multi-spectral sensor; determining the multi-channel reflectance data corresponding to the multiple reflective materials according to the reflectance data of the multiple reflective materials in the open source data set and the spectral response curve of the multi-spectral sensor; determining the multiple sample reflectance spectrum data according to the first light source spectrum data corresponding to the multiple light sources and the multi-channel reflectance data corresponding to the multiple reflective materials; constructing the reflectance spectrum data set according to the multiple sample reflectance spectrum data and the calibration matrix; and obtaining the light source prediction model based on the training of the reflectance spectrum data set. Among them, the light source prediction model is used to process the second reflectance spectrum data, predict at least one light source prediction type and the corresponding confidence level of each light source prediction type; at least one light source prediction type and the corresponding confidence level are used to determine the white balance parameters corresponding to the first image collected by the camera; the white balance parameters are used to process the first image to obtain the second image.

[0015] In this way, it is only necessary to collect the first light source spectrum data corresponding to multiple light sources through the multi-spectral sensor in the electronic device, and combine it with the open source data set including the reflectance data of multiple reflective materials to generate a series of reflectance spectrum data of the light source irradiating the reflective material, so as to construct a reflectance spectrum data set, and use the constructed reflectance spectrum data set to train the light source prediction model. Therefore, this way of constructing the reflectance spectrum data set reduces the amount of manual collection, making the construction of the reflectance spectrum data set more efficient; and, in theory, as long as the reflectance data of more reflective materials can be expanded, and / or the first light source spectrum data corresponding to more light sources can be collected, the data volume of the reflectance spectrum data set can be further expanded, making the data volume of the reflectance spectrum data set richer, so that the prediction accuracy of the trained light source prediction model is higher.

[0016] In addition, the light source prediction model obtained through training can be used to predict the light source prediction type of the ambient light source when the camera captures the first image, and the white balance parameters can be determined based on the light source prediction type to perform white balance processing on the first image to reduce the problem of image color cast, thereby improving the white balance processing effect of the image.

[0017] In one possible implementation, multi-channel reflectance data corresponding to the multiple reflective materials are determined based on the reflectance data of the multiple reflective materials in the open source data set and the spectral response curve of the multispectral sensor, including: calculating the overlapping integral of the reflectance data of the multiple reflective materials and the spectral response curve of the multispectral sensor to obtain the multi-channel reflectance data corresponding to the multiple reflective materials.

[0018] In a possible implementation, multiple sample reflection spectrum data are determined based on the first light source spectrum data corresponding to multiple light sources and the multi-channel reflectance data corresponding to multiple reflective materials, including: multiplying the first light source spectrum data corresponding to the multiple light sources and the multi-channel reflectance data corresponding to the multiple reflective materials according to the corresponding channels to obtain the multiple sample reflection spectrum data. In this way, the reflection spectrum data of the light source irradiating the reflective material and collected by the multi-spectral sensor can be constructed.

[0019] In a possible implementation, a reflectance spectrum data set is constructed based on multiple sample reflectance spectrum data and a calibration matrix, including: multiplying each sample reflectance spectrum data by the calibration matrix to obtain a reflectance spectrum data set. In this way, the spectral dimension information of the reflectance spectrum data in the reflectance spectrum data set can be enriched to improve the prediction accuracy of the trained light source prediction model.

[0020] In a third aspect, an embodiment of the present application proposes an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program to execute the above-mentioned image processing method, or to execute the above-mentioned model training method.

[0021] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are executed, the above-mentioned image processing method is implemented, or the above-mentioned model training method is implemented.

[0022] In a fifth aspect, an embodiment of the present application proposes a computer program product, including a computer program, which, when executed, enables a computer to execute the above-mentioned image processing method, or enables a computer to execute the above-mentioned model training method.

[0023] The possible implementation methods of the third to fifth aspects have effects similar to those of the first aspect and its possible designs, or similar to those of the second aspect and its possible designs, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the location of the multispectral sensor provided in the embodiment of the present application;

[0025] Figure 2 A schematic diagram of imaging of a multispectral sensor provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a spectral response curve of a multi-spectral sensor provided in an embodiment of the present application;

[0027] Figure 4A schematic diagram of an electronic device provided in an embodiment of the present application photographing an object;

[0028] Figure 5 A schematic diagram of the hardware system structure of an electronic device provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the software system structure of an electronic device provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of an application scenario provided by an embodiment of the present application

[0031] Figure 8 A flowchart of an image processing method provided in an embodiment of the present application;

[0032] Fig. 9 A schematic diagram of second reflection spectrum data provided in an embodiment of the present application;

[0033] Fig.10 A schematic diagram of the structure of a light source prediction model provided in an embodiment of the present application;

[0034] Fig.11 A flowchart of a model training method provided in an embodiment of the present application;

[0035] Fig.12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0036] Fig.13 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0037] Fig.14 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0039] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0040] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0041] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0042] 1. Multispectral Sensor

[0043] Compared with the image sensor in the camera, the multispectral sensor has more photosensitive channels, the spectral response range of the multispectral sensor is wider, and it can sense more light information. Among them, the multispectral sensor can include n-channel detection and response units, each channel has a different spectral response, and n is an integer greater than 3.

[0044] In an electronic device, the multispectral sensor and the camera may be placed on the same side of the electronic device. For example, Figure 1 As shown, the electronic device includes multiple cameras 193 and a multispectral sensor 180, and the multispectral sensor 180 and the camera 193 can be located on the same side, for example, the multispectral sensor 180 can be located on the same side as the rear camera. In some embodiments, the field of view (FOV) of the multispectral sensor 180 can be greater than the field of view of the camera 193.

[0045] In some embodiments, the multispectral sensor 180 may be located under the lampshade, that is, the lampshade may be located on the light incident side of the multispectral sensor 180. The lampshade serves as an optical element for focusing ambient light onto the multispectral sensor 180. Of course, the multispectral sensor 180 may also be located at other locations of the electronic device, which is not limited in the embodiments of the present application.

[0046] For example, Figure 2 This is a schematic diagram of imaging of a multi-spectral sensor provided in an embodiment of the present application. Figure 2 As shown, multispectral sensor 180 may include detection response units of eight channels, such as a detection response unit of an F1 color channel, a detection response unit of an F2 color channel, a detection response unit of an F3 color channel, a detection response unit of an F4 color channel, a detection response unit of an F5 color channel, a detection response unit of an F6 color channel, a detection response unit of an F7 color channel, and a detection response unit of an F8 color channel.

[0047] The F1 color channel, the F2 color channel, the F3 color channel, the F4 color channel, the F5 color channel, the F6 color channel, the F7 color channel, and the F8 color channel are all color channels within the visible light wavelength range. In addition, the colors of any two color channels among the F1 color channel, the F2 color channel, the F3 color channel, the F4 color channel, the F5 color channel, the F6 color channel, the F7 color channel, and the F8 color channel are different.

[0048] It is understood that the multispectral sensor 180 in the embodiment of the present application includes but is not limited to the following: Figure 2 The multispectral sensor shown in FIG. 1 may also be other types of multispectral sensors. In one example, the multispectral sensor 180 may also be an RGBCMY multispectral sensor, that is, the RGBCMY multispectral sensor includes six channels, which are red channel, green channel, blue channel, cyan channel, magenta channel and yellow channel. In another example, the multispectral sensor 180 may also be an RYYB multispectral sensor, that is, the RYYB multispectral sensor includes four channels, which are red channel, yellow channel, yellow channel and blue channel.

[0049] In addition, the multispectral sensor 180 in the embodiment of the present application may also include an ultraviolet channel, a near infrared (NIR) channel, a short-wave infrared (SWIR) channel, an infrared (IR) channel, a long-wave infrared (LWIR) channel or an all-pass channel, in addition to the color channel within the visible light wavelength range.

[0050] 2. Spectral response

[0051] Spectral responsivity can also be called spectral sensitivity. Spectral responsivity can be used to indicate the ability of a multispectral sensor to convert incident light energy of different wavelengths into electrical energy. If the light energy of a certain wavelength incident on a multispectral sensor is converted into the number of photons, and the current generated by the multispectral sensor and transmitted to the external circuit is expressed in the number of electrons, then the ability of each incident photon to be converted into electrons transmitted to the external circuit is called quantum efficiency (QE), which is expressed in percentage. The spectral responsivity of a multispectral sensor depends on the quantum efficiency, as well as parameters such as wavelength and integration time.

[0052] For example, Figure 3 for Figure 2 Schematic diagram of the spectral response curve of the multi-spectral sensor shown. The horizontal axis represents the wavelength in nm (nanometers), and the vertical axis represents the spectral response corresponding to different spectra. Among them, the first curve 31 represents the spectral response curve corresponding to the F1 color channel, the second curve 32 represents the spectral response curve corresponding to the F2 color channel, the third curve 33 represents the spectral response curve corresponding to the F3 color channel, the fourth curve 34 represents the spectral response curve corresponding to the F4 color channel, the fifth curve 35 represents the spectral response curve corresponding to the F5 color channel, the sixth curve 36 represents the spectral response curve corresponding to the F6 color channel, the seventh curve 37 represents the spectral response curve corresponding to the F7 color channel, and the eighth curve 38 represents the spectral response curve corresponding to the F8 color channel.

[0053] 3. Auto white balance (AWB)

[0054] Automatic white balance is the automatic correction of the color of the image captured by the camera. The so-called "white balance" is to correct the color difference caused by different color temperatures. White objects captured under light sources of different color temperatures can appear as original white, while other color objects can be as close to their original colors as possible, so that the overall image presentation effect conforms to the visual and cognitive habits of the human eye.

[0055] 4. Correlated color temperature (CCT)

[0056] Correlated color temperature refers to the temperature of the black body radiator with the closest color to the same brightness stimulus, expressed in Kelvin temperature, and is used to describe the color of light near the Planckian locus. Other light sources except thermal radiation light sources have linear spectra, and their radiation characteristics are quite different from those of black body radiation. Therefore, the light colors of these light sources may not fall exactly on the black body locus on the chromaticity diagram. For such light sources, CCT is usually used to describe the color characteristics of the light source.

[0057] Usually, when shooting with natural light, the color temperature of the light is different at different times, so the color of the photos taken is also different. For example, when shooting under a clear blue sky, the color temperature of the light is higher, so the photo is cooler; if shooting at dusk, the color temperature of the light is lower, so the photo is warmer. When shooting with artificial light, the color of the photos taken will also be different depending on the type of light source.

[0058] 5. Color coordinates

[0059] Color coordinates are also called color systems. Color coordinates can accurately identify colors. Usually, the horizontal coordinate of the color coordinate is x and the vertical coordinate of the color coordinate is y. A color coordinate can determine a point on the chromaticity diagram, which can be called a color point. A color point can be represented by color coordinates (x, y).

[0060] Reference Figure 4 As shown, when a user uses the electronic device 100 to take a photo, the light emitted by the light source 401 irradiates the object 402 , and the light reflected by the object 402 is incident on the camera and the multi-spectral sensor in the electronic device 100 .

[0061] During the shooting process, the light received by the camera and the multi-spectral sensor in the electronic device 100 is the reflected spectrum S x，y (λ), the reflection spectrum S x，y (λ) is the light source spectrum L(λ) of the light source 401 after being reflected by the reflective material corresponding to the object 402, which is equivalent to the reflectivity R corresponding to the light source spectrum L(λ) of the light source 401 and the reflectivity R corresponding to the reflective material of the object 402. x，y (λ) is obtained after overlap integration.

[0062] Therefore, when the light source is different, the light source spectrum corresponding to the light source is also different. When an electronic device uses a camera to capture an image, it is affected by the color of the light source, which causes a difference between the color of the object in the image captured by the camera and the actual color of the object.

[0063] For example, when a white object is placed under a fluorescent lamp, the color of the image captured by the camera is close to the color of the object itself. When the white object is placed under a red light emitting diode (LED) lamp, the color of the image captured by the camera is affected by the red LED light and becomes reddish due to the change in light source.

[0064] Currently, electronic devices can use automatic white balance algorithms to perform white balance processing on images captured by cameras to reduce the impact of light source color on images captured by the camera.

[0065] However, in some pure color scenes (pure color scenes other than black, white or gray), such as scenes where the subject is illuminated by light sources such as yellow light, red light or green light, the automatic white balance algorithm cannot find effective statistics, making it impossible for the automatic white balance algorithm to effectively perform white balance processing on the image captured by the camera, resulting in color cast problems in the image.

[0066] Based on this, an embodiment of the present application provides an image processing method, which obtains a first image captured by a camera and first reflection spectrum data captured by a multispectral sensor, the first reflection spectrum data and the first image correspond to the same shooting scene, determines second reflection spectrum data according to the first reflection spectrum data and a calibration matrix, processes the second reflection spectrum data using a light source prediction model, predicts at least one light source prediction type and a confidence level corresponding to each light source prediction type, determines white balance parameters corresponding to the first image according to the at least one light source prediction type and the corresponding confidence level, and processes the first image using the white balance parameters to obtain a second image.

[0067] Therefore, the embodiment of the present application can predict the light source prediction type of the ambient light source when the camera collects the first image based on the first reflection spectrum data collected by the multi-spectral sensor and in combination with the light source prediction model, and determine the white balance parameters based on the light source prediction type to perform white balance processing on the first image. In this way, in certain pure color scenes (pure color scenes other than black, white and gray), white balance processing can be performed on the first image collected by the camera to improve the consistency between the color of the object in the second image obtained after processing and the actual color of the object, that is, the color restoration degree of the object in the second image obtained after processing is higher, so as to reduce the problem of image color cast, thereby improving the white balance processing effect of the image.

[0068] The image processing method provided in the embodiment of the present application can be applied to electronic devices with cameras and multispectral sensors. Electronic devices include terminal devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices can be mobile phones, smart TVs, wearable devices, tablet computers (Pad), computers with wireless transceiver functions, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technology and specific device form used by the electronic devices.

[0069] In order to better understand the embodiments of the present application, the structure of the electronic device according to the embodiments of the present application is introduced below.

[0070] Figure 5 The structure diagram of the electronic device 100 is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a multi-spectral sensor 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195.

[0071] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0072] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0073] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0074] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0075] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0076] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0077] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0078] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0079] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0080] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0081] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0082] The display screen 194 is used to display images, display videos, and receive sliding operations, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or more display screens 194.

[0083] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0084] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the image sensor of the camera through the lens. The light signal is converted into an electrical signal, and the image sensor of the camera transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0085] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the image sensor. The image sensor can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include one or more cameras 193.

[0086] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0087] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0088] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0089] In some embodiments, the multi-spectral sensor 180 may be used to collect first reflection spectrum data and first light source spectrum data corresponding to multiple light sources.

[0090] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0091] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0092] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or removing the SIM card from the SIM card interface 195 .

[0093] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0094] Figure 6 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Androidruntime) and the system library, the hardware abstraction layer and the kernel layer.

[0095] The application layer can include a series of application packages. Figure 6 As shown, the application package may include applications such as camera, settings, and calendar.

[0096] The camera application is an application with the functions of photographing and recording, and the electronic device can respond to the user's operation of opening the camera application to take photos or record videos. It is understandable that the camera application's photo taking and video recording functions can also be called by other applications.

[0097] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0098] Among them, Figure 6As shown, the application framework layer may also include a camera service, which may be called by a camera application to implement functions such as taking photos or recording videos.

[0099] In addition, if Figure 6 As shown, the application framework layer may also include a window manager, a content provider, a resource manager, a view system, and the like.

[0100] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0101] Content providers are used to store and retrieve data and make it accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0102] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0103] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0104] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0105] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0106] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0107] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0108] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0109] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG2, H.262, MP3, AAC, AMR, JPG, PNG, etc.

[0110] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0111] The hardware abstraction layer is an abstract structure between the kernel layer and the Android runtime. The hardware abstraction layer can be a package of the hardware driver of the kernel layer, providing a calling interface for the application framework layer.

[0112] In an embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction module (camera hardware abstraction layer, camera HAL).

[0113] The kernel layer is a layer between hardware and software. The kernel layer includes at least a camera driver, a sensor driver, and a display driver. In some embodiments, the camera driver is used to control the operation of the camera, the sensor driver is used to control the operation of the multispectral sensor, and the display driver is used to control the display screen to display images.

[0114] The hardware may be a camera, a multispectral sensor, a display screen, etc. In the embodiment of the present application, the camera may be a front camera or a rear camera.

[0115] In some embodiments, when the electronic device is executing the image processing method of the embodiment of the present application, the camera can capture the first image and send the first image to the camera hardware abstract module through the camera driver, and the multispectral sensor can capture the first reflection spectrum data and send the first reflection spectrum data to the camera hardware abstract module through the sensor driver. In this way, the camera hardware abstract module can obtain the first image captured by the camera and the first reflection spectrum data collected by the multispectral sensor. The camera hardware abstract module can be used to determine the second reflection spectrum data based on the first reflection spectrum data and the calibration matrix; use the light source prediction model to process the second reflection spectrum data, and predict at least one light source prediction type and the corresponding confidence level of each light source prediction type; determine the white balance parameters corresponding to the first image based on the at least one light source prediction type and the corresponding confidence level; use the white balance parameters to process the first image to obtain the second image.

[0116] Therefore, the embodiment of the present application can execute the image processing method provided by the embodiment of the present application through the camera hardware abstract module. Of course, the process corresponding to the image processing method provided by the embodiment of the present application can also be set in the software of the kernel layer or the application framework layer, or set in independent dedicated hardware, and the embodiment of the present application does not limit this.

[0117] It should be noted that although the embodiments of the present application are described using the Android system, the principles of the image processing method are also applicable to electronic devices with operating systems such as iOS or Windows.

[0118] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0119] For ease of understanding, the embodiment of the present application takes a mobile phone as an electronic device, and first illustrates the application scenarios of the image processing method in combination with some user interfaces shown in the embodiment of the present application.

[0120] When the user turns on the screen of the electronic device and controls the electronic device to be in an unlocked state, the electronic device may display the following Figure 7 The first interface 701 shown in (a) of FIG. The first interface 701 may be the desktop of the electronic device, on which icons of multiple installed applications are displayed, such as a file management application icon, an email application icon, a weather application icon, a calculator application icon, a clock application icon, a recorder application icon, a music application icon, a setting application icon, an address book application icon, a phone application icon, a message application icon, and a camera application icon 7011.

[0121] The user can perform a touch operation on the camera application icon 7011, and the touch operation can be a click operation, a long press operation, etc., so that the electronic device receives the user's touch operation on the camera application icon 7011, and the electronic device starts the camera application in response to the touch operation.

[0122] After the camera application is started, the electronic device may display the following Figure 7 The second interface 702 shown in (b) of FIG. The second interface 702 may be a preview interface provided by a camera application for implementing a shooting function, which includes a preview box 7021, a shooting control 7022, and function controls corresponding to a plurality of shooting modes.

[0123] In certain pure color scenes (pure color scenes other than black, white or gray), such as scenes where the subject is illuminated by light sources such as yellow light, red light or green light, the preview box 7021 can be used to display a second image obtained by performing white balance processing on the first image captured by the camera using the image processing method provided in an embodiment of the present application, so as to improve the consistency between the color of the subject in the second image and the actual color of the subject.

[0124] The shooting control 7022 is used to trigger the shooting operation of the electronic device. Function controls corresponding to various shooting modes may include night scene mode controls, portrait mode controls, photo mode controls, video mode controls, professional mode controls, and more controls for opening more functions in the camera application.

[0125] Therefore, the embodiment of the present application can start the camera application by performing a touch operation on the camera application icon 7011. After the camera application is started, and when the camera captures an image corresponding to the object in a certain pure color scene, the electronic device can execute the process corresponding to the image processing method provided in the embodiment of the present application.

[0126] Understandably, Figure 7 (a) and Figure 7 The interface shown in (b) is merely an example of a user interface in the process of launching a camera application by touching the camera application icon of an electronic device, and does not constitute a limitation on the embodiments of the present application.

[0127] In another scenario, the user can also call the corresponding interface through a third-party application installed on the electronic device to access the camera application of the electronic device to start the camera application. After the camera application is started, and when the camera captures an image corresponding to the object in a certain pure color scene, the electronic device can execute the process corresponding to the image processing method provided in the embodiment of the present application.

[0128] In addition, when the electronic device is in the lock screen state, the user can instruct the electronic device to open the camera application by sliding right on the display screen of the electronic device. Alternatively, when the electronic device is in the lock screen state, the lock screen interface includes an icon of the camera application, and the user instructs the electronic device to open the camera application by clicking the icon of the camera application.

[0129] It should be understood that the above is an example of the operation of opening the camera application; the camera application can also be opened by instructing the electronic device through voice instructions or other operations, and this application does not impose any limitation on this.

[0130] The image processing method provided in the embodiment of the present application can be applied to the preview scene of the camera application in addition to the above-mentioned. It can also be applied to the shooting scene of the camera application. Specifically, the user can perform a touch operation on the shooting control 7022, such as a click operation, so that the camera can shoot the subject in certain pure color scenes, and use the image processing method provided in the embodiment of the present application to perform white balance processing on the captured image.

[0131] In addition, the image processing method provided in the embodiment of the present application can also be applied to video recording scenarios or video call scenarios, etc.

[0132] For example, Figure 8 The flowchart of an image processing method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device can include a camera and a multi-spectral sensor. Figure 8 As shown, the image processing method may specifically include the following steps:

[0133] S801, the electronic device obtains a first image captured by a camera.

[0134] In some embodiments, the camera may be a color camera, such as an RGB camera, and the first image captured by the camera may be a color image. Furthermore, the first image may be an image in RAW format.

[0135] S802, the electronic device obtains first reflection spectrum data collected by the multi-spectral sensor; the first reflection spectrum data and the first image correspond to the same shooting scene.

[0136] When an electronic device is used for shooting, the light emitted by the light source is irradiated onto the object, and the light reflected by the object is incident on the multispectral sensor. After the multispectral sensor receives the light reflected back by the object, each channel of the multispectral sensor has a different spectral response, so that the multispectral sensor can collect first reflection spectrum data.

[0137] The first reflection spectrum data collected by the multi-spectral sensor may be a one-dimensional sequence of 1×n dimensions, where n is the number of channels of the multi-spectral sensor. The n data in the first reflection spectrum data are reflection spectrum response data corresponding to multiple channels in the multi-spectral sensor, and the reflection spectrum response data corresponding to multiple channels may also be referred to as multi-channel raw data.

[0138] The number of data in the first reflection spectrum data is related to the number of channels of the multi-spectral sensor. For example, the number of data in the first reflection spectrum data and the number of channels of the multi-spectral sensor can both be 8, or 10, or can be less or more.

[0139] Furthermore, the first image collected by the camera and the first reflection spectrum data collected by the multispectral sensor are collected for the same object and at the same time, so that the first reflection spectrum data and the first image correspond to the same shooting scene.

[0140] For example, the first image collected by the camera and the first reflection spectrum data collected by the multispectral sensor are for the same tree and are collected at 12 noon, so that the first reflection spectrum data and the first image correspond to the same shooting scene. In this case, the corresponding light source is the sunlight at 12 noon outdoors.

[0141] S803: The electronic device determines second reflection spectrum data according to the first reflection spectrum data and the calibration matrix.

[0142] The electronic device is provided with a calibration matrix, and the matrix dimension of the calibration matrix may be n×m, where n is the number of channels of the multispectral sensor, and m is the number of wavelengths in a preset spectral range, and m is greater than n. For example, the preset spectral range may be a visible light wavelength range, and the visible light wavelength range may be 380 nm to 780 nm, and the number m of wavelengths in the visible light wavelength range may be 401.

[0143] The calibration matrix is ​​used to expand the first reflection spectrum data of 1×n dimension to the second reflection spectrum data of 1×m dimension, that is, the second reflection spectrum data can be a one-dimensional series of 1×m dimension, and the m data in the second reflection spectrum data are the reflection spectrum response data within the spectral range composed of the expanded m wavelengths.

[0144] For example, n is equal to 8, and m is equal to 401, then the calibration matrix is ​​used to expand the first reflection spectrum data of 1×8 dimensions to the second reflection spectrum data of 1×401 dimensions.

[0145] Therefore, the electronic device can obtain the second reflection spectrum data according to the first reflection spectrum data and in combination with the calibration matrix. The second reflection spectrum data can also be called a low-precision reflection spectrum.

[0146] In some embodiments, a monochromator, an integrating sphere, a spectrometer, etc. may be used to calibrate the multi-spectral sensor to obtain a calibration matrix, and the spectrometer may be a high-precision spectrometer.

[0147] Specifically, the two ports of the integrating sphere are connected to the monochromator and the spectrometer respectively, and the multispectral sensor in the electronic device is attached to the photosensitive port of the integrating sphere. The monochromator can use each nanometer of light in the visible light wavelength range to sequentially irradiate the integrating sphere. Each time the monochromator uses a certain nanometer of light to irradiate the integrating sphere, the multispectral sensor can obtain the first spectral response data, and the spectrometer can obtain the second spectral response data. Therefore, when the monochromator uses each nanometer of light in the visible light wavelength range to sequentially irradiate the integrating sphere, the multispectral sensor can obtain multiple groups of first spectral response data, and the spectrometer can obtain multiple groups of second spectral response data. Then, the least squares method can be used to process the multiple groups of first spectral response data obtained by the multispectral sensor and the multiple groups of second spectral response data obtained by the spectrometer to obtain a calibration matrix.

[0148] For example, the wavelength range of visible light can be 380nm to 780nm. When the monochromator uses a certain nanometer of light to irradiate the integrating sphere, the first spectral response data obtained by the multi-spectral sensor can be a one-dimensional series of 1×8 dimensions, and the second spectral response data obtained by the spectrometer can be a one-dimensional series of 1×401 dimensions. Therefore, when the monochromator uses each nanometer of light within 380nm to 780nm to irradiate the integrating sphere in sequence, the multi-spectral sensor obtains 401 sets of first spectral response data, and the spectrometer obtains 401 sets of second spectral response data, that is, all the first spectral response data obtained by the multi-spectral sensor can be 8×401 data, and all the second spectral response data obtained by the spectrometer can be 401×401 data. After the 401 sets of first spectral response data obtained by the multi-spectral sensor and the 401 sets of second spectral response data obtained by the spectrometer are processed by the least squares method, the matrix dimension of the obtained calibration matrix can be 8×401.

[0149] In some embodiments, the electronic device multiplies the first reflection spectrum data by a calibration matrix to obtain second reflection spectrum data.

[0150] By multiplying the first reflection spectrum data of 1×n dimensions by a calibration matrix of n×m dimensions, second reflection spectrum data of 1×m dimensions can be obtained.

[0151] Exemplarily, by multiplying the first reflection spectrum data with the calibration matrix, the second reflection spectrum data obtained can be as follows: Fig. 9 The second reflection spectrum data shown is shown in FIG. , wherein the horizontal axis represents the wavelength in nm, and the vertical axis represents the spectral responsivity.

[0152] S804: The electronic device processes the second reflection spectrum data using a light source prediction model to predict at least one light source prediction type and a confidence level corresponding to each light source prediction type.

[0153] The electronic device may be provided with a light source prediction model. After obtaining the second reflection spectrum data, the electronic device may input the second reflection spectrum data into the light source prediction model, and after being processed by the light source prediction model, output at least one light source prediction type and the confidence corresponding to each light source prediction type.

[0154] In this way, the electronic device can predict at least one light source prediction type and the confidence level corresponding to each light source prediction type based on the second reflection spectrum data and in combination with a pre-trained light source prediction model, that is, predict the light source type of the environment when the first image was collected, so as to predict the white balance parameters and achieve the purpose of assisting white balance.

[0155] It should be noted that the embodiment of the present application is to input the second reflection spectrum data into the light source prediction model after determining the second reflection spectrum data based on the first reflection spectrum data collected by the multi-spectral sensor and the calibration matrix, instead of directly inputting the first reflection spectrum data collected by the multi-spectral sensor into the light source prediction model. The reason is that the spectral dimension information of the second reflection spectrum data is richer, and the reflection spectrum data under different light sources can be classified through the light source prediction model, and the reflection spectrum data under the same light source can be classified into one category. Therefore, it is possible to predict the light source type of the environment in the process of collecting the first image through the light source prediction model, so as to improve the accuracy of the light source prediction type predicted by the light source prediction model. If the first reflection spectrum data collected by the multi-spectral sensor is directly input into the light source prediction model, due to the less spectral dimension information of the first reflection spectrum data, the prediction effect of the light source type of the environment in the process of collecting the first image predicted by the light source prediction model will be poor.

[0156] In some embodiments, the light source prediction model is obtained by training based on a reflectance spectrum dataset, wherein the light source prediction model comprises a fully connected neural network model.

[0157] Taking the light source prediction model including the fully connected neural network model as an example, refer to Fig.10 As shown, the light source prediction model may include an input layer, multiple hidden layers, an output layer, and a softmax (normalization) layer. In the light source prediction model, the layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer.

[0158] The working principle of the fully connected neural network model can be expressed by the following linear relationship expression: z = f (w × a + b), where a represents the input vector, z represents the output vector, b represents the offset vector, w represents the weight matrix, and f () represents the activation function. Each layer operates the input vector a through a linear expression to obtain the output vector z.

[0159] When the second reflection spectrum data is input into the fully connected neural network model, the output layer in the fully connected neural network model can output at least one light source prediction type and the confidence corresponding to each light source prediction type. At this time, the value of the confidence is not limited and can range from negative infinity to positive infinity. Therefore, a softmax layer is set after the output layer in the fully connected neural network model, and the confidence corresponding to each light source prediction type output by the output layer can be normalized to between 0 and 1.

[0160] Understandably, Fig.10 The fully connected neural network model shown is only an example of a light source prediction model and can also be replaced by other classification network models as a light source prediction model, which is not limited in the embodiments of the present application.

[0161] S805: The electronic device determines a first color temperature according to at least one light source prediction type and a corresponding confidence level.

[0162] The electronic device can determine the color temperature of the light source in the environment where the first image is captured based on at least one light source prediction type output by the light source prediction model and the confidence level corresponding to each light source prediction type. The first color temperature is actually the correlated color temperature.

[0163] In some embodiments, the electronic device obtains a second color temperature corresponding to each light source prediction type; the electronic device uses the confidence of each light source prediction type to perform weighted summation on the second color temperatures corresponding to the light source prediction type to obtain the first color temperature.

[0164] The light source corresponding to each light source prediction type has a second color temperature, which is actually a correlated color temperature. The electronic device can use the confidence of each light source prediction type as a weight value, and perform weighted summation on the second color temperatures corresponding to the light source prediction type to calculate the first color temperature.

[0165] For example, the light source prediction model outputs three light source prediction types, the second color temperature corresponding to the first light source prediction type is T1, the second color temperature corresponding to the second light source prediction type is T2, and the second color temperature corresponding to the third light source prediction type is T3, and the confidence corresponding to the first light source prediction type is w1, the confidence corresponding to the second light source prediction type is w2, and the confidence corresponding to the third light source prediction type is w3. Therefore, the first color temperature T0 = w1×T1+w2×T2+w3×T3.

[0166] The embodiment of the present application may adopt the following two implementation methods to obtain the second color temperature corresponding to each light source prediction type.

[0167] In one implementation, the electronic device queries the second color temperature corresponding to each light source prediction type from the first storage table. The first storage table stores second color temperatures corresponding to multiple light source types; the second color temperatures corresponding to multiple light source types are obtained by using an illuminometer to collect light sources corresponding to the multiple light source types.

[0168] A illuminometer may be used to collect light sources corresponding to a plurality of light source types in advance to obtain second color temperatures corresponding to the plurality of light source types, and store the second color temperatures in the electronic device. After obtaining at least one light source prediction type output by the light source prediction model, the electronic device directly queries the second color temperature corresponding to the light source type matching each light source prediction type from a first storage table of the electronic device by looking up the table.

[0169] In another implementation, the electronic device queries the first light source spectrum data corresponding to each light source prediction type from the second storage table; the electronic device determines the second light source spectrum data corresponding to each light source prediction type based on the first light source spectrum data corresponding to each light source prediction type and the calibration matrix; the electronic device determines the second color temperature corresponding to each light source prediction type based on the second light source spectrum data corresponding to each light source prediction type and the tristimulus value curve. The second storage table stores the first light source spectrum data corresponding to multiple light source types; the first light source spectrum data corresponding to multiple light source types is obtained by using a multi-spectral sensor to collect light sources corresponding to multiple light source types.

[0170] A multi-spectral sensor in an electronic device may be used in advance to collect light sources corresponding to multiple light source types to obtain first light source spectrum data corresponding to the multiple light source types, and store the first light source spectrum data in the electronic device. The first light source spectrum data is spectrum data corresponding to the light source, and the first light source spectrum data may be a one-dimensional sequence of 1×n dimensions, where n is the number of channels of the multi-spectral sensor.

[0171] After acquiring at least one light source prediction type output by the light source prediction model, the electronic device directly queries the first light source spectrum data corresponding to the light source type matching each light source prediction type from the second storage table of the electronic device by looking up the table.

[0172] The electronic device may multiply the first light source spectrum data corresponding to each light source prediction type by the calibration matrix to obtain the second light source spectrum data corresponding to each light source prediction type. The matrix dimension of the calibration matrix may be n×m, and the second light source spectrum data may be a one-dimensional sequence of 1×m dimensions, where m is the number of wavelengths within the preset spectrum range.

[0173] The electronic device can integrate the second light source spectrum data corresponding to each light source prediction type with the three stimulus value curve respectively to obtain the three stimulus values ​​corresponding to each light source prediction type; then, according to the three stimulus values ​​corresponding to each light source prediction type, calculate the color coordinates corresponding to each light source prediction type; and then, according to the color coordinates corresponding to each light source prediction type, calculate the second color temperature corresponding to each light source prediction type.

[0174] Specifically, the tristimulus values ​​X, Y, and Z corresponding to each light source prediction type can be calculated using the following formula:

[0175]

[0176]

[0177]

[0178] Among them, the three stimulus value curves are X(λ), Y(λ) and Z(λ). represents the spectrum data of the second light source, and k is a constant. The tristimulus value curve may be a tristimulus value curve under CIE1931 or other standards, which is not limited in the present embodiment.

[0179] In the color coordinates (x, y), x=X / (X+Y+Z), y=Y / (X+Y+Z). Exemplarily, the second color temperature may be calculated using the following formula:

[0180] CCT1=-449×h 3 +3525×h 2 -6823.3×h+5520.33;

[0181] h=(x-0.3320) / (y-0.1858)

[0182] Wherein, CCT1 represents the second color temperature, and x and y are color coordinates.

[0183] S806: The electronic device determines a white balance parameter corresponding to the first image according to the first color temperature.

[0184] In the color temperature curve of the light source, the horizontal axis is R / G and the vertical axis is B / G. R / G represents the ratio of the pixel value of the red channel to the pixel value of the green channel, and B / G represents the ratio of the pixel value of the blue channel to the pixel value of the green channel.

[0185] In some embodiments, the white balance parameter corresponding to the first image may include a first white balance parameter and a second white balance parameter. The first white balance parameter is Rgain, and Rgain=G / R, G / R represents the ratio of the pixel value of the green channel to the pixel value of the red channel; the second white balance parameter is Bgain, and Bgain=G / B, G / B represents the ratio of the pixel value of the green channel to the pixel value of the blue channel.

[0186] Therefore, after calculating the first color temperature, the horizontal and vertical coordinate values ​​corresponding to the position of the first color temperature on the color temperature curve of the light source can be determined, and the reciprocal of the horizontal coordinate value can be used as the first white balance parameter, and the reciprocal of the vertical coordinate value can be used as the second white balance parameter.

[0187] In summary, according to the above steps S805 and S806, the electronic device determines the white balance parameters corresponding to the first image according to at least one light source prediction type and the corresponding confidence level.

[0188] S807: The electronic device processes the first image using the white balance parameter to obtain a second image.

[0189] After calculating and obtaining the white balance parameters corresponding to the first image, the electronic device uses the white balance parameters to perform white balance processing on the first image to obtain a second image to achieve white balance.

[0190] In some embodiments, the electronic device multiplies the pixel values ​​of the first color channel in the first image by the first white balance parameter, and multiplies the pixel values ​​of the second color channel in the first image by the second white balance parameter to obtain the second image.

[0191] For example, the first white balance parameter is Rgain, and the second white balance parameter is Bgain. The first white balance parameter is used to adjust the pixel value of the red channel in the first image, and the second white balance parameter is used to adjust the pixel value of the blue channel in the first image. Specifically, the pixel value of the red channel in the first image is multiplied by the first white balance parameter Rgain, and the pixel value of the blue channel in the first image is multiplied by the second white balance parameter Bgain, so as to achieve white balance processing of the first image.

[0192] In summary, the embodiment of the present application can predict the light source prediction type of the ambient light source when the camera collects the first image based on the first reflection spectrum data collected by the multi-spectral sensor and in combination with the light source prediction model, and determine the white balance parameters based on the light source prediction type to perform white balance processing on the first image. In this way, in certain pure color scenes (pure color scenes other than black, white and gray), white balance processing can be performed on the first image collected by the camera to improve the consistency between the color of the object in the second image obtained after processing and the actual color of the object, that is, the color restoration degree of the object in the second image obtained after processing is higher, so as to reduce the problem of image color cast, thereby improving the white balance processing effect of the image.

[0193] It should be noted that the image processing method of the embodiment of the present application is mainly used to perform white balance processing on the first image captured by the camera in certain pure color scenes (pure color scenes other than black, white and gray), while in other scenes (such as pure color scenes of black, white and gray, or non-pure color scenes), the original automatic white balance algorithm is still used to perform white balance processing on the first image captured by the camera.

[0194] When the electronic device performs white balance processing on the first image captured by the camera, it can use the following method to determine whether to use the image processing method of the embodiment of the present application to perform white balance processing, or to use the original automatic white balance algorithm to perform white balance processing: divide the first image captured by the camera into multiple image blocks; calculate the mean of the pixel values ​​of the red channel in each image block to obtain the red pixel mean, calculate the mean of the pixel values ​​of the green channel in each image block to obtain the green pixel mean, and calculate the mean of the pixel values ​​of the blue channel in each image block to obtain the blue pixel mean; calculate the ratio of the red pixel mean to the green pixel mean in each image block, and calculate the ratio of the blue pixel mean to the green pixel mean in each image block; in this way, the ratio of the red pixel mean to the green pixel mean, and the ratio of the blue pixel mean to the green pixel mean in each image block can be used as a landing point in the coordinate system corresponding to the color temperature curve of the light source; detect whether the landing point corresponding to each image block is located within the gray point area corresponding to the color temperature curve of the light source. If the landing points corresponding to at least some of the image blocks are located within the gray point area corresponding to the color temperature curve of the light source, the original automatic white balance algorithm is used to perform white balance processing; if the landing points corresponding to all image blocks are located outside the gray point area corresponding to the color temperature curve of the light source, the image processing method of the embodiment of the present application is used to perform white balance processing.

[0195] The image processing method of the embodiment of the present application needs to rely on a pre-trained light source prediction model when performing white balance processing on the first image, so as to process the input second reflection spectrum data through the pre-trained light source prediction model to predict the type of light source in the environment when the first image is captured, that is, to predict at least one light source prediction type. When training the light source prediction model, it needs to rely on a large amount of reflection spectrum data.

[0196] For example, Fig.11 Flow chart of the model training method provided in the embodiment of the present application. Fig.11 As shown, the model training method may specifically include the following steps:

[0197] S1101, obtaining first light source spectrum data corresponding to multiple light sources collected by a multi-spectral sensor.

[0198] When the electronic device is in the complete state, a multispectral sensor in the electronic device can be used to collect first light source spectrum data corresponding to multiple light sources. The first light source spectrum data can be a one-dimensional sequence of 1×n dimensions, where n is the number of channels of the multispectral sensor.

[0199] The types of light sources collected by the multispectral sensor can cover common light sources in life, such as outdoor sunlight in the morning, outdoor sunlight at noon, outdoor sunlight in the evening, outdoor light on cloudy days, LED lights with different color temperatures, fluorescent lights with different color temperatures, incandescent lamps with different color temperatures, cool white fluorescent lamps (CWF) and TL84 light source (a narrow-band fluorescent light source, a three-primary color fluorescent lamp), etc.

[0200] For example, the number of channels of the multispectral sensor is 8, and the types of light sources collected by the multispectral sensor may be 50. Therefore, first light source spectrum data corresponding to the 50 light sources may be obtained, and each first light source spectrum data may be a one-dimensional series of 1×8 dimensions.

[0201] S1102: Determine multi-channel reflectance data corresponding to the multiple reflective materials according to the reflectance data of the multiple reflective materials in the open source data set and the spectral response curve of the multispectral sensor.

[0202] The open source data set includes reflectance data of various reflective materials, such as reflectance data of Munsell colors, etc. In some embodiments, the overlap integral of the reflectance data of various reflective materials and the spectral response curve of the multi-spectral sensor is calculated to obtain multi-channel reflectance data corresponding to the various reflective materials. The multi-channel reflectance data corresponding to each reflective material can be a one-dimensional sequence of 1×n dimensions.

[0203] For example, the open source data set contains reflectance data for 2,000 reflective materials. After overlapping integration with the spectral response curve of the multispectral sensor, multi-channel reflectance data corresponding to the 2,000 reflective materials can be obtained. The multi-channel reflectance data corresponding to each reflective material can be a one-dimensional series of 1×8 dimensions.

[0204] S1103, determining a plurality of sample reflection spectrum data according to the first light source spectrum data corresponding to the plurality of light sources and the multi-channel reflectance ratio data corresponding to the plurality of reflection materials.

[0205] In some embodiments, the first light source spectrum data corresponding to the various light sources and the multi-channel reflectance data corresponding to the various reflective materials are multiplied according to the corresponding channels to obtain various sample reflectance spectrum data.

[0206] Each sample reflection spectrum data is actually the reflection spectrum data collected by the multi-spectral sensor when a light source irradiates a reflective material, wherein each sample reflection spectrum data can be a one-dimensional sequence of 1×n dimensions.

[0207] For example, the multispectral sensor may collect 50 types of light sources and obtain multi-channel reflectance data corresponding to 2000 types of reflective materials. Then, after multiplying the first light source spectrum data corresponding to the 50 light sources with the multi-channel reflectance data corresponding to the 2000 reflective materials according to the corresponding channels, 50×2000 types of sample reflectance spectrum data can be obtained, and each type of sample reflectance spectrum data can be a one-dimensional series of 1×8 dimensions.

[0208] S1104: construct a reflection spectrum data set according to the reflection spectrum data of multiple samples and a calibration matrix.

[0209] In some embodiments, each sample reflectance spectrum data is multiplied by a calibration matrix to obtain a reflectance spectrum data set.

[0210] The matrix dimension of the calibration matrix may be n×m, where m is the number of wavelengths in a preset spectral range. The calibration matrix may be used to expand the sample reflectance spectrum data of 1×n dimensions to 1×m dimensions to construct a reflectance spectrum data set.

[0211] For example, the matrix dimension of the calibration matrix can be 8×401, and 50×2000 types of sample reflection spectrum data are obtained, and each type of sample reflection spectrum data can be a one-dimensional series of 1×8 dimensions. Therefore, the constructed reflection spectrum data set includes 50×2000 types of reflection spectrum data, and each type of reflection spectrum data in the reflection spectrum data set can be a one-dimensional series of 1×401 dimensions.

[0212] S1105, obtaining a light source prediction model based on the reflection spectrum dataset training.

[0213] In one feasible method, each type of reflection spectrum data in the reflection spectrum data set can be input into the light source prediction model to be trained to predict the light source type; then, the weight matrix in the light source prediction model to be trained is updated according to the predicted light source type and the actual light source type of the light source to train the light source prediction model.

[0214] Among them, the light source prediction model is used to process the second reflection spectrum data to predict at least one light source prediction type and the confidence level corresponding to each light source prediction type; at least one light source prediction type and the corresponding confidence level are used to determine the white balance parameters corresponding to the first image captured by the camera; the white balance parameters are used to process the first image to obtain the second image.

[0215] The prediction accuracy of the light source prediction model depends largely on the amount and quality of the reflectance spectrum data set. Therefore, the richer the reflectance spectrum data included in the reflectance spectrum data set, the higher the prediction accuracy of the trained light source prediction model.

[0216] In some related technologies, generally, under the illumination of the same light source, multiple objects of different reflective materials are replaced in turn, and the objects are collected through a multispectral sensor to obtain the reflective spectrum data corresponding to different reflective materials under the same light source; then, other types of light sources are replaced in turn, and multiple objects of different reflective materials are replaced in turn under the illumination of each light source to obtain the reflective spectrum data corresponding to different reflective materials under multiple light sources, so as to construct a reflective spectrum data set.

[0217] However, this method of constructing a reflectance spectrum dataset requires constructing a large number of real-life scenes by replacing different types of light sources and different reflective materials, which results in time-consuming and labor-intensive construction of the reflectance spectrum dataset and low efficiency. In addition, due to the limited number of replaceable reflective materials, the amount of data in the reflectance spectrum dataset is limited, which affects the prediction accuracy of the trained light source prediction model.

[0218] The embodiment of the present application only needs to collect the first light source spectrum data corresponding to multiple light sources through the multi-spectral sensor in the electronic device, and combine it with the open source data set including the reflectance data of multiple reflective materials to generate a series of reflectance spectrum data of the light source irradiating the reflective material, so as to construct a reflectance spectrum data set, and use the constructed reflectance spectrum data set to train the light source prediction model. Therefore, this way of constructing the reflectance spectrum data set reduces the amount of manual collection, making the construction of the reflectance spectrum data set more efficient; and, in theory, as long as the reflectance data of more reflective materials can be expanded, and / or the first light source spectrum data corresponding to more light sources can be collected, the data volume of the reflectance spectrum data set can be further expanded, making the data volume of the reflectance spectrum data set richer, so that the prediction accuracy of the trained light source prediction model is higher.

[0219] It should be noted that the above Fig.11 The electronic device used to perform the model training process can be used with the above Figure 8 The electronic device used to perform the image processing process is the same as that described above. Figure 8 The electronic devices used to perform the image processing process are different. For example, in some embodiments, the above Figure 8 The electronic device used to perform the image processing process may be a terminal device, such as a mobile phone, a tablet computer, etc., and the above Fig.11 The electronic device used to perform the model training process can be a server.

[0220] With the above Figure 8 The electronic device used to perform the image processing process is the terminal device, and the above Fig.11 Taking the server as an example, the electronic device used to execute the model training process, the terminal device can download the trained light source prediction model from the server to save the light source prediction model to the terminal device, or the light source prediction model can be directly saved to the terminal device before the terminal device leaves the factory.

[0221] Combined with the above Figures 7 to 10 The image processing method provided in the embodiment of the present application is described, and combined with Fig.11 The model training method provided in the embodiment of the present application is explained, and the device for executing the above method provided in the embodiment of the present application is described below.

[0222] Fig.12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be an electronic device in an embodiment of the present application, or a chip or chip system in an electronic device.

[0223] like Fig.12As shown, the image processing apparatus 1200 may include a processing unit 1201. The processing unit 1201 is used to support the image processing apparatus 1200 to perform the above processing steps.

[0224] Specifically, the processing unit 1201 is used to obtain a first image captured by a camera; the processing unit 1201 is used to obtain first reflection spectrum data captured by a multispectral sensor, the first reflection spectrum data and the first image corresponding to the same shooting scene; the processing unit 1201 is used to determine second reflection spectrum data based on the first reflection spectrum data and a calibration matrix; the processing unit 1201 is used to process the second reflection spectrum data using a light source prediction model, and predict at least one light source prediction type and a corresponding confidence level for each light source prediction type; the processing unit 1201 is used to determine white balance parameters corresponding to the first image based on at least one light source prediction type and the corresponding confidence level; the processing unit 1201 is used to process the first image using the white balance parameters to obtain a second image.

[0225] In a possible implementation, the image processing device 1200 further includes a storage unit 1202. The storage unit 1202 is connected to the processing unit 1201 via a line. The storage unit 1202 may include one or more memories, and the memory may be a device used to store programs or data in one or more devices or circuits. The storage unit 1202 may exist independently and be connected to the processing unit 1201 via a communication bus. The storage unit 1202 may also be integrated with the processing unit 1201.

[0226] The storage unit 1202 can store computer-executable instructions of the method in the electronic device so that the processing unit 1201 executes the image processing method in the above embodiment. The storage unit 1202 can be a register, a cache, or a random access memory (RAM), etc. The storage unit 1202 can be integrated with the processing unit 1201. The storage unit 1202 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 1202 can be independent of the processing unit 1201.

[0227] Fig.13 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. Fig.13As shown, the model training device 1300 may include a training module 1301. The training module 1301 is used to obtain the first light source spectrum data corresponding to the multiple light sources collected by the multi-spectral sensor; the training module 1301 is used to determine the multi-channel reflectance data corresponding to the multiple reflective materials according to the reflectance data of the multiple reflective materials in the open source data set and the spectral response curve of the multi-spectral sensor; the training module 1301 is used to determine the multiple sample reflective spectrum data according to the first light source spectrum data corresponding to the multiple light sources and the multi-channel reflectance data corresponding to the multiple reflective materials; the training module 1301 is used to construct a reflective spectrum data set according to the multiple sample reflective spectrum data and the calibration matrix; the training module 1301 is used to train the light source prediction model based on the reflective spectrum data set. The light source prediction model is used to process the second reflective spectrum data to predict at least one light source prediction type and the confidence corresponding to each light source prediction type; at least one light source prediction type and the corresponding confidence are used to determine the white balance parameters corresponding to the first image collected by the camera; the white balance parameters are used to process the first image to obtain the second image.

[0228] Fig.14 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig.14 As shown, the chip 1400 includes one or more (including two) processors 1401 , a communication line 1402 and a communication interface 1403 . Optionally, the chip 1400 also includes a memory 1404 .

[0229] In some implementations, the memory 1404 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0230] The image processing method or model training method described in the above embodiment of the present application can be applied to the processor 1401, or implemented by the processor 1401. The processor 1401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware or software instructions in the processor 1401. The above-mentioned processor 1401 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1401 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0231] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 1404, and the processor 1401 reads the information in the memory 1404 and completes the steps of the above method in combination with its hardware.

[0232] The processor 1401 , the memory 1404 , and the communication interface 1403 may communicate with each other via the communication line 1402 .

[0233] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0234] The present application embodiment also provides a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part to perform the above-mentioned image processing method or model training method. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrated. For example, available media may include magnetic media (e.g., floppy disks, hard disks or tapes), optical media (e.g., digital versatile discs (digital versatile discs, DVDs)), or semiconductor media (e.g., solid state disks (solid state disks, SSDs)), etc.

[0235] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above-mentioned image processing method or model training method.

[0236] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the above-mentioned image processing method or model training method. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, and will not be repeated here.

[0237] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program or instruction. When the computer program or instruction is executed by the processor, the above-mentioned image processing method or model training method is implemented. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0238] As a possible design, the computer-readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer-readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of the medium. Disks and optical discs as used herein include compact discs (CDs), laser discs, optical discs, DVDs, floppy disks and Blu-ray discs, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0239] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0240] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a camera and a multi-spectral sensor, the method includes: The electronic device acquires a first image captured by the camera; The electronic device acquires first reflection spectrum data collected by the multispectral sensor; the first reflection spectrum data and the first image correspond to the same shooting scene; The electronic device multiplies the first reflection spectrum data by a calibration matrix to obtain second reflection spectrum data, wherein the calibration matrix is ​​used to expand the first reflection spectrum data of 1×n dimensions to the second reflection spectrum data of 1×m dimensions, where n is the number of channels of the multi-spectral sensor, m is the number of wavelengths within a preset spectral range, and m is greater than n; The electronic device processes the second reflection spectrum data using a light source prediction model to predict at least one light source prediction type and a confidence level corresponding to each light source prediction type; The electronic device determines, according to the at least one light source prediction type and the corresponding confidence level, a white balance parameter corresponding to the first image; The electronic device processes the first image using the white balance parameter to obtain a second image.

2. The method according to claim 1, characterized in that: The electronic device determines, according to the at least one light source prediction type and the corresponding confidence level, a white balance parameter corresponding to the first image, including: The electronic device determines a first color temperature according to the at least one light source prediction type and the corresponding confidence level; The electronic device determines the white balance parameter according to the first color temperature.

3. The method according to claim 2, characterized in that The electronic device determines a first color temperature according to the at least one light source prediction type and the corresponding confidence level, including: The electronic device acquires a second color temperature corresponding to each of the predicted light source types; The electronic device uses the confidence of each of the light source prediction types to perform weighted summation on the second color temperatures corresponding to the light source prediction types to obtain the first color temperature.

4. The method according to claim 3, characterized in that The electronic device obtains the second color temperature corresponding to each of the light source prediction types, including: The electronic device queries the first storage table for a second color temperature corresponding to each of the light source prediction types; The first storage table stores second color temperatures corresponding to a plurality of light source types; the second color temperatures corresponding to the plurality of light source types are obtained by collecting light sources corresponding to the plurality of light source types using an illuminometer.

5. The method according to claim 3, characterized in that: The electronic device obtains the second color temperature corresponding to each of the light source prediction types, including: The electronic device queries the first light source spectrum data corresponding to each light source prediction type from the second storage table; The electronic device determines the second light source spectrum data corresponding to each of the light source prediction types according to the first light source spectrum data corresponding to each of the light source prediction types and the calibration matrix; The electronic device determines the second color temperature corresponding to each of the predicted light source types according to the second light source spectrum data and the tristimulus value curve corresponding to each of the predicted light source types; The second storage table stores first light source spectrum data corresponding to a plurality of light source types; the first light source spectrum data corresponding to the plurality of light source types are obtained by collecting light sources corresponding to the plurality of light source types using the multi-spectral sensor.

6. The method according to claim 1, characterized in that The white balance parameters include a first white balance parameter and a second white balance parameter; and the electronic device processes the first image using the white balance parameters to obtain a second image, including: The electronic device obtains the second image by multiplying the pixel value of the first color channel in the first image by the first white balance parameter, and by multiplying the pixel value of the second color channel in the first image by the second white balance parameter.

7. A model training method, characterized in that: include: Acquire first light source spectrum data corresponding to multiple light sources collected by the multi-spectral sensor; Determine multi-channel reflectance data corresponding to the multiple reflective materials according to the reflectance data of the multiple reflective materials in the open source data set and the spectral response curve of the multi-spectral sensor; Determining a plurality of sample reflection spectrum data according to the first light source spectrum data corresponding to the plurality of light sources and the multi-channel reflectance ratio data corresponding to the plurality of reflection materials; Multiplying each of the sample reflection spectrum data by a calibration matrix respectively to construct a reflection spectrum data set, wherein the calibration matrix is ​​used to expand the 1×n-dimensional sample reflection spectrum data to 1×m-dimensional, where n is the number of channels of the multispectral sensor, m is the number of wavelengths within a preset spectral range, and m is greater than n; A light source prediction model is obtained based on the reflection spectrum data set training; Among them, the light source prediction model is used to process the second reflection spectrum data to predict at least one light source prediction type and the confidence level corresponding to each light source prediction type; the at least one light source prediction type and the corresponding confidence level are used to determine the white balance parameters corresponding to the first image captured by the camera; the white balance parameters are used to process the first image to obtain the second image.

8. The method according to claim 7, characterized in that Determining multi-channel reflectance data corresponding to the multiple reflective materials according to the reflectance data of the multiple reflective materials in the open source data set and the spectral response curve of the multi-spectral sensor includes: The overlap integral of the reflectance data of the multiple reflective materials and the spectral response curve of the multi-spectral sensor is calculated to obtain multi-channel reflectance data corresponding to the multiple reflective materials.

9. The method according to claim 7, characterized in that: The determining of a plurality of sample reflection spectrum data according to the first light source spectrum data corresponding to the plurality of light sources and the multi-channel reflectance data corresponding to the plurality of reflection materials comprises: The first light source spectrum data corresponding to the multiple light sources and the multi-channel reflectance data corresponding to the multiple reflective materials are multiplied according to the corresponding channels to obtain the multiple sample reflective spectrum data.

10. An electronic device, characterized in that: It comprises a memory and a processor, the memory is used to store a computer program, and the processor is used to call the computer program to execute the image processing method as described in any one of claims 1 to 6, or to execute the model training method as described in any one of claims 7 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction. When the computer program or instruction is executed, the image processing method as described in any one of claims 1 to 6 is implemented, or the model training method as described in any one of claims 7 to 9 is implemented.

12. A computer program product, characterized in that It comprises a computer program, which, when executed, enables a computer to execute the image processing method as claimed in any one of claims 1 to 6, or enables a computer to execute the model training method as claimed in any one of claims 7 to 9.

Images