Image processing method, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of computers, and provides an image processing method, electronic equipment and a readable storage medium, a scene light source is predicted through response data of a spectrum device, automatic white balance is carried out on the predicted scene light source, scene conditions do not need to be limited, and the application scene is wider. The method comprises the following steps: in a shooting process of the electronic equipment, acquiring target response data output by a spectrum device based on a current illumination environment when the electronic equipment shoots a current shooting scene; matching a light source type for the target response data to obtain a target light source type corresponding to the current illumination environment; according to the target light source type, determining a target white balance parameter corresponding to the target light source type from a mapping relationship comprising a plurality of light source types and a plurality of white balance parameters; and according to the target white balance parameter, performing white balance processing on an image shot by the electronic equipment for the current shooting scene to obtain a target image.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of computer technology, and in particular, to an image processing method, an electronic device, and a readable storage medium. Background Art

[0002] White balance is an important concept in photography, and its purpose is to restore white objects to white regardless of the light source. In real life, the white objects people see are always white, which is the result of the visual correction of the human eye. However, for an image sensor, it is impossible to accurately judge the color of an image under different color temperatures. Therefore, it is necessary to rely on the white balance function for adjustment.

[0003] Traditional white balance methods mainly include the automatic white balance method based on the gray world method and the automatic white balance method based on the perfect reflection method. The former assumes that the average reflection of all object surfaces in the scene is colorless, and the latter assumes that the brightest point in the image is the white point, that is, the R, G, and B values of the brightest point are all 255. However, when the prerequisite conditions they assume do not hold, such as obvious color cast with large areas of color blocks in the image, or when the brightest point in the image also deviates significantly from white, the processing results of the above methods will show obvious deviations and cannot achieve the effect of white balance adjustment. Summary of the Invention

[0004] Embodiments of the present application provide an image processing method, an electronic device, and a readable storage medium, which predict the scene light source through the response data of the spectral device, and perform automatic white balance for the predicted scene light source without restricting the scene conditions, making the applicable scenarios more extensive.

[0005] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device, and the electronic device includes a spectral device; the method includes: during the shooting process of the electronic device, obtaining target response data output by the spectral device based on the current lighting environment when the electronic device shoots the current shooting scene; matching a light source type for the target response data to obtain a target light source type corresponding to the current lighting environment; determining target white balance parameters corresponding to the target light source type from a mapping relationship including multiple light source types and multiple white balance parameters; and performing white balance processing on the image shot by the electronic device for the current shooting scene according to the target white balance parameters to obtain a target image.

[0007] As described above, the present application combines the target response data output by the spectral device based on the current illumination environment when the electronic device captures the current shooting scene, determines the target light source corresponding to the current illumination environment, and determines the target white balance parameter matching the target light source based on the target light source. Then, the white balance processing is performed on the image captured for the current shooting scene through the white balance parameter. Compared with the traditional white balance method that requires gray points or white points in the original image to calibrate the original image, the embodiment of the present application estimates the light source of the current scene through the output value of the spectral device, and then directly determines the corresponding white balance parameter according to the scene light source, without any restrictions on the shooting scene. Even if there is no white point (such as pure red) in the shooting scene, the accurate white balance parameter can be determined to achieve white balance adjustment in this scene.

[0008] In a possible implementation manner, matching a light source type for the target response data to obtain the target light source type corresponding to the current illumination environment includes: inputting the target response data into a pre-trained light source prediction model to obtain the target light source type corresponding to the current illumination environment; the light source prediction model is trained based on multiple sample response data and multiple sample light source types, and a sample response data is the response data of the spectral device for a sample light source type.

[0009] As described above, the present application can directly call the neural network model during the shooting process to predict the light source of the current illumination environment of the shooting scene, without relying on the white point position information in the shooting scene, and then perform white balance processing on the shooting picture based on the predicted light source type.

[0010] In a possible implementation manner, the light source prediction model includes at least a first-layer classification model and a second-layer classification model. The classification result of the first-layer classification model includes multiple first-type light sources, and the classification result of the second-layer classification model includes multiple second-type light sources. The multiple second-type light sources are subtypes of the multiple first-type light sources. Inputting the target response data into the pre-trained light source prediction model to obtain the target light source type corresponding to the current illumination environment includes: inputting the target response data into the first-layer classification model to predict the first predicted light source type; the first predicted light source type is the light source with the largest prediction weight among the multiple first-type light sources; when the weight corresponding to the first predicted light source type is greater than or equal to the preset weight, inputting the response data into the second-layer classification model to obtain the second predicted light source type, and taking the second predicted light source type as the target light source type.

[0011] As can be seen from the above, in order to avoid inaccurate prediction results of the light source prediction model in such scenarios, the present application can design the structure of the light source prediction model as the structure of a multi-layer classification model. After each layer of the classification model, the electronic device can determine whether to execute the escape program according to the result output by the current layer, that is, no longer use the light source prediction model for light source prediction, so as to improve the robustness of the light source prediction model.

[0012] In a possible implementation, the method further includes: when the weight corresponding to the first predicted light source type is less than a preset weight, determining the target light source type corresponding to the current light environment as a mixed light source type, and performing white balance processing on the image captured by the electronic device for the current shooting scene according to the preset white balance parameters.

[0013] As can be seen from the above, the present application can first call the first layer of the classification model for light source prediction. When the weight distribution of the prediction result output by the first layer model is not concentrated enough (for example, the weights of the light sources of each first type are relatively close or the light source with the largest weight is still less than the preset weight), it is considered that the light source corresponding to the current light environment may be a mixed light source scene, and the escape process is executed.

[0014] In a possible implementation, the current shooting scene includes a shooting object, and the target response data is the response data output by the electronic device based on the current light environment and the material of the shooting object when shooting; the method further includes: constructing a training set; the training set includes the actual response data output by the spectral device when the electronic device shoots in N*M scenarios, N is the number of sample light source types, M is the number of material types of the shooting object, and both N and M are integers greater than 1; using the actual response data in the training set as the model input and the sample light source types in the training set as labels, training a preset neural network model to obtain a light source prediction model.

[0015] As can be seen from the above, through the above training process, the neural network model can have the function of predicting the light source type according to the response data of the spectral device. Compared with estimating the light source through a color sensor (RGB sensor) and color distance, the color information obtained by the color sensor is limited, while the training set data information used to train the neural network model is richer, improving the accuracy of the light source prediction of the neural network model.

[0016] In a possible implementation, constructing a training set includes: obtaining the target actual response data output by the spectral device when the electronic device shoots the target material under N sample light source types; fitting the theoretical response data of the spectral device when the electronic device shoots in N*M scenarios; based on the target actual response data and the theoretical response data, determining the actual response data of the spectral device when the electronic device shoots in N*M scenarios to obtain a training set.

[0017] As described above, after obtaining the target actual response data of the spectral device for the target material, the present application can directly determine the actual response data of the spectral device for other materials by establishing the migration relationship between the target material and other materials and fitting the theoretical response data of the spectral device during shooting in N*M scenarios, without the need to test each of the other materials one by one, and can efficiently and quickly construct a training set.

[0018] In a possible implementation, fitting the theoretical response data of the spectral device when the electronic device shoots in N*M scenarios includes: obtaining the spectral information of each sample light source type among N sample light source types, the spectral reflectance of each material among M materials, and the spectral channel response curve of the spectral device; determining the spectral information received by the spectral device when the electronic device shoots in N*M scenarios according to the spectral information of each sample light source type and the spectral reflectance of each material; and fitting to obtain the theoretical response data of the spectral device when the electronic device shoots in N*M scenarios according to the spectral information received by the spectral device and the spectral channel response curve.

[0019] As described above, after the scenario is determined, the present application can determine the spectral information that the spectral device can actually receive according to the spectral information of the light source and the spectral reflectance of the material in the scenario, and can determine the theoretical output value of the spectral device through this spectral information and the spectral channel response curve of the spectral device.

[0020] In a possible implementation, determining the actual response data of the spectral device when the electronic device shoots in N*M scenarios based on the target actual response data and the theoretical response data includes: determining the deviation between the target theoretical response data and the target actual response data of the spectral device when the electronic device shoots the target material under N sample light source types; and adjusting the theoretical response data of the spectral device when the electronic device shoots M materials under N sample light source types based on the deviation to obtain the actual response data of the spectral device when the electronic device shoots in N*M scenarios.

[0021] As described above, with the light source as the reference, the present application can correspond these N target actual response data to the N target theoretical response data obtained by fitting one by one, determine the deviation between the target actual response data and the target theoretical response data under the same light source, and use this deviation as the deviation between the actual response data and the theoretical response data of the spectral device for any material under this light source. Furthermore, after the electronic device fits the theoretical response data of the spectral device when shooting in N*M scenarios, it can obtain the actual response data of the spectral device when the electronic device shoots in N*M scenarios through the fitting data and the deviation corresponding to different light sources.

[0022] In a possible implementation, the proportion of the area of the solid-color scene in the current shooting scene is greater than or equal to a preset proportion, the proportion of the area of the connected region of the solid-color scene in the current shooting scene is greater than or equal to a preset ratio, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or there is no white point in the current shooting scene.

[0023] As can be seen from the above, the present application can be applied to shooting scenes with a single color and / or no significant white points. For example, the proportion of the area of the solid-color scene in the current shooting scene is greater than or equal to a preset proportion, the proportion of the area of the connected region of the solid-color scene in the current shooting scene is greater than or equal to a preset ratio, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or there is no white point in the current shooting scene, etc.

[0024] In a possible implementation, the method further includes: shooting a color card by an electronic device under multiple light source types to obtain color card images under different light source types; for the color card images under each light source type, performing color adjustment to obtain standard color card images under each light source type; marking that the color distortion degree of the color card image is less than that of the color card image; obtaining the color adjustment parameters between the color card images and the standard color card images under each light source type, and using the color adjustment parameters as white balance parameters to obtain a mapping relationship.

[0025] As can be seen from the above, the present application can record the color adjustment parameters between the color card image and the standard color card image, so that when the electronic device shoots under the same light source, the color adjustment parameters can still be used to perform white balance on the captured image.

[0026] In a second aspect, a method for training a light source prediction model is provided. The method includes: obtaining a training set; the training set includes actual response data output by a spectral device in an electronic device when the electronic device shoots under N*M types of scenes, where N is the number of sample light source types and M is the material of the shooting object, and both N and M are integers greater than 1; using the actual response data in the training set as the model input and the sample light source types in the training set as labels to train a preset neural network model to obtain a light source prediction model; the light source prediction model is used to predict the light source type matched by the response data output by the spectral device based on the lighting environment and the material of the shooting object when the electronic device shoots for any shooting scene.

[0027] In a possible implementation, using the actual response data in the training set as the model input and the sample light source types in the training set as labels to train a preset neural network model to obtain a light source prediction model includes: using the actual response data in the training set as the model input and the sample light source types in the training set as labels to perform iterative training on the preset neural network model until the number of training times is greater than or equal to a preset number, and then determining the obtained light source prediction model.

[0028] As can be seen from the above, through the above training process, the neural network model can be enabled to have the function of predicting the light source type according to the response data of the spectral device. Compared with estimating the light source through a color sensor (RGB sensor) and color distance, the color information obtained by the color sensor is limited, while the training set data information used to train the neural network model is richer, improving the accuracy of the light source prediction of the neural network model. Further, the electronic device can directly call the neural network model during the shooting process to predict the light source of the current lighting environment of the shooting scene, without relying on the white point position information in the shooting scene, and then perform white balance processing on the captured image based on the predicted light source type.

[0029] In a third aspect, there is provided an image processing device, including: a functional unit for performing the image processing method as in the first aspect; wherein, the actions performed by the functional unit are implemented by hardware or by hardware executing corresponding software.

[0030] In a fourth aspect, the present application provides an electronic device, including a memory and a processor, the memory is used for storing a computer program, and the processor is used for calling the computer program to implement the image processing method as in the first aspect.

[0031] In a fifth aspect, the present application provides a computer-readable storage medium, including computer program instructions, which when executed by a processor in an electronic device, implement the image processing method as in the first aspect.

[0032] In a sixth aspect, the present application provides a computer program product, which when running on a computer, causes the computer to execute the image processing method described in the first aspect. The computer may be the above-mentioned electronic device.

[0033] It can be understood that the beneficial effects that can be achieved in the above second to sixth aspects can refer to the beneficial effects in the image processing method described in any one of the ways in the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is an effect diagram of a traditional white balance method for a single-color scene;

[0035] Figure 2 It is a schematic structural diagram of an electronic device provided in some embodiments of the present application;

[0036] Figure 3 It is a schematic diagram of a shooting scene provided in some embodiments of the present application;

[0037] Figure 4 It is one of the flow schematic diagrams of the image processing method provided in some embodiments of the present application;

[0038] Figure 5 Schematic diagram of reflectance of white / grey objects provided for some embodiments of the present application;

[0039] Figure 6 Schematic diagram of white balance target provided for some embodiments of the present application;

[0040] Figure 7 One of the schematic diagrams of the light source prediction model provided for some embodiments of the present application;

[0041] Figure 8 One of the schematic diagrams of the light source prediction model provided for some embodiments of the present application;

[0042] Figure 9 One of the comparison diagrams of white balance effects provided for some embodiments of the present application;

[0043] Figure 10 One of the comparison diagrams of white balance effects provided for some embodiments of the present application;

[0044] Figure 11 One of the comparison diagrams of white balance effects provided for some embodiments of the present application;

[0045] Figure 12 One of the schematic flowcharts of the image processing method provided for some embodiments of the present application;

[0046] Figure 13 One of the schematic flowcharts of the image processing method provided for some embodiments of the present application;

[0047] Figure 14 Schematic diagram of the structure of an image processing device provided for some embodiments of the present application;

[0048] Figure 15 Schematic diagram of the hardware structure of an electronic device provided for some embodiments of the present application. Detailed implementation manners

[0049] Next, the technical solutions of the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. do not necessarily mean different. Also, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" means two or more.

[0050] Before introducing the method provided by the embodiments of the present application in detail, the nouns or terms involved in the embodiments of the present application will be explained first.

[0051] White Balance: An important concept in photography, the purpose of which is to restore white objects to white regardless of any light source. It can be extended to compensate for the color cast phenomenon that occurs when shooting under a specific light source by strengthening the corresponding complementary color. That is, the white balance setting of the camera can calibrate the deviation of the color temperature, and the desired picture effect can be achieved through the white balance during shooting.

[0052] Spectrum: After a composite light passes through a dispersion system (such as a prism, grating, etc.) and is dispersed, the monochromatic lights dispersed are arranged in order of wavelength (or frequency) size, and the full name is the optical spectrum.

[0053] Color temperature: A measurement unit representing the color component contained in light. Theoretically speaking, the blackbody temperature refers to the color presented by an absolute blackbody after being heated from absolute zero (-273 °C). After the blackbody is heated, it gradually changes from black to red, then to yellow, white, and finally emits blue light. When heated to a certain temperature, the spectral components contained in the light emitted by the blackbody are called the color temperature at this temperature, and the measurement unit is "K" (Kelvin).

[0054] For an image sensor, it is usually impossible to accurately judge the color of an image under different color temperatures. For example, the image taken in a room with fluorescent lights will appear greenish, the image taken under incandescent lights indoors will be yellowish, and the image taken in the shadow of sunlight will be bluish.

[0055] However, in real life, whether it is sunny, cloudy, indoors under incandescent lights or fluorescent lights, the white objects people see are always white, which is the result of the visual correction of the human eye. Therefore, for the images generated by the image sensor, the white balance method can be relied on for adjustment.

[0056] Some traditional white balance methods, such as the gray world algorithm, will treat all pixel points in the captured live scene as white points for statistics to obtain the average RGB components of all pixel points (denoted as Ravg, Bavg, Gavg), and then take Gavg / Ravg and Gavg / Bavg as the gain compensations for the R and B channels respectively. Another example is the perfect reflection algorithm, which will treat a part of pixel points with relatively large brightness values in the captured live scene as white points for statistics to obtain the Ravg, Bavg, Gavg of these pixel points, and then take Gavg / Ravg and Gavg / Bavg as the gain compensations for the R and B channels respectively.

[0057] It can be seen that traditional white balance methods have harsh application conditions and require the presence of white / gray objects in the captured scene, or the overall captured scene is not completely uniform (i.e., the overall color difference is large), and there is a high-light area to correctly predict the light source color temperature and then perform correct white balance correction.

[0058] However, in practical applications, many captured scenes do not have a significant white point, such as green plants, red walls, leather bags, wooden tables, blue skies, pennants, etc. At this time, the accuracy of traditional white balance methods needs to be improved.

[0059] As Figure 1 described, when the color in the captured scene is single and there is no significant white point, although traditional white balance processing is used, there is still an obvious color difference between the captured live image and the actual observation effect of the human eye. After placing a white board in this scene and then using traditional white balance processing, the color difference between the captured live image and the actual observation effect of the human eye is significantly reduced.

[0060] In view of the above problems, the embodiments of the present application provide an image processing method, an electronic device, and a readable storage medium, which predict the scene light source through the response data of the spectral device and perform automatic white balance for the predicted scene light source, without restricting the captured scene, making the applicable scene of white balance wider.

[0061] In particular, in the case where there is no significant white point in the captured scene (such as pure red), it is not necessary to refer to the white point in the scene to determine the white balance parameters, so that the present application can still achieve accurate white balance when taking pictures in a scene without a significant white point.

[0062] The following will describe in detail the image processing method provided by the embodiments of the present application with reference to the accompanying drawings. It should be noted that the embodiments of the present application can be borrowed or referenced from each other. For example, for the same or similar steps, they can be referenced from each other between method embodiments, system embodiments, and device embodiments without limitation.

[0063] The image processing method provided in this application can be applied to an electronic device. In some embodiments, the electronic device can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a notebook computer, an Ultra-mobile Personal Computer (UMPC), a handheld computer, a netbook, a Personal Digital Assistant (PDA), a wearable electronic device, a smart watch, etc. This application does not make special limitations on the specific form of the electronic device.

[0064] As Figure 2 shown, the electronic device may include: a spectral device 11, a camera 12, a processor 13, and a display screen 14. Among them, the processor 13 is respectively connected to the spectral device 11, the camera 12, and the display screen 14.

[0065] The spectral device 11 is used to collect spectral information in the captured scene. Specifically, the spectral device 11 is provided with one or more spectral channel arrays, each spectral channel array includes spectral channels arranged in an array, and each spectral channel in one of the spectral channel arrays is used to pass light signals of different wavelengths.

[0066] The spectral device 11 can be a single-point / imaging spectral device, a filter / grating spectroscopy / interference spectral device, a time modulation / snapshot spectral device, a multi-spectral / hyperspectral / ultra-spectral, etc. The embodiments of this application do not limit the type of the specific spectral device.

[0067] The processor 13 can control the camera 12 to take pictures and display the captured images through the display screen 14. When the camera 12 takes pictures, the spectral device 11 can output response data based on the current lighting environment, and the processor 13 can also perform white balance processing on the captured picture according to the response data output by the spectral device 11. The specific processing process can refer to the image processing method described in the following method embodiments and will not be elaborated here.

[0068] Exemplarily, as Figure 3 shown, after the user starts the camera of the electronic device, a preview interface of the camera is displayed on the electronic device, and the preview interface can be an interface after white balance processing.

[0069] In some embodiments, the electronic device may further include an external memory interface, an internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, a sensor module, keys, a motor, an indicator, a camera, and a Subscriber Identification Module (SIM) card interface, etc. The audio module may include, but is not limited to, one or more of the following, such as a speaker, a receiver, a microphone, a headphone interface, etc. The sensor module may include, but is not limited to, one or more sensors, such as a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0070] The processor 13 may include one or more processing units. For example, the processor may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and / or a Neural-network Processing Unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. The processor is the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0071] The display screen 14 can be used to display images, videos, a series of Graphical User Interfaces (GUIs), etc.

[0072] The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to achieve the data storage function. For example, files such as music and videos are saved in the external memory card. The internal memory can be used to store computer-executable program code, and the executable program code includes instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. For example, in this application, the processor makes the electronic device execute the image processing method provided in this application by running the instructions stored in the internal memory.

[0073] The camera 12 is used to capture still images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a Charge Coupled Device (CCD) or a Complementary Metal-Oxide-Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signals in formats such as standard optical primary colors (RGB) and color encoding (YUV).

[0074] In some embodiments, the electronic device may include one or N cameras, where N is a positive integer greater than 1. Among the N cameras, m cameras are disposed under the display screen. The cameras disposed under the display screen can be used as front cameras, and the other cameras can be disposed on the back of the electronic device and used as rear cameras, where m <= N. The image processing method of the embodiments of this application can process the images captured by both the front camera and the rear camera.

[0075] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0076] The following introduces the image processing method provided by the embodiments of this application.

[0077] Figure 4 The flowchart of the image processing method provided by the embodiments of this application may include the following steps:

[0078] S201. During the shooting process of the electronic device, obtain the target response data output by the spectral device based on the current lighting environment when the electronic device shoots the current shooting scene.

[0079] It should be noted that during the process of the user using the electronic device to shoot, the electronic device can perform spectral measurement on the scene to be shot through the spectral device to obtain the environmental lighting information of the scene to be shot. In this process, the spectral device can output a digital response signal based on the lighting environment of the scene to be shot, such as the raw value (RAW), and the electronic device can use this value as the response data of the spectral device.

[0080] From the spectral dimension, if an object evenly reflects the energy of visible light, it can be considered that such an object is "white" or "gray", and the degree of white or gray depends on the reflectivity of the object. As Figure 5 shown, the higher the degree of whiteness of an object, the higher the reflectivity of the object. And if an object is "white" or "gray", the object can evenly reflect the energy of visible light, that is, the object has the same reflectivity for visible light of different wavelengths.

[0081] During the shooting process of the electronic device, the light information of the current shooting scene received by the electronic device is a mixture of the light source and the reflectance ratio of the object material. Therefore, the spectral information (S camera ) received by the spectral device can be regarded as the Hadamard product of the light source spectrum (S source ) and the object material reflectance ratio (R object ):

[0082] S camera = S source * R object

[0083] Correspondingly, the spectral device will generate different response data according to the different spectral information received.

[0084] Under normal circumstances, when the electronic device shoots different shooting scenes, the RAW values output by the spectral device are different. When the electronic device shoots the same scene under different lighting environments, the RAW values output by the spectral device are also different. For example, when the electronic device shoots leather bags of different colors, the RAW values output by the spectral device in the electronic device are different. When the electronic device shoots the same leather bag under sunlight and under an incandescent lamp, the RAW values output by the spectral device in the electronic device are also different.

[0085] As a possible implementation, in response to a user's shooting request, the electronic device turns on the camera and displays a preview screen corresponding to the current shooting scene. The spectral device in the electronic device can output RAW values based on the lighting environment of the current shooting scene. The electronic device can obtain the RAW values output by the spectral device and perform white balance processing on the current preview screen based on the RAW values.

[0086] As another possible implementation, in response to a user's shooting operation, the electronic device takes a picture of the current shooting scene to obtain an initial image. The spectral device in the electronic device can output RAW values based on the lighting environment of the current shooting scene. The electronic device can obtain the RAW values output by the spectral device and perform white balance processing on the initial image based on the RAW values.

[0087] As Figure 6 shown, in a shooting scene without a significant white point, the electronic device (such as a camera) can obtain an unwhite-balanced raw image (denoted as Non-AWB) according to the spectral information received by the spectral device. According to the above introduction, different object materials have different reflectance ratios (denoted as material reflectance), and the color temperature of an object with material reflectance actually observed by the human eye can be used as the image that is expected to be obtained by white balance (i.e., objective white balance). Therefore, after estimating the light source spectrum of the shooting scene, the electronic device can execute the image processing method provided in the embodiments of the present application to perform white balance on the raw image so that the raw image is adjusted to the effect of the customer white balance image that is expected to be obtained.

[0088] In addition, the image processing method of the embodiments of the present application does not limit the specific shooting scene. Optionally, the image processing method provided in the embodiments of the present application can be applied to shooting scenes with a single color and / or without a significant white point. For example, the area ratio of the pure color scene in the current shooting scene is greater than or equal to a preset ratio, the area ratio of the connected region of the pure color scene in the current shooting scene is greater than or equal to a preset ratio, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or there is no white point in the current shooting scene.

[0089] In some embodiments, during the shooting process of the electronic device, the electronic device can perform frame analysis on the current shooting scene. If the area ratio of the pure color scene in the current shooting scene is greater than or equal to a preset ratio, the area ratio of the connected region of the pure color scene in the current shooting scene is greater than or equal to a preset ratio, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or there is no white point in the current shooting scene, the electronic device can execute the image processing method of the embodiments of the present application to perform white balance processing on the image of the current shooting scene. Otherwise, the electronic device can perform white balance processing on the image of the current shooting scene through a traditional white balance method.

[0090] S202. Match a light source type for the target response data to obtain the target light source type corresponding to the current lighting environment.

[0091] As a possible implementation, the electronic device can input the target response data into a pre-trained light source prediction model, output the light source type, and use this light source type as the target light source type corresponding to the current lighting environment.

[0092] It should be noted that the light source prediction model is trained based on multiple sample response data and multiple sample light source types. A sample response data is the response data of the spectral device for a sample light source type. The input of the light source prediction model can be set as the RAW value, and the output is the light source type. The light source prediction model can be any classification model, such as a decision tree model, a random forest model, or a gradient boosting tree model, etc. The specific type of the light source prediction model is not limited in the embodiments of the present application.

[0093] Such as Figure 7 , for the material and light source of the current shooting scene, the spectral device can output the target response data from the spectral response table configured by itself. Among them, different numbers represent different object materials, and different colors represent different light sources. The electronic device inputs the target response data output by the spectral device (denoted as input S) into the light source prediction model, processes the target response data through the hidden layer in the light source prediction model, predicts the light source type matching the target response data, and outputs this light source type (denoted as output L).

[0094] Among them, the target light source type corresponding to the current lighting environment includes but is not limited to sunlight, fluorescence, incandescent lamp, or light-emitting diode (LED), etc.

[0095] In some embodiments, the captured scene is a scene mixed with different types of light sources, such as by the window, inside the train, etc. At this time, the light source corresponding to the lighting environment of the captured scene includes both sunlight and incandescent lamp. To avoid inaccurate prediction results of the light source prediction model in such scenes, the embodiments of the present application can design the structure of the light source prediction model as the structure of a multi-layer classification model. After passing through each layer of the classification model, the electronic device can judge whether to execute the escape program according to the result output by the current layer, that is, no longer use the light source prediction model for light source prediction, so as to improve the robustness of the light source prediction model.

[0096] Such as Figure 8 As shown, the light source prediction model includes at least a first-layer classification model and a second-layer classification model. The classification result of the first-layer classification model includes multiple light sources of the first type, and the classification result of the second-layer classification model includes multiple light sources of the second type. The multiple light sources of the second type are subtypes of the multiple light sources of the first type.

[0097] In practical applications, the labels of the classification results of the first-layer classification model can be set as n types of major light sources, such as natural light sources, artificial light sources, etc. The labels of the classification results of the second-layer classification model can be set as k types of minor light sources, such as sunlight, fluorescence, incandescent lamps, LEDs, etc. The electronic device can first call the first-layer classification model to predict the light source. When the weight distribution of the prediction results output by the first-layer model is not concentrated enough (for example, the weights of the first-type light sources are similar or the weight of the light source with the largest weight is still less than the preset weight), it is considered that the light source corresponding to the current lighting environment may be a mixed light source scenario, and the escape process is executed.

[0098] Optionally, before the electronic device predicts the light source type using the light source prediction model, it can first determine whether the image of the captured scene is a solid color image (the specific determination method can refer to the relevant content of S201 above and will not be elaborated here). If the image of the captured scene is not a solid color image, the electronic device can perform automatic white balance (Auto White Balance, AWB) on the image of the current captured scene through traditional white balance methods. If the image of the captured scene is a solid color image, the electronic device can call the first-layer classification model for light source prediction. Specifically, the electronic device can obtain the RAW value output by the spectral device and preprocess the RAW value (including spectral reconstruction, normalization, and / or differentiation, etc.). Further, the electronic device inputs the preprocessed RAW value into the first-layer classification model. The first-layer classification model determines the matching degree (which can be measured by weight) between each light source of the first type (label 1-n) and the RAW value based on the input RAW value. The electronic device can determine the light source type with the largest predicted weight among multiple light sources of the first type as the first predicted light source type. The electronic device determines whether the weight corresponding to the first predicted light source type is less than the preset weight to determine whether to filter the RAW value based on this determination result. When the weight corresponding to the first predicted light source type is less than the preset weight, the electronic device determines the target light source type corresponding to the current lighting environment as the mixed light source type, that is, determines to filter the RAW value and execute the escape procedure. For example, in this case, the electronic device no longer continues to use the RAW value for light source prediction, but directly performs white balance processing (AWB) on the current image according to the preset white balance parameters, and then executes the subsequent image signal processing (Image Signal Processing, ISP) process. When the weight corresponding to the first predicted light source type is greater than or equal to the preset weight, the electronic device continues to call the second-layer classification model for light source prediction. Specifically, the electronic device continues to input the RAW value output by the spectral device into the second-layer classification model. The second-layer classification model determines the matching degree (which can be measured by weight) between each light source of the second type (label 1-k) and the RAW value based on the input RAW value. The electronic device can determine the light source type with the largest predicted weight among multiple light sources of the second type as the second predicted light source type and use the second predicted light source type as the target light source type. Further, the electronic device can determine the target white balance parameters according to the target light source type and perform white balance processing (AWB) on the current image according to the target white balance parameters, and then execute the subsequent image signal processing (Image Signal Processing, ISP) process.

[0099] As another possible implementation, the electronic device can determine the light source type that matches the target response data from the mapping relationships including multiple response data and multiple light source types based on the target response data, and use this light source type as the target light source type corresponding to the current lighting environment.

[0100] Exemplarily, the electronic device can collect the RAW values of the spectral device under multiple prior light source types, and sequentially construct the mapping relationships including multiple response data and multiple light source types. During the shooting process of the electronic device, the electronic device can query in this mapping relationship according to the target RAW value output by the spectral device based on the current lighting environment to obtain the light source type that matches the target RAW value.

[0101] S203. Determine the target white balance parameter corresponding to the target light source type from the mapping relationships including multiple light source types and multiple white balance parameters.

[0102] As a possible implementation, the electronic device queries the target white balance parameter corresponding to the target light source type from the mapping relationships including multiple light source types and multiple white balance parameters according to the target light source type.

[0103] In some embodiments, in order to obtain the mapping relationships including multiple light source types and multiple white balance parameters, the electronic device can shoot a color card under multiple light source types to obtain color card images under different light source types. For the color card images under each light source type, the electronic device performs color adjustment to obtain standard images under each light source type. Among them, the color distortion degree of the annotated image is less than that of the color card image. Further, the electronic device obtains the color adjustment parameters between the color card images and the standard color card images under each light source type, and uses the color adjustment parameters as white balance parameters to obtain the mapping relationship.

[0104] It should be noted that the color card is the manifestation of the colors existing in nature on a certain material, and the specific material of the color card in the embodiments of the present application is not limited.

[0105] Optionally, for the color card image captured by the electronic device under a certain light source, the operation and maintenance personnel can compare the real color card (that is, the color card observed by the human eye) with the color card image captured by the electronic device. Taking the real color card as the standard, control the electronic device to perform color adjustment on the color card image until there is no obvious visual difference between the color card image adjusted by the electronic device and the real color card image, and obtain the standard color card image under this light source. The electronic device can record the color adjustment parameters between the color card image and the standard color card image, so that when the electronic device shoots under the same light source, it can still use this color adjustment parameter to perform white balance on the captured picture.

[0106] S204. Perform white balance processing on the image captured by the electronic device for the current shooting scene according to the target white balance parameter to obtain a target image.

[0107] As a possible implementation, the target white balance parameter is used to indicate the adjustment of color temperature. The electronic device can adjust the color temperature of the image captured in the current shooting scene according to the target white balance parameter to obtain an adjusted target image to achieve white balance. Among them, compared with the image before adjustment captured in the current shooting scene, the target image is closer to the picture of the current shooting scene actually seen by the human eye.

[0108] As another possible implementation, the target white balance parameter is used for the gain coefficients of the R, G, and B channels. The electronic device can adjust the R, G, and B gain coefficients of the image captured in the current shooting scene according to the target white balance parameter to obtain an adjusted target image to achieve white balance. Similarly, compared with the image before adjustment captured in the current shooting scene, the target image is closer to the picture of the current shooting scene actually seen by the human eye.

[0109] As Figure 9 shown, for the image captured in a large-area solid color scene, the image processed by the traditional white balance method is severely distorted. After placing a whiteboard in the scene, the severe image distortion is improved. By using the image processing method provided in the embodiments of the present application, when the user uses the electronic device to capture a large-area solid color scene under different lighting conditions, the effect is similar to that of the traditional white balance method after placing a whiteboard. Compared with the traditional white balance method without a whiteboard, the white balance result is more accurate, improving the accuracy of the color of the photos taken by the electronic device.

[0110] It can be understood that in the embodiments of the present application, in combination with the target response data output by the spectral device based on the current lighting environment when the electronic device captures the current shooting scene, the target light source corresponding to the current lighting environment is determined, and the target white balance parameter matching it is determined based on this target light source. Then, white balance processing is performed on the image captured for the current shooting scene through this white balance parameter. Compared with the traditional white balance method that requires a gray point or a white point in the original image to calibrate the original image, in the embodiments of the present application, the output value of the spectral device is used to estimate the light source of the current scene, and then the corresponding white balance parameter is directly determined according to the scene light source, without any limitation on the shooting scene. Even if there is no white point (such as pure red) in the shooting scene, the accurate white balance parameter can be determined to achieve white balance adjustment in this scene.

[0111] It should be noted that the image processing method provided in the embodiments of the present application can not only perform white balance during the shooting process of the electronic device, but also perform white balance on the image captured by the electronic device (i.e., perform white balance after the electronic device shoots).

[0112] Exemplarily, when a user uses an electronic device to photograph a certain scene, the electronic device can obtain the RAW value output by the spectral device and perform white balance processing on the current preview screen based on the RAW value, so that the preview screen on the screen becomes closer to the effect observed by the human eye.

[0113] As Figure 10 shown, the user uses mobile phone 1 and mobile phone 2 to photograph a certain green plant scene respectively. Among them, mobile phone 1 uses the traditional white balance method, and mobile phone 2 uses the white balance method of the embodiment of the present application. Mobile phone 1 and mobile phone 2 respectively turn on their respective cameras and display the preview screen. It can be seen from the display results that compared with the preview screen displayed by mobile phone 2, the color of the green plants in the preview screen displayed by mobile phone 1 is seriously distorted and the white balance effect is poor.

[0114] Another exemplarily, for a certain green plant image captured by an electronic device, if the user believes that the color of the green plant image is seriously distorted, the user can instruct the electronic device to perform white balance on the green plant image. Correspondingly, in response to the white balance instruction issued by the user, the electronic device can obtain the RAW value output by the spectral device when the green plant image is captured from the relevant information of the image, and perform white balance processing on the green plant image based on the RAW value to obtain an adjusted green plant image, where the degree of color distortion of the adjusted green plant image is lower than that of the original green plant image.

[0115] As Figure 11 shown, the user selects a certain green plant image from the album of the electronic device and instructs the electronic device to perform white balance on the green plant image. Correspondingly, in response to the white balance instruction issued by the user, the electronic device can obtain the RAW value output by the spectral device when the green plant image is captured from the relevant information of the image, and perform white balance processing on the green plant image based on the RAW value to obtain an adjusted green plant image. By comparing the green plant images before and after adjustment, it can be obtained that the degree of color distortion of the adjusted green plant image is lower than that of the original green plant image.

[0116] In a design, in order to train a light source prediction model, as Figure 12 shown, the image processing method provided by the embodiment of the present application may further include the following steps:

[0117] S301. Construct a training set.

[0118] Among them, the training set includes the actual response data output by the spectral device during shooting for N*M scenarios by the electronic device, where N is the number of sample light source types, M is the type of materials of the shooting object, and both N and M are integers greater than 1. For example, the sample light source types at least include sunlight, fluorescence, incandescent lamp, and LED, etc., and the types of materials of the shooting object at least include whiteboard, green plant, red wall, leather bag, wooden table, and silk banner, etc. Optionally, in the embodiments of the present application, M materials can also be constructed through some preset material color representation databases, such as the Munsell dataset, the ink dataset, etc.

[0119] As a possible implementation manner, the electronic device shoots for N*M scenarios respectively to test the actual response data output by the spectral device, and then obtains the training set.

[0120] As another possible implementation manner, the electronic device can obtain the target actual response data output by the spectral device when the electronic device shoots the target material under N sample light source types. Further, the electronic device fits the theoretical response data of the spectral device when shooting under N*M scenarios, and based on the target actual response data and the theoretical response data, determines the actual response data of the spectral device when shooting under N*M scenarios, and obtains the training set.

[0121] Among them, the target material can be any one of the M material types. For example, the target material can be a whiteboard.

[0122] It can be understood that after obtaining the target actual response data of the spectral device for the target material, in the embodiments of the present application, by establishing the migration relationship between the target material and other materials, and fitting the theoretical response data of the spectral device when shooting under N*M scenarios, the actual response data of the spectral device for other materials can be directly determined, without having to test other materials one by one, and the training set can be constructed efficiently and quickly.

[0123] In some embodiments, in order to fit the theoretical response data of the spectral device when shooting under N*M scenarios, the electronic device can first obtain the spectral information of each sample light source type among the N sample light source types, the spectral reflectance of each material among the M materials, and the spectral channel response curve of the spectral device. Further, the electronic device determines the spectral information received by the spectral device when shooting under N*M scenarios according to the spectral information of each sample light source type and the spectral reflectance of each material, and, according to the spectral information received by the spectral device and the spectral channel response curve, fits to obtain the theoretical response data of the spectral device when shooting under N*M scenarios.

[0124] It should be noted that the spectral channel response curve, also known as the spectral response function, is an inherent system parameter of the spectral device, which is used to reflect the relationship between the incident spectrum of the spectral device and the output value.

[0125] Therefore, after the scene is determined, the electronic device can determine the spectral information that the spectral device can actually receive according to the spectral information of the light source and the spectral reflectance of the material in the scene, and can determine the theoretical output value of the spectral device through this spectral information and the spectral channel response curve of the spectral device.

[0126] Exemplarily, after obtaining the spectral information of each sample light source type in N sample light source types and the spectral reflectance of each material in M materials, according to the formula S in S201 camera = S source * R object , the electronic device can calculate the spectral information received by the spectral device in N * M scenarios. Further, the electronic device substitutes the calculated spectral information into the spectral channel response curve, and can obtain the theoretical response data of the spectral device in the corresponding scenario.

[0127] In some embodiments, in order to obtain the actual response data of the spectral device during shooting in N * M scenarios, the electronic device can determine the deviation between the target theoretical response data and the target actual response data of the spectral device when shooting the target material under N sample light source types, and adjust the theoretical response data of the spectral device when shooting M materials under N sample light source types based on the deviation, so as to obtain the actual response data of the spectral device when shooting in N * M scenarios.

[0128] Optionally, the electronic device can determine the target actual response data output by the spectral device when shooting the target material under N sample light source types through actual shooting. For example, when the electronic device shoots a whiteboard under N sample light source types, N target actual response data output by the spectral device can be obtained.

[0129] Further, with the light source as the reference, the electronic device can correspond these N target actual response data to the N target theoretical response data obtained by fitting one by one, and can determine the deviation between the target actual response data and the target theoretical response data under the same light source, and use this deviation as the deviation between the actual response data and the theoretical response data of the spectral device for any material under this light source. Then, after the electronic device fits the theoretical response data of the spectral device when shooting in N * M scenarios, it can obtain the actual response data of the spectral device when shooting in N * M scenarios through the fitting data and the deviations corresponding to different light sources.

[0130] S302. Use the actual response data in the training set as the model input and the sample light source types in the training set as labels to train a preset neural network model to obtain a light source prediction model.

[0131] As a possible implementation, the electronic device can use the actual response data in the training set as the model input and the sample light source types in the training set as labels to perform iterative training on the preset neural network model until the number of training times is greater than or equal to the preset number of times, and then determine the obtained light source prediction model.

[0132] As another possible implementation, the electronic device can use the actual response data in the training set as the model input and the sample light source types in the training set as labels to perform iterative training on the preset neural network model until the model loss function converges, and then determine the obtained light source prediction model. Among them, the loss function is used to reflect the difference between the predicted light source type output by the model and the sample light source type.

[0133] In some embodiments, to reduce the processing pressure on the electronic device, the above model training process can also be performed on the server. The server trains to obtain a light source prediction model and deploys the trained light source prediction model to the electronic device with a photographing function. Correspondingly, during the photographing process, the electronic device can call the pre-deployed light source prediction model to execute the above image processing process to achieve white balance for the captured scene image.

[0134] In practical applications, such as Figure 13As shown in the figure, the process of constructing the training set may include S1 - S7. The electronic device can first capture the whiteboard under N prior light sources to obtain the actual RAW values output by the spectral device (S1), and determine the actual RAW values output by the N*M scene spectral device (S7) through the theoretical RAW values output by the constructed N*M scene spectral device (S6). To obtain the theoretical RAW values output by the constructed N*M scene spectral device, the electronic device can first construct N*M scenarios (S5). Among them, the N*M scenarios are composed of any combination of N prior light sources and M object materials. The spectral information of different prior light sources is different (S3), and the spectral reflectance of different object materials is different (S4). Further, a monochromator can be used to calibrate the spectral channel response curve of the spectral device in the electronic device (S2). Based on the spectral channel response curve of the spectral device and the constructed N*M scenarios, the theoretical RAW values output by the N*M scene spectral device are fitted. When training the neural network model, the input of the model is set to the RAW value, and the output of the model is set to the light source type, and a light source prediction model is trained (S8). In actual application, the electronic device can input the RAW value output by the spectral device in the previously captured scene into the light source prediction model to obtain the target light source type corresponding to the current lighting environment. The electronic device can pre - construct a mapping relationship including multiple light source types and multiple white balance parameters (S9). The electronic device can determine the target white balance parameter corresponding to the target light source type from this mapping relationship according to the target light source type (S10), and then perform white balance processing on the image captured by the electronic device for the current shooting scene according to the target white balance parameter to obtain the target image.

[0135] It can be understood that through the above - mentioned training process, the neural network model can be equipped with the function of predicting the light source type according to the response data of the spectral device. Compared with estimating the light source through a color sensor (RGB sensor) and color distance, the color information obtained by the color sensor is limited, while the training set data information used to train the neural network model is more abundant, improving the accuracy of the neural network model for light source prediction. Further, the electronic device can directly call this neural network model during the shooting process to predict the light source of the current lighting environment of the shooting scene, without relying on the white - point position information in the shooting scene, and then realize white balance processing on the captured image based on the predicted light source type.

[0136] The above embodiments mainly introduce the solutions provided by the embodiments of the present application from the perspective of devices (equipment). It can be understood that, in order to implement the above methods, the devices or equipment include the corresponding hardware structures and / or software modules for executing each method process, and these hardware structures and / or software modules for executing each method process can constitute an electronic device. Those skilled in the art should easily realize that, in combination with the algorithm steps of the examples described in the embodiments of the present application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described function for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0137] The embodiments of the present application can divide the function modules of the device or equipment according to the above method examples. For example, the device or equipment can correspond to each function and divide each function module, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software function module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0138] In the case of dividing each function module corresponding to each function, Figure 14 shows a schematic structural diagram of an image processing device 40 of a function module. The image processing device 40 of the function module can be used to execute the image processing method provided by the above embodiments. As Figure 14 shown, the image processing device 40 includes an acquisition unit 401, a determination unit 402, and a processing unit 403.

[0139] The acquisition unit 401 is configured to acquire, during the shooting process of the electronic device, the target response data output by the spectral device based on the current illumination environment when the electronic device shoots the current shooting scene. The determination unit 402 is configured to match a light source type for the target response data to obtain the target light source type corresponding to the current illumination environment. The determination unit 402 is further configured to determine, according to the target light source type, the target white balance parameter corresponding to the target light source type from the mapping relationship including multiple light source types and multiple white balance parameters; the processing unit 403 is configured to perform white balance processing on the image shot by the electronic device for the current shooting scene according to the target white balance parameter to obtain a target image.

[0140] In a possible implementation, the determining unit 402 is specifically configured to: input the target response data into a pre-trained light source prediction model to obtain the target light source type corresponding to the current illumination environment; the light source prediction model is trained based on a plurality of sample response data and a plurality of sample light source types, and one sample response data is the response data of the spectral device for a sample light source type.

[0141] In a possible implementation, the light source prediction model includes at least a first-layer classification model and a second-layer classification model. The classification result of the first-layer classification model includes a plurality of first-type light sources, and the classification result of the second-layer classification model includes a plurality of second-type light sources. The plurality of second-type light sources are subtypes of the plurality of first-type light sources. The determining unit 402 is specifically configured to: input the target response data into the first-layer classification model to predict the first predicted light source type; the first predicted light source type is the light source with the largest prediction weight among the plurality of first-type light sources; in the case where the weight corresponding to the first predicted light source type is greater than or equal to the preset weight, input the response data into the second-layer classification model to obtain the second predicted light source type, and use the second predicted light source type as the target light source type.

[0142] In a possible implementation, the determining unit 402 is further configured to: in the case where the weight corresponding to the first predicted light source type is less than the preset weight, determine the target light source type corresponding to the current illumination environment as the mixed light source type, and perform white balance processing on the image captured by the electronic device for the current shooting scene according to the preset white balance parameters.

[0143] In a possible implementation, the current shooting scene includes a shooting object, and the target response data is the response data output by the electronic device based on the current illumination environment and the material of the shooting object during shooting; the processing unit 403 is further configured to: construct a training set; the training set includes the actual response data output by the spectral device when the electronic device shoots in N*M scenarios, where N is the number of sample light source types, M is the number of material types of the shooting object, and both N and M are integers greater than 1; use the actual response data in the training set as the model input and the sample light source types in the training set as labels to train a preset neural network model to obtain the light source prediction model.

[0144] In a possible implementation, the processing unit 403 is specifically configured to: obtain the target actual response data output by the spectral device when the electronic device shoots the target material under N sample light source types; fit the theoretical response data of the spectral device when the electronic device shoots in N*M scenarios; based on the target actual response data and the theoretical response data, determine the actual response data of the spectral device when the electronic device shoots in N*M scenarios to obtain the training set.

[0145] In a possible implementation, the processing unit 403 is specifically configured to: obtain the spectral information of each sample light source type among N sample light source types, the spectral reflectance of each material among M materials, and the spectral channel response curve of the spectral device; determine the spectral information received by the spectral device when the electronic device takes pictures in N*M scenarios according to the spectral information of each sample light source type and the spectral reflectance of each material; and fit the theoretical response data of the spectral device when the electronic device takes pictures in N*M scenarios according to the spectral information received by the spectral device and the spectral channel response curve.

[0146] In a possible implementation, the processing unit 403 is specifically configured to: determine the deviation between the target theoretical response data and the target actual response data of the spectral device when the electronic device takes pictures of the target material under N sample light source types; and adjust the theoretical response data of the spectral device when the electronic device takes pictures of M materials under N sample light source types based on the deviation to obtain the actual response data of the spectral device when the electronic device takes pictures in N*M scenarios.

[0147] In a possible implementation, the proportion of the area of the solid color scene in the current shooting scene is greater than or equal to a preset proportion, the proportion of the area of the connected region of the solid color scene in the current shooting scene is greater than or equal to a preset ratio, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or there is no white point in the current shooting scene.

[0148] In a possible implementation, the processing unit 403 is further configured to: obtain the color card images under different light source types by the electronic device taking pictures of the color card under multiple light source types; perform color adjustment on the color card images under each light source type to obtain the standard color card images under each light source type; label that the color distortion degree of the color card image is less than that of the color card image; obtain the color adjustment parameters between the color card images and the standard color card images under each light source type, and use the color adjustment parameters as the white balance parameters to obtain the mapping relationship.

[0149] In the case of implementing the functions of the above integrated module in the form of hardware, the embodiments of the present application provide a possible structure of the electronic device involved in the above embodiments. As Figure 15 shown, the electronic device 50 includes: a processor 502, a bus 504. In an exemplary implementation, the electronic device 50 may further include a memory 501; optionally, the electronic device 50 may further include a communication interface 503.

[0150] The processor 502 can implement or execute various exemplary logic blocks, modules, and circuits described in connection with the embodiments of the present application. The processor 502 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in connection with the embodiments of the present application. The processor 502 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0151] The communication interface 503 is used to connect to other devices through a communication network. The communication network can be an Ethernet, a radio access network, a wireless local area network (WLAN), etc.

[0152] The memory 501 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), or other types of dynamic storage devices that can store information and instructions. It can also be an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium, or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0153] As a possible implementation, the memory 501 can exist independently of the processor 502. The memory 501 can be connected to the processor 502 through the bus 504 for storing instructions or program code. When the processor 502 calls and executes the instructions or program code stored in the memory 501, the image processing method provided by the embodiments of the present application can be implemented.

[0154] In another possible implementation, the memory 501 can also be integrated with the processor 502.

[0155] The bus 504 can be an extended industry standard architecture (EISA) bus, etc. The bus 504 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0156] Some embodiments of the present application provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions, which, when running on a computer, cause the computer to execute the image processing method described in any one of the above embodiments.

[0157] Exemplarily, the above computer-readable storage medium may include, but is not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical discs (e.g., Compact Discs (CDs), Digital Versatile Discs (DVDs), etc.), smart cards, and flash memory devices (e.g., Erasable Programmable Read-Only Memories (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present application may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0158] An embodiment of the present application provides a computer program product containing instructions, which, when running on a computer, cause the computer to execute the image processing method described in any one of the above embodiments.

[0159] As described above, the above are only the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device including a spectral device; the method includes: During the shooting process of the electronic device, obtaining target response data output by the spectral device based on the current lighting environment when the electronic device shoots the current shooting scene; Matching a light source type for the target response data to obtain a target light source type corresponding to the current lighting environment; According to the target light source type, determining a target white balance parameter corresponding to the target light source type from a mapping relationship including multiple light source types and multiple white balance parameters; Performing white balance processing on an image captured by the electronic device for the current shooting scene according to the target white balance parameter to obtain a target image.

2. The method according to claim 1, wherein The matching a light source type for the target response data to obtain a target light source type corresponding to the current lighting environment includes: Inputting the target response data into a pre-trained light source prediction model to obtain a target light source type corresponding to the current lighting environment; the light source prediction model is trained based on multiple sample response data and multiple sample light source types, and one sample response data is the response data of the spectral device for a sample light source type.

3. The method according to claim 2, characterized in that, The light source prediction model includes at least a first-layer classification model and a second-layer classification model. The classification result of the first-layer classification model includes multiple first-type light sources, and the classification result of the second-layer classification model includes multiple second-type light sources. The multiple second-type light sources are subtypes of the multiple first-type light sources; The inputting the target response data into a pre-trained light source prediction model to obtain a target light source type corresponding to the current lighting environment includes: Inputting the target response data into the first-layer classification model to predict a first predicted light source type; the first predicted light source type is the light source with the largest prediction weight among the multiple first-type light sources; When the weight corresponding to the first predicted light source type is greater than or equal to a preset weight, inputting the response data into the second-layer classification model to obtain a second predicted light source type, and using the second predicted light source type as the target light source type.

4. The method according to claim 3, wherein The method further includes: When the weight corresponding to the first predicted light source type is less than the preset weight, determining the target light source type corresponding to the current lighting environment as a mixed light source type, and performing white balance processing on an image captured by the electronic device for the current shooting scene according to a preset white balance parameter.

5. The method according to claim 2, characterized in that, The current shooting scene includes a shooting object, and the target response data is the response data output by the electronic device during shooting based on the current lighting environment and the material of the shooting object; the method further includes: Constructing a training set; the training set includes the actual response data output by the spectral device when the electronic device shoots in N*M scenarios, N is the number of sample light source types, M is the type of the material of the shooting object, and both N and M are integers greater than 1; Using the actual response data in the training set as the model input and the sample light source types in the training set as labels, train a preset neural network model to obtain the light source prediction model.

6. The method according to claim 5, characterized in that, The constructing of the training set includes: Obtain the target actual response data output by the spectral device when the electronic device captures the target material under the N sample light source types. Fit the theoretical response data of the spectral device when the electronic device captures under the N*M scenarios. Based on the target actual response data and the theoretical response data, determine the actual response data of the spectral device when the electronic device captures under the N*M scenarios, and obtain the training set.

7. The method according to claim 6, wherein The fitting of the theoretical response data of the spectral device when the electronic device captures under the N*M scenarios includes: Obtain the spectral information of each sample light source type among the N sample light source types, the spectral reflectance of each material among the M materials, and the spectral channel response curve of the spectral device. According to the spectral information of each sample light source type and the spectral reflectance of each material, determine the spectral information received by the spectral device when the electronic device captures under the N*M scenarios. According to the spectral information received by the spectral device and the spectral channel response curve, fit to obtain the theoretical response data of the spectral device when the electronic device captures under the N*M scenarios.

8. The method according to claim 6, wherein The determining of the actual response data of the spectral device when the electronic device captures under the N*M scenarios based on the target actual response data and the theoretical response data includes: Determine the deviation between the target theoretical response data and the target actual response data of the spectral device when the electronic device captures the target material under the N sample light source types. Based on the deviation, adjust the theoretical response data of the spectral device when the electronic device captures the M materials under the N sample light source types to obtain the actual response data of the spectral device when the electronic device captures under the N*M scenarios.

9. The method according to any one of claims 1 - 8, characterized in that, The proportion of the area of the pure color scene in the current capture scene is greater than or equal to a preset proportion, the proportion of the area of the connected region of the pure color scene in the current capture scene is greater than or equal to a preset ratio, the number of color categories in the current capture scene is less than or equal to a preset threshold, and / or there is no white point in the current capture scene.

10. The method according to any one of claims 1-8, characterized in that, The method further includes: Capture a color card by the electronic device under the multiple light source types to obtain color card images under different light source types. For the color card images under each light source type, perform color adjustment to obtain standard color card images under each light source type; mark that the color distortion degree of the color card image is less than that of the color card image. Obtain the color adjustment parameters between the color card images and the standard color card images under each light source type, and use the color adjustment parameters as white balance parameters to obtain the mapping relationship.

11. A training method for a light source prediction model, characterized in that, The method includes: Obtain a training set; the training set includes actual response data output by a spectral device in the electronic device when the electronic device takes pictures in N*M scenarios, where N is the number of sample light source types and M is the material of the shooting object, and both N and M are integers greater than 1; Use the actual response data in the training set as the model input and the sample light source type in the training set as the label to train a preset neural network model to obtain a light source prediction model; the light source prediction model is used to predict the light source type matched by the response data output by the spectral device based on the illumination environment and the material of the shooting object when the electronic device takes pictures in any shooting scenario.

12. The method according to claim 11, wherein The step of using the actual response data in the training set as the model input and the sample light source type in the training set as the label to train a preset neural network model to obtain a light source prediction model includes: Use the actual response data in the training set as the model input and the sample light source type in the training set as the label to perform iterative training on the preset neural network model until the number of training times is greater than or equal to the preset number of times, and then determine the obtained light source prediction model.

13. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of any one of claims 1-10 or the steps of any one of claims 11-12.

14. A computer-readable storage medium having computer programs / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, the steps of any one of claims 1-10 or the steps of any one of claims 11-12 are implemented.

15. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of any one of claims 1-10 or the steps of any one of claims 11-12 are implemented.

Images