Image processing methods, electronic devices and readable storage media

Abstract

This application relates to the field of computer technology, providing an image processing method, an electronic device, and a readable storage medium. It predicts scene light sources using response data from a spectral device and performs automatic white balance on the predicted scene light sources, without limiting scene conditions, thus broadening its applicability. The method includes: during the electronic device's shooting process, acquiring target response data output by the spectral device based on the current lighting environment when the electronic device shoots the current scene; matching the target response data to a light source type to obtain the target light source type corresponding to the current lighting environment; determining the target white balance parameter corresponding to the target light source type from a mapping relationship including multiple light source types and multiple white balance parameters; and performing white balance processing on the image captured by the electronic device for the current scene based on the target white balance parameter to obtain the target image.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to an image processing method, an electronic device, and a readable storage medium. Background Technology

[0002] White balance is an important concept in photography, its purpose being to reproduce white objects as white regardless of the light source. In real life, white objects always appear white to us due to the visual correction of the human eye. However, image sensors cannot accurately determine the color of an image at different color temperatures; therefore, white balance is necessary for adjustment.

[0003] Traditional white balance methods mainly include automatic white balance methods based on the gray-world method and automatic white balance methods based on the perfect reflection method. The former assumes that the average reflection of all object surfaces in the scene has no color difference, while the latter assumes that the brightest point in the image is the white point, that is, the R, G, and B values ​​of the brightest point are all 255. However, when these assumptions are not met, such as when there are large areas of color in the image or obvious color casts, or when the brightest point in the image deviates significantly from white, the processing results of the above methods will show obvious deviations and fail to achieve the desired white balance adjustment. Summary of the Invention

[0004] This application provides an image processing method, electronic device, and readable storage medium that predicts scene light sources using response data from a spectral device and performs automatic white balance for the predicted scene light sources, without limiting scene conditions, thus making it applicable to a wider range of scenarios.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an image processing method is provided, applied to an electronic device, the electronic device including a spectral device; the method includes: during the shooting process of the electronic device, acquiring target response data output by the spectral device based on the current lighting environment when the electronic device shoots for the current shooting scene; matching the target response data with a light source type to obtain the target light source type corresponding to the current lighting environment; determining the target white balance parameter corresponding to the target light source type from a mapping relationship including multiple light source types and multiple white balance parameters; and performing white balance processing on the image shot by the electronic device for the current shooting scene based on the target white balance parameter to obtain the target image.

[0007] As can be seen from the above, this application combines the target response data output by the spectral device based on the current lighting environment when the electronic device captures the current shooting scene, determines the target light source corresponding to the current lighting environment, and determines the matching target white balance parameters based on the target light source. Then, white balance processing is performed on the image captured in the current shooting scene using these white balance parameters. Compared to traditional white balance methods that require the presence of gray or white points in the original image for calibration, this application's embodiment estimates the light source of the current scene through the output value of the spectral device, and then directly determines the corresponding white balance parameters based on the scene light source. This has no limitations on the shooting scene; even if there are no white points in the shooting scene (such as pure red), accurate white balance parameters can still be determined, achieving white balance adjustment in that scene.

[0008] In one possible implementation, matching the target response data with the light source type to obtain the target light source type corresponding to the current lighting environment includes: inputting the target response data into a pre-trained light source prediction model to obtain the target light source type corresponding to the current lighting environment; the light source prediction model is trained based on multiple sample response data and multiple sample light source types, and one sample response data is the response data of a spectral device for a sample light source type.

[0009] As can be seen from the above, this application can directly call the neural network model during the shooting process to predict the light source of the current lighting environment of the shooting scene without relying on the white point position information in the shooting scene, thereby realizing white balance processing of the shooting image based on the predicted light source type.

[0010] In one possible implementation, the light source prediction model includes at least a first-layer classification model and a second-layer classification model. The classification result of the first-layer classification model includes multiple light sources of the first type, and the classification result of the second-layer classification model includes multiple light sources of the second type. The multiple light sources of the second type are subtypes of the multiple light sources of the first type. The target response data is input into the pre-trained light source prediction model to obtain the target light source type corresponding to the current lighting environment, including: inputting the target response data into the first-layer classification model to predict a first predicted light source type; the first predicted light source type is the light source with the largest prediction weight among the multiple light sources of the first type; if the weight corresponding to the first predicted light source type is greater than or equal to a preset weight, the response data is input into the second-layer classification model to obtain a second predicted light source type, and the second predicted light source type is used as the target light source type.

[0011] As can be seen from the above, in order to avoid inaccurate prediction results of the light source prediction model in such scenarios, this application can design the structure of the light source prediction model as a multi-layer classification model. After each layer of the classification model, the electronic device can determine whether to execute the escape procedure based on the output of the current layer, that is, no longer use the light source prediction model to predict the light source, thereby improving the robustness of the light source prediction model.

[0012] In one possible implementation, the method further includes: if the weight corresponding to the first predicted light source type is less than the preset weight, determining the target light source type corresponding to the current lighting environment as a mixed light source type, and performing white balance processing on the image captured by the electronic device for the current shooting scene according to the preset white balance parameters.

[0013] As can be seen from the above, this application can first call the first-layer classification model to predict the light source. When the weight distribution of the prediction results output by the first-layer model is not concentrated enough (such as the weights of each first type of light source are small or the light source with the largest weight is still less than the preset weight), it is considered that the light source corresponding to the current lighting environment may be a mixed light source scene, and the escape process is executed.

[0014] In one possible implementation, the current shooting scene includes the shooting object, and the target response data is the response data output by the electronic device based on the current lighting environment and the material of the shooting object when shooting; the method further includes: constructing a training set; the training set includes the electronic device's response data for N... The actual response data output by the spectral device during shooting in M ​​scenarios, where N is the number of sample light source types and M is the material type of the shooting object, and both N and M are integers greater than 1; the actual response data in the training set is used as the model input, and the sample light source types in the training set are used as labels to train the preset neural network model to obtain the light source prediction model.

[0015] As can be seen from the above, through the training process described above, the neural network model can be equipped with the function of predicting the type of light source based on the response data of the spectral device. Compared with light source estimation through color sensors (RGB sensors) and color distance, the color information obtained by color sensors is limited, while the training set data used to train the neural network model is richer in information, thus improving the accuracy of the light source prediction by the neural network model.

[0016] One possible implementation involves constructing a training set, including: acquiring the actual target response data output by a spectral device when an electronic device photographs the target material under N sample light source types; and fitting the electronic device to the N sample light source types. Theoretical response data of the spectral device during shooting in M ​​scenarios; based on the actual response data of the target and the theoretical response data, determine the electronic device in N scenarios. The training set is obtained by taking actual response data of the spectral device under M different scenarios.

[0017] As can be seen from the above, after obtaining the actual response data of the spectral device to the target material, this application establishes the migration relationship between the target material and other materials, and fits N... The theoretical response data of the spectral device when shooting in M ​​different scenarios can directly determine the actual response data of the spectral device for other materials, eliminating the need to test each material individually and enabling the efficient and rapid construction of a training set.

[0018] In one possible implementation, the fitted electronic device is in N The theoretical response data of the spectral device during shooting in M ​​scenarios includes: acquiring the spectral information of each sample light source type among N sample light source types, the spectral reflectance of each material among M materials, and the spectral channel response curves of the spectral device; based on the spectral information of each sample light source type and the spectral reflectance of each material, determining the response data of the electronic device in N scenarios. The spectral information received by the spectrometer during shooting in M ​​different scenarios; based on the spectral information received by the spectrometer and the spectral channel response curves, the electronic device's performance in N scenarios is fitted to obtain the spectral information. Theoretical response data of spectral devices when shooting in M ​​different scenarios.

[0019] As can be seen from the above, after the scene is determined, this application can determine the actual spectral information that the spectral device can receive based on the spectral information of the light source and the spectral reflectance of the material in the scene, and determine the theoretical output value of the spectral device through the spectral information and the spectral channel response curve of the spectral device.

[0020] In one possible implementation, the electronic device's response at N is determined based on the target's actual response data and theoretical response data. The actual response data of the spectral device when shooting in M ​​scenarios includes: determining the deviation between the theoretical target response data and the actual target response data of the spectral device when the electronic device shoots the target material under N sample light source types; adjusting the theoretical response data of the spectral device when the electronic device shoots the M materials under N sample light source types based on the deviation, to obtain the actual response data of the spectral device when the electronic device shoots the M materials under N sample light source types. Actual response data of the spectrometer during shooting in M ​​different scenarios.

[0021] As shown above, this application uses the light source as a reference, and maps the actual response data of these N targets one-to-one with the fitted theoretical response data of the N targets. This allows for the determination of the deviation between the actual response data and the theoretical response data of the targets under the same light source, and this deviation is used as the deviation between the actual response data and the theoretical response data of the spectral device for any material under that light source. Furthermore, the electronic device, after fitting the N... After obtaining the theoretical response data of the spectral device when shooting in M ​​different scenarios, the electronic device's performance in N scenarios can be obtained by fitting the data and the deviations corresponding to different light sources. Actual response data of the spectrometer during shooting in M ​​different scenarios.

[0022] In one possible implementation, the area ratio of solid color scenes in the current shooting scene is greater than or equal to a preset ratio, the area ratio of connected regions of solid color scenes in the current shooting scene is greater than or equal to a preset proportion, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or, there are no white dots in the current shooting scene.

[0023] As can be seen from the above, this application can be applied to shooting scenarios with a single color and / or no significant white spots. For example, the area ratio of the solid color scene in the current shooting scene is greater than or equal to a preset ratio, the area ratio of the connected region of the solid color scene in the current shooting scene is greater than or equal to a preset proportion, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or, there are no white spots in the current shooting scene, etc.

[0024] In one possible implementation, the method further includes: taking pictures of the color card under multiple light source types using an electronic device to obtain color card images under different light source types; adjusting the colors of the color card images under each light source type to obtain standard color card images under each light source type; indicating that the color distortion of the color card images is less than that of the color card images; obtaining the color adjustment parameters between the color card images under each light source type and the standard color card images, and using the color adjustment parameters as white balance parameters to obtain a mapping relationship.

[0025] As can be seen from the above, this application can record the color adjustment parameters between the color chart image and the standard color chart image, so that when the electronic device takes pictures under the same light source, it can still use the color adjustment parameters to perform white balance on the picture.

[0026] Secondly, a training method for a light source prediction model is provided, the method comprising: acquiring a training set; the training set including electronic devices targeting N The actual response data output by the spectral device in the electronic device during shooting in M ​​different scenarios, where N is the number of sample light source types and M is the material of the shooting object, and both N and M are integers greater than 1; the actual response data in the training set is used as the model input, and the sample light source types in the training set are used as labels to train the preset neural network model to obtain the light source prediction model; the light source prediction model is used to predict the light source type matched by the response data output by the spectral device of the electronic device for any shooting scenario based on the lighting environment and the material of the shooting object.

[0027] In one possible implementation, the actual response data in the training set is used as the model input, and the sample light source types in the training set are used as labels to train a preset neural network model to obtain a light source prediction model. This includes: using the actual response data in the training set as the model input and the sample light source types in the training set as labels to iteratively train the preset neural network model until the number of training iterations is greater than or equal to the preset number of iterations, thereby determining the obtained light source prediction model.

[0028] As shown above, through the training process described, the neural network model can predict the type of light source based on the response data of spectral devices. Compared to estimating light sources using color sensors (RGB sensors) and color distance, where the color information obtained by the color sensor is limited, the training set data used to train the neural network model is much richer, thus improving the accuracy of the neural network model's light source prediction. Furthermore, electronic devices can directly call this neural network model during the shooting process to predict the light source in the current lighting environment of the shooting scene, without relying on the white point position information in the shooting scene, thereby achieving white balance processing of the captured image based on the predicted light source type.

[0029] Thirdly, an image processing apparatus is provided, comprising: a functional unit for performing the image processing method of the first aspect; wherein the actions performed by the functional unit are implemented by hardware or by hardware executing corresponding software.

[0030] Fourthly, this application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to implement the image processing method as described in the first aspect.

[0031] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a processor in an electronic device, implement the image processing method as described in the first aspect.

[0032] Sixthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the image processing method as described in the first aspect. The computer may be the aforementioned electronic device.

[0033] Understandably, the beneficial effects that can be achieved by the second to sixth aspects mentioned above can be referred to as the beneficial effects of the image processing methods described in any of the ways in the first aspect, and will not be repeated here. Attached Figure Description

[0034] Figure 1 The image shows the effect of traditional white balance methods in a single-color scene.

[0035] Figure 2 Schematic diagrams of electronic device structures provided for some embodiments of this application;

[0036] Figure 3 This application provides schematic diagrams of shooting scenarios for some embodiments;

[0037] Figure 4 This is one of the schematic flowcharts of an image processing method provided in some embodiments of this application;

[0038] Figure 5 A schematic diagram illustrating the reflectance of a white / gray object provided in some embodiments of this application;

[0039] Figure 6 This is a schematic diagram of white balance targets provided for some embodiments of this application;

[0040] Figure 7 This is one of the schematic diagrams of a light source prediction model provided in some embodiments of this application;

[0041] Figure 8 This is one of the schematic diagrams of a light source prediction model provided in some embodiments of this application;

[0042] Figure 9 This is one of the white balance effect comparisons provided for some embodiments of this application;

[0043] Figure 10 This is one of the white balance effect comparisons provided for some embodiments of this application;

[0044] Figure 11 This is one of the white balance effect comparisons provided for some embodiments of this application;

[0045] Figure 12 This is one of the schematic flowcharts of an image processing method provided in some embodiments of this application;

[0046] Figure 13 This is one of the schematic flowcharts of an image processing method provided in some embodiments of this application;

[0047] Figure 14 This application provides a schematic diagram of the structure of an image processing apparatus according to some embodiments;

[0048] Figure 15 This is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. Furthermore, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., are not necessarily different. Also, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0050] Before providing a detailed description of the methods provided in the embodiments of this application, the nouns or terms involved in the embodiments of this application will be explained.

[0051] White balance is an important concept in photography, aiming to reproduce white objects as white regardless of the light source. It can also be used to compensate for color casts that occur when shooting under specific light sources by enhancing the corresponding complementary color. In other words, the camera's white balance setting can calibrate color temperature deviations, allowing you to achieve the desired image effect during shooting.

[0052] Spectrum: The pattern of monochromatic light scattered by a dispersive system (such as a prism or grating) after polychromatic light is dispersed is called the optical spectrum.

[0053] Color temperature is a unit of measurement that indicates the color components in light. Theoretically, the blackbody temperature refers to the color of an absolute blackbody after it has been heated from absolute zero (-273°C). When heated, a blackbody gradually changes from black to red, then to yellow, then to white, and finally emits blue light. The spectral composition of the light emitted by a blackbody at a certain temperature is called the color temperature at that temperature, and the unit of measurement is "K" (Kelvin).

[0054] Image sensors typically cannot accurately determine the colors of an image at all different color temperatures. For example, an image taken in a room with fluorescent lighting will appear greenish, an image taken under indoor tungsten lighting will appear yellowish, and an image taken in the shade of sunlight will appear bluish.

[0055] However, in real life, regardless of whether it's a sunny day, a cloudy day, or under indoor incandescent or fluorescent lights, white objects always appear white to people; this is a result of visual correction by the human eye. Therefore, images generated by image sensors can be adjusted using white balance methods.

[0056] Traditional white balance methods, such as the gray-world algorithm, treat all pixels in the captured image as white points to calculate the average RGB components of all pixels (denoted as Ravg, Bavg, and Gavg). Then, Gavg / Ravg and Gavg / Bavg are used as gain compensations for the R and B channels, respectively. Another example is the perfect reflection algorithm, which treats a subset of pixels with relatively high brightness values ​​in the captured image as white points to calculate their Ravg, Bavg, and Gavg values. Gavg / Ravg and Gavg / Bavg are then used as gain compensations for the R and B channels, respectively.

[0057] It is evident that traditional white balance methods have stringent applicable conditions. They require the presence of white / gray objects in the scene being photographed, or the scene not being completely uniform (i.e., having a large overall color difference) and the presence of highlight areas in order to accurately predict the color temperature of the light source and thus perform correct white balance correction.

[0058] However, in practical applications, many scenes being photographed do not have significant white spots, such as green plants, red walls, leather bags, wooden tables, blue skies, and banners. In these cases, the accuracy of traditional white balance methods still needs to be improved.

[0059] like Figure 1 As stated, when the scene being photographed is monochromatic and lacks significant white spots, even with traditional white balance processing, the captured live image still exhibits a noticeable color difference compared to the actual visual observation. However, by placing a whiteboard in the scene and then applying traditional white balance processing, the color difference between the captured live image and the actual visual observation is significantly reduced.

[0060] In view of the above problems, this application provides an image processing method, electronic device and readable storage medium, which predicts scene light source by using response data of spectral device and performs automatic white balance for the predicted scene light source, without limiting the shooting scene, making the applicable scenarios of white balance more extensive.

[0061] In particular, in shooting scenes without significant white spots (such as pure red), there is no need to refer to white spots in the scene to determine the white balance parameters, so that this application can still achieve accurate white balance when taking pictures in scenes without significant white spots.

[0062] The image processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that the various embodiments of this application can be mutually referenced or consulted. For example, the same or similar steps, method embodiments, system embodiments and device embodiments can all be referred to each other without limitation.

[0063] The image processing method provided in this application can be applied to electronic devices. In some embodiments, the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, personal digital assistant (PDA), wearable electronic device, smartwatch, etc. This application does not impose any special limitations on the specific form of the electronic device.

[0064] like Figure 2 As shown, the electronic device may include: a spectrometer 11, a camera 12, a processor 13, and a display screen 14. The processor 13 is connected to the spectrometer 11, the camera 12, and the display screen 14.

[0065] The spectral device 11 is used to acquire spectral information in the scene being photographed. Specifically, the spectral device 11 is provided with one or more spectral channel arrays, each spectral channel array including spectral channels arranged in an array, and each spectral channel in one of the spectral channel arrays is used to transmit light signals of different wavelengths.

[0066] The spectroscopic device 11 can be a single-point / imaging spectroscopic device, a filter / grating spectral / interference spectroscopic device, a time-modulated / snapshot spectroscopic device, a multispectral / hyperspectral / hyperspectral device, etc. This application does not limit the specific type of spectroscopic device in its embodiments.

[0067] The processor 13 can control the camera 12 to take pictures and display the captured images on the display screen 14. When the camera 12 takes pictures, the spectral device 11 can output response data based on the current lighting environment. The processor 13 can also perform white balance processing on the captured image based on the response data output by the spectral device 11. The specific processing process can be referred to the image processing method described in the following method embodiment, and will not be repeated here.

[0068] For example, such as Figure 3 As shown, after the user activates the camera on the electronic device, a preview interface of the camera is displayed on the electronic device. This preview interface can be an interface that has undergone white balance processing.

[0069] In some embodiments, the electronic device may further include an external memory interface, internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, a sensor module, buttons, a motor, an indicator, a camera, and a Subscriber Identification Module (SIM) card interface, etc. The audio module may include, but is not limited to, one or more of the following: a speaker, a receiver, a microphone, a headphone jack, etc. The sensor module may include, but is not limited to, one or more sensors: a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0070] Processor 13 may include one or more processing units, such as: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor. Network Processing Unit (NPU), etc. Different processing units can be independent devices or integrated into one or more processors. The processor is the nerve center and command center of an electronic device. The controller generates operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0071] The display screen 14 can be used to display images, videos, a series of graphical user interfaces (GUIs), etc.

[0072] An external memory interface can be used to connect external memory cards, such as Micro SD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the external memory interface to perform data storage functions. For example, music, video, and other files can be saved on the external memory card. Internal memory can be used to store computer executable program code, which includes instructions. The processor executes the instructions stored in the internal memory to perform various functional applications and data processing of the electronic device. For example, in this application, the processor executes the instructions stored in the internal memory to cause the electronic device to perform the image processing method provided in this application.

[0073] Camera 12 is used to capture still images or videos. An optical image of an object is generated through the lens and projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) semiconductor. Oxide Semiconductor (CMOS) phototransistors. The photosensitive element converts light signals into electrical signals, which are then passed to the ISP (Image Signal Processor) for conversion into digital image signals. The ISP outputs the digital image signals to the DSP (Digital Signal Processor) for further processing. The DSP converts the digital image signals into image signals in standard optical color (RGB) or color-coded (YUV) formats.

[0074] In some embodiments, the electronic device may include one or N cameras, where N is a positive integer greater than 1. Of the N cameras, m cameras are positioned below the display screen. The cameras positioned below the display screen can be used as front-facing cameras, and the other cameras can be positioned on the back of the electronic device as rear-facing cameras, where m <= N. The image processing method of this application embodiment can process images captured by both the front-facing camera and the rear-facing camera.

[0075] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0076] The image processing method provided in the embodiments of this application will be described below.

[0077] Figure 4 A flowchart illustrating the image processing method provided in this application embodiment may include the following steps:

[0078] S201. During the shooting process of the electronic device, acquire the target response data output by the spectral device based on the current lighting environment when the electronic device shoots the current shooting scene.

[0079] It should be noted that during the process of taking pictures with an electronic device, the device can use a spectrometer to measure the spectrum of the scene being photographed to obtain ambient lighting information. In this process, the spectrometer can output a digital response signal, such as a raw (RAW) value, based on the lighting environment of the scene being photographed. The electronic device can then use this value as the spectrometer's response data.

[0080] From a spectral perspective, an object that reflects visible light energy equally can be considered "white" or "gray," with the degree of whiteness or grayness depending on the object's reflectivity. For example... Figure 5 As shown, the whiter an object is, the higher its reflectivity. Furthermore, if an object is "white" or "gray," it reflects visible light energy equally, meaning it has the same reflectivity for different wavelengths of visible light.

[0081] During the shooting process of an electronic device, the light information of the current shooting scene received by the electronic device is a mixture of the reflectance of the light source and the object's material. Therefore, the spectral information received by the spectral device ( ) can be regarded as the spectrum of the light source ( ) and the reflectance of the object's material ( The Hadamarda accumulation:

[0082]

[0083] Correspondingly, the spectral device will generate different response data based on the different spectral information received.

[0084] Typically, electronic devices output different RAW values ​​from their spectral sensors when shooting in different scenarios. Similarly, when shooting the same scene under different lighting conditions, the RAW values ​​will also differ. For example, the RAW values ​​output by the spectral sensor will differ when shooting a handbag of different colors. Furthermore, the RAW values ​​will also differ depending on whether the handbag is shot in sunlight or under incandescent light.

[0085] As one possible implementation, in response to a user's shooting request, the electronic device opens its camera and displays a preview of the current shooting scene. The spectral analyzer in the electronic device can output RAW values ​​based on the lighting environment of the current shooting scene. The electronic device can acquire the RAW values ​​output by the spectral analyzer and perform white balance processing on the current preview based on these RAW values.

[0086] As another possible implementation, in response to the user's shooting action, the electronic device captures an image of the current shooting scene to obtain an initial image. The spectral device in the electronic device can output RAW values ​​based on the lighting environment of the current shooting scene. The electronic device can acquire the RAW values ​​output by the spectral device and perform white balance processing on the initial image based on these RAW values.

[0087] like Figure 6 As shown, in shooting scenarios without significant white spots, electronic devices (such as cameras) can obtain the original image without white balance (denoted as Non-AWB) based on the spectral information received by the spectral device. As described above, different object materials have different reflectance ratios (denoted as material reflection), and the color temperature of objects with material reflection actually observed by the human eye can be used as the desired image for white balance (i.e., objective white balance). Therefore, after estimating the light source spectrum of the shooting scene, the electronic device can execute the image processing method provided in this application embodiment to perform white balance on the original image, so that the original image is adjusted to achieve the desired customer white balance image effect.

[0088] Furthermore, the image processing method in this application embodiment is not limited to specific shooting scenarios. Optionally, the image processing method provided in this application embodiment can be applied to shooting scenarios with a single color and / or no significant white spots. For example, the area ratio of the solid color scene in the current shooting scenario is greater than or equal to a preset ratio, the area ratio of the connected region of the solid color scene in the current shooting scenario is greater than or equal to a preset proportion, the number of color categories in the current shooting scenario is less than or equal to a preset threshold, and / or, there are no white spots in the current shooting scenario, etc.

[0089] In some embodiments, during the shooting process of the electronic device, the electronic device can perform image analysis on the current shooting scene. If the area ratio of the solid color scene in the current shooting scene is greater than or equal to a preset ratio, the area ratio of the connected region of the solid color scene in the current shooting scene is greater than or equal to a preset proportion, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or there are no white spots in the current shooting scene, then the electronic device can execute the image processing method of this application embodiment to perform white balance processing on the image of the current shooting scene. Conversely, the electronic device can perform white balance processing on the image of the current shooting scene using traditional white balance methods.

[0090] S202. Match the light source type to the target response data to obtain the target light source type corresponding to the current lighting environment.

[0091] As one possible implementation, electronic devices can input target response data into a pre-trained light source prediction model, output the light source type, and use this light source type as the target light source type corresponding to the current lighting environment.

[0092] It should be noted that the light source prediction model is trained based on multiple sample response data and multiple sample light source types. A single sample response data point represents the response data of a spectral device to a specific light source type. The input to the light source prediction model can be set to RAW values, and the output is the light source type. The light source prediction model can be any classification model, such as a decision tree model, a random forest model, or a gradient boosting tree model. This application does not limit the specific type of light source prediction model in its embodiments.

[0093] like Figure 7 For the materials and light sources of the current shooting scene, the spectral device can output target response data from its configured spectral response table. Different numbers represent different object materials, and different colors represent different light sources. The electronic device inputs the target response data (denoted as input S) from the spectral device into the light source prediction model. The hidden layer in the light source prediction model processes the target response data, predicts the light source type matching the target response data, and outputs that light source type (denoted as output L).

[0094] The target light source type corresponding to the current lighting environment includes, but is not limited to, sunlight, fluorescence, incandescent lamps, or light-emitting diodes (LEDs).

[0095] In some embodiments, the scene being filmed is a scene with a mixture of different types of light sources, such as near a window or inside a train. In this case, the light source corresponding to the lighting environment of the scene being filmed includes both sunlight and incandescent light. To avoid inaccurate prediction results from the light source prediction model in such scenes, embodiments of this application can design the structure of the light source prediction model as a multi-layer classification model. After each layer of the classification model, the electronic device can determine whether to execute the escape procedure based on the output of the current layer, that is, no longer use the light source prediction model for light source prediction, thereby improving the robustness of the light source prediction model.

[0096] like Figure 8 As shown, the light source prediction model includes at least a first-layer classification model and a second-layer classification model. The classification results of the first-layer classification model include multiple light sources of the first type, and the classification results of the second-layer classification model include multiple light sources of the second type. The multiple light sources of the second type are subtypes of the multiple light sources of the first type.

[0097] In practical applications, the classification results of the first-layer classification model can be labeled with n major light source categories, such as natural light sources and artificial light sources. The classification results of the second-layer classification model can be labeled with k minor light source categories, such as sunlight, fluorescence, incandescent lamps, and LEDs. Electronic devices can first call the first-layer classification model to predict the light source. When the weight distribution of the prediction results output by the first-layer model is not concentrated enough (e.g., the weights of each first-type light source are small, or the light source with the largest weight is still less than the preset weight), it is considered that the current lighting environment may correspond to a mixed light source scene, and an escape procedure is executed.

[0098] Optionally, before using the light source prediction model to predict the light source type, the electronic device can first determine whether the image of the scene being photographed is a solid color image (the specific judgment method can be referred to the relevant content of S201 above, and will not be repeated here). If the image of the scene being photographed is not a solid color image, the electronic device can perform automatic white balance (AWB) on the image of the current scene using the traditional white balance method. If the image of the scene being photographed is a solid color image, the electronic device can call the first-layer classification model to predict the light source. Specifically, the electronic device can obtain the RAW value output by the spectral device and preprocess the RAW value (including spectral reconstruction, normalization, and / or differencing, etc.). Further, the electronic device inputs the preprocessed RAW value into the first-layer classification model. Based on the input RAW value, the first-layer classification model determines the degree of matching (which can be measured by weight) between each first-type light source (label 1-n) and the RAW value. The electronic device can determine the light source type with the highest prediction weight among multiple first-type light sources as the first predicted light source type. The electronic device judges whether the weight corresponding to the first predicted light source type is less than the preset weight, and determines whether to filter the RAW value based on the judgment result. If the weight corresponding to the first predicted light source type is less than the preset weight, the electronic device determines the target light source type corresponding to the current lighting environment as a mixed light source type, that is, it determines to filter the RAW value and executes the escape procedure. For example, in this case, the electronic device no longer uses the RAW value for light source prediction, but directly performs white balance processing (AWB) on the current image according to the preset white balance parameters, and then executes the subsequent Image Signal Processing (ISP) process. If the weight corresponding to the first predicted light source type is greater than or equal to the preset weight, the electronic device continues to call the second-layer classification model for light source prediction. Specifically, the electronic device continues to input the RAW value output by the spectral device into the second-layer classification model. Based on the input RAW value, the second-layer classification model determines the degree of matching (which can be measured by weight) between each second type of light source (label 1-k) and the RAW value. The electronic device can determine the light source type with the highest prediction weight among multiple second-type light sources as the second predicted light source type, and use the second predicted light source type as the target light source type. Furthermore, the electronic device can determine the target white balance parameters based on the target light source type, and perform white balance processing (AWB) on the current image according to the target white balance parameters, thereby executing the subsequent image signal processing (ISP) process.

[0099] As another possible implementation, the electronic device can determine the light source type that matches the target response data from a mapping relationship between multiple response data and multiple light source types based on the target response data, and use that light source type as the target light source type corresponding to the current lighting environment.

[0100] For example, an electronic device can acquire RAW values ​​from a spectral device under multiple prior light source types, and sequentially construct a mapping relationship between multiple response data and multiple light source types. During the shooting process of the electronic device, the electronic device can query this mapping relationship based on the target RAW value output by the spectral device according to the current lighting environment to obtain the light source type that matches the target RAW value.

[0101] S203. Based on the target light source type, determine the target white balance parameter corresponding to the target light source type from the mapping relationship between multiple light source types and multiple white balance parameters.

[0102] As one possible implementation, the electronic device queries the target white balance parameter corresponding to the target light source type from a mapping relationship that includes multiple light source types and multiple white balance parameters, based on the target light source type.

[0103] In some embodiments, to obtain a mapping relationship between multiple light source types and multiple white balance parameters, the electronic device can photograph a color chart under multiple light source types to obtain color chart images under different light source types. For each color chart image under a different light source type, the electronic device performs color adjustment to obtain a standard image for that light source type. The color distortion of the standard image is less than that of the color chart image. Further, the electronic device acquires the color adjustment parameters between the color chart images under each light source type and the standard color chart image, and uses these color adjustment parameters as white balance parameters to obtain the mapping relationship.

[0104] It should be noted that the color chart is the representation of colors that exist in nature on a certain material, and the specific material of the color chart is not limited in the embodiments of this application.

[0105] Optionally, for color chart images captured by electronic devices under a certain light source, maintenance personnel can compare the actual color chart (i.e., the color chart as observed by the human eye) with the color chart image captured by the electronic device. Using the actual color chart as the standard, the electronic device can adjust the colors of the color chart image until there is no significant visual difference between the adjusted color chart image and the actual color chart image, thus obtaining the standard color chart image under that light source. The electronic device can record the color adjustment parameters between the color chart image and the standard color chart image, so that when the electronic device captures images under the same light source, it can still use these color adjustment parameters to perform white balance on the captured image.

[0106] S204. Based on the target white balance parameters, perform white balance processing on the image captured by the electronic device for the current shooting scene to obtain the target image.

[0107] As one possible implementation, a target white balance parameter is used to indicate the color temperature adjustment. The electronic device can adjust the color temperature of the image captured in the current shooting scene according to the target white balance parameter, obtaining the adjusted target image to achieve white balance. The target image is closer to the actual scene seen by the human eye compared to the image captured before adjustment in the current shooting scene.

[0108] As another possible implementation, the target white balance parameter is used for the gain coefficients of the R, G, and B channels. The electronic device can adjust the R, G, and B gain coefficients of the image captured in the current shooting scene according to the target white balance parameter to obtain the adjusted target image, thereby achieving white balance. Similarly, compared to the image captured in the current shooting scene before adjustment, the target image is closer to the scene actually seen by the human eye.

[0109] like Figure 9 As shown, images captured in large-area solid-color scenes suffer from severe distortion after processing using traditional white balance methods. Placing a whiteboard in the scene significantly reduces this distortion. The image processing method provided in this application, when used by users with electronic devices to capture large-area solid-color scenes under different lighting conditions, achieves similar results to the traditional white balance method with a whiteboard. Compared to the traditional white balance method without a whiteboard, the white balance result is more accurate, improving the color accuracy of images produced by the electronic device.

[0110] Understandably, this application embodiment combines the target response data output by the spectral device based on the current lighting environment when the electronic device captures the current shooting scene, determines the target light source corresponding to the current lighting environment, and determines the matching target white balance parameters based on the target light source. Then, white balance processing is performed on the image captured in the current shooting scene using these white balance parameters. Compared to traditional white balance methods that require the presence of gray or white points in the original image for calibration, this application embodiment estimates the light source of the current scene through the output value of the spectral device, and then directly determines the corresponding white balance parameters based on the scene light source. It has no restrictions on the shooting scene; even if there are no white points in the shooting scene (such as pure red), accurate white balance parameters can still be determined, achieving white balance adjustment in that scene.

[0111] It should be noted that the image processing method provided in this application embodiment can perform white balance not only during the shooting process of the electronic device, but also on the image captured by the electronic device (i.e., perform white balance after the electronic device has taken the picture).

[0112] For example, when a user takes a picture of a scene using an electronic device, the electronic device can obtain the RAW value output by the spectral device and perform white balance processing on the current preview image based on the RAW value, so that the preview image on the screen is closer to the effect observed by the human eye.

[0113] like Figure 10 As shown, a user takes photos of a green scene using mobile phone 100 and mobile phone 200 respectively. Mobile phone 100 uses a traditional white balance method, while mobile phone 200 uses the white balance method of this embodiment. Mobile phones 100 and 200 each open their respective cameras and display preview images. The results show that, compared to the preview image displayed by mobile phone 200, the colors of the green plants are severely distorted, indicating a poor white balance effect.

[0114] As another example, if a user believes that an image of a green plant captured by an electronic device is severely distorted in color, they can instruct the electronic device to perform white balance on the image. Accordingly, in response to the user's white balance command, the electronic device can obtain the RAW value output by the spectral device when the image was captured from the relevant information of the image, and perform white balance processing on the image based on this RAW value to obtain an adjusted image of the green plant, wherein the degree of color distortion in the adjusted image is lower than that of the original image.

[0115] like Figure 11 As shown, the user selects a green plant image from the electronic device's photo album and instructs the device to perform white balance on the image. Accordingly, in response to the user's white balance command, the electronic device obtains the RAW value output by the spectral analyzer when the image was captured from the image's relevant information, and performs white balance processing based on this RAW value to obtain the adjusted image. Comparing the images before and after adjustment reveals that the adjusted image exhibits less color distortion than the original image.

[0116] In one design, in order to train a light source prediction model, such as Figure 12 As shown, the image processing method provided in this application embodiment may further include the following steps:

[0117] S301. Construct the training set.

[0118] The training set includes electronic devices targeting N The actual response data output by the spectral device during shooting in M ​​different scenarios, where N is the number of sample light source types and M is the material type of the object being photographed, and both N and M are integers greater than 1. For example, the sample light source types include at least sunlight, fluorescence, incandescent lamps, and LEDs, and the material types of the objects being photographed include at least whiteboards, green plants, red walls, leather bags, wooden tables, and pennants. Optionally, embodiments of this application can also construct M material types using some preset material color representation databases, such as Munsell datasets and ink datasets.

[0119] As one possible implementation, the electronic device targets N respectively. M different scenarios were used to capture images in order to test the actual response data output by the spectral device, and thus obtain the training set.

[0120] As another possible implementation, the electronic device can acquire the actual response data of the target material output by the spectral device when the electronic device photographs the target material under N sample light source types. Furthermore, the electronic device can fit the electronic device under N... The theoretical response data of the spectral device were collected during shooting in M ​​scenarios. Based on the actual response data of the target and the theoretical response data, the electronic device's performance in N scenarios was determined. The training set is obtained by taking actual response data of the spectral device under M different scenarios.

[0121] The target material can be any one of the M material types. For example, the target material can be a whiteboard.

[0122] Understandably, after acquiring the actual response data of the spectral device to the target material, this embodiment of the application establishes the migration relationship between the target material and other materials, and fits N. The theoretical response data of the spectral device when shooting in M ​​different scenarios can directly determine the actual response data of the spectral device for other materials, eliminating the need to test each material individually and enabling the efficient and rapid construction of a training set.

[0123] In some embodiments, in order to fit N The electronic device can first acquire the spectral information of each of the N sample light source types, the spectral reflectance of each of the M materials, and the spectral channel response curves of the spectral device when shooting in M ​​different scenarios. Further, based on the spectral information of each sample light source type and the spectral reflectance of each material, the electronic device can determine the theoretical response data of the spectral device in N scenarios. The spectral information received by the spectrometer during shooting in M ​​different scenarios, and the electronic device's performance in N scenarios, are fitted based on the spectral information received by the spectrometer and the spectral channel response curves. Theoretical response data of spectral devices when shooting in M ​​different scenarios.

[0124] It should be noted that the spectral channel response curve, also known as the spectral response function, is an inherent system parameter of the spectral device, used to reflect the relationship between the incident spectrum and the output value of the spectral device.

[0125] Therefore, once the scene is determined, the electronic device can determine the actual spectral information that the spectral device can receive based on the spectral information of the light source in the scene and the spectral reflectance of the material. Then, through this spectral information and the spectral channel response curve of the spectral device, the theoretical output value of the spectral device can be determined.

[0126] For example, after obtaining the spectral information of each of the N sample light source types and the spectral reflectance of each of the M materials, according to the formula in S201... Electronic devices can calculate in N The spectral information received by the spectral device under M scenarios. Furthermore, the electronic device substitutes the calculated spectral information into the spectral channel response curve to obtain the theoretical response data of the spectral device under the corresponding scenario.

[0127] In some embodiments, in order to obtain N The electronic device can determine the deviation between the theoretical and actual response data of the spectral device when shooting the target material under N sample light source types, based on the actual response data of the spectral device in M ​​scenarios. This deviation is then used to adjust the theoretical response data of the spectral device when shooting the target material under N sample light source types, thus obtaining the electronic device's performance under N scenarios. Actual response data of the spectrometer during shooting in M ​​different scenarios.

[0128] Optionally, the electronic device can determine the actual target response data output by the spectral device when photographing the target material under N different sample light source types through actual shooting. For example, the electronic device can obtain N actual target response data output by the spectral device by photographing a whiteboard under N different sample light source types.

[0129] Furthermore, using the light source as a reference, the electronic device maps the actual response data of these N targets one-to-one with the fitted theoretical response data of the N targets. This allows it to determine the deviation between the actual and theoretical response data of the targets under the same light source, and this deviation is used as the difference between the actual and theoretical response data of the spectral device for any material under that light source. Subsequently, the electronic device, after fitting the N... After obtaining the theoretical response data of the spectral device when shooting in M ​​different scenarios, the electronic device's performance in N scenarios can be obtained by fitting the data and the deviations corresponding to different light sources. Actual response data of the spectrometer during shooting in M ​​different scenarios.

[0130] S302. Using the actual response data in the training set as model input and the sample light source types in the training set as labels, train the preset neural network model to obtain the light source prediction model.

[0131] As one possible implementation, the electronic device can use the actual response data in the training set as model input, use the sample light source types in the training set as labels, and iteratively train the preset neural network model until the number of training iterations is greater than or equal to the preset number of iterations, thereby determining the light source prediction model.

[0132] As another possible implementation, the electronic device can use actual response data from the training set as model input, and the sample light source types from the training set as labels to iteratively train a pre-defined neural network model until the model's loss function converges, thus determining the light source prediction model. The loss function reflects the difference between the predicted light source type output by the model and the sample light source type.

[0133] In some embodiments, to reduce the processing load on electronic devices, the model training process described above can also be performed on a server. The server trains the light source prediction model, and then deploys the trained light source prediction model to an electronic device with a camera function. Accordingly, during the shooting process, the electronic device can call the pre-deployed light source prediction model to execute the above image processing process to achieve white balance for the captured scene image.

[0134] In practical applications, such as Figure 13 As shown, the process of constructing the training set can include S1-S7. The electronic device can first photograph a white board under N prior light sources to obtain the actual RAW values ​​output by the spectral device (S1), and then use the constructed N... The theoretical RAW value (S6) output by the M-scene spectral device determines N. The actual RAW value (S7) output by the M scene spectral device. To construct the N... The theoretical RAW value output by the M-scene spectral device can be used by the electronic device to first construct N. M scenarios (S5). Among them, N The M scenarios are composed of any combination of N prior light sources and M object materials. Different prior light sources have different spectral information (S3), and different object materials have different spectral reflectance (S4). Furthermore, a monochromator can be used to calibrate the spectral channel response curves of the spectral devices in the electronic device (S2). Based on the spectral channel response curves of the spectral devices and the constructed N... M scenarios, fitting N The theoretical RAW value output by the spectrometer in scene M. When training the neural network model, the input is set to the RAW value, and the output is set to the light source type, thus training a light source prediction model (S8). In practical applications, the electronic device can input the RAW value output by the spectrometer in the current shooting scene into the light source prediction model to obtain the target light source type corresponding to the current lighting environment. The electronic device can pre-construct a mapping relationship between multiple light source types and multiple white balance parameters (S9). Based on the target light source type, the electronic device can determine the target white balance parameter corresponding to the target light source type from this mapping relationship (S10), and then perform white balance processing on the image captured by the electronic device for the current shooting scene based on the target white balance parameter to obtain the target image.

[0135] Understandably, through the aforementioned training process, the neural network model gains the ability to predict light source types based on the response data of spectral devices. Compared to estimating light sources using color sensors (RGB sensors) and color distance, where the color information obtained by the color sensor is limited, the training set data used to train the neural network model is much richer, thus improving the accuracy of the neural network model's light source prediction. Furthermore, electronic devices can directly call upon this neural network model during the shooting process to predict the light source in the current lighting environment of the shooting scene, without relying on the white point position information in the shooting scene, thereby enabling white balance processing of the captured image based on the predicted light source type.

[0136] The above embodiments mainly describe the solutions provided by the embodiments of this application from the perspective of an apparatus (device). It is understood that, in order to implement the above methods, the apparatus or device includes hardware structures and / or software modules corresponding to the execution of each method flow, and these hardware structures and / or software modules corresponding to the execution of each method flow can constitute an electronic device. Those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the various examples described in the embodiments applied herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a certain function is executed in a hardware or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] This application embodiment can divide the device or equipment into functional modules according to the above method examples. For example, the device or equipment can be divided into functional modules corresponding to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0138] When dividing each function into modules according to its corresponding function. Figure 14 A schematic diagram of the structure of an image processing device 40 with functional modules is shown. This image processing device 40 can be used to execute the image processing methods provided in the above embodiments. Figure 14 As shown, the image processing apparatus 40 includes an acquisition unit 401, a determination unit 402, and a processing unit 403.

[0139] Acquisition unit 401 is used to acquire target response data output by the spectral device based on the current lighting environment when the electronic device captures a picture of the current shooting scene. Determination unit 402 is used to match the target response data with a light source type to obtain the target light source type corresponding to the current lighting environment. Determination unit 402 is also used to determine the target white balance parameter corresponding to the target light source type from a mapping relationship including multiple light source types and multiple white balance parameters. Processing unit 403 is used to perform white balance processing on the image captured by the electronic device for the current shooting scene based on the target white balance parameter to obtain the target image.

[0140] In one possible implementation, the determining unit 402 is specifically used to: input the target response data into a pre-trained light source prediction model to obtain the target light source type corresponding to the current lighting environment; the light source prediction model is trained based on multiple sample response data and multiple sample light source types, and one sample response data is the response data of a spectral device for a sample light source type.

[0141] In one possible implementation, the light source prediction model includes at least a first-layer classification model and a second-layer classification model. The classification result of the first-layer classification model includes multiple light sources of the first type, and the classification result of the second-layer classification model includes multiple light sources of the second type. The multiple light sources of the second type are subtypes of the multiple light sources of the first type. The determining unit 402 is specifically used for: inputting the target response data into the first-layer classification model to predict the first predicted light source type; the first predicted light source type is the light source with the largest prediction weight among the multiple light sources of the first type; if the weight corresponding to the first predicted light source type is greater than or equal to the preset weight, inputting the response data into the second-layer classification model to obtain the second predicted light source type, and using the second predicted light source type as the target light source type.

[0142] In one possible implementation, the determining unit 402 is further configured to: determine the target light source type corresponding to the current lighting environment as a mixed light source type when the weight corresponding to the first predicted light source type is less than the preset weight, and perform white balance processing on the image captured by the electronic device for the current shooting scene according to the preset white balance parameters.

[0143] In one possible implementation, the current shooting scene includes the shooting object, and the target response data is the response data output by the electronic device based on the current lighting environment and the material of the shooting object when shooting; the processing unit 403 is further configured to: construct a training set; the training set includes the electronic device targeting N The actual response data output by the spectral device during shooting in M ​​scenarios, where N is the number of sample light source types and M is the material type of the shooting object, and both N and M are integers greater than 1; the actual response data in the training set is used as the model input, and the sample light source types in the training set are used as labels to train the preset neural network model to obtain the light source prediction model.

[0144] In one possible implementation, the processing unit 403 is specifically used to: acquire the actual target response data output by the spectral device when the electronic device photographs the target material under N sample light source types; and fit the electronic device under N sample light source types. Theoretical response data of the spectral device during shooting in M ​​scenarios; based on the actual response data of the target and the theoretical response data, determine the electronic device in N scenarios. The training set is obtained by taking actual response data of the spectral device under M different scenarios.

[0145] In one possible implementation, the processing unit 403 is specifically used to: acquire the spectral information of each of the N sample light source types, the spectral reflectance of each of the M materials, and the spectral channel response curves of the spectral device; and determine the electronic device's response in the N sample light source types based on the spectral information of each sample light source type and the spectral reflectance of each material. The spectral information received by the spectrometer during shooting in M ​​different scenarios; based on the spectral information received by the spectrometer and the spectral channel response curves, the electronic device's performance in N scenarios is fitted to obtain the spectral information. Theoretical response data of spectral devices when shooting in M ​​different scenarios.

[0146] In one possible implementation, the processing unit 403 is specifically used to: determine the deviation between the theoretical response data and the actual response data of the spectral device when the electronic device photographs the target material under N types of sample light sources; and adjust the theoretical response data of the spectral device when the electronic device photographs M types of materials under N types of sample light sources based on the deviation, thereby obtaining the theoretical response data of the spectral device when the electronic device photographs the target material under N types of sample light sources. Actual response data of the spectrometer during shooting in M ​​different scenarios.

[0147] In one possible implementation, the area ratio of solid color scenes in the current shooting scene is greater than or equal to a preset ratio, the area ratio of connected regions of solid color scenes in the current shooting scene is greater than or equal to a preset proportion, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or, there are no white dots in the current shooting scene.

[0148] In one possible implementation, the processing unit 403 is further configured to: capture color cards under multiple light source types using an electronic device to obtain color card images under different light source types; perform color adjustment on the color card images under each light source type to obtain standard color card images under each light source type; indicate that the color distortion of the color card images is less than that of the color card images; obtain the color adjustment parameters between the color card images under each light source type and the standard color card images, and use the color adjustment parameters as white balance parameters to obtain a mapping relationship.

[0149] When implementing the functions of the integrated modules described above in hardware, this application provides a possible structure for the electronic device involved in the above embodiments. For example... Figure 15 As shown, the electronic device 50 includes a processor 502 and a bus 504. In an exemplary embodiment, the electronic device 50 may further include a memory 501; optionally, the electronic device 50 may further include a communication interface 503.

[0150] Processor 502 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with embodiments of this application. Processor 502 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with embodiments of this application. Processor 502 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0151] Communication interface 503 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0152] The memory 501 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0153] In one possible implementation, the memory 501 can exist independently of the processor 502. The memory 501 can be connected to the processor 502 via a bus 504 and is used to store instructions or program code. When the processor 502 calls and executes the instructions or program code stored in the memory 501, it can implement the image processing method provided in the embodiments of this application.

[0154] In another possible implementation, the memory 501 can also be integrated with the processor 502.

[0155] Bus 504 can be an extended industry standard architecture (EISA) bus, etc. Bus 504 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0156] Some embodiments of this application provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer, cause the computer to perform the image processing method as described in any of the above embodiments.

[0157] For example, the aforementioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this application may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0158] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the image processing method described in any of the above embodiments.

[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method is applied to an electronic device, the electronic device including a spectral device, and the electronic device further including a pre-trained light source prediction model. The light source prediction model includes a first-layer classification model and a second-layer classification model. The classification result of the first-layer classification model includes multiple light sources of a first type, and the classification result of the second-layer classification model includes multiple light sources of a second type. The multiple second-type light sources are subtypes of the multiple first-type light sources. The method includes: During the shooting process of the electronic device, target response data output by the spectral device based on the current lighting environment is acquired when the electronic device shoots for the current shooting scene. The target response data is input into the first-layer classification model to predict the first predicted light source type; the first predicted light source type is the light source with the largest prediction weight among the plurality of first-type light sources. If the weight corresponding to the first predicted light source type is greater than or equal to the preset weight, the response data is input into the second-layer classification model to obtain the second predicted light source type, and the second predicted light source type is used as the target light source type. Based on the target light source type, the target white balance parameter corresponding to the target light source type is determined from a mapping relationship that includes multiple light source types and multiple white balance parameters; Based on the target white balance parameters, the image captured by the electronic device for the current shooting scene is subjected to white balance processing to obtain the target image.

2. The method according to claim 1, characterized in that, The method further includes: If the weight corresponding to the first predicted light source type is less than the preset weight, the target light source type corresponding to the current lighting environment is determined to be a mixed light source type, and the image captured by the electronic device for the current shooting scene is processed for white balance according to the preset white balance parameters.

3. The method according to claim 1, characterized in that, The current shooting scene includes the shooting object, and the target response data is the response data output by the electronic device based on the current lighting environment and the material of the shooting object when shooting; the method further includes: Construct a training set; the training set includes the actual response data output by the spectral device when the electronic device takes pictures in N*M scenarios, where N is the number of sample light source types, M is the material type of the object being photographed, and N and M are both integers greater than 1; The actual response data in the training set is used as the model input, and the sample light source types in the training set are used as labels to train the preset neural network model, thereby obtaining the light source prediction model.

4. The method according to claim 3, characterized in that, The construction of the training set includes: Acquire the actual target response data output by the spectral device when the electronic device photographs the target material under the N sample light source types; Fit the theoretical response data of the spectral device when the electronic device takes pictures in the N*M scenarios; Based on the actual response data of the target and the theoretical response data, the actual response data of the spectral device when the electronic device takes pictures in the N*M scenarios is determined, and the training set is obtained.

5. The method according to claim 4, characterized in that, The fitting of the theoretical response data of the spectral device when the electronic device captures images in the N*M scenarios includes: Obtain the spectral information of each of the N sample light source types, the spectral reflectance of each of the M materials, and the spectral channel response curve of the spectral device; Based on the spectral information of each sample light source type and the spectral reflectance of each material, determine the spectral information received by the spectral device when the electronic device takes pictures in the N*M scenarios; Based on the spectral information received by the spectral device and the spectral channel response curve, the theoretical response data of the spectral device when the electronic device takes pictures in the N*M scenarios is obtained by fitting.

6. The method according to claim 4, characterized in that, The process of determining the actual response data of the spectral device when the electronic device captures images in the N*M scenarios, based on the actual response data of the target and the theoretical response data, includes: Determine the deviation between the theoretical target response data and the actual target response data of the spectral device when the electronic device photographs the target material under N sample light source types; Based on the aforementioned deviation, the theoretical response data of the spectral device of the electronic device is adjusted when shooting M materials under N sample light source types, thereby obtaining the actual response data of the spectral device of the electronic device when shooting under the N*M scenarios.

7. The method according to any one of claims 1-6, characterized in that, The area ratio of solid color scenes in the current shooting scene is greater than or equal to a preset ratio, the area ratio of connected regions of solid color scenes in the current shooting scene is greater than or equal to a preset proportion, the number of color categories in the current shooting scene is less than or equal to a preset threshold, and / or, there are no white spots in the current shooting scene.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: The electronic device captures color charts under multiple light source types to obtain color chart images under different light source types. For color chart images under various light source types, color adjustments are performed to obtain standard color chart images under each light source type; the color distortion of the annotated color chart images is less than that of the standard color chart images. Obtain the color adjustment parameters between the color chart image and the standard color chart image under each light source type, and use the color adjustment parameters as white balance parameters to obtain the mapping relationship.

9. A training method for a light source prediction model, characterized in that, The method includes: Obtain a training set; the training set includes the actual response data output by the spectral device in the electronic device when the electronic device takes pictures in N*M scenarios, where N is the number of sample light source types, M is the material of the object being photographed, and N and M are both integers greater than 1; The actual response data in the training set is used as the model input, and the sample light source types in the training set are used as labels to train the preset neural network model to obtain a light source prediction model. The light source prediction model is used to predict the light source type matched by the response data output by the spectral device based on the lighting environment and the material of the shooting object when the electronic device shoots for any shooting scene. The light source type includes multiple first types. The light source prediction model includes a first-layer classification model and a second-layer classification model. The classification result of the first-layer classification model includes multiple light sources of the first type, and the classification result of the second-layer classification model includes multiple light sources of the second type. The multiple light sources of the second type are subtypes of the multiple light sources of the first type.

10. The method according to claim 9, characterized in that, The step of training a preset neural network model using actual response data from the training set as model input and sample light source types from the training set as labels to obtain a light source prediction model includes: The actual response data in the training set is used as the model input, and the sample light source types in the training set are used as labels. The preset neural network model is iteratively trained until the number of training iterations is greater than or equal to the preset number of iterations, and the light source prediction model is obtained.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of any of the methods of claims 1-8 or any of the methods of claims 9-10.

12. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When executed by a processor, the computer program / instructions implement the steps of any of the methods described in claims 1-8 or any of the methods described in claims 9-10.

13. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instructions implement the steps of any of the methods described in claims 1-8 or any of the methods described in claims 9-10.

Images