Image processing method and electronic equipment

Abstract

The embodiment of the invention is applied to the technical field of image processing, and provides an image processing method and electronic equipment. The electronic device obtains a first image frame and multispectral data corresponding to an area where a target object is located in the first image frame. Wherein the reflectivity of the target object is within a preset reflection interval. Afterwards, the electronic equipment carries out light source prediction on the shooting environment of the first image frame according to the multispectral data corresponding to the area where the target object is located in the first image frame, and light source color information corresponding to the first image frame is obtained. And then, the electronic equipment performs color restoration on the first image frame according to the light source color information corresponding to the first image frame to obtain a second image frame. According to the invention, accurate restoration of the image color can be realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] With the development of electronic device (such as mobile phone) technology, the shooting function of electronic devices has also developed rapidly, and more and more users like to use electronic devices to shoot images (such as photos or videos).

[0003] However, when shooting images, electronic devices may be affected by colored light sources (such as LED lights), causing the images shot by the electronic devices to have color casts, which not only affects the image quality, but also affects the user's shooting experience. Therefore, how to reduce the color cast of the image and achieve accurate restoration of the image color to improve the image quality and the user's shooting experience is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and an electronic device for achieving accurate restoration of image colors, thereby improving image quality and user shooting experience.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, an image processing method is provided, which is applied to an electronic device, in which the electronic device obtains a first image frame and multispectral data corresponding to the area where a target object is located in the first image frame. The reflectivity of the target object is within a preset reflection range. Afterwards, the electronic device can perform a light source prediction on the shooting environment of the first image frame based on the multispectral data corresponding to the area where the target object is located in the first image frame, and obtain the light source color information corresponding to the first image frame. Afterwards, the electronic device can perform color restoration on the first image frame based on the light source color information corresponding to the first image frame, and obtain a second image frame.

[0006] In the embodiment of the present application, since the multispectral data can reflect the reflectivity of the target object and the color information of the light source, if the reflectivity of the target object is within the preset reflection range, it means that the reflectivity of the target object is relatively stable, and the electronic device can perform color restoration on the first image frame according to the color information of the light source predicted by the multispectral data. In this way, accurate restoration of the image color can be achieved, so that the second image frame after color restoration can reflect the real picture content, reducing the occurrence of low image color restoration due to the inability to distinguish the color of the target object and the color of the ambient light source, thereby improving the display effect of the second image frame and enhancing the user's visual experience.

[0007] In a possible implementation of the first aspect, the process of the electronic device predicting the light source color information corresponding to the first image frame may specifically include: the electronic device predicts the light source of the first image frame according to the multispectral data of the target object in the first image frame to obtain the XYZ tristimulus values ​​of the first color space. Afterwards, the electronic device may extract the coordinates of the XYZ tristimulus values ​​to obtain the xy color coordinates. The xy color coordinates are the light source color information of the first color space.

[0008] In the embodiment of the present application, since the XYZ tristimulus value of the first color space includes the light source intensity and the xy color coordinates, and the light source intensity is affected by the distance between the ambient light source and the user. Therefore, in order to ensure that the xy color coordinates are not affected by other factors (i.e., the distance between the ambient light source and the user), the electronic device can extract the coordinates of the XYZ tristimulus value to obtain the xy color coordinates that can characterize the light source color information. In this way, the accurate prediction of the light source color information can be achieved, which provides a basis for the subsequent accurate restoration of the image color.

[0009] In a possible implementation of the first aspect, the process of color restoration of the first image frame by the electronic device may specifically include: the electronic device performs spatial conversion on the light source color information of the first color space to obtain the light source color information of the second color space. The color mode corresponding to the second color space is the same as the color mode corresponding to the first image frame. Afterwards, the electronic device may perform color restoration on the first image frame according to the light source color information of the second color space to obtain the second image frame.

[0010] The first image frame is an RGB image, the first color space is a CIE color space, and the second color space is an RGB color space.

[0011] In the embodiment of the present application, in order to ensure that the light source color information is consistent with the expression form of the first image frame, the electronic device can perform spatial conversion on the light source color information of the CIE color space to obtain the light source color information of the RGB color space. In this way, the accurate restoration of the RGB image can be achieved, so that the second image frame after color restoration can reflect the real picture content, and the situation of low image color restoration due to the inability to distinguish the color of the target object and the color of the ambient light source is reduced, thereby improving the display effect of the second image frame and enhancing the user's visual experience.

[0012] In a possible implementation of the first aspect, the process of the electronic device acquiring multispectral data corresponding to the area where the target object is located in the first image frame may specifically include: the electronic device acquires the multispectral data of the first image frame. Each pixel in the first image frame corresponds to a multispectral data. Afterwards, the electronic device may perform regional segmentation on the first image frame to obtain an image of the area where the target object is located in the first image frame. Afterwards, the electronic device may acquire the multispectral data of the pixel points corresponding to the image of the area where the target object is located from the multispectral data of the first image frame.

[0013] In the embodiment of the present application, the electronic device performs region segmentation on the first image frame to obtain a region image where the target object is located in the first image frame. In this way, the electronic device can conveniently extract multispectral data of the region where the target object is located, providing a basis for subsequent accurate prediction of light source color information.

[0014] In a possible implementation of the first aspect, when the first image frame is a portrait image frame, and the target object of the portrait image frame includes human skin, the process of the electronic device acquiring multispectral data corresponding to the area where the target object is located in the first image frame may specifically include: the electronic device performs regional segmentation on the portrait image frame to obtain a skin mask image corresponding to the area where the human skin is located. Afterwards, the electronic device may acquire multispectral data of pixel points corresponding to the skin mask image from the multispectral data of the portrait image frame.

[0015] In the embodiment of the present application, accurate segmentation of the human skin area can be achieved through regional segmentation, and then multi-spectral data of the area where the human skin is located in the first image frame can be obtained, which provides a basis for subsequent accurate prediction of the light source color information of the human skin area.

[0016] In a possible implementation of the first aspect, the process of the electronic device predicting the light source color information corresponding to the first image frame may specifically include: when the multispectral data of the pixel points in the area where the target object is located in the first image frame meets the preset conditions, the electronic device can perform light source prediction on the shooting environment of the first image frame based on the multispectral data of any pixel point in the area where the target object is located in the first image frame, and obtain the light source color information corresponding to the first image frame.

[0017] The preset conditions include: a difference between multispectral data of any two pixels in the area where the target object is located is less than a preset difference, and / or a ratio between the number of pixels with the same multispectral data and the total number of pixels in the area where the target object is located is greater than a preset ratio.

[0018] In the embodiment of the present application, if the multispectral data of the pixel point in the area where the target object is located in the first image frame meets the preset conditions, it means that there is only a single light source in the shooting environment of the first image frame. Therefore, in order to reduce the waste of computing resources, the electronic device can perform a light source prediction on the shooting environment of the first image frame based only on the multispectral data of any pixel point in the area where the target object is located in the first image frame. In this way, the utilization rate of computing resources can be improved.

[0019] In a possible implementation of the first aspect, the process of the electronic device predicting the light source color information corresponding to the first image frame may also include: when the multispectral data of the pixel points in the area where the target object is located in the first image frame does not meet the preset conditions, the electronic device predicts the light source of the shooting environment of the target object in the first image frame based on the multispectral data of each pixel point in the area where the target object is located, and obtains the light source color information corresponding to the target object in the first image frame. Afterwards, the electronic device performs color restoration on the area where the target object is located in the first image frame based on the light source color information corresponding to the target object in the first image frame, and obtains the second image frame.

[0020] In an embodiment of the present application, if the multispectral data of the pixel points in the area where the target object is located in the first image frame meets the preset conditions, it means that there may be multiple light sources in the shooting environment of the first image frame. Therefore, in order to improve the accuracy of the prediction of the light source color information, the electronic device can predict the light source for the shooting environment of the target object in the first image frame based on the multispectral data of each pixel point in the area where the target object is located. In this way, a basis can be provided for the subsequent accurate restoration of the image color. In addition, the electronic device only predicts the light source for the target object in the first image frame, and can perform color restoration in a targeted manner, further improving the color restoration degree of the first image frame, reducing the occurrence of the situation where the color restoration of the first image frame is affected by the image background interfering with the light source prediction accuracy, and improving the display effect of the first image frame.

[0021] In a possible implementation of the first aspect, the process of the electronic device predicting the light source color information corresponding to the first image frame may specifically include: the electronic device inputs the multispectral data corresponding to the area where the target object is located in the first image frame into a target light source prediction network to obtain the light source color information corresponding to the first image frame. The target light source prediction network is used to predict the relationship between the light source color information of the image frame and the multispectral data corresponding to the area where the target object is located in the image frame.

[0022] In an embodiment of the present application, by predicting the light source color information corresponding to the first image frame through the target light source prediction network, accurate prediction of the light source color information can be achieved, so that the predicted light source color information is close to the actual light source color information, thereby providing a basis for the subsequent color restoration of the first image frame.

[0023] In a possible implementation of the first aspect, the process of training a target light source prediction network by an electronic device may specifically include: the electronic device obtains a first sample data set. The first sample data set includes multiple groups of first sample data, and the first sample data includes multispectral data of the area where the target object is located in the sample image frame and real light source color information, and the acquisition time of the multispectral data is the same as the acquisition time of the real light source color information. For each group of first sample data, the electronic device may input the multispectral data of the area where the target object is located in the sample image frame into a pre-constructed light source prediction network to obtain predicted light source color information. Afterwards, the electronic device may adjust the parameters of the pre-constructed light source prediction network according to the predicted light source color information and the real light source color information to obtain the target light source prediction network.

[0024] In the embodiment of the present application, the target light source prediction network can be trained through the multispectral data of the area where the target object is located in the sample image frame and the real light source color information, which can provide a basis for the subsequent accurate prediction of the light source color information. In addition, since the acquisition time of the multispectral data is the same as the acquisition time of the real light source color information, the situation where the target light source prediction network predicts the light source poorly due to the asynchronous acquisition time can be reduced, thereby improving the accuracy of the light source color information prediction.

[0025] In a possible implementation of the first aspect, before the electronic device obtains the first sample data set, it may also include: the electronic device may obtain multiple sample image frames and multispectral data of each sample image frame from a shooting device, and obtain real light source color information from a spectrometer. The shooting device is a device including a multispectral sensor, the multispectral sensor is used to collect multispectral data of the sample image frame, the spectrometer is placed opposite to the shooting device, and the distance between the spectrometer and the target object is less than a preset distance. Afterwards, for each sample image frame, the electronic device may align the sample image frame with the multispectral data of the sample image frame to obtain the multispectral data corresponding to each pixel in the sample image frame. Afterwards, the electronic device may obtain the multispectral data corresponding to the area where the target object is located in the sample image frame from the multispectral data corresponding to each pixel in the sample image frame.

[0026] In the embodiment of the present application, the electronic device can make each pixel in the sample image frame correspond to a multispectral data by data alignment. In this way, the multispectral data corresponding to the area where the target object is located can be accurately extracted, which provides a basis for the subsequent accurate training of the target light source prediction network.

[0027] In a possible implementation of the first aspect, the color attributes of the shooting environments corresponding to the plurality of sample image frames are different, wherein the color attribute of the shooting environment may include at least one of brightness, hue, and saturation.

[0028] In the embodiment of the present application, the electronic device can collect multiple first sample image frames with different color attributes, that is, the color attributes of the ambient light sources corresponding to different groups of first sample data are different. In this way, the target light source prediction network can predict ambient light sources with different color attributes, ensuring the diversity of light source color information prediction, improving the accuracy of light source prediction, and providing a basis for subsequent restoration of image color.

[0029] In a possible implementation of the first aspect, the above-mentioned multiple sample image frames include multiple image frames including the target object at one position or multiple positions acquired by the photographing device when there is one light source in the shooting environment, and / or, when there are multiple light sources in the shooting environment, image frames including the target object at the same position acquired by the photographing device at different shooting positions.

[0030] In the embodiment of the present application, if there is only one light source in the shooting environment, the camera device can collect multiple image frames of the target object at one position or multiple positions. If there are multiple light sources in the shooting environment, the camera device collects image frames of the target object at the same position at different shooting positions. In this way, the target light source prediction network can perform light source prediction for sample image frames at different shooting positions, ensuring the diversity of light source color information prediction, improving the accuracy of light source prediction, and providing a basis for subsequent restoration of image color.

[0031] In a possible implementation of the first aspect, the process of color restoration of a first image frame by an electronic device may specifically include: the electronic device inputs light source color information corresponding to the first image frame and the first image frame into a target color restoration network to obtain a second image frame; wherein the target color restoration network is obtained by adjusting parameters based on a predicted restored image and a real image during iterative training, the predicted restored image is obtained by color restoration of the image to be restored based on the image to be restored and the light source color information, and the real image is an image not affected by the colored light source.

[0032] In the embodiment of the present application, by restoring the color of the first image frame through the target color restoration network, accurate restoration of the image color can be achieved, so that the first image frame (i.e., the second image frame) after color restoration is more consistent with the real scene, thereby improving the user's shooting experience.

[0033] In a possible implementation of the first aspect, the real image is obtained by color restoration of the image to be restored by a preset processing method. The image to be restored is acquired by a camera. The light source color information is acquired by a spectrometer when the camera acquires the image to be restored.

[0034] In the embodiment of the present application, the real image can be obtained by color restoration of the image to be restored using a preset processing method. In this way, the real restored image can be more consistent with the real shooting scene, that is, the target color restoration network trained by the real image can restore the image consistent with the real shooting scene, thereby improving the user's visual experience.

[0035] In a possible implementation of the first aspect, the real image is captured by a camera device under target light. The target light is used to simulate natural lighting. The image to be restored is rendered based on the real image according to preset color information. The light source color information is preset color information.

[0036] In the embodiment of the present application, since the real image is an image frame captured by the shooting device under the target light, and the target light can simulate natural light. Therefore, the occurrence of color deviation due to light differences can be reduced, so that the real image can display real colors. Afterwards, the electronic device can render the real image according to the preset color information to obtain the image to be restored. In this way, the collection efficiency of the second sample data (i.e., the image to be restored, the light source color information, and the real image) can be improved, thereby improving the training efficiency of the color restoration network.

[0037] In a second aspect, the present application provides an electronic device, comprising a camera, a multispectral sensor, a memory and one or more processors; the camera, the multispectral sensor, the memory and the processor are coupled; the camera is used to capture a first image frame, the multispectral sensor is used to capture multispectral data, the memory is used to store computer program code, and the computer program code comprises computer instructions; when the processor executes the computer instructions, the electronic device executes the method described above.

[0038] In a third aspect, the present application provides a computer-readable storage medium, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method described above.

[0039] In a fourth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method described above.

[0040] In a fifth aspect, a chip is provided, comprising: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method as described above.

[0041] Among them, the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application; Figure 2 A schematic diagram showing a front camera and a rear camera of a mobile phone provided in an embodiment of the present application; Figure 3 A flowchart of an image processing method provided in an embodiment of the present application; Figure 4 A schematic diagram of a mobile phone camera interface provided in an embodiment of the present application; Figure 5 A schematic diagram of an interface for taking photos under a color restoration function provided in an embodiment of the present application; Figure 6 A schematic diagram of a mobile phone performing semantic segmentation on a first image frame provided in an embodiment of the present application; Figure 7 A schematic diagram of multispectral data of a target object provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of a target light source prediction network provided in an embodiment of the present application; Fig. 9 A schematic diagram of a training target light source prediction network provided in an embodiment of the present application; Fig.10 A schematic diagram of a training target color restoration network provided in an embodiment of the present application; Fig.11 A schematic diagram of color restoration for a first image frame provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0044] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0045] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0046] In some embodiments, when an electronic device is shooting, the captured image frames may have color deviation due to some reasons (such as lighting, scene, etc.), resulting in poor shooting effect of the electronic device, thereby affecting the user's shooting experience.

[0047] For example, for portrait shooting scenes, electronic devices may be affected by colored light sources (such as LED lights), causing the captured portrait image frames to have color casts and fail to present portrait images with better viewing experience. In other words, the portrait effect in the portrait image frames captured by the electronic device may differ from the characteristics of the characters in the actual scene. For example, the skin color of the character in the portrait image frame captured by the electronic device is darker than the character's real skin color. For another example, the color of the character's clothes in the portrait image frame captured by the electronic device is different from the real color of the character's clothes (such as a pure white shirt appears grayish white in the image frame). It can be understood that if the portrait image frame generated by the electronic device cannot truly reflect the characteristics of the character, it will affect the presentation effect of the portrait image frame, and thus affect the user's visual experience.

[0048] In some embodiments, in order to restore the color of the image and present the content of the picture realistically, the electronic device may capture the first image frame through an RGB image sensor. The first image frame is an RGB image, and the RGB image may be composed of three color channels. The color channel may be a red (red, R) channel, a green (green, G) channel, and a blue (blue, B) channel. In other words, the electronic device may obtain the color information of the first image frame by superimposing the three color channels on each other. Afterwards, the electronic device may perform color restoration on the first image frame according to the three color channel information corresponding to the first image frame to obtain a second image frame. That is, the second image frame generated by the electronic device is an image frame obtained after color restoration based on the first image frame.

[0049] However, since the electronic device cannot determine whether the color information of the target object in the first image frame is the color information of the target object itself or the color information generated by the ambient light source based on the three color channel information corresponding to the first image frame, this results in the electronic device being unable to restore the color of the target object itself in a targeted manner, that is, unable to accurately restore the color, thereby limiting the color restoration capability of the electronic device and ultimately affecting the user's shooting experience.

[0050] Therefore, in order to improve the color restoration of the image frame and enhance the user's shooting experience, an embodiment of the present application provides an image processing method, in which, in response to the user's shooting operation, the electronic device obtains a first image frame and multispectral data corresponding to the area where the target object is located in the first image frame. Wherein, the reflectivity of the target object is within a preset reflection range. Afterwards, the electronic device can predict the light source of the shooting environment of the first image frame based on the multispectral data corresponding to the area where the target object is located in the first image frame, and obtain the light source color information corresponding to the first image frame. Afterwards, the electronic device can perform color restoration on the first image frame based on the light source color information corresponding to the first image frame to obtain a second image frame.

[0051] In the embodiment of the present application, since the multispectral data can reflect the reflectivity of the target object and the color information of the light source, if the reflectivity of the target object is within the preset reflection range, it means that the reflectivity of the target object is relatively stable, and the electronic device can perform color restoration on the first image frame according to the color information of the light source predicted by the multispectral data. In this way, accurate restoration of the image color can be achieved, so that the second image frame after color restoration can reflect the real picture content, reducing the occurrence of low image color restoration due to the inability to distinguish the color of the target object and the color of the ambient light source, thereby improving the display effect of the second image frame and enhancing the user's visual experience.

[0052] In some examples, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a smart watch, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, and other devices that include a camera and a multi-spectral sensor. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0053] For example, Figure 1 FIG. 2 shows a schematic diagram of the hardware structure of the electronic device 200. Figure 1 As shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 211, a power management module 212, a battery 213, an antenna 1, an antenna 2, a mobile communication module 240, a wireless communication module 250, an audio module 270, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 1-N 293, a display screen 294, and a subscriber identification module (SIM) card interface 1-N 295, etc.

[0054] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0055] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0056] In some embodiments, the electronic device 200 can implement the image processing method provided in this application through the processor 210.

[0057] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 240, the wireless communication module 250, the modem processor and the baseband processor.

[0058] The electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0059] The display screen (or screen) 294 is used to display images, videos, etc. For example, the display screen may be a touch screen, etc. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1. In an embodiment of the present application, the display screen 294 may be used to display a preview interface and a shooting interface, etc. in a video recording mode.

[0060] The electronic device 200 can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0061] ISP is used to process the data fed back by camera 293. For example, when an electronic device takes a photo, the shutter is opened, and light is transmitted to the camera photosensitive element (or image sensor) through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 293. In some embodiments, camera 293 includes a shutter. The shutter is a device in the camera used to control the time that light irradiates the photosensitive element.

[0062] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0063] In some embodiments, the camera 293 may include a lens, which is an optical component for generating an image.

[0064] Exemplarily, the N cameras 293 may include: one or more front cameras and one or more rear cameras. Figure 2 , taking the above-mentioned electronic device 200 as a mobile phone as an example. Figure 2 The (a) interface shows a front camera, such as the front camera 20. Figure 2 The interface (b) in FIG. 1 shows three rear cameras, such as rear camera 21, rear camera 22 and rear camera 23. Of course, the number of cameras in the mobile phone includes but is not limited to the number described in the above embodiment.

[0065] Among them, the above-mentioned N cameras 293 may include one or more of the following cameras: a main camera, a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, a macro camera, a fisheye camera, an infrared camera, a depth camera and a black and white camera.

[0066] In some embodiments, the camera 293 may further include an image sensor, which is an RGB image sensor having three RGB channels. That is, the electronic device 200 can collect an RGB image (ie, the first image frame) through the RGB image sensor.

[0067] The sensor module 280 may include a multi-spectral sensor, etc. The multi-spectral sensor can capture more information about the light source, so that the electronic device 200 can capture image frames with accurate colors. Specifically, the multi-spectral sensor can capture spectral information (or spectral signals) of multiple bands from ultraviolet light to near-infrared light. In other words, the multi-spectral sensor is a device for sensing spectral information in multiple wavelength ranges such as infrared, visible light and ultraviolet light.

[0068] In some embodiments, the multispectral sensor may be used to collect multispectral data when the electronic device 200 captures the first image frame, wherein the multispectral data includes spectral information of multiple bands.

[0069] The image processing method of the embodiment of the present application can be used in a scene where an electronic device takes an image. For example, the electronic device takes a photo through the front camera or rear camera of the electronic device. The method of the embodiment of the present application will be described below by taking the electronic device as a mobile phone as an example. Specifically, Figure 3 As shown, the image processing method may include S301 to S306.

[0070] S301, in response to a user's shooting operation, a mobile phone collects a first image frame and multi-spectral data, wherein the first image frame includes a target object.

[0071] In some embodiments, after detecting the user's shooting operation, the mobile phone can collect a first image frame through an RGB image sensor, and collect multispectral data through a multispectral sensor at the same time. That is to say, the acquisition time of the first image frame is the same as the acquisition time of the multispectral data. Among them, the first image frame may include a target object. The target object is an object with a relatively stable reflectivity, such as an object whose reflectivity is in a preset reflection interval. The preset reflection interval can be pre-set according to the reflectivity of the type to which the target object belongs. The multispectral data may include spectral information of multiple bands. The multiple bands may be infrared, visible light, ultraviolet light, etc.

[0072] For example, the target object may be a human skin area, a facial skin area, a green plant, a paper book with the same color (such as white, yellow, etc.) inner pages, etc. Any object with a relatively stable reflectivity may be used as a target object, and there is no specific limitation. It is understandable that since most of the human face is skin, the target object may also be a facial skin area.

[0073] It should be noted that in order for the subsequent mobile phone to distinguish the multispectral data of the area where the target object is located in the first image frame from the multispectral data of the background area in the first image frame to improve the accuracy of the prediction of the light source color information, the resolution of the multispectral sensor in the mobile phone needs to be greater than or equal to the preset resolution. It can be understood that a high-resolution multispectral sensor can not only obtain more spectral and spatial information, thereby providing more accurate multispectral data, reduce errors and interference, and improve the reliability and accuracy of multispectral data, but also capture richer color information, providing a basis for the subsequent restoration of image color.

[0074] In some embodiments of the present application, it is considered that the reflectivity of objects of different types may vary greatly, while the reflectivity of objects of the same type has certain similarities, that is, the difference between the reflectivity of two objects of the same type will not be too large. Therefore, when the reflectivity of the above-mentioned target object is in a preset reflection interval, the mobile phone can determine that the target object belongs to the type to which the preset reflection interval belongs. In other words, the preset reflection interval corresponds to the type of the target object, and the reflectivity of objects of the same type will all be in the same preset reflection interval. Among them, the preset reflection interval can be pre-set according to the reflectivity of objects of the same type.

[0075] Exemplarily, taking a portrait shooting scene, and the target objects include target object A and target object B as an example, if the reflectivity of target object A is in the preset reflection range corresponding to the human skin area, and the reflectivity of target object B is in the preset reflection range corresponding to the human skin area, it means that the difference between the reflectivity of target object A and the reflectivity of target object B is small, that is, the reflectivity of target object A and the reflectivity of target object B are similar. Therefore, the mobile phone can determine that the target object A and the target object B are the same type of objects, that is, the target object A and the target object B are both human skin.

[0076] In some cases, taking the above target object as human skin as an example, the reflectivity of different human skin has certain similarities. However, considering that the skin color of different users may be different, that is, the user may have yellow skin or white skin. The reflectivity of human skin of different skin colors may have large differences. For example, the difference between the reflectivity of white skin and the reflectivity of black skin is greater. Therefore, the mobile phone can set different preset reflection intervals for human skin of different skin colors.

[0077] In one case, the shooting operation may be an operation in which the user clicks a shooting control in a shooting mode, for example, see Figure 4 , the phone is displaying Figure 4The first photographing interface shown in (a). The first photographing interface includes a photographing control 4A. If the photographing control 4A is clicked by the user, it means that the user wants to take a photo in the current scene, that is, the user triggers the photographing operation. Therefore, the mobile phone can photograph the current scene in the photographing mode to obtain the first image frame. In one example, the mobile phone can automatically turn on the color restoration function in the default photographing mode. That is, in response to the user's photographing operation in the photographing mode, the mobile phone can perform color restoration on the collected image frames so that the generated image frames meet the user's viewing experience. In another example, the mobile phone can not turn on the color restoration function in the default photographing mode. That is, in response to the user's photographing operation in the photographing mode, the mobile phone can directly store the collected image frames in the album, that is, the mobile phone does not need to perform color restoration on the collected image frames.

[0078] In another case, the shooting operation may also be an operation in which the user clicks a shooting control in a shooting mode in which the color restoration function is turned on. For example, see Figure 5 , user C holds the mobile phone to shoot user D, that is, user C points the mobile phone lens toward user D to shoot, that is, the mobile phone can display Figure 5 The camera interface in the camera mode shown in (a) of the interface. The camera interface in the camera mode includes a preview image and a "more" control. The preview image includes the user D and the light source 501. Afterwards, when a user touches the "more" control, the mobile phone can display a Figure 5 The setting interface shown in interface (b) in the figure. The setting interface includes multiple function controls, such as a "slow motion" control, a "delay" control, and a "color restoration" control. Afterwards, when a user touches the "color restoration" control, the mobile phone can turn on the color restoration function in response to the touch operation, and after the color restoration function is turned on, the mobile phone can display Figure 5 The photo taking interface under the color restoration function shown in the interface (c) in the figure. The photo taking interface under the color restoration function may include a preview image and a photo taking control 5A. Afterwards, when a user touches the photo taking control 5A, the mobile phone may use the color restoration function to take a photo in the scene in response to the touch operation to generate a color restored image frame (i.e., Figure 5 (d) The image 510 is included in the interface. Afterwards, the mobile phone can save the color restored image frame to the mobile phone album so that the user can view the color restored image frame.

[0079] In another case, the shooting operation may also be an operation in which the user clicks on a recording control in a video recording mode. That is, if a user clicks on a recording control, the mobile phone may perform a video recording operation to generate a corresponding video file. It is understood that the video file may be composed of multiple image frames. That is, if a user clicks on a recording control, the mobile phone may perform color restoration on each captured image frame, and obtain a video file based on the multiple image frames after color restoration. That is, each image frame in the video file is a color restored image frame.

[0080] In some embodiments, if the target object is a user, that is, the shooting scene of the mobile phone is a portrait scene, the shooting operation may also be an operation in which the user clicks a photo control in the portrait mode. For example, please refer to Figure 4 , the phone is displaying Figure 4 The second photo taking interface shown in (b) of FIG. 4 includes a photo taking control 4B. If the photo taking control 4B is clicked by the user, the mobile phone can take a photo of the current scene in the portrait mode to obtain a first image frame.

[0081] It can be understood that the first image frame is an image frame including the target object directly generated by the mobile phone camera after shooting, that is, the first image frame is an image frame that has not been color restored. Exemplarily, the camera can be the front camera or the rear camera of the mobile phone. The number of cameras used by the mobile phone to shoot the first image frame can be one or more, and there is no specific limitation.

[0082] Correspondingly, the above-mentioned first image frame can also be an image frame including the target object stored in the mobile phone album, that is, the first image frame can be an image frame previously captured by the mobile phone camera, or an image frame downloaded by the mobile phone from the target application (application, APP), etc., without specific limitation. Exemplarily, the target APP can be a social type APP, a news type APP, an entertainment type APP, etc. It should be noted that if the first image frame is an image frame stored in the mobile phone album, the multispectral data of the first image frame can be the data carried by the first image frame, that is, the first image frame can be stored in the mobile phone album together with the multispectral data.

[0083] In one implementation, the mobile phone will perform color restoration on the first image frame captured only when receiving the user's shooting operation. In other words, the mobile phone will not perform color restoration on the preview image when displaying the preview image. In this way, unnecessary power consumption loss can be reduced and the usage time of the mobile phone can be increased.

[0084] S302: The mobile phone aligns the first image frame and the multispectral data to obtain the multispectral data corresponding to the first image frame.

[0085] In some embodiments, after acquiring the first image frame and the multispectral data, the mobile phone can align the first image frame and the multispectral data to obtain the multispectral data corresponding to the first image frame. In this way, each pixel in the first image frame corresponds to a piece of multispectral data, thereby providing a basis for the subsequent prediction of the light source color information.

[0086] Specifically, after obtaining the multispectral data corresponding to the first image frame, the mobile phone may obtain the multispectral data of the area where the target object in the first image frame is located from the multispectral data corresponding to the first image frame.

[0087] S303, the mobile phone performs region segmentation on the first image frame to obtain a region image where the target object is located.

[0088] In some embodiments, after acquiring the first image frame, the mobile phone may segment the area where the target object is located in the first image frame to obtain an area image where the target object is located. The area image where the target object is located is used to characterize the area where the target object is located in the first image frame.

[0089] Specifically, the mobile phone can perform region segmentation on the first image frame according to the region segmentation network to obtain a region image where the target object is located. The region image can be represented by a mask image. The mask image is an image used to mark and segment a specific target or region in an image. Specifically, the mobile phone inputs the first image frame into the region segmentation network to obtain a mask image of the target object.

[0090] In some cases, if the first image frame is a portrait image frame, that is, the first image frame includes a target object of human skin, the mobile phone can perform regional segmentation on the portrait image frame to obtain a skin mask image corresponding to the area where the human skin is located. Alternatively, if the first image frame is a portrait image frame including a target object of facial skin, the mobile phone can perform regional segmentation on the portrait image frame to obtain a facial mask image corresponding to the area where the facial skin is located.

[0091] In other cases, if the first image frame is a green plant image frame, that is, the first image frame includes a target object of green plants, the mobile phone can perform region segmentation on the green plant image frame to obtain a mask image corresponding to the region where the green plants are located.

[0092] For example, Figure 6As shown, the first image frame 601 is an image frame including a target object of human skin. The mobile phone inputs the first image frame 601 into the region segmentation network, and can obtain a skin mask image 602 corresponding to the region where the human skin is located. It can be understood that the skin mask image 602 replaces the background and clothes in the first image frame 601 with black, and replaces the face and skin in the first image frame 601 with white. The size of the skin mask image 602 is the same as that of the first image frame 601.

[0093] It can be understood that the mobile phone can separate the foreground and background of the first image frame by performing region segmentation on the first image frame, that is, realizing the cutout technology for separating the target object from the background, which provides a basis for the subsequent mobile phone to predict the light source color information.

[0094] It should be noted that the above data alignment process and the region segmentation process can be performed simultaneously or in a sequence, and there is no specific limitation. For example, the mobile phone can perform steps S302 and S303 simultaneously. For another example, the mobile phone can first perform step S303 and then perform step S302.

[0095] S304: The mobile phone obtains multispectral data corresponding to the regional image of the target object from the multispectral data corresponding to the first image frame.

[0096] Specifically, after obtaining the regional image of the target object, the mobile phone can extract the multispectral data of the target object from the multispectral data adapted to the first image according to the regional image, that is, extract the multispectral data of the pixel points in the region where the target object is located in the first image frame. For example, taking the target object as human skin as an example, the mobile phone can only extract the multispectral data of the region where the human skin is located in the first image frame from the multispectral data.

[0097] It should be noted that the present application takes into account that multispectral data can reflect the reflectivity of the target object and the light source color information of the shooting environment, that is, the multispectral data can be the product of the reflectivity and the light source color information. Therefore, if the reflectivity of the target object in the above-mentioned first image frame is relatively stable, the mobile phone can use the target object as a reference object with a known reflectivity to predict the light source color information corresponding to the first image frame based on the multispectral data of the area where the target object is located in the first image frame. In this way, the accuracy of the light source color information prediction can be improved, thereby providing a basis for the subsequent restoration of the color of the first image frame.

[0098] For example, the multispectral data of the target object is Figure 7 The multispectral data shown is used as an example. Figure 7Curve a in FIG. 1 is used to represent the multispectral data of band 1, curve b is used to represent the multispectral data of band 2, and curve c is used to represent the multispectral data of band 3. In other words, the multispectral data of the target object may include the multispectral data of band 1, the multispectral data of band 2, and the multispectral data of band 3. It can be seen that Figure 7 The horizontal axis of each curve is the band length, and the vertical axis is the band intensity.

[0099] In some embodiments, after acquiring the multispectral data of the target object, the mobile phone can delete all the multispectral data corresponding to the first image frame except the multispectral data of the target object. That is, the mobile phone only includes the multispectral data of the target object in the first image frame. In this way, the situation where useless multispectral data occupies the mobile phone memory can be reduced, and the utilization rate of the mobile phone memory space can be improved.

[0100] S305 , the mobile phone predicts the light source of the shooting environment of the first image frame according to the multispectral data corresponding to the regional image of the target object, and obtains the light source color information corresponding to the first image frame.

[0101] Specifically, after obtaining the multispectral data corresponding to the regional image of the target object, the mobile phone can predict the light source of the shooting environment of the first image frame according to the multispectral data of the target object to obtain the light source color information corresponding to the first image frame. The light source color information is used to characterize the light source of the environment in which the target object is located when the mobile phone shoots the first image frame, that is, the light source of the shooting environment of the first image frame. The light source color information can be characterized by xy chromaticity coordinates. The xy chromaticity coordinates are used to characterize the color information of the ambient light source. The xy chromaticity coordinates are a way of representing color in the CIE color space (or first color space). The CIE color space is used to formulate a standard for color measurement. The CIE color space can define color according to the three primary colors (X, Y, Z), or according to the human eye's perception of color (i.e., brightness (L), red component (a), and green component (b)).

[0102] In some cases, the mobile phone predicts the light source of the shooting environment of the first image frame based on the multispectral data corresponding to the regional image of the target object, and can obtain the XYZ tristimulus values ​​(tristimulus values) of the CIE color space. Among them, the XYZ tristimulus values ​​are used to characterize the degree of stimulation of the three primary colors by the human eye. The XYZ tristimulus values ​​may include light source intensity and xy color coordinates. The light source intensity is affected by the distance between the ambient light source and the user, that is, the closer the distance between the ambient light source and the user, the stronger the light source intensity. Therefore, in order to ensure that the above-mentioned xy color coordinates are not affected by other factors (i.e., the distance between the ambient light source and the user), the mobile phone can extract the coordinates of the XYZ tristimulus values ​​to obtain the xy color coordinates, that is, to obtain the light source color information in the CIE color space.

[0103] In one implementation, the mobile phone can determine the number of light sources present in the shooting environment of the first image frame based on the multispectral data of the pixel points in the regional image. Afterwards, when the number of light sources is 1, the mobile phone can predict the light source of the shooting environment of the first image frame based on the multispectral data of any pixel point in the regional image, and obtain a light source color information that can characterize the first image frame. In this way, the waste of computing resources can be reduced, thereby improving the utilization rate of computing resources. In the case of multiple light sources, the mobile phone can predict the light source of the shooting environment of the first image frame based on the multispectral data of each pixel point in the regional image, and obtain the light source color information of each pixel point of the target object in the first image frame, that is, obtain multiple light source color information that can characterize the target object in the first image frame. In this way, the accuracy of the light source color information prediction can be improved, providing a basis for the subsequent accurate restoration of image color.

[0104] In some embodiments, the mobile phone can determine whether the multispectral data of the pixel points in the above-mentioned regional image meets the preset conditions. If the multispectral data of the pixel points in the regional image meets the preset conditions, it means that there is only a single light source in the shooting environment of the first image frame, and the mobile phone can determine that the number of the light sources is 1. If the multispectral data of the pixel points in the regional image does not meet the preset conditions, it means that there may be multiple light sources in the shooting environment of the first image frame, and the mobile phone can determine that the number of light sources is multiple. Among them, the preset conditions may include that the difference between the multispectral data of any two pixel points in the regional image is less than the preset difference, and / or that the ratio between the number of pixel points with the same multispectral data and the total number of pixel points in the regional image is greater than the preset ratio.

[0105] It can be understood that if the difference between the multispectral data of any two pixels in the regional image is less than the preset difference, it means that the multispectral data of each pixel in the regional image are relatively close, that is, the multispectral data of each pixel in the regional image can be approximately the same. In addition, if the ratio between the number of pixels with the same multispectral data and the total number of pixels in the regional image is greater than the preset ratio, it means that the number of pixels with the same multispectral data in the regional image is large, that is, there is only a single light source in the shooting environment of the first image frame. Therefore, in order to reduce the waste of computing resources, the mobile phone can predict the light source of the shooting environment of the first image frame based only on the multispectral data of any pixel in the regional image, and obtain a light source color information that can characterize the first image frame.

[0106] In other embodiments, the mobile phone can calculate the multispectral data of the pixel points in the above-mentioned regional image by a statistical method to obtain the target multispectral data. The statistical method may include a method for solving the mean, a method for solving the mode, and a method for solving the median. Afterwards, when the difference between the target multispectral data and the multispectral data of each pixel point in the regional image is less than a preset difference, the mobile phone can determine that the number of the above-mentioned light sources is 1. When the difference between the target multispectral data and the multispectral data of any pixel point in the regional image is greater than or equal to the preset difference, the mobile phone can determine that the number of light sources is multiple.

[0107] In some other embodiments, the mobile phone can group the pixels with the same multispectral data in the above-mentioned regional image into at least one pixel set. Afterwards, the mobile phone can determine the target pixel set with the largest number of pixels from the at least one pixel set. Afterwards, when the ratio between the number of pixels in the target pixel set and the total number of pixels in the regional image is greater than a preset ratio, and / or the difference between the multispectral data of the target pixel in the target pixel set and the multispectral data of each pixel in the regional image is less than a preset difference, the mobile phone can determine that the number of light sources is 1. When the ratio between the number of pixels in the target pixel set and the total number of pixels in the regional image is less than or equal to a preset ratio, or when the difference between the multispectral data of the target pixel in the target pixel set and the multispectral data of any pixel in the regional image is greater than or equal to a preset difference, the mobile phone can determine that the number of light sources is multiple.

[0108] In one implementation, the mobile phone can predict the light source of the shooting environment of the first image frame based on the target light source prediction network and the multispectral data corresponding to the area where the target object is located in the first image frame. The target light source prediction network is used to predict the relationship between the light source color information of the image frame and the multispectral data corresponding to the area where the target object is located in the image frame, so that the predicted light source color information is close to the real light source color information, thereby providing a basis for the subsequent color restoration of the first image frame. For example, Figure 8 As shown, the target light source prediction network can be a deep neural network (DNN), which can include an input layer, multiple hidden layers, and an output layer. The input layer is used to convert the input data into a format that can be processed inside the neural network, that is, to convert the first image frame into a vector form. The hidden layer is used to convert the input data into a higher-level feature representation. The output layer is used to output the processing result of the input data.

[0109] Specifically, the mobile phone can input the multispectral data of the target object corresponding to the area in the above-mentioned first image frame into the target light source prediction network to obtain the light source color information corresponding to the first image frame. It can be understood that if there is only a single light source in the shooting environment of the first image frame, it means that the multispectral data of each pixel in the first image frame are the same, that is, the light source color information of the target object in the first image frame is equivalent to the light source color information of the first image frame. Therefore, the mobile phone can input the multispectral data of any pixel in the regional image into the target light source prediction network to obtain the light source color information of the first image frame. If there are multiple light sources in the shooting environment of the first image frame, it means that the multispectral data of the pixel in the first image frame is different, that is, the light source color information of the target object in the first image frame may be different from the light source color information of other areas in the first image frame. Therefore, in order to achieve accurate restoration of image color, the mobile phone can input the multispectral data of each pixel in the regional image into the target light source prediction network to obtain the light source color information of each pixel of the target object in the first image frame.

[0110] In some embodiments, Fig. 9As shown, taking the above-mentioned target light source prediction network as an example of a neural network for predicting the light source color information of the human skin area in the first image frame, the training process of the target light source prediction network may specifically include: the mobile phone may obtain a first sample data set. Among them, the first sample data set includes multiple groups of first sample data, and each group of first sample data may include multispectral data of the area where the human skin is located in the sample image frame and real light source color information. The sample image frame is an image frame including a target object of human skin. The multispectral data is collected by the shooting device through a multispectral sensor. The real light source color information is the light source color information collected by the spectrometer when the shooting device collects the sample image frame. The spectrometer is a scientific instrument that decomposes light with complex components into spectral lines. The spectrometer is used to record the real light source color information near the user when the shooting device collects the sample image frame.

[0111] It can be understood that the spectrometer is placed opposite to the camera, and the distance between the spectrometer and the target object can be less than the preset distance. That is to say, when the camera collects sample image frames and multispectral data, the spectrometer can collect the real light source color information at the same time. The camera is a device including a multispectral sensor, and the multispectral sensor is used to collect multispectral data of the sample image frame. Exemplarily, the camera can be a mobile phone, a computer, etc., as long as it is a device with a multispectral sensor, and there is no specific limitation.

[0112] It should be noted that if there is one or more light sources in the shooting environment of the above-mentioned sample image frame, the shooting device can only collect one sample image frame and the real light source color information corresponding to the sample image frame, thereby generating a set of first sample data. If there are multiple light sources in the shooting environment of the first sample image frame, the shooting device can collect multiple sample image frames of the target object at the same position and the real light source color information corresponding to each sample image frame according to different shooting positions, thereby generating multiple sets of first sample data. In this way, the mobile phone can train the light source prediction network according to different light source color information, ensure the diversity of the first sample data, and provide a basis for the subsequent mobile phone to accurately predict the light source color information. Among them, the number of sample image frames collected is positively correlated with the number of light sources in the shooting environment. That is, the more light sources there are in the shooting environment, the more sample image frames are collected. For example, if the number of light sources in the shooting environment is 2, the number of sample image frames collected can be 4, 6, etc., as long as the number of collections is greater than the number of light sources, and there is no specific limitation.

[0113] In one implementation, before acquiring the first sample data set, the mobile phone may acquire multiple sample image frames and multispectral data of the sample image frames from the shooting device. Afterwards, for each sample image frame, the mobile phone may align the sample image frame with the multispectral data of the sample image frame to obtain multispectral data adapted to the sample image frame, that is, to obtain multispectral data corresponding to each pixel in the sample image frame. Afterwards, the mobile phone may acquire multispectral data of the pixel points in the area where the human skin is located from the multispectral data adapted to the sample image frame, that is, to obtain multispectral data corresponding to the area where the human skin is located in the sample image frame.

[0114] In another implementation method, the multispectral data corresponding to the area where the human skin is located in the above-mentioned sample image frame can also be directly obtained from the shooting device. That is, after the shooting device extracts the multispectral data corresponding to the area where the human skin is located in the sample image frame from the multispectral data adapted to the sample image frame, the multispectral data corresponding to the area where the human skin is located in the sample image frame can be directly sent to the mobile phone.

[0115] In some cases, in order to ensure the diversity of light source color information prediction, the mobile phone can collect multiple first sample image frames with different color attributes, that is, the color attributes of the ambient light sources corresponding to different groups of first sample data are different. In this way, the target light source prediction network can predict ambient light sources with different color attributes, improve the accuracy of light source prediction, and provide a basis for subsequent restoration of image color. Among them, the color attributes may include brightness, hue, saturation, etc. Among them, brightness is used to characterize the lightness and darkness of the color. Hue is used to characterize the type of color (such as red, green, blue, etc.). Saturation is used to characterize the purity and intensity of the color in the color.

[0116] In other cases, in order to ensure the diversity of light source color information prediction, the mobile phone can also obtain sample image frames and real light source color information at different shooting positions. That is, the human skin area of ​​the sample image frames in different groups of first sample data is the same, but the shooting positions of the camera devices that collect the sample image frames are different. In this way, the target light source prediction network can perform light source prediction for sample image frames at different shooting positions, which improves the accuracy of light source prediction and provides a basis for subsequent restoration of image color.

[0117] Specifically, Fig. 9As shown, after acquiring the above-mentioned first sample database, for each group of first sample data in the first sample database, the mobile phone can input the multispectral data of the human skin area in the sample image frame into the pre-built light source prediction network to obtain the predicted light source color information. Afterwards, the mobile phone can adjust the parameters of the light source prediction network according to the predicted light source color information and the above-mentioned real light source color information to obtain a trained light source prediction network (or called a target light source prediction network).

[0118] In some embodiments, Fig. 9 As shown, the mobile phone can determine the loss value of the light source prediction network based on the above-mentioned predicted light source color information and the real light source color information. Afterwards, the mobile phone adjusts the parameters of the light source prediction network according to the loss value of the light source prediction network until the light source prediction network meets the first training requirement and obtains the target light source prediction network. Among them, the first training requirement may include that the loss value of the light source prediction network is less than the second preset loss value, or that the predicted light source color information is the same as the real light source color information, or that the first training number reaches the first preset number. The first training number is the number of times the mobile phone trains the light source prediction network.

[0119] It should be noted that the training process of the target light source prediction network can also be performed by a cloud service, that is, after the cloud service completes the training of the light source prediction network, that is, after obtaining the target light source prediction network, the target light source prediction network can be sent to the mobile phone. Afterwards, the mobile phone can predict the light source color information of the first image frame based on the target light source prediction network.

[0120] S306: The mobile phone performs color restoration on the first image frame according to the light source color information corresponding to the first image frame to obtain a second image frame.

[0121] Specifically, after obtaining the light source color information corresponding to the first image frame, the mobile phone can perform color restoration on the first image frame according to the light source color information corresponding to the first image frame to obtain the first image frame after color restoration, that is, to obtain the second image frame.

[0122] In some embodiments, when a light source color information is obtained, it means that there is only a single light source in the shooting environment of the above-mentioned first image frame, and the mobile phone can perform color restoration on the entire first image frame based on the one light source color information. It can be understood that if there is only a single light source in the shooting environment of the first image frame, it means that the multi-spectral data of each pixel in the first image frame are the same, that is, the light source color information of the target object can be equivalent to the light source color information of the first image frame. Therefore, the mobile phone can achieve color restoration of the entire image based on only one light source color information, thereby improving the accuracy of image color restoration.

[0123] In other embodiments, when multiple light source color information is obtained, it means that there are multiple light sources in the shooting environment of the above-mentioned first image frame, and the mobile phone can perform color restoration on the target object in the first image frame according to the multiple light source color information. It can be understood that if there are multiple light sources in the shooting environment of the first image frame, it means that the multi-spectral data of each pixel in the first image frame may be different, that is, the light source color information of the target object may be different from the light source color information of other areas in the first image frame. Therefore, in order to ensure the accuracy of color restoration, the mobile phone can perform color restoration on the target object in the first image frame according to the multiple light source color information. In this way, the color restoration accuracy of the target object in the first image frame can be achieved, the occurrence of color restoration errors in other areas of the first image frame can be reduced, the color restoration accuracy of the first image frame can be ensured, and the user's visual experience is improved.

[0124] It can be understood that the above light source color information is a form of expression of the CIE color space, and the above first image frame is an RGB image. Therefore, in order to ensure that the light source color information is consistent with the expression of the first image frame, the mobile phone can perform spatial conversion on the light source color information of the CIE color space to obtain the light source color information of the RGB color space (or the second color space). Among them, the spatial conversion refers to converting the light source color information from the CIE color space to the RGB color space. The RGB color space is used to generate colors. The CIE color space can generate colors by superimposing different proportions of the three colors of red, green and blue. Afterwards, the mobile phone can restore the color of the first image frame according to the light source color information of the RGB color space.

[0125] In some cases, if there is a single light source in the shooting environment of the first image frame, the light source color information in the RGB color space can be represented by the RGB value of any pixel in the area where the target object is located in the first image frame. If there are multiple light sources in the shooting environment of the first image frame, the light source color information in the RGB color space can be represented by an RGB image with the same resolution as the first image frame. Each pixel in the area where the target object is located in the RGB image corresponds to an RGB value, and the RGB values ​​of all pixels in the RGB image except the area where the target object is located are empty.

[0126] In one implementation, the mobile phone can perform color restoration on the first image frame according to the target color restoration network and the light source color information. The target color restoration network is used to perform color restoration on the first image frame, so that the first image frame after color restoration is more consistent with the real scene, and the image color restoration accuracy is improved. Specifically, the mobile phone can input the light source color information and the first image frame into the target color restoration network to obtain the second image frame.

[0127] In one example, the target color restoration network may be a codec. The codec may include an encoder and a decoder. In another example, the target color restoration network may also be a deep neural network, etc., which is not specifically limited.

[0128] In some embodiments, Fig.10 As shown, the training process of the above-mentioned target color restoration network may specifically include: the mobile phone may obtain a second sample data set. The second sample data set includes multiple groups of second sample data, and each group of second sample data may include an image to be restored, light source color information, and a real image. The image to be restored and the real image both include a target object, and the image to be restored is an image frame that has not been color restored, and the real image is an image frame that is not affected by a colored light source. Afterwards, for each group of second sample data in the second sample data set, the mobile phone may input the image to be restored and the light source color information into a pre-constructed color restoration network to obtain a predicted restored image. Afterwards, the mobile phone may adjust the parameters of the color restoration network according to the predicted restored image and the real image to obtain a trained color restoration network (or referred to as a target color restoration network).

[0129] Specifically, Fig.10 As shown, the mobile phone can determine the loss value of the color restoration network based on the above-mentioned predicted restored image and the real image. Afterwards, the mobile phone can adjust the parameters of the color restoration network according to the loss value of the color restoration network until the color restoration network meets the second training requirement, and obtain a trained color restoration network (or target color restoration network). The second training requirement may include that the loss value of the color restoration network is less than a second preset loss value, or that the predicted restored image is the same as the real image, or that the second training number reaches a second preset number. The second training number is the number of times the mobile phone trains the color restoration network.

[0130] In some cases, the real image in the second sample data may be obtained by color restoration of the image to be restored in the second sample data using a preset processing method. The preset processing method is a photo retouching method. The image to be restored may be acquired by a camera. The light source color information in the second sample data may be acquired by a spectrometer when the camera acquires the image to be restored. In this way, the real restored image can be more consistent with the real shooting scene, that is, the target color restoration network trained by the real image can restore the image that conforms to the real shooting scene, thereby improving the user's visual experience.

[0131] In other cases, the real image in the second sample data may be an image frame captured by the shooting device under the target light. The target light is D65 light. The spectrum of the D65 light is similar to natural daylight. In other words, the D65 light can simulate natural lighting, which can reduce the occurrence of color deviations due to lighting differences, so that the real image can display real colors. Afterwards, the mobile phone can perform degradation processing on the real image to obtain the image to be restored in the second sample data. Specifically, the mobile phone can render the real image according to the preset color information to obtain the image to be restored. It can be understood that the preset color information is the light source color information in the second sample data. In this way, the collection efficiency of the second sample data can be improved, thereby improving the training efficiency of the color restoration network.

[0132] In one implementation, the second sample data set and the first sample data set may be the same sample data set. That is, the sample data in the sample data set may simultaneously include real light source color information, the image to be restored, the real restored image, and multispectral data corresponding to the area where the human skin is located in the image to be restored. Afterwards, when training the target light source prediction network, the mobile phone may obtain the real light source color information in the sample data and the multispectral data corresponding to the area where the human skin is located in the image to be restored. That is, the sample image frame in the first sample data may be equivalent to the image to be restored in the second sample data. When training the target color restoration network, the mobile phone may obtain the real light source color information, the image to be restored, and the real image in the sample data. That is, the real light source color information in the first sample data may be equivalent to the light source color information in the second sample data.

[0133] It should be noted that the training process of the target color restoration network can also be performed by a cloud service, that is, after the cloud service completes the training of the color restoration network, that is, after obtaining the target color restoration network, the target color restoration network can be sent to the mobile phone. Afterwards, the mobile phone can restore the color of the first image frame according to the target color restoration network to obtain the second image frame.

[0134] The above describes how the mobile phone performs color restoration on the first image frame. The following will further describe a possible implementation process of the mobile phone performing color restoration on the first image frame. Fig.11 As shown, the color restoration process may specifically include: the mobile phone collects a first image frame and multispectral data. The first image frame includes the target object (i.e., the user's skin). Afterwards, the mobile phone may perform regional segmentation on the first image frame to obtain a regional image where the target object is located (i.e., a regional image of the user's skin S). Afterwards, the mobile phone may obtain multispectral data corresponding to the regional image of the target object from the multispectral data of the first image frame based on the regional image of the target object.

[0135] Afterwards, the mobile phone can input the multispectral data corresponding to the regional image of the above-mentioned target object into the target light source prediction network to obtain XYZ tristimulus values. Afterwards, the mobile phone can extract the coordinates of the XYZ tristimulus values ​​to obtain xy color coordinates. Afterwards, the mobile phone can perform spatial transformation on the xy color coordinates to obtain light source color information. Among them, the light source color information is used to characterize the color information in the RGB color space. Afterwards, the mobile phone can input the light source color information and the above-mentioned first image frame into the target color restoration network to obtain a second image frame. Among them, the second image frame is the first image frame after color restoration.

[0136] In one implementation, since the multispectral data is composed based on the reflectivity of the target object and the light source color information of the shooting environment, the mobile phone can predict the light source color information of the shooting environment when the reflectivity of the target object is in a stable state. When the light source color information of the shooting environment is in a stable state, the mobile phone can also predict the reflectivity of the target object. Afterwards, the mobile phone can determine the type of the target object based on the reflectivity of the target object and the preset reflection range. For example, if the reflectivity of the target object is within the preset reflection range of the user, the mobile phone can determine that the target object is human skin.

[0137] In some embodiments, the mobile phone can collect a third image frame through an image sensor under D65 light, and collect multispectral data through a multispectral sensor. The third image frame includes human skin. Afterwards, the mobile phone can perform regional segmentation on the third image frame to obtain multispectral data corresponding to the area where the human skin is located. Afterwards, the mobile phone can perform skin quality detection on the user based on the multispectral data corresponding to the area where the human skin is located to obtain the user's skin condition. For example, the mobile phone can input the multispectral data corresponding to the area where the human skin is located into the skin quality detection network to obtain the user's skin condition.

[0138] In some embodiments, the present application provides a computer storage medium including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method for adjusting usage parameters as described above.

[0139] In some embodiments, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method for adjusting usage parameters as described above.

[0140] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0141] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0145] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: The electronic device acquires a first image frame and multispectral data corresponding to a region where a target object is located in the first image frame; wherein the reflectivity of the target object is within a preset reflectivity range; The electronic device predicts the light source of the shooting environment of the first image frame according to the multispectral data corresponding to the area where the target object is located in the first image frame, and obtains the light source color information corresponding to the first image frame; The electronic device performs color restoration on the first image frame according to the light source color information corresponding to the first image frame to obtain a second image frame.

2. The method according to claim 1, characterized in that The electronic device predicts the light source of the shooting environment of the first image frame according to the multispectral data corresponding to the area where the target object is located in the first image frame, and obtains the light source color information corresponding to the first image frame, including: The electronic device performs light source prediction on the first image frame according to the multispectral data of the target object in the first image frame to obtain XYZ tristimulus values ​​in a first color space; The electronic device extracts coordinates of the XYZ tristimulus values ​​to obtain xy color coordinates; wherein the xy color coordinates are light source color information of the first color space.

3. The method according to claim 2, characterized in that The electronic device performs color restoration on the first image frame according to the light source color information corresponding to the first image frame to obtain a second image frame, including: The electronic device performs space conversion on the light source color information of the first color space to obtain the light source color information of the second color space; wherein the color mode corresponding to the second color space is the same as the color mode corresponding to the first image frame; The electronic device performs color restoration on the first image frame according to the light source color information in the second color space to obtain the second image frame.

4. The method according to claim 3, characterized in that: The first image frame is an RGB image, the first color space is a CIE color space, and the second color space is an RGB color space.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device acquires a first image frame and multispectral data corresponding to an area where a target object is located in the first image frame, including: The electronic device acquires the first image frame and multispectral data of the first image frame; wherein each pixel point in the first image frame corresponds to a multispectral data; The electronic device performs region segmentation on the first image frame to obtain a region image where the target object in the first image frame is located; The electronic device obtains multispectral data of pixel points corresponding to the regional image where the target object is located from the multispectral data of the first image frame.

6. The method according to claim 5, characterized in that The electronic device predicts the light source of the shooting environment of the first image frame according to the multispectral data corresponding to the area where the target object is located in the first image frame, and obtains the light source color information corresponding to the first image frame, including: When the multispectral data of the pixel points in the area where the target object is located in the first image frame meets a preset condition, the electronic device predicts the light source of the shooting environment of the first image frame according to the multispectral data of any pixel point in the area where the target object is located in the first image frame, and obtains the light source color information corresponding to the first image frame; Among them, the preset conditions include: the difference between the multispectral data of any two pixels in the area where the target object is located is less than a preset difference, and / or the ratio between the number of pixels with the same multispectral data and the total number of pixels in the area where the target object is located is greater than a preset ratio.

7. The method according to claim 5, characterized in that The electronic device predicts the light source of the shooting environment of the first image frame according to the multispectral data corresponding to the area where the target object is located in the first image frame, and obtains the light source color information corresponding to the first image frame, including: When the multispectral data of the pixel points in the area where the target object is located in the first image frame does not meet the preset condition, the electronic device predicts the light source of the shooting environment of the target object in the first image frame according to the multispectral data of each pixel point in the area where the target object is located, and obtains the light source color information corresponding to the target object in the first image frame; The electronic device performs color restoration on the first image frame according to the light source color information corresponding to the first image frame to obtain a second image frame, including: The electronic device performs color restoration on the area where the target object in the first image frame is located according to the light source color information corresponding to the target object in the first image frame to obtain the second image frame.

8. The method according to claim 5, characterized in that The first image frame is a portrait image frame, the target object of the portrait image frame includes human skin, and the electronic device performs region segmentation on the first image frame to obtain a region image where the target object in the first image frame is located, including: The electronic device performs region segmentation on the portrait image frame to obtain a skin mask image corresponding to the region where the human skin is located; The electronic device acquires multispectral data of pixel points corresponding to the regional image of the target object from the multispectral data of the first image frame, including: The electronic device obtains multispectral data of pixel points corresponding to the skin mask image from the multispectral data of the portrait image frame.

9. The method according to claim 1, characterized in that: The electronic device predicts the light source of the shooting environment of the first image frame according to the multispectral data corresponding to the area where the target object is located in the first image frame, and obtains the light source color information corresponding to the first image frame, including: The electronic device inputs the multispectral data corresponding to the area where the target object is located in the first image frame into the target light source prediction network to obtain the light source color information corresponding to the first image frame; wherein the target light source prediction network is used to predict the relationship between the light source color information of the image frame and the multispectral data corresponding to the area where the target object is located in the image frame.

10. The method according to claim 9, characterized in that The target light source prediction network is trained through the following process: Acquire a first sample data set; wherein the first sample data set includes multiple groups of first sample data, the first sample data includes multispectral data of the area where the target object is located in the sample image frame and real light source color information, and the acquisition time of the multispectral data is the same as the acquisition time of the real light source color information; For each group of the first sample data, the multispectral data of the area where the target object is located in the sample image frame is input into a pre-built light source prediction network to obtain predicted light source color information; According to the predicted light source color information and the real light source color information, the parameters of the pre-built light source prediction network are adjusted to obtain the target light source prediction network.

11. The method according to claim 10, characterized in that Before obtaining the first sample data set, the method further includes: Acquire multiple sample image frames and multispectral data of each sample image frame from a shooting device, and acquire the color information of the real light source from a spectrometer; wherein the shooting device is a device including a multispectral sensor, the multispectral sensor is used to collect the multispectral data of the sample image frames, the spectrometer is placed opposite to the shooting device, and the distance between the spectrometer and the target object is less than a preset distance; For each of the sample image frames, aligning the sample image frame with the multispectral data of the sample image frame to obtain multispectral data corresponding to each pixel in the sample image frame; From the multispectral data corresponding to each pixel point in the sample image frame, the multispectral data corresponding to the area where the target object is located in the sample image frame is obtained.

12. The method according to claim 11, characterized in that The color attributes of the shooting environments corresponding to the multiple sample image frames are different, and the color attribute of the shooting environment includes at least one of brightness, hue and saturation; and / or, The plurality of sample image frames include a plurality of image frames including the target object at one position or multiple positions acquired by the photographing device when there is a light source in the photographing environment; and / or, The plurality of sample image frames include image frames including the target object at the same position, which are acquired when the shooting device is located at different shooting positions when there are multiple light sources in the shooting environment.

13. The method according to claim 1, characterized in that The electronic device performs color restoration on the first image frame according to the light source color information corresponding to the first image frame to obtain a second image frame, including: The electronic device inputs the light source color information corresponding to the first image frame and the first image frame into a target color restoration network to obtain the second image frame; wherein the target color restoration network is obtained by adjusting parameters based on a predicted restored image and a real image during an iterative training process, the predicted restored image is obtained by color restoration of the image to be restored based on the image to be restored and the light source color information, and the real image is an image not affected by a colored light source.

14. The method according to claim 13, characterized in that The real image is obtained by color restoration of the image to be restored by a preset processing method, the image to be restored is acquired by a camera device, and the light source color information is acquired by a spectrometer when the camera device acquires the image to be restored; and / or, The real image is captured by the camera device under target light, and the target light is used to simulate natural lighting. The image to be restored is obtained by rendering according to preset color information based on the real image, and the light source color information is the preset color information.

15. An electronic device, characterized in that: The electronic device includes a camera, a multispectral sensor, a memory and one or more processors; the camera, the multispectral sensor, the memory and the processor are coupled; the camera is used to capture a first image frame, the multispectral sensor is used to capture multispectral data, and the memory is used to store computer program code, wherein the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 14.

17. A computer program product, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 14.

Images