Multispectral camera, electronic device and image processing method

Abstract

The present application discloses a multi-spectral camera, an electronic device and an image processing method, which belong to the field of image processing technology. The multi-spectral camera includes: M single-channel pixel units and at least one multi-channel pixel unit, each single-channel pixel unit is used to generate image data according to a light signal of a spectral band, and each multi-channel pixel unit is used to generate image data according to light signals of all spectral bands, and M is a positive integer; wherein the M single-channel pixel units include at least one single-channel pixel unit whose light transmittance is greater than a threshold value.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a multispectral camera, an electronic device and an image processing method. Background Art

[0002] With the increasing popularity of electronic devices and the continuous advancement of camera technology, users often use electronic devices to take photos and share the captured images with friends through social networking platforms, which makes the image capture effect the focus of users.

[0003] At present, when electronic devices capture images, they generally process the images based on a main camera and a telephoto camera, that is, the imaging cameras in the electronic device. However, in actual situations, the shooting scenes are diverse. For complex shooting scenes or low-light environments, the image quality of images captured by existing electronic devices is poor. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a multispectral camera, an electronic device and an image processing method, which can improve the image quality of captured images.

[0005] In a first aspect, an embodiment of the present application provides a multi-spectral camera, which includes: M single-channel pixel units and at least one multi-channel pixel unit, each single-channel pixel unit is used to generate image data based on the light signal of a spectral band, and each multi-channel pixel unit is used to generate image data based on the light signals of all spectral bands, and M is a positive integer; wherein the M single-channel pixel units include at least one single-channel pixel unit whose light transmittance is greater than a threshold value.

[0006] In a second aspect, an embodiment of the present application provides an electronic device, comprising: an imaging camera and a multispectral camera as in the first aspect; the field of view of the multispectral camera is larger than the field of view of the imaging camera.

[0007] In a third aspect, an embodiment of the present application provides an image processing method, which is applied to the electronic device of the second aspect, the method comprising: collecting RGB images through an imaging camera, and collecting multispectral images through a multispectral camera; and performing image processing on the RGB image based on the multispectral image.

[0008] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the third aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the third aspect are implemented.

[0010] In a sixth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the third aspect.

[0011] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the third aspect.

[0012] In an embodiment of the present application, a multi-spectral camera includes: M single-channel pixel units and at least one multi-channel pixel unit, each single-channel pixel unit is used to generate image data according to the light signal of a spectral band, and each multi-channel pixel unit is used to generate image data according to the light signals of all spectral bands, and M is a positive integer; wherein the M single-channel pixel units include at least one single-channel pixel unit whose light transmittance is greater than a threshold value. In this solution, on the one hand, since the multispectral camera contains two different pixel units, the multispectral camera can not only collect light in different bands, but also collect light in the full-pass band. In this way, compared with the existing method that cannot collect light in the full-pass band, the multispectral image collected by the multispectral camera in a dark light environment can have richer image detail information, making the RGB image collected in the dark light environment after multispectral image processing clearer; on the other hand, since the light transmittance of the pixel unit in the multispectral camera can be set according to user needs, the multispectral image collected by the multispectral camera can have more accurate image spectral information. Therefore, the multispectral image can provide rich color information and image detail information, making the color performance of the RGB image after multispectral image processing more accurate and clearer; thereby improving the image quality of the RGB image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of a pixel array provided by some embodiments of the present application;

[0014] Figure 2 is a schematic diagram of a pixel array provided by some embodiments of the present application;

[0015] Figure 3 is a schematic diagram of a pixel array provided by some embodiments of the present application;

[0016] Figure 4is a schematic diagram of a pixel array provided by some embodiments of the present application;

[0017] Figure 5 is a schematic diagram of a pixel array provided by some embodiments of the present application;

[0018] Figure 6 is a flowchart of an image processing method provided by some embodiments of the present application;

[0019] Figure 7 is a flowchart of an image processing method provided by some embodiments of the present application;

[0020] Figure 8 is a flowchart of an image processing method provided by some embodiments of the present application;

[0021] Fig. 9 is a flowchart of an image processing method provided by some embodiments of the present application;

[0022] Fig.10 is a structural schematic diagram of an image processing device provided by some embodiments of the present application;

[0023] Fig.11 is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application;

[0024] Fig.12 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0028] The nouns and terms involved in the embodiments of the present application are explained below.

[0029] 1. Multispectral camera: A multispectral camera is a camera that can capture spectral information in multiple wavelength ranges at the same time. The principle is to use optical filters or spectroscopic elements to separate and capture light from different wavelength ranges, thereby realizing the collection and analysis of spectral information of multiple wavelengths. Traditional color cameras use three sensors to capture light in the three wavelength ranges of red, green and blue respectively, while multispectral cameras can capture a wider range of wavelengths, usually including visible light, infrared and ultraviolet spectra. Multispectral cameras can use multiple narrowband filters or spectroscopic elements, such as prisms or gratings, to separate light of different wavelengths and then transmit them to the corresponding pixel arrays respectively. By capturing spectral information in multiple wavelength ranges at the same time, multispectral cameras can provide richer spectral information, which can be applied to many fields such as agriculture, environmental monitoring, geological exploration, etc.

[0030] 2. Binocular stereo vision: Binocular stereo vision is an important form of machine vision. It is a method based on the principle of parallax and obtains the three-dimensional geometric information of an object from multiple images. The binocular stereo vision system generally uses two cameras to obtain two digital images of the object being measured from different angles at the same time, or uses a single camera to obtain two digital images of the object being measured from different angles at different times, and restores the three-dimensional geometric information of the object based on the principle of parallax, and reconstructs the three-dimensional contour and position of the object. The binocular stereo vision system has broad application prospects in the field of machine vision.

[0031] 3. There are two principles of multi-spectral cameras commonly used in the consumer field: narrow-band filtering and spectral calculation reconstruction. The following diagrams illustrate them respectively:

[0032] (1) A narrowband filter multispectral camera is a camera that can obtain multiple narrowband spectral information. The principle is to place a specific narrowband filter on each pixel of the image sensor (CMOS & CCD). These narrowband filters can be designed to only allow light within a specific wavelength range to pass through, thereby separating and capturing spectral information of different wavelengths.

[0033] When light passes through a narrowband filter, only light within a specific wavelength range can reach the sensor's pixel array. Therefore, the spectral information captured by each pixel is for a specific band, and spectral information in multiple bands can be captured, thereby obtaining the spectral characteristics of the target object at different wavelengths.

[0034] In order to obtain more spatial information, the filter on the pixel is usually designed in an array type, and the common ones are 3*3 and 4*4 arrays. Figure 1 The filter pattern of the multi-spectral sensor with a 4*4 array and the spectral transmittance reference diagram of each filter are shown below. Figure 1 As shown, channels 0-15 can pass light of different wavelengths respectively, and after being collected by the sensor, spectral images of different bands are formed.

[0035] (2) Computational reconstruction multispectral cameras place specially designed filters on the surface of sensor pixels to modulate the incident light in the spectral domain, use CMOS image sensors to complete the mapping measurement from the spectral domain to the electrical domain, and then use compressed sensing algorithms to perform computational reconstruction of the spectrum. By integrating sensor pixels and filters on a large scale, real-time spectral imaging is finally achieved.

[0036] The filter placed on the pixel surface of the computational reconstruction multispectral camera is different from that of the narrowband filter multispectral sensor. The narrowband filter only transmits a very narrow band, while the filter of the computational reconstruction multispectral camera is specially customized and has a modulated transmittance curve. The filter design of the computational reconstruction multispectral camera is usually also array-type. Common ones include 3*3, 4*4 arrays, etc. Figure 2 The following is an example diagram of the filter pattern of a 3*3 array computationally reconstructed multispectral sensor and the spectral transmittance of a filter.

[0037] 4. RGB image: contains only information of three bands: red, green and blue. It is usually used for visualization purposes and is the color image we see in daily life.

[0038] 5. Multispectral images: Contains information from multiple bands, usually including red, green, blue, and near-infrared bands. Multispectral images can provide more spectral information than RGB images, which helps to distinguish the characteristics of different objects.

[0039] The multispectral camera, electronic device and image processing method provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in combination with the accompanying drawings.

[0040] In the multispectral camera, electronic device and image processing method provided in the embodiments of the present application, on the one hand, since the multispectral camera includes two different pixel units, the multispectral camera can not only collect light of different bands, but also collect light of the full-pass band. In this way, compared with the existing method that cannot collect light of the full-pass band, the multispectral image collected by the multispectral camera in a dark light environment can have richer image detail information, so that the RGB image collected in the dark light environment after the multispectral image processing is clearer; on the other hand, since the light transmittance of the pixel unit in the multispectral camera can be set according to the needs of the user, the multispectral image collected by the multispectral camera can have more accurate image spectral information. Therefore, the multispectral image can provide rich color information and image detail information, so that the color performance of the RGB image after the multispectral image processing is more accurate and clearer; thereby improving the image quality of the RGB image.

[0041] An embodiment of the present application provides a multi-spectral camera, which includes: M single-channel pixel units and at least one multi-channel pixel unit, each single-channel pixel unit is used to generate image data according to the light signal of a spectral band, and each multi-channel pixel unit is used to generate image data according to the light signals of all spectral bands, and M is a positive integer; wherein the M single-channel pixel units include at least one single-channel pixel unit whose light transmittance is greater than a threshold value.

[0042] In some embodiments of the present application, Figure 3 As shown, the multi-spectral camera may include: C1-C8 pixel units and 1 Mono pixel unit, wherein the C1-C8 pixel units are 8 single-channel pixel units included in the multi-spectral camera, and the Mono pixel unit is 1 multi-channel pixel unit included in the multi-spectral camera, each single-channel pixel unit is used to generate image data according to the light signal of one spectral band, and the multi-channel pixel unit is used to generate image data according to the light signals of all spectral bands, and M is a positive integer.

[0043] It should be noted that the above-mentioned Mono pixel unit can be set at any position in the pixel array, for example, at each vertex position of the pixel array, such as Figure 3 The lower right corner shown in ; Another example: the middle position of the pixel array, such as Figure 3 The position of the C5 pixel unit in the figure can be determined according to actual usage requirements, and the embodiment of the present application does not limit this.

[0044] In some embodiments of the present application, Figure 4As shown, the multi-spectral camera may include: C1-C7 pixel units and 2 Mono pixel units, wherein the C1-C7 pixel units are 7 single-channel pixel units included in the multi-spectral camera, and the 2 Mono pixel units are 2 multi-channel pixel units included in the multi-spectral camera, each single-channel pixel unit is used to generate image data according to the light signal of one spectral band, and each multi-channel pixel unit is used to generate image data according to the light signals of all spectral bands, and M is a positive integer.

[0045] In some examples, two Mono pixel units may also be arranged at the lower left corner and the upper right corner of the pixel array, that is, Figure 4 The positions of the C6 pixel unit and the C2 pixel unit can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0046] In some examples, two Mono pixel units can also be set Figure 4 The positions of the C1 pixel unit and the C7 pixel unit in the pixel array shown in FIG. Figure 4 The positions of the C3 pixel unit and the C5 pixel unit in the pixel array shown are specifically determined according to actual usage requirements, and are not limited in this embodiment of the present application.

[0047] In some embodiments of the present application, Figure 5 As shown, the multi-spectral camera may include: C1-C5 pixel units and 4 Mono pixel units, wherein the C1-C5 pixel units are 5 single-channel pixel units included in the multi-spectral camera, and the 4 Mono pixel units are 4 multi-channel pixel units included in the multi-spectral camera, each single-channel pixel unit is used to generate image data according to the light signal of one spectral band, and each multi-channel pixel unit is used to generate image data according to the light signals of all spectral bands, and M is a positive integer.

[0048] In some examples, the four Mono pixel units may also be arranged in the middle of the edge of the pixel array, for example: Figure 5 The positions of the C1 pixel unit, C2 pixel unit, C4 pixel unit and C5 pixel unit shown in the figure are located; the specific positions can be determined according to actual usage requirements, and the embodiments of the present application are not limited to this.

[0049] In some embodiments of the present application, when the multi-spectral camera includes at least two multi-channel pixel units, the positions of the multi-channel pixel units in the pixel array can be dispersed as much as possible, so that the multi-spectral camera can obtain more comprehensive scene information and capture more details to better improve the image quality of the RGB image.

[0050] In some embodiments of the present application, the filter provided on the pixel unit of the multi-spectral camera may be any of the following: an absorption filter, an FP thin film filter, an ultra-surface filter, etc. The specific filter may be determined according to actual use requirements, and the embodiments of the present application do not limit this.

[0051] In some embodiments of the present application, when the multispectral camera is a narrow-band filtering multispectral camera, the filter set on the pixel unit of the multispectral camera only passes a spectrum with a narrower wavelength, for example, only passes a spectrum of 10-20nm.

[0052] In some embodiments of the present application, the filter set on the pixel unit of the multi-spectral camera can be an array type, for example, the filter can be a 3*3 array or a 4*4 array, and the present application embodiment takes the 3*3 array as an example for explanation. The specific can be determined according to actual use requirements, and the present application embodiment does not limit this.

[0053] In some embodiments of the present application, the filter set on each single-channel pixel unit can transmit light of different bands, so that each single-channel pixel unit can receive a light signal of a spectral band to generate image data corresponding to the spectral band.

[0054] In some embodiments of the present application, the filter set on the multi-channel pixel unit can transmit light of all bands, so that the multi-channel pixel unit can receive light signals of all spectral bands to generate image data corresponding to all spectral bands.

[0055] In some examples, when the filter set on the multi-channel pixel unit transmits light of all bands, the filter set on the multi-channel pixel unit can be called an all-pass filter, and the multi-channel pixel unit set with the all-pass filter is sensitive to the entire spectrum.

[0056] For example, as shown in Table 1 below, when the multispectral camera is a narrow-band filtering multispectral camera, the filter set on the pixel unit of the multispectral camera is a 3*3 array, and the multispectral camera includes 1 multi-channel pixel unit, an example of the light of different central wavelengths that can be transmitted by the filter set on each pixel unit, and the full width at half maximum (FWHM) corresponding to the different central wavelengths. It should be noted that the filter set on each pixel unit can transmit light of any wavelength band, and the specific value can be determined according to actual use requirements, and the embodiments of the present application are not limited to this.

[0057] Table 1

[0058] Band Center wavelength(nm) FWHM(nm) 1 410 10 2 450 14 3 480 15 4 520 15 5 560 15 6 600 15 7 650 15 8 940 20 9 All Access Full-through type

[0059] In some embodiments of the present application, due to different skin colors, there are large differences in reflectivity at 390-410nm and 450-520nm. Therefore, when designing the filters of the multi-spectral camera, the filters corresponding to 390-410nm and 450-520nm can be optimized and enhanced so that the light has better transmittance in this band, that is, the M single-channel pixel units mentioned above include at least one single-channel pixel unit whose light transmittance is greater than the threshold value.

[0060] In some examples, in combination with the above Table 1, the single-channel pixel units whose light transmittance is greater than the threshold value may be C1 single-channel pixel units to C4 single-channel pixel units corresponding to 390-410 nm and 450-520 nm.

[0061] In some examples, the electronic device can precisely customize the two wavelength bands of 390-410nm and 450-520nm so that the transmittance of light in these two wavelength bands is above 90%.

[0062] In some embodiments of the present application, when the multispectral camera is a computationally reconstructed multispectral camera, the electronic device can also enhance the accuracy of the two bands according to the algorithm, so that the difference between the accuracy of the two bands and the professional spectrum is within a preset range, for example: the difference can be within the range of ±5.

[0063] The present invention provides an electronic device, which includes an imaging camera and Figure 1 A multispectral camera is shown. The field of view of a multispectral camera is larger than that of an imaging camera.

[0064] In some embodiments of the present application, the imaging camera may be any of the following: a main camera, a telephoto camera, etc. The specific one may be determined according to actual use requirements, and the embodiments of the present application do not limit this.

[0065] The image processing method in the embodiment of the present application can be applied to the scene of shooting images.

[0066] Scenario 1: In an outdoor scenario, if user Xiaoyu wants to take a photo of Xiaoxi, Xiaoyu can open the camera application in the electronic device, select the portrait shooting mode, and take a photo of Xiaoxi.

[0067] Scenario 2: In an indoor scenario, if user Xiaoyu wants to take a photo of Xiaoxi, Xiaoyu can open the camera application in the electronic device, select the portrait shooting mode, and take a photo of Xiaoxi.

[0068] In the solution provided in the embodiment of the present application, on the one hand, since the multispectral camera includes two different pixel units, the multispectral camera can not only collect light in different bands, but also collect light in the full-pass band. In this way, compared with the existing method that cannot collect light in the full-pass band, the multispectral image collected by the multispectral camera in a dark light environment can have richer image detail information, so that the RGB image collected in the dark light environment after multispectral image processing is clearer; on the other hand, since the light transmittance of the pixel unit in the multispectral camera can be set according to user needs, the multispectral image collected by the multispectral camera can have more accurate image spectral information. Therefore, the multispectral image can provide rich color information and image detail information, so that the color performance of the RGB image after multispectral image processing is more accurate and clearer; thereby improving the image quality of the RGB image.

[0069] The image processing method provided in the embodiment of the present application may be executed by an image processing device, which may be an electronic device, or a functional module or entity in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.

[0070] The present application provides an image processing method. Figure 6 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present application, which can be applied to electronic devices. Figure 6 As shown, the image processing method provided in the embodiment of the present application may include the following steps 201 and 202.

[0071] Step 201: The electronic device collects RGB images through an imaging camera and collects multispectral images through a multispectral camera.

[0072] In some embodiments of the present application, the above-mentioned RGB image can be a preview image captured by an imaging camera, or an image captured by an imaging camera, which can be specifically determined according to the actual scene, and the embodiments of the present application are not limited to this.

[0073] Step 202: The electronic device performs image processing on the RGB image according to the multispectral image.

[0074] In some embodiments of the present application, the electronic device can perform white balance restoration processing on the RGB image based on the multispectral image, or adjust the skin color of the face area in the RGB image, or perform focus processing on the RGB image, or perform background blur processing on the RGB image.

[0075] Example 1, combined with scene 1, in portrait shooting mode, the electronic device can collect an RGB image containing "Xiao Xi" through a telephoto camera, and collect a multispectral image containing "Xiao Xi" through a multispectral camera, and then perform image processing on the RGB image based on the multispectral image, so that Xiao Yu can take high-quality images through the electronic device.

[0076] Example 2, combined with scene 2, in portrait shooting mode, the electronic device can collect an RGB image containing "Xiao Xi" through the main camera, and collect a multispectral image containing "Xiao Xi" through the multispectral camera, and then perform image processing on the RGB image based on the multispectral image, so that Xiao Yu can take high-quality images through the electronic device.

[0077] It should be noted that for the detailed steps of the electronic device performing image processing on the RGB image based on the multispectral image, reference may be made to the specific description in the following embodiments, which will not be repeated here.

[0078] In the image processing method provided in the embodiment of the present application, on the one hand, since the multi-spectral camera includes two different pixel units, the multi-spectral camera can not only collect light in different bands, but also collect light in the full-pass band. In this way, compared with the existing method that cannot collect light in the full-pass band, the multi-spectral image collected by the multi-spectral camera in a dark light environment can have richer image detail information, so that the RGB image collected in the dark light environment after the multi-spectral image processing is clearer; on the other hand, since the light transmittance of the pixel unit in the multi-spectral camera can be set according to the needs of the user, the multi-spectral image collected by the multi-spectral camera can have more accurate image spectral information. Therefore, the multi-spectral image can provide rich color information and image detail information, so that the color performance of the RGB image after the multi-spectral image processing is more accurate and clearer; thereby improving the image quality of the RGB image.

[0079] In some embodiments of the present application, before the above step 202, the image processing method provided by the embodiment of the present application further includes the following step 301.

[0080] Step 301: The electronic device performs internal calibration processing on the imaging camera and the multispectral camera respectively, and performs external calibration processing on the imaging camera and the multispectral camera respectively, to generate camera calibration data.

[0081] In some embodiments of the present application, the above internal parameters may include at least one of the following: focal length, principal point coordinates, distortion coefficient, etc.; external parameters may include at least one of the following: rotation matrix and translation vector, etc. The specific parameters may be determined according to actual use requirements, and the embodiments of the present application do not limit this.

[0082] In some embodiments of the present application, after the electronic device generates camera calibration data, the camera calibration data can be stored.

[0083] In some embodiments of the present application, the electronic device may perform internal parameter calibration processing on the imaging camera by: calibrating the internal parameters of the RGB image captured by the imaging camera.

[0084] In some embodiments of the present application, the electronic device may perform internal parameter calibration processing on the multispectral camera by: calibrating the internal parameters of the multispectral image captured by the multispectral camera.

[0085] In some examples, the multispectral image captured by the multispectral camera includes: an image corresponding to a pixel unit of each channel of the multispectral camera and an image obtained by fusing the pixel units of each channel of the multispectral camera.

[0086] In some embodiments of the present application, the electronic device performs external parameter calibration processing on the imaging camera and the multispectral camera respectively, and specifically can be: the image after the spectral fusion of the imaging camera and the multispectral camera is calibrated with external parameters to generate calibration external parameters 1. The external parameters of the fused images of different imaging cameras and multispectral cameras are recorded as 1-a, 1-b, 1-c, ... in sequence.

[0087] It should be noted that for the detailed steps of performing internal and external calibration processing on the imaging camera and the multispectral camera by the electronic device to generate camera calibration data, please refer to the description of performing internal and external calibration processing on the two cameras to generate camera calibration data in the relevant technology, which will not be repeated here.

[0088] In some embodiments of the present application, Figure 6 ,like Figure 7 As shown, the multispectral image includes a face area; the above step 202 can be specifically implemented through the following steps 202a to 202c.

[0089] Step 202a: The electronic device obtains spectral information of the face area in the multi-spectral image.

[0090] In the embodiment of the present application, the above-mentioned spectrum information is used to indicate the intensity information of the reflected light of each pixel in the face area at different wavelengths.

[0091] In some embodiments of the present application, when the electronic device is in a portrait shooting mode, the electronic device can obtain spectral information of a face area in a multi-spectral image.

[0092] In some embodiments of the present application, the electronic device may first determine the face area from the RGB image, and then determine the face area in the multispectral image based on the camera calibration data.

[0093] In some embodiments of the present application, when receiving a first input from a user to an RGB image, the electronic device may determine a face region from the RGB image based on an input position of the first input.

[0094] It can be understood that the first input is used to focus on the face in the RGB image. When the RGB image is displayed, the user can click on the face in the RGB image so that the electronic device determines the face area from the RGB image.

[0095] In some embodiments of the present application, when a plurality of face regions are included in an RGB image and the first input from the user is not received, the electronic device may use the region with the largest area occupied by the face as the above-mentioned face region.

[0096] In some embodiments of the present application, when acquiring spectral information of a face region in a multispectral image, the electronic device may filter out regions with excessively large spectral differences, such as eye regions, lip regions, or decorative regions.

[0097] Step 202b: The electronic device determines a first color temperature corresponding to the spectrum information.

[0098] In some embodiments of the present application, the above step 202b can be specifically implemented through the following steps 202b1 and 202b2.

[0099] Step 202b1: The electronic device reconstructs the spectrum information to obtain a spectrum curve corresponding to the spectrum information, and converts the spectrum curve into a first XYZ tristimulus value.

[0100] In some embodiments of the present application, the electronic device may convert the spectrum curve into the first XYZ value according to the tristimulus value method.

[0101] It should be noted that the detailed steps for the electronic device to perform spectral reconstruction on the spectral information, obtain the spectral curve corresponding to the spectral information, and convert the spectral curve into the first XYZ value can be found in the related technology, where the electronic device performs spectral reconstruction on the spectral information, obtains the spectral curve corresponding to the spectral information, and converts the spectral curve into the XYZ value, which will not be repeated here.

[0102] Step 202b2: The electronic device determines a first color temperature corresponding to the first XYZ value according to the correspondence between the XYZ value and the color temperature.

[0103] In some embodiments of the present application, the electronic device can obtain the first color temperature corresponding to the first XYZ value through a skin spectrum color temperature lookup table, and feed the first color temperature back to the image processing link to perform white balance restoration processing on the RGB image.

[0104] In some embodiments of the present application, the skin spectrum color temperature lookup table stores XYZ values ​​of different skin colors at different color temperatures.

[0105] In some embodiments of the present application, after converting the spectral curve into a first XYZ value, the electronic device may look up the first XYZ value in a skin spectrum color temperature lookup table to determine a first color temperature corresponding to the first XYZ value.

[0106] In some embodiments of the present application, the electronic device can obtain the spectral curves of different typical skin colors at typical color temperatures through experiments, and then further calculate the XYZ values ​​of different skin colors at typical color temperatures to obtain the above-mentioned skin spectrum color temperature lookup table.

[0107] In some examples, the typical skin color may be any of the following: light skin, medium skin, dark skin, etc. The specific skin color may be determined according to actual use requirements, and the embodiments of the present application do not limit this.

[0108] In some examples, the typical color temperature may be any of the following: D65, A, TL84, F12, etc. The specific color temperature may be determined according to actual use requirements, and the embodiments of the present application do not limit this.

[0109] Exemplarily, taking skin colors including: light skin color, medium skin color and dark skin color, and color temperatures including: D65, A and F12 as an example, the skin spectrum color temperature lookup table in the present application can be as shown in Table 2 below, which stores the XYZ values ​​of light skin color at the above three color temperatures, the XYZ values ​​of medium skin color at the three color temperatures, and the XYZ values ​​of dark skin color at the three color temperatures.

[0110] Table 2

[0111]

[0112] For example: when the first XYZ value is (17.7, 16.5, 9.6), the electronic device can look up (17.7, 16.5, 9.6) in the above Table 2. Since (17.7, 16.5, 9.6) is the XYZ value of dark skin color at the D65 color temperature, the D65 color temperature can be determined as the first color temperature corresponding to the first XYZ value.

[0113] It should be noted that the XYZ values ​​of the three different skin colors at different color temperatures included in Table 2 are only examples, and can be determined according to actual use requirements, and the present application embodiment does not limit this.

[0114] In some embodiments of the present application, when the electronic device fails to find the first XYZ value in the skin spectrum color temperature lookup table, the color temperature corresponding to the fourth XYZ value can be used as the first color temperature, and the difference between the first XYZ value and the fourth XYZ value is within a preset range.

[0115] Step 202c: The electronic device performs white balance restoration processing on the RGB image based on the first color temperature, or adjusts the skin color of the face area in the RGB image based on the first color temperature.

[0116] It can be understood that the multi-spectral white balance (Automatic White Balance, AWB) function can restore the scene color well in most scenes with white spots, with white spots as the restoration target. However, in complex color scenes without white spots, because the received reflection spectra of objects of different colors are received, direct AWB restoration is likely to cause color cast. Therefore, when there are white spots in the RGB image, the electronic device can use the AWB algorithm in the relevant technology to perform white balance restoration processing on the RGB image, that is, white balance correction. When there are no white spots in the RGB image, the electronic device can perform white balance restoration processing on the RGB image based on the first color temperature.

[0117] In some embodiments of the present application, the electronic device can determine whether there are white spots in the RGB image acquired by the imaging camera in the current scene based on the AWB algorithm.

[0118] In some examples, a white point may be understood as a pixel having an RGB value of (255, 255, 255).

[0119] In some examples, the electronic device may divide the RGB image into M*N regions of interest (ROIs), and then obtain the R single-channel average, G single-channel average, and B single-channel average of each ROI, and determine whether there are white spots in the RGB image based on the R single-channel average, G single-channel average, and B single-channel average of each ROI.

[0120] Exemplarily, when the R single-channel average value, the G single-channel average value, and the B single-channel average value of any ROI are equal, the electronic device may determine that the any ROI is a white point area, that is, white points exist in the RGB image.

[0121] Exemplarily, when the R single channel average value of any ROI is ∈ (G single channel average value - Tdn, G single channel average value + Tdn), and the B single channel average value is ∈ (G single channel average value - Tdn, G single channel average value + Tdn), the electronic device can determine that any ROI is a white point area, that is, there are white points in the RGB image. Wherein Tdn is a preset threshold.

[0122] In some embodiments of the present application, in the above step 202c, "the electronic device performs white balance restoration processing on the RGB image based on the first color temperature" can be specifically implemented through the following step 202c1.

[0123] Step 202c1: When there is no white point in the RGB image, the electronic device performs white balance restoration processing on the RGB image based on the first color temperature.

[0124] In some embodiments of the present application, the electronic device may perform white balance restoration processing on the RGB image based on the first color temperature and the AWB algorithm.

[0125] Example 3, in combination with Example 1, in portrait shooting mode, the electronic device can collect an RGB image containing "Xiao Xi" through a telephoto camera, and collect a multispectral image containing "Xiao Xi" through a multispectral camera, and then when there are no white spots in the RGB image, the electronic device can first determine the facial area of ​​"Xiao Xi", that is, the above-mentioned facial area, in the multispectral image, and then obtain the spectral information of the facial area of ​​"Xiao Xi" in the multispectral image to determine the true color temperature of the current outdoor scene based on the spectral information, and finally, based on the color temperature, perform white balance restoration processing on the RGB image containing "Xiao Xi" to make the color performance of the captured image more natural.

[0126] It should be noted that for the detailed steps of how the electronic device performs white balance restoration processing on the RGB image based on the first color temperature and the AWB algorithm, please refer to the description of how the electronic device performs white balance restoration processing on the captured image based on the color temperature and the AWB algorithm in the related art, which will not be repeated here.

[0127] In this way, since the M single-channel pixel units of the multispectral camera include at least one single-channel pixel unit whose light transmittance is greater than the threshold value, the electronic device can obtain accurate spectral information of the face area through the multispectral image collected by the multispectral camera, and then obtain the real color temperature of the scene based on the accurate spectral information of the face area, and further perform accurate white balance restoration processing on the RGB image based on the real color temperature of the scene, ensuring that the color performance of the RGB image is more natural in different environments, thereby improving the image quality of the RGB image.

[0128] In some embodiments of the present application, in the above step 202c, "the electronic device adjusts the skin color of the face area in the RGB image based on the first color temperature" can be specifically implemented through the following steps 202c2 and 202c3.

[0129] Step 202c2: The electronic device determines a first skin color level based on the first XYZ value.

[0130] In an embodiment of the present application, the first skin color level is a skin color level corresponding to the skin color of the face area in the multispectral image.

[0131] It can be understood that when designing the filters of the multi-spectral camera, the filters of the corresponding partial bands are optimized and enhanced so that the light has better transmittance and spectral accuracy in this band. Therefore, the multi-spectral camera can distinguish the differences in different skin colors. Therefore, the electronic device can divide the skin color into levels to optimize the skin color in combination with the first color temperature.

[0132] In some embodiments of the present application, the electronic device may adopt the commonly used Fitzpatrick Skin Type's 6-color skin color level, or the Google Monk Skin Tone Scale's 10-color skin color level, or customize the skin color level according to the image preference style, and then determine which skin color level corresponds to the skin color of the facial area based on the first XYZ value.

[0133] In some embodiments of the present application, the electronic device may compare the first XYZ value corresponding to the face area in the multispectral image with the XYZ values ​​corresponding to different defined skin color levels to obtain the skin color level corresponding to the skin color of the face area in the multispectral image.

[0134] Step 202c3: The electronic device maps the second XYZ value corresponding to the face area in the RGB image to a third XYZ value corresponding to the first skin color level based on the first mapping parameter.

[0135] In an embodiment of the present application, the first mapping parameter is a parameter generated by the imaging camera based on the first skin color level and the first color temperature.

[0136] In some embodiments of the present application, an image signal processor (ISP) of an imaging camera may generate a first mapping parameter based on a first skin color level and a first color temperature.

[0137] It can be understood that after obtaining the first color temperature and the first skin color level, the imaging camera can generate a first mapping parameter based on the first skin color level and the first color temperature to adjust the second XYZ value corresponding to the face area in the RGB image to the third XYZ value corresponding to the first skin color level to optimize the skin color of the face area in the RGB image.

[0138] In some embodiments of the present application, the first skin color level may correspond to an XYZ value range, for example, the X value range corresponding to the light skin color level is 90-100, the Y value range is 80-90, and the Z value range is 70-80. The electronic device may map the second XYZ value corresponding to the face area in the RGB image to a third XYZ value in the XYZ value range corresponding to the first skin color level based on the first mapping parameter.

[0139] For example, when the first skin color level is a light skin color level, the X value range corresponding to the light skin color level is 90-100, the Y value range is 80-90, the Z value range is 70-80, and the second XYZ value corresponding to the face area in the RGB image is (60, 50, 40), the electronic device can adjust the second XYZ value to (95, 80, 70) based on the first mapping parameter, that is, adjust it to the XYZ value corresponding to the light skin color level.

[0140] Example 4, combined with Example 2, in the portrait shooting mode, the electronic device can collect an RGB image containing "Xiaoxi" through the main camera, and collect a multispectral image containing "Xiaoxi" through the multispectral camera, and then the electronic device can first determine the facial area of ​​"Xiaoxi" in the multispectral image, that is, the above-mentioned facial area, and then obtain the spectral information of the facial area of ​​"Xiaoxi" in the multispectral image to determine the true color temperature of the current indoor scene according to the spectral information, and then the electronic device can adjust the skin color of the facial area of ​​"Xiaoxi" in the RGB image based on the color temperature and the facial skin color level of "Xiaoxi", that is, optimize the facial skin color of "Xiaoxi", so that the facial skin color of "Xiaoxi" in the captured image is more natural and the character image is more beautiful.

[0141] In this way, since the M single-channel pixel units of the multispectral camera include at least one single-channel pixel unit whose light transmittance is greater than the threshold value, the electronic device can obtain accurate spectral information of the face area through the multispectral image collected by the multispectral camera, and then obtain the real color temperature of the scene, and then generate a first mapping parameter by combining the color temperature and the skin color level corresponding to the skin color of the face area, and the second XYZ value corresponding to the face area in the RGB image collected by the imaging camera can be adjusted to the third XYZ value corresponding to the skin color level corresponding to the skin color of the face area, thereby realizing the optimization of the face skin color when shooting the face, making the face skin color in the RGB image more natural and the character image more beautiful, thereby improving the image quality of the captured RGB image.

[0142] It should be noted that the XYZ values ​​mentioned in the above embodiments may be CIE XYZ values.

[0143] In some embodiments of the present application, Figure 6 ,like Figure 8As shown, the above step 202 can be specifically implemented through the following steps 202d to 202f.

[0144] Step 202d: The electronic device generates a first depth image based on the camera calibration data, the RGB image and the multispectral image.

[0145] In an embodiment of the present application, the camera calibration data includes: internal parameters of the imaging camera, internal parameters of the multispectral camera, external parameters of the imaging camera and external parameters of the multispectral camera.

[0146] In some embodiments of the present application, the electronic device can obtain a first depth image of the current scene based on camera calibration data, RGB images and multi-spectral images according to a parallax method of binocular stereo vision.

[0147] In some embodiments of the present application, before the electronic device generates a first depth image based on camera calibration data, an RGB image, and a multispectral image, the electronic device may map the multispectral image to an RGB color space, and perform appropriate image optimization processing such as noise reduction and edge enhancement, and may map the RGB image to an RGB space, and perform appropriate image optimization processing such as noise reduction and edge enhancement.

[0148] Step 202e: The electronic device fuses the first depth image with the RGB image to obtain a second depth image.

[0149] In an embodiment of the present application, the second depth image includes depth information of the photographed object.

[0150] In some embodiments of the present application, the electronic device may align the first depth image with the RGB image, that is, perform the above-mentioned fusion process, to obtain the second depth image.

[0151] In some embodiments of the present application, the second depth image is an RGBD image.

[0152] Step 202f: The electronic device performs focusing processing on the RGB image based on the depth information of the photographed object, or performs background blur processing on the RGB image based on the depth information of the photographed object.

[0153] It can be understood that, since the second depth image includes depth information of the photographed object, the RGB image can be focused or the background blurred based on the depth information of the photographed object.

[0154] In some embodiments of the present application, after acquiring the depth information of the photographed object, the electronic device may reversely infer the VCM adjustment amount based on the depth information of the photographed object to complete the focus on the photographed object.

[0155] In some embodiments of the present application, after acquiring the depth information of the subject, the electronic device can feedback the depth information of the subject with the phase focus (Phase Detection Auto Focus, PDAF) or automatic focus (Automatic Focus, AF) of the imaging camera to assist in focusing on the subject.

[0156] In some embodiments of the present application, in a shooting scene with a large flow of people and when the RGB image includes many objects, the electronic device can set the focus logic to "focus on the nearest frontal face", that is, the object in the RGB image that is closest to the camera and has a frontal face is determined as the shooting object, and then the shooting object is focused to avoid the problem of the camera frequently switching the focus object during shooting, resulting in unclear shooting of the shooting object in the RGB image.

[0157] In some embodiments of the present application, after acquiring the depth information of the subject, the electronic device can determine the distance information between the front background and the back background of the subject based on the depth information of the subject to blur the background of the subject.

[0158] In some embodiments of the present application, when a large number of objects are included in an RGB image, the electronic device can receive user input, and determine the photographed object in the RGB image based on the input position of the input to blur the background of the photographed object.

[0159] In some embodiments of the present application, when an RGB image includes a large number of objects, the electronic device can determine the photographed object from the RGB image based on the posture information corresponding to each object, the position of each object in the RGB image, and the distance between each object and the electronic device, and then blur the background of the photographed object through the depth information of the photographed object.

[0160] Specifically, the electronic device can determine the weight value of each object based on the posture information corresponding to each object, the position of each object in the RGB image, and the distance between each object and the electronic device, and then select the object with the highest weight value as the shooting object.

[0161] In some examples, when the posture information of the object indicates that the object faces the imaging camera from the side, the weight value of the object is lower; when the posture information of the object indicates that the object faces the imaging camera from the front, the weight value of the object is higher.

[0162] In some examples, when the position of the object in the RGB image is at the center of the shooting, the weight value of the object is higher; when the position of the object in the RGB image is at the edge, the weight value of the object is lower.

[0163] In some examples, when the distance between the object and the electronic device is farther, the weight value of the object is lower; when the distance between the object and the electronic device is closer, the weight value of the object is higher.

[0164] For example: assuming that the RGB image includes object A and object B, object A faces the imaging camera head-on, object B faces the imaging camera sideways, object A is at the center of the shooting, object B is at the edge, object A is far from the electronic device, and object B is close to the electronic device, the electronic device can determine that the weight value of object A is 0.7 and the weight value of object B is 0.3, and then take object A as the shooting object.

[0165] Example 5, combined with Example 1, when there is a large flow of people in an outdoor scene, the electronic device collects an RGB image containing "Xiaoxi" through a telephoto camera, and collects a multispectral image containing "Xiaoxi" through a multispectral camera. The electronic device can generate a depth image of the current scene based on the camera calibration data, the RGB image containing "Xiaoxi" and the multispectral image containing "Xiaoxi", and then further fuse the depth image of the current scene with the RGB image containing "Xiaoxi" to obtain an RGBD image, which includes the depth information of "Xiaoxi", so that the electronic device can accurately focus on the face of "Xiaoxi" based on the depth information, so that the face of "Xiaoxi" in the captured image is clear.

[0166] Example 6. In combination with Example 1, the electronic device collects an RGB image containing "Xiaoxi" through a telephoto camera, and collects a multispectral image containing "Xiaoxi" through a multispectral camera. When there is a large flow of people in an outdoor scene, Xiaoyu can trigger the electronic device to enter the background blur mode, so that the electronic device generates a depth image of the current scene based on the camera calibration data, the RGB image containing "Xiaoxi" and the multispectral image containing "Xiaoxi". Then, the depth image of the current scene can be further fused with the RGB image containing "Xiaoxi" to obtain an RGBD image, which includes the depth information of "Xiaoxi". Therefore, the electronic device can accurately blur the background of "Xiaoxi" based on the depth information, so that the captured image can accurately highlight "Xiaoxi".

[0167] In this way, since the scene depth information can be obtained through the binocular stereo vision system formed by the multi-spectral camera and the imaging camera, the subject in the RGB image can be accurately focused or the background can be blurred, thereby significantly improving the focus effect and blur effect of the RGB image, thereby improving the image quality of the RGB image. In addition, through the depth information of the subject, real people and advertising paintings in the RGB image can be distinguished, so that in a multi-object scene, the electronic device can accurately select the subject to be photographed.

[0168] In some embodiments of the present application, Figure 6 ,like Fig. 9 As shown, the above step 202 can be specifically implemented through the following steps 202g to 202i.

[0169] Step 202g: When no face is recognized in the RGB image, the electronic device determines the face area in the multispectral image.

[0170] In some embodiments of the present application, the multispectral image may be obtained based on image data generated by at least one multi-channel pixel unit.

[0171] In some embodiments of the present application, in a portrait shooting mode, the electronic device can identify whether a face is included in an RGB image. When a face is not identified in the RGB image, the electronic device can determine a face area in the multispectral image.

[0172] It is understandable that in dark conditions, if the imaging camera initially estimates that the exposure is insufficient, there is no face AE or face AE cannot be started due to being too dark, the exposure brightness of the face will be low. Therefore, in the face shooting mode, the electronic device can first identify whether the RGB image includes a face.

[0173] Step 202h: The electronic device determines coordinate information of the face area in the RGB image based on the camera calibration data and the face area in the multispectral image.

[0174] In some embodiments of the present application, the coordinate information of the face area in the RGB image includes the coordinate information of all pixels in the face area.

[0175] In some embodiments of the present application, the electronic device can map the face area in the multispectral image to the face area corresponding to the RGB image based on the camera calibration data to determine the coordinate information of the face area in the RGB image.

[0176] Step 202i: The electronic device adjusts the exposure parameters of the face area in the RGB image based on the coordinate information of the face area in the RGB image.

[0177] In some embodiments of the present application, the electronic device may determine the face region in the RGB image based on the coordinate information of the face region in the RGB image to adjust the exposure parameters of the face region in the RGB image.

[0178] In some embodiments of the present application, the electronic device can adjust the exposure parameters of the face area in the RGB image through the image processing link of the imaging camera. For example, the image processing link of the imaging camera calculates the regional brightness of the face area in the RGB image to adjust the exposure parameters of the face area in the RGB image to a reasonable range.

[0179] For example, in a 10-bit image, the maximum exposure parameter is 1023. The exposure parameter value of the face area in the original RGB image may be less than 70, and the face cannot be recognized. After adjustment, the exposure parameter value of the face area in the RGB image can be 300.

[0180] Example 7, combined with Example 2, in the portrait shooting mode, the electronic device collects an RGB image containing "Xiaoxi" through a telephoto camera, and collects a multispectral image containing "Xiaoxi" through a multispectral camera. The electronic device can then identify whether the RGB image includes the face of "Xiaoxi". It can be understood that due to the dark indoor environment, the electronic device may not be able to recognize the face of "Xiaoxi" in the RGB image. When it is recognized that the face of "Xiaoxi" is not included in the RGB image, the electronic device can determine the face area of ​​"Xiaoxi" in the RGB image based on the camera calibration data and the face area of ​​"Xiaoxi" in the multispectral image to adjust the exposure parameters of the face area of ​​"Xiaoxi" in the RGB image, so that the captured image achieves the best low-light portrait effect.

[0181] In this way, since the multispectral camera includes a multi-channel pixel unit, that is, the above-mentioned mono pixel unit, the multispectral image captured by the multispectral camera can have richer facial detail information in a dark light environment. Therefore, the multispectral camera can be used to assist the imaging camera in reasonably exposing the face area in the captured RGB image, that is, the face area in the RGB image is reasonably exposed through the multispectral image, so that the face in the RGB image captured in a dark light environment after multispectral image processing is clearer, thereby improving the image quality of the RGB image.

[0182] In some embodiments of the present application, before the above step 202i, the image processing method provided by the embodiment of the present application further includes the following step 401.

[0183] Step 401: The electronic device performs image fusion processing on the RGB image after adjusting the exposure parameters and the multispectral image to obtain an RGB image after image fusion processing.

[0184] In some embodiments of the present application, when the RGB image has large noise, the electronic device can fuse the RGB image with the multispectral image to obtain an RGB image after image fusion processing, so as to achieve the purpose of noise removal and edge enhancement of the RGB image.

[0185] It should be noted that for the detailed steps of image fusion processing of RGB images and multispectral images, reference may be made to the description of fusion processing of RGB images and multispectral images in the related art, which will not be repeated here.

[0186] In this way, in a dark light environment, after the multispectral camera assists the imaging camera to reasonably expose the face area in the RGB image, the multispectral image can be fused with the RGB image, thereby removing the RGB image noise and enhancing the RGB image edge, further improving the image quality of the RGB image taken in a dark light environment.

[0187] It should be noted that the CIE XYZ color space is used to describe the color coordinates in the embodiment of the present application, which is only an example. In actual use, other color spaces may also be used to describe the color coordinates, such as CIE Lab, etc. The specific color space can be determined according to actual use requirements, and the embodiment of the present application does not limit this.

[0188] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0189] Fig.10 A possible structural diagram of an image processing device involved in an embodiment of the present application is shown. Fig.10 As shown, the image processing device 70 may include: a collection module 71 and a processing module 72 .

[0190] The acquisition module 71 is used to acquire RGB images through an imaging camera and to acquire multispectral images through a multispectral camera;

[0191] The processing module 72 is used to perform image processing on the RGB image collected by the collection module according to the multispectral image collected by the collection module 71.

[0192] An embodiment of the present application provides an image processing device. On the one hand, since the multispectral camera included in the image processing device includes two different pixel units, the multispectral camera can not only collect light in different bands, but also collect light in the full-pass band. In this way, compared with the existing method that cannot collect light in the full-pass band, the multispectral image collected by the multispectral camera in a dark light environment can have richer image detail information, so that the RGB image collected in the dark light environment after the multispectral image processing is clearer; on the other hand, since the light transmittance of the pixel unit in the multispectral camera can be set according to the needs of the user, the multispectral image collected by the multispectral camera can have more accurate image spectral information. Therefore, the multispectral image can provide rich color information and image detail information, so that the color performance of the RGB image after the multispectral image processing is more accurate and clearer; thereby improving the image quality of the RGB image.

[0193] In one possible implementation, the multispectral image includes a face region; the processing module 72 is specifically used to obtain spectral information of the face region in the multispectral image, the spectral information is used to indicate the intensity information of the reflected light of each pixel in the face region at different wavelengths; and determine a first color temperature corresponding to the spectral information; and based on the first color temperature, perform white balance restoration processing on the RGB image, or adjust the skin color of the face region in the RGB image based on the first color temperature.

[0194] In a possible implementation, the processing module 72 is specifically used to perform spectral reconstruction on the spectral information, obtain a spectral curve corresponding to the spectral information, and convert the spectral curve into a first XYZ tristimulus value; and determine the first color temperature corresponding to the first XYZ value according to the correspondence between the XYZ value and the color temperature.

[0195] In a possible implementation, the processing module 72 is specifically configured to perform white balance restoration processing on the RGB image based on the first color temperature when no white point exists in the RGB image.

[0196] In one possible implementation, the processing module 72 is specifically used to determine a first skin color level based on the first XYZ value, where the first skin color level is the skin color level corresponding to the skin color of the face area in the multispectral image; and based on a first mapping parameter, map the second XYZ value corresponding to the face area in the RGB image to a third XYZ value corresponding to the first skin color level; wherein the first mapping parameter is a parameter generated by the imaging camera based on the first skin color level and the first color temperature.

[0197] In one possible implementation, the processing module 72 is specifically used to generate a first depth image based on camera calibration data, an RGB image and a multispectral image, the camera calibration data including: internal parameters of an imaging camera, internal parameters of a multispectral camera, external parameters of an imaging camera and external parameters of a multispectral camera; and fuse the first depth image with the RGB image to obtain a second depth image, the second depth image including depth information of a photographed object; and focus the RGB image based on the depth information of the photographed object, or blur the background of the RGB image based on the depth information of the photographed object.

[0198] In one possible implementation, the processing module 72 is specifically used to determine the face area in the multispectral image when no face is recognized in the RGB image; and determine the coordinate information of the face area in the RGB image based on the camera calibration data and the face area in the multispectral image; and adjust the exposure parameters of the face area in the RGB image based on the coordinate information of the face area in the RGB image.

[0199] In a possible implementation, the processing module 72 is further configured to, after adjusting the exposure parameters of the face region in the RGB image, perform image fusion processing on the RGB image after the exposure parameters are adjusted and the multispectral image to obtain the RGB image after the image fusion processing.

[0200] In one possible implementation, the processing module 72 is also used to perform internal parameter calibration processing on the imaging camera and the multispectral camera respectively before performing image processing on the RGB image according to the multispectral image, and to perform external parameter calibration processing on the imaging camera and the multispectral camera respectively to generate camera calibration data.

[0201] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0202] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0203] The image processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0204] Alternatively, if Fig.11 As shown, an embodiment of the present application also provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores programs or instructions that can be executed on the processor 901, and when the program or instructions are executed by the processor 901, the various steps of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0205] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0206] Fig.12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0207] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0208] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0209] The processor 110 is used to collect RGB images through an imaging camera and to collect multispectral images through a multispectral camera; and to perform image processing on the RGB images according to the multispectral images.

[0210] An embodiment of the present application provides an electronic device. On the one hand, since the multispectral camera included in the electronic device includes two different pixel units, the multispectral camera can not only collect light in different bands, but also collect light in the full-pass band. In this way, compared with the existing light that cannot collect light in the full-pass band, the multispectral image collected by the multispectral camera in a dark light environment can have richer image detail information, so that the RGB image collected in the dark light environment after multispectral image processing is clearer; on the other hand, since the light transmittance of the pixel unit in the multispectral camera can be set according to user needs, the multispectral image collected by the multispectral camera can have more accurate image spectral information. Therefore, the multispectral image can provide rich color information and image detail information, so that the color performance of the RGB image after multispectral image processing is more accurate and clearer; thereby improving the image quality of the RGB image.

[0211] In some embodiments of the present application, the multispectral image includes a face region; the processor 110 is specifically used to obtain spectral information of the face region in the multispectral image, the spectral information being used to indicate intensity information of reflected light of each pixel in the face region at different wavelengths; and determine a first color temperature corresponding to the spectral information; and based on the first color temperature, perform white balance restoration processing on the RGB image, or adjust the skin color of the face region in the RGB image based on the first color temperature.

[0212] In some embodiments of the present application, the processor 110 is specifically used to perform spectral reconstruction on the spectral information, obtain a spectral curve corresponding to the spectral information, and convert the spectral curve into a first XYZ tristimulus value; and determine a first color temperature corresponding to the first XYZ value based on the correspondence between the XYZ value and the color temperature.

[0213] In some embodiments of the present application, the processor 110 is specifically configured to perform white balance restoration processing on the RGB image based on the first color temperature when there is no white point in the RGB image.

[0214] In some embodiments of the present application, the processor 110 is specifically used to determine a first skin color level based on a first XYZ value, where the first skin color level is the skin color level corresponding to the skin color of a face area in a multispectral image; and based on a first mapping parameter, map a second XYZ value corresponding to the face area in the RGB image to a third XYZ value corresponding to the first skin color level; wherein the first mapping parameter is a parameter generated by the imaging camera based on the first skin color level and the first color temperature.

[0215] In some embodiments of the present application, the processor 110 is specifically used to generate a first depth image based on camera calibration data, an RGB image and a multispectral image, the camera calibration data including: internal parameters of an imaging camera, internal parameters of a multispectral camera, external parameters of an imaging camera and external parameters of a multispectral camera; and fuse the first depth image with the RGB image to obtain a second depth image, the second depth image including depth information of a photographed object; and focus the RGB image based on the depth information of the photographed object, or blur the background of the RGB image based on the depth information of the photographed object.

[0216] In some embodiments of the present application, the processor 110 is specifically used to determine a face area in a multispectral image when no face is recognized in the RGB image; and determine coordinate information of the face area in the RGB image based on camera calibration data and the face area in the multispectral image; and adjust exposure parameters of the face area in the RGB image based on the coordinate information of the face area in the RGB image.

[0217] In some embodiments of the present application, the processor 110 is further configured to, after adjusting the exposure parameters of the face area in the RGB image, perform image fusion processing on the RGB image after the exposure parameters are adjusted and the multispectral image to obtain the RGB image after the image fusion processing.

[0218] In some embodiments of the present application, the processor 110 is also used to perform internal parameter calibration processing on the imaging camera and the multispectral camera respectively before performing image processing on the RGB image according to the multispectral image, and to perform external parameter calibration processing on the imaging camera and the multispectral camera respectively to generate camera calibration data.

[0219] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0220] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.

[0221] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0222] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0223] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0224] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0225] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0226] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0227] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0228] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0229] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0230] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0231] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A multispectral camera, characterized in that: include: M single-channel pixel units and at least one multi-channel pixel unit, each single-channel pixel unit is used to generate image data according to a light signal of a spectral band, and each multi-channel pixel unit is used to generate image data according to light signals of all spectral bands, and M is a positive integer; The M single-channel pixel units include at least one single-channel pixel unit whose light transmittance is greater than a threshold value.

2. An electronic device, characterized in that: It comprises an imaging camera and the multispectral camera as claimed in claim 1, wherein the field of view of the multispectral camera is larger than the field of view of the imaging camera.

3. An image processing method, executed by the electronic device according to claim 2, characterized in that: The method comprises: Collecting RGB images through an imaging camera, and collecting multispectral images through the multispectral camera; The RGB image is processed according to the multispectral image.

4. The method according to claim 3, characterized in that: The multispectral image includes a face area; The step of performing image processing on the RGB image according to the multispectral image comprises: Acquire spectral information of a face region in the multispectral image, where the spectral information is used to indicate intensity information of reflected light of each pixel in the face region at different wavelengths; Determining a first color temperature corresponding to the spectral information; Based on the first color temperature, white balance restoration processing is performed on the RGB image, or based on the first color temperature, the skin color of the face area in the RGB image is adjusted.

5. The method according to claim 4, characterized in that The determining a first color temperature corresponding to the spectral information includes: Performing spectral reconstruction on the spectral information to obtain a spectral curve corresponding to the spectral information, and converting the spectral curve into a first XYZ tristimulus value; A first color temperature corresponding to the first XYZ value is determined according to a correspondence between the XYZ value and the color temperature.

6. The method according to claim 4 or 5, characterized in that: The performing white balance restoration processing on the RGB image based on the first color temperature includes: When there is no white point in the RGB image, a white balance restoration process is performed on the RGB image based on the first color temperature.

7. The method according to claim 5, characterized in that The adjusting the skin color of the face area in the RGB image based on the first color temperature includes: Determine a first skin color level based on the first XYZ value, where the first skin color level is a skin color level corresponding to the skin color of a face area in the multispectral image; Based on the first mapping parameter, mapping the second XYZ value corresponding to the face area in the RGB image to the third XYZ value corresponding to the first skin color level; The first mapping parameter is a parameter generated by the imaging camera based on the first skin color level and the first color temperature.

8. The method according to claim 3, characterized in that The step of performing image processing on the RGB image according to the multispectral image comprises: Generate a first depth image based on camera calibration data, the RGB image and the multispectral image, the camera calibration data including: an internal parameter of the imaging camera, an internal parameter of the multispectral camera, an external parameter of the imaging camera and an external parameter between the multispectral cameras; Fusing the first depth image with the RGB image to obtain a second depth image, where the second depth image includes depth information of the photographed object; Based on the depth information of the photographed object, the RGB image is subjected to focusing processing, or based on the depth information of the photographed object, the RGB image is subjected to background blur processing.

9. The method according to claim 3, characterized in that: The step of performing image processing on the RGB image according to the multispectral image comprises: If no human face is recognized in the RGB image, determining a human face region in the multispectral image; Determine coordinate information of the face area in the RGB image based on the camera calibration data and the face area in the multispectral image; Based on the coordinate information of the face area in the RGB image, the exposure parameter of the face area in the RGB image is adjusted.

10. The method according to claim 9, characterized in that After adjusting the exposure parameters of the face area in the RGB image, the method further includes: The RGB image after adjusting the exposure parameters is subjected to image fusion processing with the multispectral image to obtain an RGB image after image fusion processing.

11. The method according to claim 8 or 9, characterized in that: Before performing image processing on the RGB image according to the multispectral image, the method further includes: The imaging camera and the multi-spectral camera are respectively subjected to internal parameter calibration processing, and the imaging camera and the multi-spectral camera are respectively subjected to external parameter calibration processing to generate camera calibration data.

12. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the image processing method according to any one of claims 3 to 11 are implemented.

Images