Image sensor, multispectral camera, electronic equipment and depth image acquisition method and device

Abstract

The invention discloses an image sensor, a multispectral camera, electronic equipment and a depth image acquisition method, and belongs to the technical field of image processing. The image sensor provided by the embodiment of the invention comprises a pixel array, wherein the pixel array comprises N single-channel pixel units and a multi-channel pixel unit; one single-channel pixel unit is used for sensing an optical signal of one spectral band, the multi-channel pixel unit is used for sensing optical signals of at least two spectral bands, and N is a positive integer greater than 1.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image sensor, a multispectral camera, an electronic device, and a depth image acquisition method and device. Background Art

[0002] Generally, when an electronic device needs to obtain a depth image of a subject, the electronic device can emit ambient light to the subject through a depth camera and generate a depth image based on reflected light of the ambient light, so that the electronic device can obtain an accurate depth image of the subject.

[0003] However, there may be a situation where the electronic device cannot capture the depth image of the subject through the depth camera. At this time, the electronic device may not be able to emit ambient light to the subject and generate a depth image based on the reflected light of the ambient light. Therefore, the electronic device cannot accurately obtain the depth image of the subject. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image sensor, a multispectral camera, an electronic device, a depth image acquisition method and an apparatus, so that when a color image of a photographed object is acquired by a rear camera, the electronic device can acquire a multispectral image of the photographed object by the multispectral camera to obtain images of the photographed object at different perspectives, and perform depth calculation based on the images of the photographed object at different perspectives to directly obtain an accurate depth image of the photographed object without acquiring the depth image of the photographed object by the depth camera. Therefore, the situation where the electronic device cannot acquire an accurate depth image of the photographed object can be avoided, thereby improving the accuracy of the depth image acquired by the electronic device.

[0005] In a first aspect, an embodiment of the present application provides an image sensor, which includes: a pixel array, which includes N single-channel pixel units and multi-channel pixel units; a single-channel pixel unit is used to sense a light signal in a spectral band, and the multi-channel pixel unit is used to sense light signals in at least two spectral bands, and N is a positive integer greater than 1.

[0006] In a second aspect, an embodiment of the present application provides a multispectral camera, which includes: an image sensor as described in the first aspect.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a rear camera and a multispectral camera as described in the second aspect; wherein the rear camera and the multispectral camera are arranged at different positions on the same side of the electronic device, and the field of view angle of the rear camera is within the field of view angle of the multispectral camera.

[0008] In a fourth aspect, an embodiment of the present application provides a depth image acquisition method, which is executed by the electronic device described in the third aspect, the method comprising: collecting a color image through a rear camera of the electronic device, and collecting a multispectral image through a multispectral camera of the electronic device; performing depth calculation based on the color image and the multispectral image, and outputting a depth image.

[0009] In a fifth aspect, an embodiment of the present application provides a depth image acquisition device, the device comprising: an acquisition module, used to acquire a color image through a rear camera of the depth image acquisition device, and to acquire a multispectral image through a multispectral camera of the depth image acquisition device. A processing module, used to perform depth calculation based on the color image and multispectral image acquired by the acquisition module, and output a depth image.

[0010] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the fourth aspect are implemented.

[0011] In a seventh aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the fourth aspect are implemented.

[0012] In an eighth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the fourth aspect.

[0013] In a ninth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the fourth aspect.

[0014] In the embodiment of the present application, since the pixel array includes N single-channel pixel units and multi-channel pixel units, when the image sensor is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to different spectral bands, and the multi-channel pixel unit can sense light signals of at least two spectral bands and generate multi-spectral channel images corresponding to at least two spectral bands. In this way, in addition to including different spectral channel images corresponding to different spectral bands, the multi-spectral image output by the image sensor can also include multi-spectral channel images corresponding to at least two spectral bands, and the multi-spectral channel image includes more object information of the photographed object. That is, the multi-spectral image output by the image sensor can include more object information of the photographed object, instead of only including different spectral channel images corresponding to different spectral bands as in the related art, that is, the multi-spectral image output by the image sensor only includes less object information of the photographed object. Therefore, after the image sensor outputs the multi-spectral image, the electronic device can perform depth calculation based on the multi-spectral channel image including more object information of the photographed object to obtain an accurate depth image of the photographed object, thereby improving the accuracy of the depth image acquired by the electronic device.

[0015] In the embodiment of the present application, since the pixel array of the image sensor of the multi-spectral camera includes N single-channel pixel units and multi-channel pixel units, when the multi-spectral camera is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to different spectral bands, and the multi-channel pixel unit can sense light signals of at least two spectral bands and generate multi-spectral channel images corresponding to at least two spectral bands. In this way, the multi-spectral image output by the multi-spectral camera can include not only different spectral channel images corresponding to different spectral bands, but also multi-spectral channel images corresponding to at least two spectral bands. The multispectral channel image includes more object information of the photographed object, that is, the multispectral image output by the multispectral camera can include more object information of the photographed object, instead of the multispectral image output by the multispectral camera in the related art only including different spectral channel images corresponding to different spectral bands, that is, the multispectral image output by the multispectral camera only includes less object information of the photographed object. Therefore, after the multispectral camera outputs the multispectral image, the electronic device can perform depth calculation based on the multispectral channel image including more object information of the photographed object to obtain an accurate depth image of the photographed object, thereby improving the accuracy of the depth image acquired by the electronic device.

[0016] In the embodiment of the present application, since the pixel array of the image sensor of the multispectral camera of the electronic device includes N single-channel pixel units and multi-channel pixel units, when the multispectral camera is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to different spectral bands, and the multi-channel pixel unit can sense light signals of at least two spectral bands and generate multispectral channel images corresponding to at least two spectral bands. In this way, in addition to including different spectral channel images corresponding to different spectral bands, the multispectral image output by the multispectral camera can also include multispectral channel images corresponding to at least two spectral bands, and the multispectral channel image includes more object information of the photographed object, that is, the multispectral image output by the multispectral camera can include more object information of the photographed object, instead of the multispectral image output by the multispectral camera generally including only different spectral channel images corresponding to different spectral bands as in the related art, that is, the multispectral image output by the multispectral camera only includes less object information of the photographed object. Therefore, after the multispectral camera outputs the multispectral image, the electronic device can perform depth calculation based on the multispectral channel image including more object information of the photographed object. Furthermore, since the rear camera and the multispectral camera are arranged at different positions on the same side of the electronic device, the field of view of the rear camera is within the field of view of the multispectral camera. This ensures that the rear camera and the multispectral camera can capture images of the same object from different perspectives. In this way, when the color image of the object is captured by the rear camera, the electronic device can capture the multispectral image of the object through the multispectral camera to obtain images of the object from different perspectives, and perform depth calculation based on the images of the object from different perspectives to directly obtain an accurate depth image of the object without the need to capture the depth image of the object through the depth camera. Therefore, it can avoid the situation where the electronic device cannot obtain an accurate depth image of the object, thereby improving the accuracy of the depth image acquired by the electronic device.

[0017] In the embodiment of the present application, since the pixel array of the image sensor of the multispectral camera of the electronic device includes N single-channel pixel units and multi-channel pixel units, when the multispectral camera is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to different spectral bands, and the multi-channel pixel unit can sense light signals of at least two spectral bands and generate multispectral channel images corresponding to at least two spectral bands. In this way, in addition to including different spectral channel images corresponding to different spectral bands, the multispectral image output by the multispectral camera can also include multispectral channel images corresponding to at least two spectral bands, and the multispectral channel image includes more object information of the photographed object, that is, the multispectral image output by the multispectral camera can include more object information of the photographed object, instead of the multispectral image output by the multispectral camera generally including only different spectral channel images corresponding to different spectral bands as in the related art, that is, the multispectral image output by the multispectral camera only includes less object information of the photographed object. Therefore, after the multispectral camera outputs the multispectral image, the electronic device can perform depth calculation based on the multispectral channel image including more object information of the photographed object. Furthermore, since the rear camera and the multispectral camera are arranged at different positions on the same side of the electronic device, the field of view of the rear camera is within the field of view of the multispectral camera. This ensures that the rear camera and the multispectral camera can capture images of the same object from different perspectives. In this way, when the color image of the object is captured by the rear camera, the electronic device can capture the multispectral image of the object through the multispectral camera to obtain images of the object from different perspectives, and perform depth calculation based on the images of the object from different perspectives to directly obtain an accurate depth image of the object without the need to capture the depth image of the object through the depth camera. Therefore, it can avoid the situation where the electronic device cannot obtain an accurate depth image of the object, thereby improving the accuracy of the depth image acquired by the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of the structure of an image sensor provided by some embodiments of the present application;

[0019] Figure 2 is a schematic diagram of the structure of an image sensor provided by some embodiments of the present application;

[0020] Figure 3 is a schematic diagram of the structure of an electronic device provided by some embodiments of the present application;

[0021] Figure 4 is a flowchart of a depth image acquisition method provided by some embodiments of the present application;

[0022] Figure 5 is a flowchart of a depth image acquisition method provided by some embodiments of the present application;

[0023] Figure 6 is a flowchart of a depth image acquisition method provided by some embodiments of the present application;

[0024] Figure 7 is a flowchart of a depth image acquisition method provided by some embodiments of the present application;

[0025] Figure 8 is a flowchart of a depth image acquisition method provided by some embodiments of the present application;

[0026] Fig. 9 is a schematic diagram of the structure of a depth image acquisition device provided in some embodiments of the present application;

[0027] Fig.10 is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application;

[0028] Fig.11 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0032] The professional terms involved in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0033] A multispectral imaging camera is a sensor that can capture spectral information in multiple wavelength ranges at the same time. The principle of a multispectral imaging camera is to use an optical filter or a spectroscopic element to separate and capture light from different wavelength ranges, and generate different image data based on the captured light. Different image data include spectral information of different wavelengths, so that electronic devices can analyze spectral information of different wavelengths through different image data. Traditional color cameras use three sensors to capture light in the three wavelength ranges of red, green and blue respectively, while multispectral imaging sensors can capture light in a wider wavelength range, usually including visible light, infrared light and ultraviolet light. Among them, a multispectral imaging camera can use multiple narrowband filters or spectroscopic elements, such as prisms or gratings, to separate light of different wavelengths and then transmit them to corresponding photosensitive pixels. By capturing spectral information in multiple wavelength ranges at the same time, a multispectral imaging camera can provide richer spectral information, which can be applied to many fields, such as agriculture, environmental monitoring, geological exploration, etc.

[0034] Typically, multispectral cameras include narrowband filtering cameras and spectral computational reconstruction cameras.

[0035] A narrowband filter camera is a sensor that can obtain multiple narrowband spectral information. The principle of a narrowband filter camera is to place a specific narrowband filter on each photosensitive pixel of the image sensor. Each narrowband filter can be designed to only allow light within a specific wavelength range to pass through, so that light in different specific wavelength ranges can pass through different narrowband filters and be received by different photosensitive pixels, generating different image data. The different image data includes spectral information of different specific wavelength ranges, so that electronic equipment can analyze spectral information of different specific wavelength ranges through the different image data.

[0036] Regarding spectral calculation and reconstruction cameras: Spectral calculation and reconstruction cameras achieve modulation of incident light in the spectral domain by setting specially designed filters on the photosensitive pixels of the pixel array, using image sensors to complete the mapping measurement from the spectral domain to the electrical domain, and then using compressed sensing algorithms to perform computational reconstruction of the spectrum. Through large-scale array integration of photosensitive pixels and filters, real-time spectral imaging is finally achieved. Spectral calculation and reconstruction cameras are different from narrow-band filtering multi-spectral sensors. Narrow-band filtering only transmits light in extremely narrow bands, while the filters on the photosensitive pixels of spectral calculation and reconstruction cameras are specially customized to form a modulated transmittance curve. The photosensitive pixel design of spectral calculation and reconstruction cameras is usually also array-type. Common ones include 3×3 and 4×4 arrays, where the distribution of photosensitive pixels on the pixel array is a 3×3 array. The 4×4 array includes photosensitive pixels C0, C1, C2, C3, C4, C5, C6, C7, C8 and C9. The filter on each photosensitive pixel can pass light of different wavelengths, and each photosensitive pixel collects the light to form image data of spectral images of different spectral bands.

[0037] The image sensor, multispectral camera, electronic device, depth image acquisition method and device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0038] Figure 1 FIG. 1 shows a schematic diagram of the structure of an image sensor provided in an embodiment of the present application. Figure 1 As shown, the image sensor provided by the embodiment of the present application includes: a pixel array, which includes N single-channel pixel units 10 and multi-channel pixel units 11; a single-channel pixel unit 10 is used to sense a light signal in a spectral band, and the multi-channel pixel unit 11 is used to sense light signals in at least two spectral bands, and N is a positive integer greater than 1.

[0039] It should be noted that Figure 1 In the figure, the number N of the N single-channel pixel units 10 is 16. Those skilled in the art can set the number of the N single-channel pixel units 10 according to requirements.

[0040] In some embodiments of the present application, for each of the N single-channel pixel units 10, a single-channel pixel unit 10 may include a photosensitive pixel and a single-channel filter, and the single-channel filter is used to pass a light signal of a spectral band. It can be understood that since the single-channel filter of a single-channel pixel unit 10 can pass a light signal of a spectral band, the photosensitive pixel of the single-channel pixel unit 10 can sense the light signal of a spectral band passed by the single-channel filter, that is, the single-channel pixel unit 10 can sense the light signal of a spectral band.

[0041] In some embodiments of the present application, the number N of the above-mentioned single-channel pixel units 10 can be any number and can be set according to usage requirements.

[0042] In some embodiments of the present application, the multi-channel pixel unit 11 may include photosensitive pixels and a multi-channel filter, and the multi-channel filter is used to pass light signals of at least two spectral bands. It can be understood that since the multi-channel filter of the multi-channel pixel unit 11 can pass light signals of at least two spectral bands, the photosensitive pixels of the multi-channel pixel unit 11 can sense the light signals of at least two spectral bands passed by the multi-channel filter, that is, the multi-channel pixel unit 11 can sense light signals of at least two spectral bands.

[0043] In some examples, the multi-channel filter may be an all-pass filter that can pass light signals in any spectral band. It is understood that the multi-channel pixel unit 11 can sense light signals in any spectral band.

[0044] In some embodiments of the present application, the number of the multi-channel pixel unit 11 may be at least one, which may be set according to usage requirements.

[0045] For example, assuming that the image sensor includes 8 pixel units, such as single-channel pixel unit C1, single-channel pixel unit C2, single-channel pixel unit C3, single-channel pixel unit C4, single-channel pixel unit C5, single-channel pixel unit C6, single-channel pixel unit C7, single-channel pixel unit C8 and multi-channel pixel unit Mono, each of the single-channel pixel unit C1, single-channel pixel unit C2, single-channel pixel unit C3, single-channel pixel unit C4, single-channel pixel unit C5, single-channel pixel unit C6, single-channel pixel unit C7 and single-channel pixel unit C8 is used to sense a light signal of a spectral band, and the multi-channel pixel unit Mono can sense light of all spectral bands. The spectral bands sensed by the 8 single-channel pixel units and the multi-channel pixel unit Mono are shown in Table 1.

[0046] Table 1

[0047]

[0048]

[0049] Among them, the central wavelength of the single-channel pixel unit C1 can be 410 nanometers nm, and the full width at half maximum (Full Width at Half Maximum, FWHM) corresponding to the single-channel pixel unit C1 is 10mn, the central wavelength of the single-channel pixel unit C2 can be 450nm, and the FWHM corresponding to the single-channel pixel unit C2 is 14mn, the central wavelength of the single-channel pixel unit C3 can be 480nm, and the FWHM corresponding to the single-channel pixel unit C3 is 15mn, the central wavelength of the single-channel pixel unit C4 can be 520nm, and the FWHM corresponding to the single-channel pixel unit C4 is 15mn, the central wavelength of the single-channel pixel unit C5 can be 560nm, and the FWHM corresponding to the single-channel pixel unit C5 is 15mn, and the The central wavelength of the single-channel pixel unit C6 can be 600nm, and the corresponding FWHM of the single-channel pixel unit C6 is 15mn. The central wavelength of the single-channel pixel unit C7 can be 650nm, and the corresponding FWHM of the single-channel pixel unit C7 is 15mn. The central wavelength of the single-channel pixel unit C8 can be 940nm, and the corresponding FWHM of the single-channel pixel unit C8 is 20mn. The central wavelength of the multi-channel pixel unit Mono is all-pass, that is, the multi-channel pixel unit Mono can sense light signals of all spectral bands, and the corresponding FWHM of the multi-channel pixel unit Mono is all-pass.

[0050] In some embodiments of the present application, the N single-channel pixel units 10 may be arranged around the multi-channel pixel unit 11; or the multi-channel pixel unit 11 may be arranged on the edge of the pixel array.

[0051] For example, Figure 2 FIG. 1 is a schematic diagram of a pixel array of an image sensor provided by an embodiment of the present application. Figure 2 As shown, the pixel array includes N single-channel pixel units, such as single-channel pixel unit C1, single-channel pixel unit C2, single-channel pixel unit C3, single-channel pixel unit C4, single-channel pixel unit C5, single-channel pixel unit C6, single-channel pixel unit C7 and single-channel pixel unit C8, and the pixel array also includes a multi-channel pixel unit Mono 11, which is arranged on the edge of the pixel array.

[0052] In some embodiments of the present application, when shooting with the image sensor provided by the embodiment of the present application, N single-channel pixel units can respectively sense light signals of different spectral bands to obtain N single-spectral channel images, and the multi-channel pixel unit can sense light signals of at least two spectral bands to obtain a multi-spectral channel image. Thus, the image sensor can fuse the N single-spectral channel images and the multi-spectral channel image to output a multi-spectral image.

[0053] An embodiment of the present application provides an image sensor, which includes a pixel array, which includes N single-channel pixel units and multi-channel pixel units; wherein a single-channel pixel unit is used to sense a light signal in a spectral band, and the multi-channel pixel unit is used to sense light signals in at least two spectral bands, and N is a positive integer greater than 1. Since the pixel array includes N single-channel pixel units and multi-channel pixel units, when the image sensor is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to the different spectral bands, and the multi-channel pixel units can sense light signals of at least two spectral bands and generate multi-spectral channel images corresponding to at least two spectral bands. In this way, the multi-spectral image output by the image sensor may include not only different spectral channel images corresponding to different spectral bands, but also multi-spectral channel images corresponding to at least two spectral bands, and the multi-spectral channel image includes more object information of the photographed object, that is, the multi-spectral image output by the image sensor may include more object information of the photographed object, instead of only including different spectral channel images corresponding to different spectral bands as in the related art, that is, the multi-spectral image output by the image sensor only includes less object information of the photographed object. Therefore, after the image sensor outputs the multi-spectral image, the electronic device may perform depth calculation based on the multi-spectral channel image including more object information of the photographed object to obtain an accurate depth image of the photographed object, thereby improving the accuracy of the depth image acquired by the electronic device.

[0054] Figure 2 FIG. 1 shows a schematic diagram of the structure of a multi-spectral camera provided in an embodiment of the present application. Figure 2 As shown, the multispectral camera provided in the embodiment of the present application includes the image sensor in the above embodiment.

[0055] An embodiment of the present application provides a multispectral camera, which includes the image sensor in the above embodiment. Since the pixel array of the image sensor of the multispectral camera includes N single-channel pixel units and multi-channel pixel units, when the multispectral camera is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to different spectral bands, and the multi-channel pixel units can sense light signals of at least two spectral bands and generate multispectral channel images corresponding to at least two spectral bands. In this way, the multispectral image output by the multispectral camera can include not only different spectral channel images corresponding to different spectral bands, but also multispectral channel images corresponding to at least two spectral bands. The channel image includes more object information of the photographed object, that is, the multispectral image output by the multispectral camera can include more object information of the photographed object, instead of the multispectral image output by the multispectral camera in the related art only including different spectral channel images corresponding to different spectral bands, that is, the multispectral image output by the multispectral camera only includes less object information of the photographed object. Therefore, after the multispectral camera outputs the multispectral image, the electronic device can perform depth calculation based on the multispectral channel image including more object information of the photographed object to obtain an accurate depth image of the photographed object, thereby improving the accuracy of the depth image acquired by the electronic device.

[0056] Figure 3 FIG. 1 is a schematic diagram showing the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 12 provided in the embodiment of the present application includes a rear camera 13 and the multispectral camera 14 in the above embodiment; wherein the rear camera 13 and the multispectral camera 14 are arranged at different positions on the same side of the electronic device 12, and the field of view angle of the rear camera 13 is within the field of view angle of the multispectral camera 14.

[0057] The electronic device 12 may further include a rear camera 15 , which may be disposed on the same surface of the electronic device 12 as the rear camera 13 and the multi-spectral camera 14 .

[0058] In some embodiments of the present application, the rear camera 13 and the multi-spectral camera 14 may be disposed at different positions on the back of the electronic device 12. Of course, the rear camera 13 and the multi-spectral camera 14 may also be disposed on other surfaces of the electronic device 12, which is not limited in the embodiments of the present application.

[0059] It should be noted that the back side of the electronic device 12 can be understood as: the side of the electronic device 12 opposite to the side on which the display screen is provided.

[0060] An embodiment of the present application provides an electronic device, which includes a rear camera and the multispectral camera in the above embodiment, and the rear camera and the multispectral camera are arranged at different positions on the same side of the electronic device, and the field of view angle of the rear camera is within the field of view angle of the multispectral camera. Since the pixel array of the image sensor of the multispectral camera of the electronic device includes N single-channel pixel units and multi-channel pixel units, when the multispectral camera is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to different spectral bands, and the multi-channel pixel units can sense light signals of at least two spectral bands and generate multispectral channel images corresponding to at least two spectral bands. In this way, in addition to including different spectral channel images corresponding to different spectral bands, the multispectral image output by the multispectral camera can also include multispectral channel images corresponding to at least two spectral bands, and the multispectral channel image includes more object information of the photographed object, that is, the multispectral image output by the multispectral camera can include more object information of the photographed object, instead of the multispectral image output by the multispectral camera generally including only different spectral channel images corresponding to different spectral bands as in the related art, that is, the multispectral image output by the multispectral camera only includes less object information of the photographed object. Therefore, after the multispectral camera outputs the multispectral image, the electronic device can perform depth calculation based on the multispectral channel image including more object information of the photographed object. Furthermore, since the rear camera and the multispectral camera are arranged at different positions on the same side of the electronic device, the field of view of the rear camera is within the field of view of the multispectral camera. This ensures that the rear camera and the multispectral camera can capture images of the same object from different perspectives. In this way, when the color image of the object is captured by the rear camera, the electronic device can capture the multispectral image of the object through the multispectral camera to obtain images of the object from different perspectives, and perform depth calculation based on the images of the object from different perspectives to directly obtain an accurate depth image of the object without the need to capture the depth image of the object through the depth camera. Therefore, it can avoid the situation where the electronic device cannot obtain an accurate depth image of the object, thereby improving the accuracy of the depth image acquired by the electronic device.

[0061] It should be noted that the depth image acquisition method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the depth image acquisition method provided in the embodiments of the present application is described by taking an electronic device as the execution subject to execute the depth image acquisition method.

[0062] Figure 4FIG. 1 is a flowchart of a depth image acquisition method provided by an embodiment of the present application, which is executed by the electronic device in the above embodiment. Figure 4 As shown, the depth image acquisition method provided in the embodiment of the present application may include the following steps 101 and 102.

[0063] Step 101: The electronic device collects a color image through a rear camera of the electronic device, and collects a multispectral image through a multispectral camera of the electronic device.

[0064] It should be noted that for the description of the multispectral camera of the electronic device, reference can be made to the specific description in the above embodiments, and the embodiments of the present application will not be repeated here.

[0065] In some embodiments of the present application, the rear camera may be any one of the following: a short-focus camera, a telephoto camera, a wide-angle camera, etc.

[0066] In some embodiments of the present application, the multispectral camera may be any one of the following: a narrow-band filtering camera, a spectral calculation and reconstruction camera.

[0067] In some embodiments of the present application, the multispectral camera is used to provide at least one of spectral information, color information and image information to a camera of an electronic device. When the electronic device is powered on, the multispectral camera can be in an on state or an off state.

[0068] In some embodiments of the present application, the multispectral image and the color image may be different types of images of the same photographed object.

[0069] In some embodiments of the present application, when a user needs to take an image of the above-mentioned subject, the user can point the electronic device toward the subject and provide a shooting input to the electronic device, so that the electronic device can turn on the rear camera of the electronic device according to the shooting input, and collect a color image of the subject through the rear camera, and collect a multispectral image of the subject through the multispectral camera of the electronic device.

[0070] In some examples, when the electronic device is powered on and the multispectral camera is powered off, the electronic device can turn on the multispectral camera based on the shooting input and collect a multispectral image of the object through the multispectral camera of the electronic device.

[0071] Step 102: The electronic device performs depth calculation based on the color image and the multispectral image, and outputs a depth image.

[0072] In a possible implementation of the present application, the electronic device may use at least one set of camera parameters to perform depth calculation based on the color image and the multi-spectral image and output a depth image.

[0073] In some embodiments of the present application, after obtaining the depth image, the electronic device may perform background blur, image enhancement, or other image processing requiring depth information on the color image based on the depth image to obtain a color image with higher image quality.

[0074] An embodiment of the present application provides a depth image acquisition method, whereby an electronic device can capture a color image through a rear camera of the electronic device, and capture a multispectral image through a multispectral camera of the electronic device, and perform depth calculation based on the color image and the multispectral image to output a depth image. Since the pixel array of the image sensor of the multispectral camera of the electronic device includes N single-channel pixel units and multi-channel pixel units, when the multispectral camera is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to different spectral bands, and the multi-channel pixel units can sense light signals of at least two spectral bands and generate multispectral channel images corresponding to at least two spectral bands. In this way, in addition to including different spectral channel images corresponding to different spectral bands, the multispectral image output by the multispectral camera can also include multispectral channel images corresponding to at least two spectral bands, and the multispectral channel image includes more object information of the photographed object, that is, the multispectral image output by the multispectral camera can include more object information of the photographed object, instead of the multispectral image output by the multispectral camera generally including only different spectral channel images corresponding to different spectral bands as in the related art, that is, the multispectral image output by the multispectral camera only includes less object information of the photographed object. Therefore, after the multispectral camera outputs the multispectral image, the electronic device can perform depth calculation based on the multispectral channel image including more object information of the photographed object. Furthermore, since the rear camera and the multispectral camera are arranged at different positions on the same side of the electronic device, the field of view of the rear camera is within the field of view of the multispectral camera. This ensures that the rear camera and the multispectral camera can capture images of the same object from different perspectives. In this way, when the color image of the object is captured by the rear camera, the electronic device can capture the multispectral image of the object through the multispectral camera to obtain images of the object from different perspectives, and perform depth calculation based on the images of the object from different perspectives to directly obtain an accurate depth image of the object without the need to capture the depth image of the object through the depth camera. Therefore, it can avoid the situation where the electronic device cannot obtain an accurate depth image of the object, thereby improving the accuracy of the depth image acquired by the electronic device.

[0075] Furthermore, in addition to providing at least one of spectral information, color information and image information for the camera of the electronic device, the multispectral camera in the embodiment of the present application can also capture a depth image corresponding to the photographed object with the rear camera, thereby improving the diversity of use of the multispectral camera.

[0076] In some embodiments of the present application, Figure 4 ,like Figure 5 As shown, before the above step 102, the depth image acquisition method provided in the embodiment of the present application may further include the following steps 201 and 202, and the above step 102 may be specifically implemented by the following step 102a.

[0077] Step 201: The electronic device obtains the shooting exposure value of the multi-spectral camera and determines the depth hole ratio.

[0078] In some embodiments of the present application, the above-mentioned depth hole ratio is used to characterize the texture details of the color image.

[0079] In some embodiments of the present application, the above-mentioned shooting exposure value may be a preset exposure value, or may be determined by the electronic device according to the shooting parameters of the multispectral camera. Among them, the shooting parameters may include at least one of the following: exposure time value, ISO sensitivity value, aperture value. Of course, the shooting parameters may include other parameters, which are not limited in the embodiments of the present application.

[0080] The following is an example of an electronic device determining a shooting exposure value based on shooting parameters of a multi-spectral camera.

[0081] In some embodiments of the present application, Figure 5 ,like Figure 6 As shown, the above step 201 can be specifically implemented through the following steps 201a and 201b.

[0082] Step 201a: The electronic device obtains the exposure time value, ISO value, and aperture value of the multispectral camera.

[0083] In some embodiments of the present application, the electronic device can read the device information of the multi-spectral camera, and obtain the exposure time value, the ISO sensitivity value, and the aperture value from the device information.

[0084] Step 201b: The electronic device calculates the shooting exposure value according to the exposure time value, ISO value and aperture value.

[0085] In some embodiments of the present application, the electronic device may use a first algorithm to calculate a first value based on the exposure time value, then use a second algorithm to calculate a second value based on the ISO value, and use a third algorithm to calculate a third value based on the aperture value, so that the electronic device may use a fourth algorithm to calculate the shooting exposure value based on the first value, the second value and the third value.

[0086] The first algorithm may be TV=log2(1 / T), where T is the exposure time value and TV is the first value. The second algorithm may be SV=log2(0.3S), where S is the ISO value and SV is the second value. The third algorithm may be AV=2*log2(Fno), where Fno is the aperture value and AV is the third value. The fourth algorithm may be EV=TV+SV+AV, where EV is the shooting exposure value.

[0087] For example, assuming that the multispectral camera is a narrow-band filter camera, combined with Figure 2 According to the characteristics of the narrowband filter camera, the sensitivity is insufficient under the condition of 100lux, the narrowband filter response of the narrowband filter camera is too low, and the noise is too large. At this time, when the multi-channel pixel unit Mono of the narrowband filter camera reaches a suitable exposure, for example, 10-bit data, when the multi-channel pixel unit Mono reaches an exposure value of 700DN, the exposure time is 133 milliseconds, the ISO value is 800, and the aperture value is 2.0, so the above-mentioned first algorithm, second algorithm, third algorithm and fourth algorithm can be used to calculate the shooting exposure value EV of 12.8.

[0088] It can be seen that since the electronic device can calculate the shooting exposure value according to the exposure time value, sensitivity ISO value and aperture value of the multispectral camera instead of using the preset exposure value, the shooting exposure value can be adapted to the characteristics of the multispectral camera. In this way, in the subsequent steps, the electronic device can accurately distinguish the shooting scenes that are compatible with the multispectral camera based on the shooting exposure value, and use appropriate camera parameters to accurately perform depth calculation to obtain an accurate depth image. In this way, the accuracy of the obtained depth image can be improved.

[0089] In some embodiments of the present application, Figure 5 ,like Figure 7 As shown, the above step 201 can be specifically implemented through the following steps 201c and 201d.

[0090] Step 201c: The electronic device performs depth calculation based on the color image and the multispectral image, and outputs an initial depth image.

[0091] In some embodiments of the present application, the electronic device can perform depth calculation based on the color image, the multispectral image, the rear camera internal parameter, the multispectral camera internal parameter, and the rear multispectral external parameter to output an initial depth image. The rear multispectral external parameter is an external parameter of the rear camera relative to the multispectral camera.

[0092] In some embodiments of the present application, the electronic device may first determine the disparity between the position of the same point in the color image and the position in the multispectral image, and then perform depth calculation based on the disparity, the rear camera internal parameter, the multispectral camera internal parameter, and the rear multispectral external parameter to output an initial depth image.

[0093] It should be noted that, for instructions on performing depth calculation based on parallax, rear camera intrinsic parameters, multispectral camera intrinsic parameters and rear multispectral extrinsic parameters and outputting an initial depth image, please refer to the specific description in the relevant technology, and the embodiments of the present application will not be repeated here.

[0094] It can be understood that the initial depth image may be an inaccurate depth image of the photographed object.

[0095] In some embodiments of the present application, the initial depth image includes at least one initial pixel point, and each initial pixel point corresponds to an initial depth value.

[0096] Step 201d: The electronic device determines the ratio of the number of hole pixels to the number of at least one initial pixel as the depth hole ratio.

[0097] In some embodiments of the present application, the number of hole pixels is the number of initial pixels whose initial depth value is 0 in at least one initial pixel.

[0098] It can be seen that since the electronic device can first perform depth calculation based on the color image and the multispectral image to obtain the initial depth image, and then accurately determine the depth hole ratio based on the number of hole pixels and the number of at least one initial pixel in the initial depth image, so as to accurately determine the texture details of the color image without having to determine it through the depth image collected by the depth camera. Therefore, it can avoid the situation where the texture information of the color image cannot be accurately determined when the depth camera cannot be used.

[0099] Step 202: The electronic device determines at least one set of camera parameters corresponding to the shooting exposure value and the depth hole ratio from the camera parameters of the rear camera and the multispectral camera.

[0100] In some embodiments of the present application, the camera parameters of the above-mentioned rear camera and multi-spectral camera may include at least one of the following: rear camera intrinsic parameters, multi-spectral camera intrinsic parameters, rear multi-spectral external parameters, N single-channel pixel intrinsic parameters, N rear single-channel pixel external parameters, multi-channel pixel intrinsic parameters, and rear multi-channel pixel external parameters; wherein, the rear multi-spectral external parameter is an external parameter of the rear camera relative to the multi-spectral camera, each single-channel pixel intrinsic parameter is an internal parameter corresponding to a single-channel pixel unit of the image sensor of the multi-spectral camera, each rear single-channel pixel external parameter is an external parameter of the rear camera relative to a single-channel pixel unit, the multi-channel pixel intrinsic parameter is an internal parameter corresponding to the multi-channel pixel unit of the image sensor of the multi-spectral camera, and each rear multi-channel pixel external parameter is an external parameter of the rear camera relative to the multi-channel pixel unit.

[0101] Among them, the image 1 captured by the rear camera and the image 2 captured by the multispectral camera can be acquired in advance, and the internal and external parameters of the rear camera and the multispectral camera can be calibrated according to the image 1 and the image, so as to respectively determine the rear camera internal parameters, the multispectral camera internal parameters, the rear multispectral external parameters, N single-channel pixel internal parameters, N rear single-channel pixel external parameters, multi-channel pixel internal parameters and rear multi-channel pixel external parameters, and store the rear camera internal parameters, the multispectral camera internal parameters, the rear multispectral external parameters, N single-channel pixel internal parameters, N rear single-channel pixel external parameters, multi-channel pixel internal parameters and rear multi-channel pixel external parameters in a predetermined storage area in the electronic device, so that the electronic device can first obtain the rear camera internal parameters, the multispectral camera internal parameters, the rear multispectral external parameters, N single-channel pixel internal parameters, N rear single-channel pixel external parameters, multi-channel pixel internal parameters and rear multi-channel pixel external parameters from the predetermined storage area, and determine at least one set of camera parameters corresponding to the shooting exposure value and the depth hole ratio.

[0102] The above-mentioned rear camera internal parameter may include at least one of the following: the focal length of the rear camera, the principal point of the rear camera, and the distortion parameter of the rear camera. The above-mentioned multispectral camera internal parameter may include at least one of the following: the focal length of the multispectral camera, the principal point of the multispectral camera, and the distortion parameter of the multispectral camera. The above-mentioned rear multispectral external parameter may include at least one of the following: a first rotation matrix and a first translation vector. The first rotation matrix is ​​used to describe the rotation relationship of the rear camera relative to the multispectral camera, and the first translation vector is used to describe the translation relationship of the rear camera relative to the multispectral camera. The above-mentioned single-channel pixel internal parameter may include at least one of the following: the focal length corresponding to the single-channel pixel unit, the principal point corresponding to the single-channel pixel unit, and the distortion parameter corresponding to the single-channel pixel unit. The above-mentioned rear single-channel pixel external parameter may include at least one of the following: a second rotation matrix and a second translation vector. The second rotation matrix is ​​used to describe the rotation relationship of the rear camera relative to the single-channel pixel unit, and the second translation vector is used to describe the translation relationship of the rear camera relative to the single-channel pixel unit. The above-mentioned multi-channel pixel internal parameters may include at least one of the following: the focal length corresponding to the multi-channel pixel unit, the principal point corresponding to the multi-channel pixel unit, and the distortion parameter corresponding to the multi-channel pixel unit. The above-mentioned rear multi-channel pixel external parameters may include at least one of the following: a third rotation matrix and a third translation vector. The third rotation matrix is ​​used to describe the rotation relationship of the rear camera relative to the multi-channel pixel unit, and the third translation vector is used to describe the translation relationship of the rear camera relative to the multi-channel pixel unit.

[0103] The focal length of the above-mentioned rear camera can be a positive integer, for example, the focal length can be 6.5. Of course, the focal length can also be other values, which is not limited in the embodiments of the present application. The principal point of the above-mentioned rear camera is used to indicate the light spot position of the rear camera, and the light spot position can be coordinate information, such as (10, 20). Of course, the light spot position can also be other coordinate values, which is not limited in the embodiments of the present application. The value of the distortion parameter of the above-mentioned rear camera can be 2. Of course, the distortion parameter can also be other values, which is not limited in the embodiments of the present application.

[0104] The focal length of the multi-spectral camera may be a positive integer, for example, the focal length may be 3.5. Of course, the focal length may also be other values, which are not limited in the embodiments of the present application. The principal point of the multi-spectral camera is used to indicate the light spot position of the multi-spectral camera, which may be coordinate information, for example, (20, 30). Of course, the light spot position may also be other coordinate values, which are not limited in the embodiments of the present application. The distortion parameter of the multi-spectral camera may be a positive integer, for example, the distortion parameter may be 4. Of course, the distortion parameter may also be other values, which are not limited in the embodiments of the present application.

[0105] The first rotation matrix may be a 3×3 matrix. Of course, the first rotation matrix may also be other matrices, which is not limited in the present embodiment. The value of the first translation vector may be a positive integer, for example, the value of the first translation vector may be 4. Of course, the value of the first translation vector may also be other values, which is not limited in the present embodiment.

[0106] The value of the focal length corresponding to the above-mentioned single-channel pixel unit can be a positive integer, for example, the value of the focal length can be 2.4. Of course, the value of the focal length can also be other values, which is not limited to the embodiments of the present application. The principal point corresponding to the above-mentioned single-channel pixel unit is used to indicate the position of the light spot corresponding to the single-channel pixel unit, and the light spot position can be coordinate information, such as (10,15). Of course, the light spot position can also be other coordinate values, which is not limited to the embodiments of the present application. The value of the distortion parameter corresponding to the above-mentioned single-channel pixel unit can be a positive integer, for example, the value of the distortion parameter can be 2. Of course, the value of the distortion parameter can also be other values, which is not limited to the embodiments of the present application.

[0107] The second rotation matrix may be a 3×3 matrix. Of course, the second rotation matrix may also be other matrices, which is not limited in the present embodiment. The value of the second translation vector may be a positive integer, for example, the value of the second translation vector may be 2. Of course, the value of the second translation vector may also be other values, which is not limited in the present embodiment.

[0108] The value of the focal length corresponding to the above-mentioned multi-channel pixel unit can be a positive integer, for example, the value of the focal length can be 1.6. Of course, the value of the focal length can also be other values, which is not limited to the embodiments of the present application. The principal point corresponding to the above-mentioned multi-channel pixel unit is used to indicate the light spot position corresponding to the single-channel pixel unit, and the light spot position can be coordinate information, such as (17,25). Of course, the light spot position can also be other coordinate values, which is not limited to the embodiments of the present application. The value of the distortion parameter corresponding to the above-mentioned multi-channel pixel unit can be a positive integer, for example, the value of the distortion parameter can be 4. Of course, the value of the distortion parameter can also be other values, which is not limited to the embodiments of the present application.

[0109] The third rotation matrix may be a 3×3 matrix. Of course, the third rotation matrix may also be other matrices, which is not limited in the embodiment of the present application. The value of the third translation vector may be a positive integer, for example, the value of the third translation vector may be 1. Of course, the value of the third translation vector may also be other values, which is not limited in the embodiment of the present application.

[0110] It should be noted that for the description of calibrating the internal and external parameters of the rear camera and the multi-spectral camera of the electronic device, reference can be made to the specific description in the relevant technology, and the embodiments of the present application will not be elaborated here.

[0111] In some embodiments of the present application, the electronic device may first determine a scene corresponding to the shooting exposure value and the depth hole ratio from at least one preset scene, each of the preset scenes corresponds to at least one set of camera parameters, and determine the at least one set of camera parameters corresponding to the one scene as at least one set of camera parameters corresponding to the shooting exposure value and the depth hole ratio.

[0112] The at least one preset scene includes at least one of the following: a bright scene with many texture details, a bright scene with few texture details, and a low-light scene.

[0113] In some embodiments of the present application, the above step 202 can be specifically implemented through the following steps 202a and 202b.

[0114] Step 202a: The electronic device determines a shooting scene according to a relationship between the shooting exposure value and the exposure threshold and a relationship between the depth hole ratio and the ratio threshold.

[0115] In some embodiments of the present application, the electronic device may first determine, from at least one preset scene, at least one scene corresponding to the size relationship between the shooting exposure value and the exposure threshold, and then, when the at least one scene is a bright scene, determine, from the at least one scene, a shooting scene corresponding to the size relationship between the depth hole ratio and the ratio threshold.

[0116] For example, assuming that at least one preset scene includes a bright scene with many texture details, a bright scene with few texture details, and a low-light scene, the electronic device can first determine at least one scene according to Table 2 and the relationship between the shooting exposure value and the exposure threshold, wherein Table 2 is illustrated with the exposure threshold of 12.8, and the exposure threshold can also be other values, which are not limited in the embodiments of the present application.

[0117] Table 2

[0118] Shooting exposure EV value Scenario ＞12.8 Bright scene ≤12.8 Low-light scenes

[0119] Among them, when the above-mentioned shooting exposure value is greater than the exposure threshold value of 12.8, the electronic device can determine the bright multi-texture detail scene and the bright few-texture detail scene from the bright multi-texture detail scene, the bright few-texture detail scene and the low-light scene; or, when the above-mentioned shooting exposure value is less than or equal to the exposure threshold value of 12.8, the electronic device can determine the low-light scene from the bright multi-texture detail scene, the bright few-texture detail scene and the low-light scene.

[0120] Then, since the bright scene with many texture details and the bright scene with few texture details are bright scenes, the electronic device can determine the shooting scene corresponding to the size relationship between the depth hole ratio and the ratio threshold according to Table 3, wherein Table 3 is illustrated with a ratio threshold of 40%, and the ratio threshold can also be other values, which is not limited in the embodiments of the present application.

[0121] Table 3

[0122] Depth Hole Ratio Scenario ＞40％ Multi-texture detail ≤40％ Less texture detail

[0123] Wherein, when the depth hole ratio is greater than the ratio threshold of 40%, the electronic device may determine the bright scene with many texture details as the shooting scene. Alternatively, when the depth hole ratio is less than or equal to the hole ratio of 40%, the electronic device may determine the bright scene with few texture details as the shooting scene.

[0124] Step 202b: The electronic device determines at least one set of camera parameters according to the correspondence between the shooting scene and the camera parameters.

[0125] In some embodiments of the present application, a shooting scene corresponds to a set of camera parameters, and the set of camera parameters includes at least two camera parameters.

[0126] It can be understood that the number of the corresponding relationships between the above-mentioned shooting scenes and camera parameters can be at least one, and each corresponding relationship is a corresponding relationship between a shooting scene and a group of camera parameters.

[0127] In some embodiments of the present application, when the shooting exposure value is greater than the exposure threshold and the depth hole ratio is greater than the ratio threshold, the shooting scene is a bright scene with many texture details; when the shooting exposure value is greater than the exposure threshold and the depth hole ratio is less than or equal to the ratio threshold, the shooting scene is a bright scene with few texture details; when the shooting exposure value is less than or equal to the exposure threshold, the shooting scene is a low-light scene.

[0128] It can be seen that since the electronic device can accurately determine the shooting scene in which the electronic device is located based on the relationship between the shooting exposure value and the exposure threshold and the relationship between the depth hole ratio and the ratio threshold, the electronic device can accurately determine at least one set of camera parameters suitable for the shooting scene in which the electronic device is located based on the correspondence between the shooting scene and the camera parameters. Therefore, in subsequent steps, the electronic device can use at least one set of camera parameters corresponding to the shooting scene in which the electronic device is located to perform depth calculation based on the color image and the multispectral image, thereby improving the accuracy of the output depth image.

[0129] Step 102a: The electronic device uses at least one set of camera parameters to perform depth calculation based on the color image and the multispectral image, and outputs a depth image.

[0130] It can be seen that since the electronic device can also accurately determine at least one set of suitable camera parameters from the camera parameters of the rear camera and the multispectral camera based on the shooting exposure value of the multispectral camera and the depth hole ratio used to characterize the texture details of the shooting scene, the electronic device can use at least one set of suitable camera parameters to perform depth calculations based on color images and multispectral images, instead of using inappropriate camera parameters to perform depth calculations based on color images and multispectral images. Therefore, the accuracy of the output depth image can be improved.

[0131] In some embodiments of the present application, Figure 5 ,like Figure 8 As shown, the above step 102a can be specifically implemented through the following steps 102a1 and 102a2.

[0132] Step 102a1: The electronic device uses each set of camera parameters to perform depth calculation according to the color image and the multispectral image, and outputs at least one intermediate depth image.

[0133] In some embodiments of the present application, the electronic device may use a set of camera parameters to perform depth calculation based on the color image and the multispectral image, and output an intermediate depth image, thereby outputting at least one intermediate depth image.

[0134] It can be understood that the at least one intermediate depth image is an inaccurate depth image.

[0135] Step 102a2: The electronic device determines a depth image based on at least one intermediate depth image.

[0136] In some embodiments of the present application, when the number of at least one intermediate depth image is one, the electronic device may directly determine the one intermediate depth image as the depth image. Alternatively, when the number of at least one intermediate depth image is at least two, the electronic device may fuse the at least two intermediate depth images and output the depth image.

[0137] It can be seen that the electronic device can use at least one set of camera parameters to perform depth calculation based on the color image and the multispectral image to obtain at least one intermediate depth image, and directly determine the accurate depth image based on the at least one intermediate depth image.

[0138] In some embodiments of the present application, after obtaining at least one intermediate depth image, the electronic device may first determine a depth image based on the at least one intermediate depth image, and then perform depth filling processing on the depth image to output the depth image. In some examples, the above 102a2 may be specifically implemented by the following steps 102a2a and 102a2b.

[0139] Step 102a2a: The electronic device determines an initial intermediate depth image based on at least one intermediate depth image.

[0140] In some embodiments of the present application, when the number of at least one intermediate depth image is one, the electronic device may directly determine the one intermediate depth image as the initial intermediate depth image. Alternatively, when the number of at least one intermediate depth image is at least two, the electronic device may fuse the at least two intermediate depth images and output the initial intermediate depth image.

[0141] Step 102a2b: The electronic device inputs the initial intermediate depth image and the color image corresponding to the color image into a depth completion model, performs depth completion processing on the initial intermediate depth image according to the color image through the depth completion model, and outputs a depth image.

[0142] In some embodiments of the present application, the color image may be an RGB image. Of course, the color image may also be other images, which is not limited in the embodiments of the present application.

[0143] In some embodiments of the present application, the depth completion model may be a trained model stored in an electronic device, or a trained model stored in a server.

[0144] It should be noted that for the description of the process of training the depth completion model, reference can be made to the specific description in the relevant technology, and the embodiments of the present application will not be repeated here.

[0145] It can be seen that since the electronic device can first determine the initial intermediate depth image based on at least one intermediate depth image, and then perform depth completion processing on the initial intermediate depth image according to the color image through the depth completion model, a more accurate depth image can be output, thereby improving the accuracy of the depth image determined by the electronic device.

[0146] The following will take different shooting scenes as examples to illustrate specific solutions for outputting depth images by electronic devices.

[0147] In some embodiments of the present application, the shooting scene is a bright multi-texture detail scene; the at least one group of camera parameters includes a first group of camera parameters and a second group of camera parameters; the first group of camera parameters includes a rear camera intrinsic parameter, a multispectral camera intrinsic parameter, and a rear multispectral extrinsic parameter; the second group of camera parameters includes a rear camera intrinsic parameter, N single-channel pixel intrinsic parameters, and N rear single-channel pixel extrinsic parameters; the rear multispectral extrinsic parameter is an extrinsic parameter of the rear camera relative to the multispectral camera, each single-channel pixel intrinsic parameter is an intrinsic parameter corresponding to a single-channel pixel unit of the image sensor of the multispectral camera, and each rear single-channel pixel extrinsic parameter is an extrinsic parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1. In some examples, the above step 102a1 can be specifically implemented by the following steps 102a1a and 102a1b.

[0148] Step 102a1a: The electronic device performs depth calculation based on the color image, the multispectral image, the rear camera internal parameters, the multispectral camera internal parameters and the rear multispectral external parameters, and outputs a first intermediate depth image.

[0149] In some embodiments of the present application, the electronic device can first determine the disparity between each pixel in the color image and the pixel corresponding to each pixel in the multispectral image, and then perform a depth calculation based on the disparity, the rear camera intrinsic parameters, the multispectral camera intrinsic parameters and the rear multispectral extrinsic parameters to obtain the depth value corresponding to each pixel, so that the electronic device can generate a first intermediate depth image based on the depth value corresponding to each pixel.

[0150] For example, taking pixel A in a color image as an example, pixel A is the pixel in the first row and first column of the color image, so that the electronic device can first determine the disparity between pixel A and pixel B, and pixel B is the pixel corresponding to pixel A in the multispectral image. It can be understood that pixel B is the pixel in the first row and first column of the multispectral image. Specifically, the electronic device can determine the disparity based on the difference between the coordinate information of pixel A in the color image and the coordinate information of pixel B in the multispectral image. Then the electronic device can calculate the depth value corresponding to pixel A based on the disparity, the focal length of the rear camera 6.5, the principal point of the rear camera (10,20), the distortion parameter 2 of the rear camera, the focal length of the multispectral camera 3.5, the principal point of the rear camera (20,30), the distortion parameter 4 of the multispectral camera, the first rotation matrix and the first translation vector 4 using a depth algorithm.

[0151] It should be noted that for the description of the above-mentioned depth algorithm, reference may be made to the specific description in the related technology, and the embodiments of the present application will not be repeated here.

[0152] Step 102a1b: The electronic device performs depth calculation based on the color image, N single-spectral channel images included in the multispectral image, the rear camera internal parameters, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters, and outputs a second intermediate depth image.

[0153] In some embodiments of the present application, each spectral channel image corresponds to a single-channel pixel unit of the multi-spectral camera. It can be understood that each spectral channel image is generated by image data obtained by sensing a light signal of a single-channel pixel.

[0154] In some embodiments of the present application, the electronic device may first divide the multispectral image into N single spectral channel images, and then perform depth calculation based on the color image, the N single spectral channel images included in the multispectral image, the rear camera intrinsic parameters, N single-channel pixel intrinsic parameters, and N rear single-channel pixel extrinsic parameters, to output a second intermediate depth image.

[0155] For example, the electronic device can divide the multispectral image into N single-spectral channel images, such as image C1, image C2, image C3, image C4, image C5, image C6, image C7 and image C8, where image C1 is generated by image data obtained by the single-channel pixel unit C1 sensing the light signal, image C2 is generated by image data obtained by the single-channel pixel unit C2 sensing the light signal, image C3 is generated by image data obtained by the single-channel pixel unit C3 sensing the light signal, image C4 is generated by image data obtained by the single-channel pixel unit C4 sensing the light signal, image C5 is generated by image data obtained by the single-channel pixel unit C5 sensing the light signal, image C6 is generated by image data obtained by the single-channel pixel unit C6 sensing the light signal, image C7 is generated by image data obtained by the single-channel pixel unit C7 sensing the light signal, and image C8 is generated by image data obtained by the single-channel pixel unit C8 sensing the light signal.

[0156] It can be seen that since the electronic device can use different sets of camera parameters to perform depth calculations based on color images and multispectral images respectively, it can obtain the first intermediate depth image and the second intermediate depth image of the shooting scene, that is, obtain more intermediate depth images of the shooting scene. Therefore, in subsequent steps, the terminal can accurately determine the depth image based on the more intermediate depth images, thereby improving the accuracy of the depth image determined by the electronic device.

[0157] In some embodiments of the present application, each of the N single spectral channel images includes at least one spectral channel pixel, and each spectral channel pixel corresponds to an edge information value. In some examples, the above step 102a1b can be specifically implemented by the following steps 102a1b1 to 102a1b4.

[0158] Step 102a1b1: The electronic device determines the number of spectral channel pixel points whose edge information values ​​are not 0 in each single spectral channel image.

[0159] In some embodiments of the present application, the electronic device can adopt at least one edge detection algorithm to perform edge detection on each single spectral channel image respectively to obtain multiple edge detection result images, and then calculate the number of spectral channel pixel points with non-zero edge information values ​​in each single spectral channel image based on the number of spectral channel pixel points with non-zero edge information values ​​in the edge detection result images corresponding to each single spectral channel image.

[0160] The above edge detection result image may be a binary image.

[0161] It can be understood that the edge information value corresponding to each spectral channel pixel point may be: the pixel value of the pixel point at the same position in the edge detection result image corresponding to each single spectral channel image.

[0162] For example, the electronic device may determine the number of spectral channel pixel points whose edge information values ​​are not 0 in each single spectral channel image by steps 1 to 8:

[0163] Step 1: The electronic device may first use two edge detection algorithms to perform edge detection on the image C1 to obtain two edge detection graphs. The two edge detection algorithms may be a canny operator algorithm and a wavelet transform algorithm. The two edge detection graphs may be an edge detection graph edgeC1-algo1 and an edge detection graph edgeC1-algo2. The edge detection graph edgeC1-algo1 is obtained by performing edge detection using the canny operator algorithm, and the edge detection graph edgeC1-algo2 is obtained by performing edge detection using the wavelet transform algorithm. Then, the pixel values ​​of the pixel points in the edge detection graph edgeC1-algo1 and the edge detection graph edgeC1-algo2 corresponding to each spectral channel pixel point in the image C1 are determined respectively to determine the edge information value corresponding to each spectral channel pixel point. Here, if the pixel values ​​of the pixel points in the edge detection graph edgeC1-algo1 and the edge detection graph edgeC1-algo2 corresponding to a spectral channel pixel point in the image C1 are both 0, then the edge information value corresponding to the spectral channel pixel point may be considered to be 0. If the pixel values ​​of the pixels corresponding to the edge detection graph edgeC1-algo1 and edge detection graph edgeC1-algo2 of a spectral channel pixel in image C1 are both 1, then the edge information value corresponding to the spectral channel pixel can be considered to be 1. If the pixel value of the pixel corresponding to the edge detection graph edgeC1-algo2 of a spectral channel pixel in image C1 is 0, and the pixel value of the pixel corresponding to the edge detection graph edgeC1-algo1 of the spectral channel pixel is 1, then the edge information value corresponding to the spectral channel pixel can be considered to be 1. If the pixel value of the pixel corresponding to the edge detection graph edgeC1-algo1 of a spectral channel pixel in image C1 is 0, and the pixel value of the pixel corresponding to the edge detection graph edgeC1-algo2 of the spectral channel pixel is 1, then the edge information value corresponding to the spectral channel pixel can be considered to be 1, so that the edge information value corresponding to each spectral channel pixel can be determined, and the number of spectral channel pixels whose corresponding edge information values ​​are not 0 can be counted to obtain the number of spectral channel pixels whose edge information values ​​in image C1 are not 0.

[0164] Step 2. The electronic device may first use the canny operator algorithm and the wavelet transform algorithm to perform edge detection on the image C2 to obtain two edge detection maps, which may be the edge detection map edgeC2-algo1 and the edge detection map edgeC2-algo2, and then respectively determine the pixel values ​​of the pixel points in the edge detection map edgeC2-algo1 and the edge detection map edgeC2-algo2 corresponding to each spectral channel pixel point in the image C2 to determine the edge information value corresponding to each spectral channel pixel point, and count the number of spectral channel pixel points whose corresponding edge information values ​​are not 0 to obtain the number of spectral channel pixel points whose edge information values ​​are not 0 in the image C2. Here, the description of determining the edge information value corresponding to each spectral channel pixel point in the image C2 may refer to the specific description of determining the edge information value corresponding to each spectral channel pixel point in the image C1.

[0165] Step 3, the electronic device can first use the canny operator algorithm and the wavelet transform algorithm to perform edge detection on the image C3 to obtain two edge detection maps, which can be the edge detection map edgeC3-algo1 and the edge detection map edgeC3-algo2, and then respectively determine the pixel values ​​of the pixel points in the edge detection map edgeC3-algo1 and the edge detection map edgeC3-algo2 corresponding to each spectral channel pixel point in the image C3 to determine the edge information value corresponding to each spectral channel pixel point, and count the number of spectral channel pixels whose corresponding edge information values ​​are not 0 to obtain the number of spectral channel pixels whose edge information values ​​are not 0 in the image C3. Here, the description of determining the edge information value corresponding to each spectral channel pixel point in the image C3 can refer to the specific description of determining the edge information value corresponding to each spectral channel pixel point in the image C1.

[0166] Step 4, the electronic device can first use the canny operator algorithm and the wavelet transform algorithm to perform edge detection on the image C4 to obtain two edge detection maps, which can be the edge detection map edgeC4-algo1 and the edge detection map edgeC4-algo2, and then respectively determine the pixel values ​​of the pixel points in the edge detection map edgeC4-algo1 and the edge detection map edgeC4-algo2 corresponding to each spectral channel pixel point in the image C4 to determine the edge information value corresponding to each spectral channel pixel point, and count the number of spectral channel pixels whose corresponding edge information values ​​are not 0 to obtain the number of spectral channel pixels whose edge information values ​​are not 0 in the image C4. Here, the description of determining the edge information value corresponding to each spectral channel pixel point in the image C4 can refer to the specific description of determining the edge information value corresponding to each spectral channel pixel point in the image C1.

[0167] Step 5. The electronic device may first use the canny operator algorithm and the wavelet transform algorithm to perform edge detection on the image C5 to obtain two edge detection maps, which may be the edge detection map edgeC5-algo1 and the edge detection map edgeC5-algo2, and then respectively determine the pixel values ​​of the pixel points in the edge detection map edgeC5-algo1 and the edge detection map edgeC5-algo2 corresponding to each spectral channel pixel point in the image C5 to determine the edge information value corresponding to each spectral channel pixel point, and count the number of spectral channel pixel points whose corresponding edge information values ​​are not 0 to obtain the number of spectral channel pixel points whose edge information values ​​are not 0 in the image C5. Here, the description of determining the edge information value corresponding to each spectral channel pixel point in the image C5 may refer to the specific description of determining the edge information value corresponding to each spectral channel pixel point in the image C1.

[0168] Step 6. The electronic device may first use the canny operator algorithm and the wavelet transform algorithm to perform edge detection on the image C6 to obtain two edge detection maps, which may be the edge detection map edgeC6-algo1 and the edge detection map edgeC6-algo2, and then respectively determine the pixel values ​​of the pixel points in the edge detection map edgeC6-algo1 and the edge detection map edgeC6-algo2 corresponding to each spectral channel pixel point in the image C6 to determine the edge information value corresponding to each spectral channel pixel point, and count the number of spectral channel pixel points whose corresponding edge information values ​​are not 0 to obtain the number of spectral channel pixel points whose edge information values ​​are not 0 in the image C6. Here, the description of determining the edge information value corresponding to each spectral channel pixel point in the image C6 may refer to the specific description of determining the edge information value corresponding to each spectral channel pixel point in the image C1.

[0169] Step 7. The electronic device may first use the canny operator algorithm and the wavelet transform algorithm to perform edge detection on the image C7 to obtain two edge detection maps, which may be the edge detection map edgeC7-algo1 and the edge detection map edgeC7-algo2, and then respectively determine the pixel values ​​of the pixel points in the edge detection map edgeC7-algo1 and the edge detection map edgeC7-algo2 corresponding to each spectral channel pixel point in the image C7 to determine the edge information value corresponding to each spectral channel pixel point, and count the number of spectral channel pixel points whose corresponding edge information values ​​are not 0 to obtain the number of spectral channel pixel points whose edge information values ​​are not 0 in the image C7. For the description of determining the edge information value corresponding to each spectral channel pixel point in the image C7, reference may be made to the specific description of determining the edge information value corresponding to each spectral channel pixel point in the image C1.

[0170] Step 8. The electronic device may first use the canny operator algorithm and the wavelet transform algorithm to perform edge detection on the image C8 to obtain two edge detection maps, which may be the edge detection map edgeC8-algo1 and the edge detection map edgeC8-algo2, and then respectively determine the pixel values ​​of the pixel points in the edge detection map edgeC8-algo1 and the edge detection map edgeC8-algo2 corresponding to each spectral channel pixel point in the image C8 to determine the edge information value corresponding to each spectral channel pixel point, and count the number of spectral channel pixel points whose corresponding edge information values ​​are not 0 to obtain the number of spectral channel pixel points whose edge information values ​​are not 0 in the image C8. Here, the description of determining the edge information value corresponding to each spectral channel pixel point in the image C8 may refer to the specific description of determining the edge information value corresponding to each spectral channel pixel point in the image C1.

[0171] It can be understood that the number of spectral channel pixels whose edge information values ​​are not 0 in each single spectral channel image is positively correlated with the edge information in each single spectral channel image. In other words, the more the number of spectral channel pixels whose edge information values ​​are not 0 in each single spectral channel image, the more edge information in each single spectral channel image can be considered.

[0172] It should be noted that the embodiment of the present application does not limit the execution order of steps 1 to 8. In one example, the electronic device can execute steps 1 to 8 in a predetermined order, which can include the arrangement order of the single-channel pixel units of the captured image on the image sensor, a random order, the order from step 1 to step 8, the order from step 8 to step 1, etc. In another example, the electronic device can execute steps 1 to 8 simultaneously.

[0173] For example, assuming that the predetermined order is the arrangement order of the single-channel pixel units of the captured image on the image sensor, the arrangement order is single-channel pixel unit C1, single-channel pixel unit C4, single-channel pixel unit C7, single-channel pixel unit C2, single-channel pixel unit C5, single-channel pixel unit C8, single-channel pixel unit C3 and single-channel pixel unit C6, then the electronic device can perform steps 1 to 8 in the order of step 1, step 4, step 7, step 2, step 5, step 8, step 3 and step 6. Of course, the arrangement order can also be other orders, and the embodiments of the present application are not limited to this.

[0174] Step 102a1b2: The electronic device determines L candidate spectral channel images having a number of pixels greater than a number threshold from the N single spectral channel images.

[0175] In some embodiments of the present application, L is a positive integer.

[0176] In some embodiments of the present application, the quantity threshold may be 100, 200, 300, 400, 500, etc. Of course, the quantity threshold may also be other numbers, which is not limited in the embodiments of the present application.

[0177] It can be understood that if the number of spectral channel pixels whose edge information values ​​in a single spectral channel image are not 0 is greater than the quantity threshold, it can be considered that there is more edge information in the single spectral channel image, that is, there are more edges in the single spectral channel image. Therefore, the electronic device can determine the single spectral channel image with more edge information as an alternative spectral channel image, so as to facilitate determination of the second intermediate depth image in the subsequent steps.

[0178] Step 102a1b3: The electronic device determines T candidate color channel images whose color wavelengths match the L spectral bands from the M color channel images.

[0179] In some embodiments of the present application, the M color channel images are color channel images included in the color image, and the L spectral bands are spectral bands corresponding to the L alternative spectral channel images; each alternative spectral channel image corresponds to at least one alternative color channel image, M is a positive integer, and T is a positive integer greater than or equal to L.

[0180] In some embodiments of the present application, the M color channel images may include at least one of the following: a red channel image, a green channel image, and a blue channel image. The color wavelength may be a central wavelength of a color band corresponding to the color channel image.

[0181] In one example, for each candidate spectral channel image in the L candidate spectral channel images, the electronic device may first calculate the difference between the central wavelength of the spectral band corresponding to one candidate spectral channel image and the color wavelength corresponding to each color channel image to obtain at least one step length. Then, the color channel image corresponding to the minimum step length in the at least one step length is determined as a candidate color channel image. It can be understood that according to this method, each candidate spectral channel image corresponds to a candidate color channel image.

[0182] In another example, for each of the L candidate spectral channel images, the electronic device may first determine the color wavelength corresponding to each color channel image, and determine at least one range with the color wavelength corresponding to each color channel image as a boundary point. Then, from the at least one range, a range is determined in which the central wavelength of the spectral band corresponding to an alternative spectral channel image is located, and two color channel images corresponding to two boundary points of the range are determined as two alternative color channel images. It can be understood that according to this method, each alternative spectral channel image corresponds to at least two alternative color channel images.

[0183] Step 102a1b4: The electronic device performs depth calculation based on L candidate spectral channel images, T candidate color channel images, rear camera internal parameters, L single-channel pixel internal parameters and L rear single-channel pixel external parameters, and outputs a second intermediate depth image.

[0184] In some embodiments of the present application, the above-mentioned L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L alternative spectral channel images among the N single-channel pixel intrinsic parameters, and the above-mentioned L rear-position single-channel pixel extrinsic parameters are the rear-position single-channel pixel extrinsic parameters corresponding to the L alternative spectral channel images among the N rear-position single-channel pixel extrinsic parameters.

[0185] In some embodiments of the present application, the post-positioned single-channel pixel external parameters corresponding to the L candidate spectral channel images can be understood as: internal parameters corresponding to the L single-channel pixel units. Here, the L single-channel pixel units can be single-channel pixel units that generate image data of the L candidate spectral channel images among the N single-channel pixel units.

[0186] It can be seen that since the number of edges in different spectral channel images may be different, the electronic device can first determine the number of pixels of the spectral channel pixels whose edge information values ​​are not 0 in each single spectral channel image to determine the number of edges in each single spectral channel image, and then determine L candidate spectral channel images whose number of pixels is greater than the number threshold from the N single spectral channel images, that is, from the N single spectral channel images, L candidate spectral channel images with a large number of edges, and based on the L candidate spectral channel images, determine T candidate color channel images with a large number of edges, so that the electronic device can perform depth calculation based on the L candidate spectral channel images, T candidate color channel images, the rear camera intrinsic parameters, the L single channel pixel intrinsic parameters and the L rear single channel pixel extrinsic parameters to obtain a more accurate second intermediate depth image.

[0187] In some embodiments of the present application, the above step 102a1b4 can be specifically implemented through the following steps 102a1b4a and 102a1b4b.

[0188] Step 102a1b4a, the electronic device performs depth calculations based on each alternative spectral channel image, each alternative color channel image corresponding to each alternative spectral channel image, the rear camera internal parameters, the single-channel pixel internal parameters corresponding to each alternative spectral channel image, and the rear single-channel pixel extrinsic parameters corresponding to each alternative spectral channel image, and outputs T depth images to be fused.

[0189] In some embodiments of the present application, for each alternative spectral channel image in the L alternative spectral channel images, the electronic device may first perform a depth calculation based on an alternative spectral channel image, an alternative color channel image corresponding to the alternative spectral channel image, a rear camera internal parameter, a single-channel pixel internal parameter corresponding to the alternative spectral channel image, and a rear single-channel pixel external parameter corresponding to the alternative color channel image to obtain a depth value corresponding to each pixel in the alternative spectral channel image, so that the electronic device can generate a depth image to be fused based on the depth value corresponding to each pixel, and perform a depth calculation based on the alternative spectral channel image, another alternative color channel image corresponding to the alternative spectral channel image, the rear camera internal parameter, the single-channel pixel internal parameter corresponding to the alternative spectral channel image, and the rear single-channel pixel external parameter corresponding to the alternative color channel image to obtain another depth value corresponding to each pixel in the alternative spectral channel image, so that the electronic device can generate another depth image to be fused based on the other depth value corresponding to each pixel, and so on, to obtain at least one depth image to be fused. Then the electronic device can perform depth calculation based on another candidate spectral channel image, an alternative color channel image corresponding to the other candidate spectral channel image, a rear camera internal parameter, a single channel pixel internal parameter corresponding to the other candidate spectral channel image, and a rear single channel pixel external parameter corresponding to the other candidate color channel image, to obtain a depth value corresponding to each pixel in the other candidate spectral channel image, so that the electronic device can generate a depth image to be fused based on a depth value corresponding to each pixel, and perform depth calculation based on the other candidate spectral channel image, another candidate color channel image corresponding to the other candidate spectral channel image, a rear camera internal parameter, a single channel pixel internal parameter corresponding to the other candidate spectral channel image, and a rear single channel pixel external parameter corresponding to the other candidate color channel image, to obtain another depth value corresponding to each pixel in the other candidate spectral channel image, so that the electronic device can generate another depth image to be fused based on another depth value corresponding to each pixel, and so on, to obtain at least one depth image to be fused. By calculating in this way, T depth images to be fused can be obtained.

[0190] For example, taking pixel C in an alternative spectral channel image as an example, pixel C is the pixel in the first row and first column of the alternative spectral channel image, so that the electronic device can first determine the disparity between pixel C and pixel D, and pixel D is the pixel corresponding to pixel C in an alternative color channel image corresponding to the alternative spectral channel image. It can be understood that pixel D is the pixel in the first row and first column of the alternative color channel image. Specifically, the electronic device can determine the disparity based on the difference between the coordinate information of pixel C in the color image and the coordinate information of pixel D in the multi-spectral image. Then, the electronic device can use a depth algorithm to calculate the depth value corresponding to the pixel point C according to the parallax, the focal length 6.5 of the rear camera, the principal point (10, 20) of the rear camera, the distortion parameter 2 of the rear camera, the focal length 2.4 corresponding to the single-channel pixel unit corresponding to the one alternative spectral channel image, the principal point (10, 15) corresponding to the single-channel pixel unit corresponding to the one alternative spectral channel image, the distortion parameter 2 corresponding to the single-channel pixel unit corresponding to the one alternative spectral channel image, the second rotation matrix corresponding to the single-channel pixel unit corresponding to the one alternative spectral channel image, and the second translation vector 2 corresponding to the single-channel pixel unit corresponding to the one alternative spectral channel image.

[0191] Step 102a1b4b: The electronic device fuses the T depth images to be fused to obtain a second intermediate depth image.

[0192] In some embodiments of the present application, the electronic device can use an iterative least squares algorithm to fuse T depth images to be fused, and analyze the hole area in the fused depth image, determine the type of the hole area in the depth image, and adaptively rotate the structural element to iteratively operate the depth image to obtain a second intermediate depth image.

[0193] In this way, it can be seen that since the electronic device can perform depth calculations based on each alternative spectral channel image, each alternative color channel image corresponding to each alternative spectral channel image, the rear camera intrinsic parameters, the single-channel pixel intrinsic parameters corresponding to each alternative spectral channel image, and the rear single-channel pixel extrinsic parameters corresponding to each alternative spectral channel image, respectively, to obtain T depth images to be fused, and perform image fusion on the T depth images to be fused, the fused second intermediate depth image can have more edges, so that in subsequent steps, the electronic device can accurately determine the depth image based on the second intermediate depth image.

[0194] In some embodiments of the present application, the above step 102a2 can be specifically implemented through the following step 102a2c.

[0195] Step 102a2c: The electronic device fuses the first intermediate depth image and the second intermediate depth image to obtain a depth image.

[0196] It can be seen that since the electronic device can respectively use the first set of camera parameters and the second set of camera parameters to perform depth calculation based on the color image and the multispectral image, and the depth calculation using the first set of camera parameters and the second set of camera parameters has different characteristics, the electronic device can obtain a first intermediate depth image and a second intermediate depth image with two characteristics, so that the electronic device can obtain an accurate depth image after fusing the first intermediate depth image and the second intermediate depth image with two characteristics.

[0197] In some embodiments of the present application, the shooting scene is a bright scene with few texture details; the at least one set of camera parameters includes a second set of camera parameters; the second set of camera parameters includes a rear camera internal parameter, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters; each single-channel pixel internal parameter is an internal parameter corresponding to a single-channel pixel unit of the image sensor of the multi-spectral camera, and each rear single-channel pixel external parameter is an external parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1. In some examples, the above step 102a1 can be specifically implemented by the following step 102a1c.

[0198] Step 102a1c: The electronic device performs depth calculation based on the color image, N single spectral channel images included in the multispectral image, the rear camera internal parameters, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters, and outputs a second intermediate depth image and a third intermediate depth image.

[0199] In some embodiments of the present application, the electronic device may use one algorithm to perform depth calculation based on the color image, N single-spectral channel images included in the multi-spectral image, the rear camera internal parameters, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters, and output a second intermediate depth image; and use another algorithm to perform depth calculation based on the color image, N single-spectral channel images included in the multi-spectral image, the rear camera internal parameters, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters, and output a third intermediate depth image.

[0200] It can be seen that since the electronic device can adopt the second set of camera parameters to perform different depth calculations based on the color image and the multispectral image respectively, it can obtain the second intermediate depth image and the third intermediate depth image of the shooting scene, that is, obtain more intermediate depth images of the shooting scene. Therefore, in the subsequent steps, the terminal can accurately determine the depth image based on the more intermediate depth images, thereby improving the accuracy of the depth image determined by the electronic device.

[0201] In some embodiments of the present application, the above step 102a1c can be specifically implemented through the following steps 102a1c1 to 102a1c4.

[0202] Step 102a1c1: The electronic device determines the number of spectral channel pixel points whose edge information values ​​are not 0 in each single spectral channel image.

[0203] It should be noted that, for the description of the electronic device determining the number of spectral channel pixels whose edge information values ​​are not 0 in each single spectral channel image, reference may be made to the specific description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0204] Step 102a1c2: The electronic device determines L candidate spectral channel images having a number of pixels greater than a number threshold from the N single spectral channel images.

[0205] In some embodiments of the present application, L is a positive integer.

[0206] It should be noted that, for the description of an electronic device determining L candidate spectral channel images having a number of pixels greater than a quantity threshold from N single spectral channel images, reference may be made to the specific description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0207] Step 102a1c3: The electronic device determines T candidate color channel images whose color wavelengths match the L spectral bands from the M color channel images.

[0208] In some embodiments of the present application, the M color channel images are color channel images included in the color image, and the L spectral bands are spectral bands corresponding to the L alternative spectral channel images; each alternative spectral channel image corresponds to at least one alternative color channel image, M is a positive integer, and T is a positive integer greater than or equal to L.

[0209] It should be noted that, for the description of an electronic device determining T candidate color channel images whose color wavelengths match L spectral bands from M color channel images, reference can be made to the specific description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0210] Step 102a1c4: The electronic device performs depth calculation based on L candidate spectral channel images, T candidate color channel images, rear camera internal parameters, L single-channel pixel internal parameters and L rear single-channel pixel external parameters, and outputs a second intermediate depth image and a third intermediate depth image.

[0211] In some embodiments of the present application, the above-mentioned L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L alternative spectral channel images among the N single-channel pixel intrinsic parameters, and the above-mentioned L rear-position single-channel pixel extrinsic parameters are the rear-position single-channel pixel extrinsic parameters corresponding to the L alternative spectral channel images among the N rear-position single-channel pixel extrinsic parameters.

[0212] In some embodiments of the present application, the electronic device may use one algorithm to perform depth calculation based on L alternative spectral channel images, T alternative color channel images, a rear camera intrinsic parameter, L single-channel pixel intrinsic parameters and L rear single-channel pixel extrinsic parameters, and output a second intermediate depth image; and use another algorithm to perform depth calculation based on the L alternative spectral channel images, T alternative color channel images, a rear camera intrinsic parameter, L single-channel pixel intrinsic parameters and L rear single-channel pixel extrinsic parameters, and output a third intermediate depth image.

[0213] It can be seen that since the electronic device can perform different depth calculations based on L alternative spectral channel images, T alternative color channel images, the rear camera intrinsic parameters, L single-channel pixel intrinsic parameters and L rear single-channel pixel extrinsic parameters, and different depth calculations have different characteristics, the electronic device can obtain a second intermediate depth image and a third intermediate depth image with two characteristics. In this way, in subsequent steps, the electronic device can obtain an accurate depth image after fusing the second intermediate depth image and the third intermediate depth image with two characteristics.

[0214] In some embodiments of the present application, the above step 102a1c4 can be specifically implemented through the following steps 102a1c4a to 102a1c4e.

[0215] Step 102a1c4a, the electronic device performs depth calculations based on each alternative spectral channel image, each alternative color channel image corresponding to each alternative spectral channel image, the rear camera internal parameters, the single-channel pixel internal parameters corresponding to each alternative spectral channel image, and the rear single-channel pixel extrinsic parameters corresponding to each alternative spectral channel image, and outputs T depth images to be fused.

[0216] It should be noted that, for the instructions on the electronic device performing depth calculation based on each alternative spectral channel image, each alternative color channel image corresponding to each alternative spectral channel image, the rear camera intrinsic parameters, the single-channel pixel intrinsic parameters corresponding to each alternative spectral channel image, and the rear single-channel pixel extrinsic parameters corresponding to each alternative spectral channel image, please refer to the specific description in the above embodiments, and the embodiments of the present application will not be repeated here.

[0217] Step 102a1c4b: The electronic device fuses the T depth images to be fused to obtain a second intermediate depth image.

[0218] It should be noted that, for the description of the electronic device performing image fusion on T depth images to be fused, reference may be made to the specific description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0219] Step 102a1c4c: the electronic device performs edge detection on each candidate spectral channel image to obtain L spectral channel detection result images, and performs edge detection on each candidate color channel image to obtain T color channel detection result images.

[0220] In some embodiments of the present application, each spectral channel detection result graph corresponds to at least one color channel detection result graph.

[0221] It can be understood that, since each candidate spectral channel image corresponds to at least one candidate color channel image, each spectral channel detection result image corresponds to at least one color channel detection result image.

[0222] In some embodiments of the present application, the above-mentioned spectral channel detection result map and color channel detection result map may be binary maps.

[0223] In some embodiments of the present application, the above step 102a1c4c can be specifically implemented through the following steps 102a1c4c1 to 102a1c4c3.

[0224] Step 102a1c4c1: The electronic device uses at least two detection algorithms to perform edge detection on each candidate spectral channel image, and generates at least two spectral channel detection process diagrams.

[0225] In some embodiments of the present application, the at least two detection algorithms may include at least one of the following: a canny operator algorithm and a wavelet transform algorithm. Of course, the at least two detection algorithms may also include other algorithms, which are not listed in the embodiments of the present application.

[0226] In some embodiments of the present application, the above-mentioned spectral channel detection process diagram may be a binary diagram.

[0227] In some embodiments of the present application, each spectral channel detection process map includes at least one detection process pixel point, and each detection process pixel point corresponds to an edge information value.

[0228] In some embodiments of the present application, the edge information value corresponding to each detection process pixel point may be the pixel value of each detection process pixel point.

[0229] Step 102a1c4c2: The electronic device marks edge pixels in the detection process pixels in the main spectral channel detection process map according to the edge information value corresponding to each detection process pixel in at least one auxiliary spectral channel detection process map.

[0230] In some embodiments of the present application, the above-mentioned main spectral channel detection process diagram is one spectral channel detection process diagram among at least two spectral channel detection process diagrams, and the above-mentioned at least one auxiliary spectral channel detection process diagram is a spectral channel detection process diagram among at least two spectral channel detection process diagrams except the main spectral channel detection process diagram.

[0231] The main spectral channel detection process diagram may be calculated using a wavelet transform algorithm. The at least one auxiliary spectral channel detection process diagram may be calculated using an algorithm other than the wavelet transform algorithm, for example, using a canny operator algorithm.

[0232] In one example, for detection process pixel point 1 whose pixel value is not 0 among the detection process pixel points included in the main spectral channel detection process map, an appropriate kernel is used for filtering in each auxiliary spectral channel detection process map, for example, 3×3 filtering is used. If a detection process pixel point whose pixel value is not 0 can be found in the neighborhood of the detection process pixel point corresponding to the detection process pixel point 1 in a certain auxiliary spectral channel detection process map, then the detection process pixel point 1 can be considered as a credible edge point, that is, the pixel value of the detection process pixel point 1 in the main spectral channel detection process map is set to 1 to mark the edge pixel point.

[0233] Step 102a1c4c3: The electronic device generates L spectral channel detection result images based on the marked main spectral channel detection process image, and performs edge detection on each candidate color channel image to obtain T color channel detection result images.

[0234] In some embodiments of the present application, the electronic device may use an edge filling algorithm to process the marked main spectral channel detection process map to generate a processed main spectral channel detection process map. In addition, the electronic device may determine whether the pixel value of the detection process pixel point 2 with a pixel value of 0 in at least one auxiliary spectral channel detection process map corresponding to the detection process pixel point in the processed main spectral channel detection process map is 0, and if not 0, the detection process pixel point corresponding to the detection process pixel point 2 in the processed main spectral channel detection process map is marked as an edge, that is, the pixel value of the detection process pixel point corresponding to the detection process pixel point 2 in the processed main spectral channel detection process map is set to 1, so as to obtain a spectral channel detection result map.

[0235] In some embodiments of the present application, the electronic device may use an edge detection algorithm to perform edge detection on each candidate color channel image to obtain a color channel detection result image.

[0236] It can be seen that since the electronic device can use at least two detection algorithms to perform edge detection on each alternative spectral channel image, and each detection algorithm has different characteristics, the electronic device can obtain a spectral channel detection process map with at least two characteristics. In this way, the electronic device can mark edge pixels in the detection process pixels in the main spectral channel detection process map according to the edge information value corresponding to each detection process pixel in at least one auxiliary spectral channel detection process map, so that the marked edge pixels in the marked main spectral channel detection process map are real edge pixels, thereby improving the accuracy of the generated spectral channel detection result map.

[0237] In some embodiments of the present application, the above step 102a1c4c can be specifically implemented through the following steps 102a1c4c4 and 102a1c4c5.

[0238] Step 102a1c4c4: the electronic device performs edge detection on each candidate spectral channel image to obtain L spectral channel detection result images, and performs edge detection on each candidate color channel image to obtain L color channel detection process images.

[0239] In some embodiments of the present application, the electronic device may use an edge detection algorithm to perform edge detection on a candidate color channel image to obtain a color channel detection process diagram.

[0240] Step 102a1c4c5: The electronic device performs edge padding processing on each color channel detection process image based on the spectral channel detection result image of the candidate spectral channel image corresponding to each candidate color channel image, to obtain L color channel detection result images.

[0241] In some embodiments of the present application, the electronic device may first use a double threshold method to perform edge padding on a color channel detection process image based on a spectral channel detection result image of an alternative spectral channel image corresponding to an alternative color channel image to obtain an intermediate image, and then perform 8-neighborhood edge point judgment based on the intermediate image and the alternative color channel image to obtain a color channel detection result image.

[0242] It should be noted that, for the description of the dual threshold method, reference may be made to the specific description in the related art, and the embodiments of the present application will not be elaborated here.

[0243] In some embodiments of the present application, after obtaining a color channel detection result image, the electronic device can also obtain the edge point depth according to a candidate color channel image and the color channel detection result image according to the parallax method. After obtaining, the scene can be segmented according to the edge point depth. Among them, it can be considered that the edge point is the first segmentation of the candidate color channel image. After calculating the edge point depth, it can be determined whether the area segmented by the edge point is located in the same plane. Combined with machine learning, the color image corresponding to the candidate color channel image is segmented to determine whether the segmentation is the same as the edge point / depth. If the same, the person or object can be segmented from the background in depth in the shooting scene.

[0244] It can be seen that since the electronic device can also perform edge padding on a color channel detection process image based on a spectral channel detection result image of an alternative spectral channel image corresponding to an alternative color channel image, the accuracy of the obtained color channel detection result image can be improved.

[0245] Step 102a1c4d, the electronic device performs depth calculations based on each spectral channel detection result image, each color channel detection result image corresponding to each spectral channel detection result image, the rear camera internal parameters, the single channel pixel internal parameters corresponding to each alternative spectral channel image, and the rear single channel pixel extrinsic parameters corresponding to each alternative spectral channel image, and outputs T depth images to be fused.

[0246] In some embodiments of the present application, the electronic device may first perform depth calculation based on a spectral channel detection result image, a color channel detection result image corresponding to the spectral channel detection result image, a rear camera internal parameter, a single-channel pixel internal parameter corresponding to each alternative spectral channel image, and a rear single-channel pixel external parameter corresponding to each alternative spectral channel image, to obtain a depth value corresponding to each pixel point in the spectral channel detection result image, so that the electronic device can generate a depth image to be fused based on the depth value corresponding to each pixel point, and perform depth calculation based on the spectral channel detection result image, another color channel detection result image corresponding to the spectral channel detection result image, the rear camera internal parameter, the single-channel pixel internal parameter corresponding to each alternative spectral channel image, and the rear single-channel pixel external parameter corresponding to each alternative spectral channel image, to obtain another depth value corresponding to each pixel point in the spectral channel detection result image, so that the electronic device can generate another depth image to be fused based on the other depth value corresponding to each pixel point, and so on, to obtain at least one depth image to be fused. Then the electronic device can perform depth calculation based on another spectral channel detection result image, a color channel detection result image corresponding to the other spectral channel detection result image, the rear camera internal parameter, the single channel pixel internal parameter corresponding to each candidate spectral channel image, and the rear single channel pixel external parameter corresponding to each candidate spectral channel image, to obtain a depth value corresponding to each pixel in the other spectral channel detection result image, so that the electronic device can generate a depth image to be fused based on a depth value corresponding to each pixel, and perform depth calculation based on the other spectral channel detection result image, another color channel detection result image corresponding to the other spectral channel detection result image, the rear camera internal parameter, the single channel pixel internal parameter corresponding to each candidate spectral channel image, and the rear single channel pixel external parameter corresponding to each candidate spectral channel image, to obtain another depth value corresponding to each pixel in the other spectral channel detection result image, so that the electronic device can generate another depth image to be fused based on another depth value corresponding to each pixel, and so on, to obtain at least one depth image to be fused. By calculating in this way, T depth images to be fused can be obtained.

[0247] It should be noted that, for the description of performing depth calculations on the electronic devices respectively and outputting T depth images to be fused, reference may be made to the specific description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0248] Step 102a1c4e: The electronic device fuses the T depth images to be fused to obtain a third intermediate depth image.

[0249] It should be noted that, for the description of the electronic device performing image fusion on T depth images to be fused, reference may be made to the specific description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0250] It can be seen that since the electronic device can perform edge detection on each alternative spectral channel image to obtain each spectral channel detection result map, and perform edge detection on each alternative color channel image to obtain each color channel detection result map, and perform depth calculation based on each spectral channel detection result map, each color channel detection result map corresponding to each spectral channel detection result map, the rear camera internal parameter, the single channel pixel internal parameter corresponding to each alternative spectral channel image, and the rear single channel pixel extrinsic parameter corresponding to each alternative spectral channel image, respectively, and output T depth images to be fused, and then fuse the T depth images to be fused. Therefore, the fused third intermediate depth image can have more edges, so that in the subsequent steps, the electronic device can accurately determine the depth image based on the third intermediate depth image.

[0251] In some embodiments of the present application, the above step 102a2 can be specifically implemented through the following step 102a2d.

[0252] Step 102a2c: The electronic device fuses the second intermediate depth image and the third intermediate depth image to obtain a depth image.

[0253] It can be seen that since the electronic device can adopt the second set of camera parameters to perform different depth calculations according to the color image and the multispectral image, and different depth calculations have different characteristics, the electronic device can obtain a second intermediate depth image and a third intermediate depth image with two characteristics. In this way, the electronic device can obtain an accurate depth image after fusing the second intermediate depth image and the third intermediate depth image with two characteristics.

[0254] In some embodiments of the present application, the shooting scene is a low-light scene; the at least one group of camera parameters includes a third group of camera parameters; the third group of camera parameters includes a rear camera internal parameter, a multi-channel pixel internal parameter, and a rear multi-channel pixel external parameter; the multi-channel pixel internal parameter is an internal parameter corresponding to a multi-channel pixel unit of an image sensor of a multi-spectral camera, and each rear multi-channel pixel external parameter is an external parameter of the rear camera relative to the multi-channel pixel unit. In some examples, the above step 102a1 can be specifically performed through the following step 102a1d.

[0255] Step 102a1d: The electronic device performs depth calculation based on the color image, the multispectral channel image included in the multispectral image, the rear camera intrinsic parameter, the multi-channel pixel intrinsic parameter, and the rear multi-channel pixel extrinsic parameter, and outputs a fourth intermediate depth image.

[0256] In some embodiments of the present application, the multi-spectral channel image may be generated by image data obtained by photosensing a multi-channel pixel unit.

[0257] In some embodiments of the present application, the electronic device may first divide the multi-spectral image into N spectral channel images, and then determine the multi-spectral channel image from the N spectral channel images.

[0258] In some embodiments of the present application, the electronic device can perform depth calculation based on the color image, the multispectral channel image included in the multispectral image, the rear camera intrinsic parameters, the multi-channel pixel intrinsic parameters, and the rear multi-channel pixel extrinsic parameters, output a fifth intermediate depth image and a sixth intermediate depth image, and fuse the fifth intermediate depth image and the sixth intermediate depth image to obtain a fourth intermediate depth image.

[0259] It should be noted that, for the description of the electronic device performing depth calculation based on the color image, the multi-spectral channel image included in the multi-spectral image, the rear camera intrinsic parameter, the multi-channel pixel intrinsic parameter, and the rear multi-channel pixel extrinsic parameter, and outputting the fifth intermediate depth image and the sixth intermediate depth image, reference can be made to the specific description in the above embodiment that the electronic device performs depth calculation based on the color image, the N single-spectral channel images included in the multi-spectral image, the rear camera intrinsic parameter, the N single-channel pixel intrinsic parameters, and the N rear single-channel pixel extrinsic parameters, and outputs the second intermediate depth image and the third intermediate depth image, and the embodiments of the present application will not be repeated here.

[0260] It can be seen that since the multi-spectral channel image has more edges than other single-spectral channel images in low-light scenes, the electronic device can determine the multi-spectral channel image from the multi-spectral image in low-light scenes, and perform depth calculation based on the multi-spectral channel image, color image, rear camera intrinsic parameters, multi-channel pixel intrinsic parameters, and rear multi-channel pixel extrinsic parameters to output an accurate fourth intermediate depth image, and then in subsequent steps, the electronic device can obtain an accurate depth image.

[0261] In some embodiments of the present application, the above step 102a2 can be specifically implemented through the following step 102a2e.

[0262] Step 102a2e: The electronic device determines the fourth intermediate depth image as the depth image.

[0263] As can be seen, the electronic device can determine the fourth intermediate depth image as the depth image to obtain an accurate depth image.

[0264] The depth image acquisition method provided in the embodiment of the present application can be executed by a depth image acquisition device. In the embodiment of the present application, the depth image acquisition device executing the depth image acquisition method is taken as an example to illustrate the depth image acquisition device provided in the embodiment of the present application.

[0265] Fig. 9 FIG. 1 is a schematic diagram showing the structure of a depth image acquisition device provided in an embodiment of the present application. Fig. 9 As shown, the depth image acquisition device 30 provided in the embodiment of the present application may include: an acquisition module 31, which is used to acquire a color image through a rear camera of the depth image acquisition device 30, and to acquire a multispectral image through a multispectral camera of the depth image acquisition device 30. A processing module 32 is used to perform depth calculation according to the color image and multispectral image acquired by the acquisition module 31, and output a depth image.

[0266] The embodiment of the present application provides a depth image acquisition device. Since the pixel array of the image sensor of the multispectral camera of the depth image acquisition device includes N single-channel pixel units and multi-channel pixel units, when the multispectral camera is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to the different spectral bands, and the multi-channel pixel units can sense light signals of at least two spectral bands and generate multispectral channel images corresponding to at least two spectral bands. In this way, in addition to including different spectral channel images corresponding to different spectral bands, the multispectral image output by the multispectral camera can also include multispectral channel images corresponding to at least two spectral bands, and the multispectral channel image includes more object information of the photographed object, that is, the multispectral image output by the multispectral camera can include more object information of the photographed object, instead of the multispectral image output by the multispectral camera generally including only different spectral channel images corresponding to different spectral bands as in the related art, that is, the multispectral image output by the multispectral camera only includes less object information of the photographed object. Therefore, after the multispectral camera outputs the multispectral image, the depth image acquisition device can perform depth calculation based on the multispectral channel image including more object information of the photographed object. Furthermore, since the rear camera and the multispectral camera are arranged at different positions on the same surface of the depth image acquisition device, the field of view of the rear camera is within the field of view of the multispectral camera, which ensures that the rear camera and the multispectral camera can capture images of the same object from different perspectives. In this way, when the color image of the object is captured by the rear camera, the depth image acquisition device can capture the multispectral image of the object through the multispectral camera to obtain images of the object from different perspectives, and perform depth calculation based on the images of the object from different perspectives to directly obtain an accurate depth image of the object without capturing the depth image of the object through the depth camera. Therefore, it can avoid the situation where the depth image acquisition device cannot obtain the accurate depth image of the object, thereby improving the accuracy of the depth image acquired by the depth image acquisition device.

[0267] In a possible implementation, the processing module 32 is further used to obtain the shooting exposure value of the multispectral camera and determine the depth hole ratio before performing depth calculation based on the color image and the multispectral image; the depth hole ratio is used to characterize the texture details of the color image; and determine at least one set of camera parameters corresponding to the shooting exposure value and the depth hole ratio from the camera parameters of the rear camera and the multispectral camera. The processing module 32 is specifically used to use at least one set of camera parameters to perform depth calculation based on the color image and the multispectral image and output the depth image.

[0268] In a possible implementation, the processing module 32 is specifically used to obtain the exposure time value, ISO value and aperture value of the multispectral camera; and calculate the shooting exposure value according to the exposure time value, ISO value and aperture value.

[0269] In a possible implementation, the processing module 32 is specifically used to perform depth calculation based on the color image and the multispectral image, and output an initial depth image; the initial depth image includes at least one initial pixel point, and each initial pixel point corresponds to an initial depth value; and the ratio of the number of hole pixels and the number of at least one initial pixel point is determined as the depth hole ratio; the number of hole pixels is the number of initial pixels whose initial depth value is 0 in at least one initial pixel point.

[0270] In a possible implementation, the processing module 32 is specifically used to determine the shooting scene according to the relationship between the shooting exposure value and the exposure threshold and the relationship between the depth hole ratio and the ratio threshold; and determine at least one set of camera parameters according to the correspondence between the shooting scene and the camera parameters; wherein one shooting scene corresponds to a set of camera parameters, and a set of camera parameters includes at least two camera parameters, and when the shooting exposure value is greater than the exposure threshold and the depth hole ratio is greater than the ratio threshold, the shooting scene is a bright scene with many texture details; when the shooting exposure value is greater than the exposure threshold and the depth hole ratio is less than or equal to the ratio threshold, the shooting scene is a bright scene with few texture details; when the shooting exposure value is less than or equal to the exposure threshold, the shooting scene is a low-light scene.

[0271] In a possible implementation, the processing module 32 is specifically used to use each set of camera parameters to perform depth calculation according to the color image and the multispectral image, output at least one intermediate depth image; and determine the depth image according to the at least one intermediate depth image.

[0272] In a possible implementation, the shooting scene is a bright multi-texture detail scene; at least one group of camera parameters includes a first group of camera parameters and a second group of camera parameters; the first group of camera parameters includes rear camera intrinsic parameters, multispectral camera intrinsic parameters and rear multispectral extrinsic parameters; the second group of camera parameters includes rear camera intrinsic parameters, N single-channel pixel intrinsic parameters, and N rear single-channel pixel extrinsic parameters; the rear multispectral extrinsic parameters are extrinsic parameters of the rear camera relative to the multispectral camera, each single-channel pixel intrinsic parameter is an intrinsic parameter corresponding to a single-channel pixel unit of the image sensor of the multispectral camera, and each rear single-channel pixel extrinsic parameter is an extrinsic parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1. The above-mentioned processing module 32 is specifically used to perform depth calculation based on the color image, the multispectral image, the rear camera internal parameters, the multispectral camera internal parameters and the rear multispectral external parameters, and output a first intermediate depth image; and perform depth calculation based on the color image, N single-spectral channel images included in the multispectral image, the rear camera internal parameters, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters, and output a second intermediate depth image.

[0273] In a possible implementation, each single spectral channel image includes at least one spectral channel pixel, and each spectral channel pixel corresponds to an edge information value. The above-mentioned processing module 32 is specifically used to determine the number of spectral channel pixels whose edge information value is not 0 in each single spectral channel image; and to determine L candidate spectral channel images whose number of pixels is greater than a quantity threshold from N single spectral channel images; L is a positive integer; and to determine T candidate color channel images whose color wavelengths match L spectral bands from M color channel images; the M color channel images are the color channel images included in the color image, and the L spectral bands are the spectral bands corresponding to the L candidate spectral channel images; each candidate spectral channel image corresponds to at least one candidate color channel image, M is a positive integer, and T is a positive integer greater than or equal to L; and, based on the L candidate spectral channel images, the T candidate color channel images, the rear camera intrinsic parameters, the L single-channel pixel intrinsic parameters and the L rear single-channel pixel extrinsic parameters, a depth calculation is performed to output a second intermediate depth image; the L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L candidate spectral channel images among the N single-channel pixel intrinsic parameters, and the L rear single-channel pixel extrinsic parameters are the rear single-channel pixel extrinsic parameters corresponding to the L candidate spectral channel images among the N rear single-channel pixel extrinsic parameters.

[0274] In one possible implementation, the processing module 32 is specifically used to perform depth calculations based on each alternative spectral channel image, each alternative color channel image corresponding to each alternative spectral channel image, the rear camera intrinsic parameters, the single-channel pixel intrinsic parameters corresponding to each alternative spectral channel image, and the rear single-channel pixel extrinsic parameters corresponding to each alternative spectral channel image, and output T depth images to be fused; and fuse the T depth images to be fused to obtain a second intermediate depth image.

[0275] In a possible implementation, the processing module 32 is specifically configured to perform image fusion on the first intermediate depth image and the second intermediate depth image to obtain a depth image.

[0276] In a possible implementation, the shooting scene is a bright scene with few texture details; at least one set of camera parameters includes a second set of camera parameters; the second set of camera parameters includes a rear camera intrinsic parameter, N single-channel pixel intrinsic parameters, and N rear single-channel pixel extrinsic parameters; each single-channel pixel intrinsic parameter is an intrinsic parameter corresponding to a single-channel pixel unit of an image sensor of a multispectral camera, and each rear single-channel pixel extrinsic parameter is an extrinsic parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1. The processing module 32 is specifically used to perform depth calculations based on the N single-spectral channel images included in the color image and the multispectral image, the rear camera intrinsic parameters, the N single-channel pixel intrinsic parameters, and the N rear single-channel pixel extrinsic parameters, and output a second intermediate depth image and a third intermediate depth image.

[0277] In a possible implementation, the processing module 32 is specifically used to determine the number of spectral channel pixels whose edge information values ​​are not 0 in each single spectral channel image; and determine L candidate spectral channel images whose number of pixels is greater than a threshold value from N single spectral channel images; L is a positive integer; and determine T candidate color channel images whose color wavelengths match the L spectral bands from M color channel images; the M color channel images are the color channel images included in the color image, and the L spectral bands are the spectral bands corresponding to the L candidate spectral channel images; each candidate spectral channel image corresponds to at least one candidate color channel image, M is a positive integer, and T is a positive integer greater than or equal to L; and, based on the L candidate spectral channel images, the T candidate color channel images, the rear camera intrinsic parameters, the L single-channel pixel intrinsic parameters and the L rear single-channel pixel extrinsic parameters, depth calculations are performed respectively to output a second intermediate depth image and a third intermediate depth image; the L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L candidate spectral channel images among the N single-channel pixel intrinsic parameters, and the L rear single-channel pixel extrinsic parameters are the rear single-channel pixel extrinsic parameters corresponding to the L candidate spectral channel images among the N rear single-channel pixel extrinsic parameters.

[0278] In a possible implementation, the processing module 32 is specifically used to perform depth calculations based on each candidate spectral channel image, each candidate color channel image corresponding to each candidate spectral channel image, a rear camera internal parameter, a single-channel pixel internal parameter corresponding to each candidate spectral channel image, and a rear single-channel pixel extrinsic parameter corresponding to each candidate spectral channel image, and output T depth images to be fused; and fuse the T depth images to be fused to obtain a second intermediate depth image; and perform edge detection on each candidate spectral channel image to obtain L spectral channel detection result images, ... Select color channel images for edge detection to obtain T color channel detection result images; each spectral channel detection result image corresponds to at least one color channel detection result image; and based on each spectral channel detection result image, each color channel detection result image corresponding to each spectral channel detection result image, a rear camera intrinsic parameter, a single channel pixel intrinsic parameter corresponding to each candidate spectral channel image, and a rear single channel pixel extrinsic parameter corresponding to each candidate spectral channel image, respectively perform depth calculations to output T depth images to be fused; and, fuse the T depth images to be fused to obtain a third intermediate depth image.

[0279] In a possible implementation, the processing module 32 is specifically used to respectively use at least two detection algorithms to perform edge detection on each candidate spectral channel image to generate at least two spectral channel detection process maps; each spectral channel detection process map includes at least one detection process pixel point, and each detection process pixel point corresponds to an edge information value; and according to the edge information value corresponding to each detection process pixel point in at least one auxiliary spectral channel detection process map, the edge pixel points are marked in the detection process pixel points in the main spectral channel detection process map; the main spectral channel detection process map is a spectral channel detection process map among the at least two spectral channel detection process maps, and at least one auxiliary spectral channel detection process map is a spectral channel detection process map among the at least two spectral channel detection process maps except the main spectral channel detection process map; and based on the marked main spectral channel detection process map, L spectral channel detection result maps are generated.

[0280] In one possible implementation, the processing module 32 is specifically used to perform edge detection on each candidate color channel image respectively to obtain L color channel detection process diagrams; and based on the spectral channel detection result diagrams of the candidate spectral channel images corresponding to each candidate color channel image, perform edge padding processing on each color channel detection process diagram to obtain L color channel detection result diagrams.

[0281] In a possible implementation, the processing module 32 is specifically configured to perform image fusion on the second intermediate depth image and the third intermediate depth image to obtain a depth image.

[0282] In a possible implementation, the shooting scene is a low-light scene; at least one group of camera parameters includes a third group of camera parameters; the third group of camera parameters includes a rear camera intrinsic parameter, a multi-channel pixel intrinsic parameter, and a rear multi-channel pixel extrinsic parameter; the multi-channel pixel intrinsic parameter is an intrinsic parameter corresponding to a multi-channel pixel unit of an image sensor of a multi-spectral camera, and each rear multi-channel pixel extrinsic parameter is an extrinsic parameter of the rear camera relative to the multi-channel pixel unit. The processing module 32 is specifically used to perform depth calculation based on the color image, the multi-spectral channel image included in the multi-spectral image, the rear camera intrinsic parameter, the multi-channel pixel intrinsic parameter, and the rear multi-channel pixel extrinsic parameter, and output a fourth intermediate depth image.

[0283] In a possible implementation, the processing module 32 is specifically configured to determine the fourth intermediate depth image as the depth image.

[0284] In one possible implementation, the processing module 32 is specifically used to determine an initial intermediate depth image based on at least one intermediate depth image; and input a color image corresponding to the initial intermediate depth image and the color image into a depth completion model, perform depth completion processing on the initial intermediate depth image according to the color image through the depth completion model, and output a depth image.

[0285] The depth image acquisition device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile internet device (mobile internet device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (network attached storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0286] The depth image acquisition device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0287] The depth image acquisition device provided in the embodiment of the present application can achieve Figures 1 to 8 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0288] In some embodiments of the present application, Fig.10 As shown, an embodiment of the present application further provides an electronic device 40, including a processor 41 and a memory 42, wherein the memory 42 stores programs or instructions that can be executed on the processor 41, and when the program or instructions are executed by the processor 41, the various process steps of the above-mentioned depth image acquisition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0289] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0290] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0291] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0292] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0293] The input unit 104 is used to collect color images through the rear camera of the electronic device, and collect multispectral images through the multispectral camera of the electronic device.

[0294] The processor 110 is used to perform depth calculation according to the color image and the multi-spectral image, and output a depth image.

[0295] An embodiment of the present application provides an electronic device. Since the pixel array of the image sensor of the multispectral camera of the electronic device includes N single-channel pixel units and multi-channel pixel units, when the multispectral camera is used for shooting, the N single-channel pixel units can sense light signals of different spectral bands and generate different spectral channel images corresponding to the different spectral bands, and the multi-channel pixel units can sense light signals of at least two spectral bands and generate multispectral channel images corresponding to at least two spectral bands. In this way, in addition to including different spectral channel images corresponding to different spectral bands, the multispectral image output by the multispectral camera can also include multispectral channel images corresponding to at least two spectral bands, and the multispectral channel image includes more object information of the photographed object, that is, the multispectral image output by the multispectral camera can include more object information of the photographed object, instead of the multispectral image output by the multispectral camera generally including only different spectral channel images corresponding to different spectral bands as in the related art, that is, the multispectral image output by the multispectral camera only includes less object information of the photographed object. Therefore, after the multispectral camera outputs the multispectral image, the electronic device can perform depth calculation based on the multispectral channel image including more object information of the photographed object. Furthermore, since the rear camera and the multispectral camera are arranged at different positions on the same side of the electronic device, the field of view of the rear camera is within the field of view of the multispectral camera. This ensures that the rear camera and the multispectral camera can capture images of the same object from different perspectives. In this way, when the color image of the object is captured by the rear camera, the electronic device can capture the multispectral image of the object through the multispectral camera to obtain images of the object from different perspectives, and perform depth calculation based on the images of the object from different perspectives to directly obtain an accurate depth image of the object without the need to capture the depth image of the object through the depth camera. Therefore, it can avoid the situation where the electronic device cannot obtain an accurate depth image of the object, thereby improving the accuracy of the depth image acquired by the electronic device.

[0296] In some embodiments of the present application, the processor 110 is further used to obtain the shooting exposure value of the multispectral camera and determine the depth hole ratio before performing depth calculation based on the color image and the multispectral image; the depth hole ratio is used to characterize the texture details of the color image; and determine at least one set of camera parameters corresponding to the shooting exposure value and the depth hole ratio from the camera parameters of the rear camera and the multispectral camera.

[0297] The processor 110 is specifically configured to use at least one set of camera parameters to perform depth calculation according to the color image and the multi-spectral image, and output a depth image.

[0298] In some embodiments of the present application, the processor 110 is specifically used to obtain the exposure time value, sensitivity ISO value and aperture value of the multispectral camera; and calculate the shooting exposure value according to the exposure time value, ISO value and aperture value.

[0299] In some embodiments of the present application, the processor 110 is specifically used to perform depth calculation based on the color image and the multispectral image, and output an initial depth image; the initial depth image includes at least one initial pixel point, and each initial pixel point corresponds to an initial depth value; and the ratio of the number of hole pixels and the number of at least one initial pixel point is determined as the depth hole ratio; the number of hole pixels is the number of initial pixels whose initial depth value is 0 in at least one initial pixel point.

[0300] In some embodiments of the present application, the processor 110 is specifically used to determine the shooting scene according to the size relationship between the shooting exposure value and the exposure threshold and the size relationship between the depth hole ratio and the ratio threshold; and determine at least one set of camera parameters according to the correspondence between the shooting scene and the camera parameters; wherein, one shooting scene corresponds to a set of camera parameters, and a set of camera parameters includes at least two camera parameters, and when the shooting exposure value is greater than the exposure threshold and the depth hole ratio is greater than the ratio threshold, the shooting scene is a bright scene with many texture details; when the shooting exposure value is greater than the exposure threshold and the depth hole ratio is less than or equal to the ratio threshold, the shooting scene is a bright scene with few texture details; when the shooting exposure value is less than or equal to the exposure threshold, the shooting scene is a low-light scene.

[0301] In some embodiments of the present application, the processor 110 is specifically used to use each set of camera parameters to perform depth calculation according to the color image and the multispectral image, output at least one intermediate depth image; and determine the depth image according to the at least one intermediate depth image.

[0302] In some embodiments of the present application, the above-mentioned shooting scene is a bright multi-texture detail scene; at least one group of camera parameters includes a first group of camera parameters and a second group of camera parameters; the first group of camera parameters includes rear camera intrinsic parameters, multispectral camera intrinsic parameters and rear multispectral extrinsic parameters; the second group of camera parameters includes rear camera intrinsic parameters, N single-channel pixel intrinsic parameters, and N rear single-channel pixel extrinsic parameters; the rear multispectral extrinsic parameters are external parameters of the rear camera relative to the multispectral camera, and each single-channel pixel intrinsic parameter is the internal parameter corresponding to a single-channel pixel unit of the image sensor of the multispectral camera, and each rear single-channel pixel extrinsic parameter is the external parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1.

[0303] The processor 110 is specifically used to perform depth calculation based on the color image, the multispectral image, the rear camera internal parameters, the multispectral camera internal parameters and the rear multispectral external parameters, and output a first intermediate depth image; and perform depth calculation based on the color image, N single-spectral channel images included in the multispectral image, the rear camera internal parameters, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters, and output a second intermediate depth image.

[0304] In some embodiments of the present application, each single spectral channel image includes at least one spectral channel pixel point, and each spectral channel pixel point corresponds to an edge information value.

[0305] The processor 110 is specifically used to determine the number of spectral channel pixels whose edge information values ​​are not 0 in each single spectral channel image; and determine L candidate spectral channel images whose number of pixels is greater than a number threshold from N single spectral channel images; L is a positive integer; and determine T candidate color channel images whose color wavelengths match L spectral bands from M color channel images; the M color channel images are color channel images included in the color image, and the L spectral bands are spectral bands corresponding to the L candidate spectral channel images; each candidate spectral channel image corresponds to at least a candidate color channel image, M is a positive integer, T is a positive integer greater than or equal to L; and, based on the L candidate spectral channel images, the T candidate color channel images, the rear camera intrinsic parameters, the L single-channel pixel intrinsic parameters and the L rear single-channel pixel extrinsic parameters, depth calculation is performed to output a second intermediate depth image; the L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L candidate spectral channel images among the N single-channel pixel intrinsic parameters, and the L rear single-channel pixel extrinsic parameters are the rear single-channel pixel extrinsic parameters corresponding to the L candidate spectral channel images among the N rear single-channel pixel extrinsic parameters.

[0306] In some embodiments of the present application, the processor 110 is specifically used to perform depth calculations based on each alternative spectral channel image, each alternative color channel image corresponding to each alternative spectral channel image, the rear camera intrinsic parameters, the single-channel pixel intrinsic parameters corresponding to each alternative spectral channel image, and the rear single-channel pixel extrinsic parameters corresponding to each alternative spectral channel image, and output T depth images to be fused; and perform image fusion on the T depth images to be fused to obtain a second intermediate depth image.

[0307] In some embodiments of the present application, the processor 110 is specifically configured to perform image fusion on the first intermediate depth image and the second intermediate depth image to obtain a depth image.

[0308] In some embodiments of the present application, the above-mentioned shooting scene is a bright scene with few texture details; at least one group of camera parameters includes a second group of camera parameters; the second group of camera parameters includes rear camera intrinsic parameters, N single-channel pixel intrinsic parameters, and N rear single-channel pixel extrinsic parameters; each single-channel pixel intrinsic parameter is an intrinsic parameter corresponding to a single-channel pixel unit of the image sensor of the multi-spectral camera, and each rear single-channel pixel extrinsic parameter is an extrinsic parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1.

[0309] The processor 110 is specifically used to perform depth calculation based on the color image, N single spectral channel images included in the multispectral image, the rear camera internal parameters, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters, and output the second intermediate depth image and the third intermediate depth image.

[0310] In some embodiments of the present application, the processor 110 is specifically used to determine the number of spectral channel pixels whose edge information values ​​are not 0 in each single spectral channel image; and determine L candidate spectral channel images whose number of pixels is greater than a number threshold from N single spectral channel images; L is a positive integer; and determine T candidate color channel images whose color wavelengths match L spectral bands from M color channel images; the M color channel images are color channel images included in the color image, and the L spectral bands are spectral bands corresponding to the L candidate spectral channel images; each candidate spectral channel image corresponds to at least one candidate color channel image, M is a positive integer, and T is a positive integer greater than or equal to L; and based on the L candidate spectral channel images, the T candidate color channel images, the rear camera intrinsic parameters, the L single-channel pixel intrinsic parameters and the L rear single-channel pixel extrinsic parameters, depth calculations are performed respectively to output a second intermediate depth image and a third intermediate depth image; the L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L candidate spectral channel images among the N single-channel pixel intrinsic parameters, and the L rear single-channel pixel extrinsic parameters are the rear single-channel pixel extrinsic parameters corresponding to the L candidate spectral channel images among the N rear single-channel pixel extrinsic parameters.

[0311] In some embodiments of the present application, the processor 110 is specifically used to perform depth calculations based on each candidate spectral channel image, each candidate color channel image corresponding to each candidate spectral channel image, a rear camera internal parameter, a single-channel pixel internal parameter corresponding to each candidate spectral channel image, and a rear single-channel pixel extrinsic parameter corresponding to each candidate spectral channel image, and output T depth images to be fused; and fuse the T depth images to be fused to obtain a second intermediate depth image; and perform edge detection on each candidate spectral channel image to obtain L spectral channel detection result images, ... Perform edge detection on the color channel image to obtain T color channel detection result images; each spectral channel detection result image corresponds to at least one color channel detection result image; and perform depth calculation based on each spectral channel detection result image, each color channel detection result image corresponding to each spectral channel detection result image, the rear camera intrinsic parameter, the single channel pixel intrinsic parameter corresponding to each alternative spectral channel image, and the rear single channel pixel extrinsic parameter corresponding to each alternative spectral channel image, and output T depth images to be fused; and perform image fusion on the T depth images to be fused to obtain a third intermediate depth image.

[0312] In some embodiments of the present application, the processor 110 is specifically used to respectively adopt at least two detection algorithms to perform edge detection on each alternative spectral channel image, and generate at least two spectral channel detection process maps; each spectral channel detection process map includes at least one detection process pixel point, and each detection process pixel point corresponds to an edge information value; and according to the edge information value corresponding to each detection process pixel point in at least one auxiliary spectral channel detection process map, the edge pixel points are marked in the detection process pixel points in the main spectral channel detection process map; the main spectral channel detection process map is a spectral channel detection process map among the at least two spectral channel detection process maps, and at least one auxiliary spectral channel detection process map is a spectral channel detection process map among the at least two spectral channel detection process maps except the main spectral channel detection process map; and based on the marked main spectral channel detection process map, L spectral channel detection result maps are generated.

[0313] In some embodiments of the present application, the processor 110 is specifically used to perform edge detection on each candidate color channel image respectively to obtain L color channel detection process images; and perform edge padding processing on each color channel detection process image based on the spectral channel detection result image of the alternative spectral channel image corresponding to each candidate color channel image, to obtain L color channel detection result images.

[0314] In some embodiments of the present application, the processor 110 is specifically configured to perform image fusion on the second intermediate depth image and the third intermediate depth image to obtain a depth image.

[0315] In some embodiments of the present application, the above-mentioned shooting scene is a low-light scene; the above-mentioned at least one group of camera parameters includes a third group of camera parameters; the third group of camera parameters includes rear camera intrinsic parameters, multi-channel pixel intrinsic parameters, and rear multi-channel pixel extrinsic parameters; the multi-channel pixel intrinsic parameters are the intrinsic parameters corresponding to the multi-channel pixel units of the image sensor of the multi-spectral camera, and each rear multi-channel pixel extrinsic parameter is an external parameter of the rear camera relative to the multi-channel pixel unit.

[0316] The processor 110 is specifically used to perform depth calculation based on the color image, the multispectral channel image included in the multispectral image, the rear camera intrinsic parameter, the multi-channel pixel intrinsic parameter, and the rear multi-channel pixel extrinsic parameter, and output a fourth intermediate depth image.

[0317] In some embodiments of the present application, the processor 110 is specifically configured to determine the fourth intermediate depth image as the depth image.

[0318] In some embodiments of the present application, the processor 110 is specifically used to determine an initial intermediate depth image based on at least one intermediate depth image; and input a color image corresponding to the initial intermediate depth image and the color image into a depth completion model, perform depth completion processing on the initial intermediate depth image according to the color image through the depth completion model, and output a depth image.

[0319] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processing unit (GPU) 1041, a microphone 1042 and an image sensor 1043, and the graphics processor 1041 processes the image data of the static picture or video obtained by the image sensor 1043 in the video capture mode or the image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0320] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct rambus RAM (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0321] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0322] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned depth image acquisition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0323] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0324] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned depth image acquisition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0325] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0326] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned depth image acquisition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0327] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0328] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0329] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image sensor, characterized in that: include: A pixel array, wherein the pixel array includes N single-channel pixel units and multi-channel pixel units; A single-channel pixel unit is used to sense a light signal of one spectral band, and the multi-channel pixel unit is used to sense light signals of at least two spectral bands, and N is a positive integer greater than 1.

2. A multispectral camera, characterized in that: Comprising the image sensor as claimed in claim 1.

3. An electronic device, characterized in that: It includes a rear camera and a multispectral camera as claimed in claim 2; wherein the rear camera and the multispectral camera are arranged at different positions on the same side of the electronic device, and the field of view of the rear camera is within the field of view of the multispectral camera.

4. A depth image acquisition method, executed by the electronic device according to claim 3, characterized in that: The method comprises: Acquiring a color image through a rear camera of the electronic device, and acquiring a multispectral image through a multispectral camera of the electronic device; Depth calculation is performed according to the color image and the multispectral image, and a depth image is output.

5. The method according to claim 1, characterized in that Before performing depth calculation according to the color image and the multispectral image, the method further includes: Acquire the shooting exposure value of the multispectral camera and determine the depth hole ratio; the depth hole ratio is used to characterize the texture details of the color image; Determine at least one set of camera parameters corresponding to the shooting exposure value and the depth hole ratio from camera parameters of the rear camera and the multispectral camera; The performing depth calculation according to the color image and the multispectral image and outputting a depth image comprises: The at least one set of camera parameters is used to perform depth calculation according to the color image and the multispectral image, and a depth image is output.

6. The method according to claim 5, characterized in that The obtaining of the shooting exposure value of the multi-spectral camera includes: Obtain the exposure time value, ISO value and aperture value of the multispectral camera; A shooting exposure value is calculated according to the exposure time value, the ISO value and the aperture value.

7. The method according to claim 5, characterized in that The determining of the depth hole ratio comprises: Performing depth calculation according to the color image and the multispectral image, and outputting an initial depth image; the initial depth image includes at least one initial pixel point, and each initial pixel point corresponds to an initial depth value; The ratio of the number of hole pixels to the number of the at least one initial pixel is determined as the depth hole ratio; the number of hole pixels is the number of initial pixels whose initial depth value is 0 among the at least one initial pixel.

8. The method according to claim 5, characterized in that The step of determining at least one set of camera parameters corresponding to the shooting exposure value and the depth hole ratio from a plurality of camera parameters includes: Determining a shooting scene according to a magnitude relationship between the shooting exposure value and the exposure threshold and a magnitude relationship between the depth hole ratio and the ratio threshold; Determine at least one set of camera parameters according to the corresponding relationship between the shooting scene and the camera parameters; Among them, one shooting scene corresponds to a group of camera parameters, and a group of camera parameters includes at least two camera parameters. When the shooting exposure value is greater than the exposure threshold and the depth hole ratio is greater than the ratio threshold, the shooting scene is a bright scene with many texture details; when the shooting exposure value is greater than the exposure threshold and the depth hole ratio is less than or equal to the ratio threshold, the shooting scene is a bright scene with few texture details; when the shooting exposure value is less than or equal to the exposure threshold, the shooting scene is a low-light scene.

9. The method according to claim 5, characterized in that The step of using the at least one set of camera parameters to perform depth calculation according to the color image and the multispectral image to obtain a depth image includes: Using each set of camera parameters respectively, performing depth calculation according to the color image and the multispectral image, and outputting at least one intermediate depth image; A depth image is determined according to the at least one intermediate depth image.

10. The method according to claim 9, characterized in that The shooting scene is a bright multi-texture detail scene; the at least one group of camera parameters includes a first group of camera parameters and a second group of camera parameters; the first group of camera parameters includes rear camera internal parameters, multispectral camera internal parameters and rear multispectral external parameters; the second group of camera parameters includes rear camera internal parameters, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters; the rear multispectral external parameters are external parameters of the rear camera relative to the multispectral camera, each single-channel pixel internal parameter is an internal parameter corresponding to a single-channel pixel unit of the image sensor of the multispectral camera, and each rear single-channel pixel external parameter is an external parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1; The method of using each group of camera parameters respectively, performing depth calculation according to the color image and the multispectral image, and outputting at least one intermediate depth image comprises: Performing depth calculation based on the color image, the multispectral image, the rear camera internal parameter, the multispectral camera internal parameter and the rear multispectral external parameter, and outputting a first intermediate depth image; Based on the color image, the N single-spectral channel images included in the multispectral image, the rear camera internal parameters, the N single-channel pixel internal parameters, and the N rear single-channel pixel external parameters, depth calculation is performed to output a second intermediate depth image.

11. The method according to claim 10, characterized in that Each single spectral channel image includes at least one spectral channel pixel point, and each spectral channel pixel point corresponds to an edge information value; The method of performing depth calculation based on the color image, the N single-spectral channel images included in the multispectral image, the rear camera internal parameters, the N single-channel pixel internal parameters, and the N rear single-channel pixel external parameters to output a second intermediate depth image includes: Determine the number of spectral channel pixel points whose edge information value is not 0 in each single spectral channel image; Determine, from the N single spectral channel images, L candidate spectral channel images whose number of pixels is greater than a number threshold, where L is a positive integer; Determine T candidate color channel images whose color wavelengths match L spectral bands from M color channel images; the M color channel images are color channel images included in the color image, and the L spectral bands are spectral bands corresponding to the L candidate spectral channel images; each candidate spectral channel image corresponds to at least one candidate color channel image, M is a positive integer, and T is a positive integer greater than or equal to L; Based on the L candidate spectral channel images, the T candidate color channel images, the rear camera intrinsic parameters, L single-channel pixel intrinsic parameters and L rear single-channel pixel extrinsic parameters, depth calculation is performed to output a second intermediate depth image; the L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L candidate spectral channel images among the N single-channel pixel intrinsic parameters, and the L rear single-channel pixel extrinsic parameters are the rear single-channel pixel extrinsic parameters corresponding to the L candidate spectral channel images among the N rear single-channel pixel extrinsic parameters.

12. The method according to claim 11, characterized in that The method of performing depth calculation based on the L candidate spectral channel images, the T candidate color channel images, the rear camera internal parameters, the L single-channel pixel internal parameters and the L rear single-channel pixel external parameters to output a second intermediate depth image includes: Based on each candidate spectral channel image, each candidate color channel image corresponding to each candidate spectral channel image, the rear camera internal parameter, the single channel pixel internal parameter corresponding to each candidate spectral channel image, and the rear single channel pixel extrinsic parameter corresponding to each candidate spectral channel image, depth calculation is performed respectively, and T depth images to be fused are output; The T depth images to be fused are fused to obtain a second intermediate depth image.

13. The method according to any one of claims 10 to 12, characterized in that The determining of the depth image according to the at least one intermediate depth image comprises: The first intermediate depth image and the second intermediate depth image are fused to obtain a depth image.

14. The method according to claim 9, characterized in that The shooting scene is a bright scene with few texture details; the at least one group of camera parameters includes a second group of camera parameters; the second group of camera parameters includes a rear camera internal parameter, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters; each single-channel pixel internal parameter is an internal parameter corresponding to a single-channel pixel unit of the image sensor of the multi-spectral camera, and each rear single-channel pixel external parameter is an external parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1; The method of using each group of camera parameters respectively, performing depth calculation according to the color image and the multispectral image, and outputting at least one intermediate depth image comprises: Based on the color image, the N single spectral channel images included in the multispectral image, the rear camera internal parameters, the N single-channel pixel internal parameters, and the N rear single-channel pixel external parameters, depth calculations are performed respectively to output a second intermediate depth image and a third intermediate depth image.

15. The method according to claim 14, characterized in that The method of performing depth calculation based on the color image, the N single-spectral channel images included in the multispectral image, the rear camera internal parameters, the N single-channel pixel internal parameters, and the N rear single-channel pixel external parameters, and outputting a second intermediate depth image and a third intermediate depth image, includes: Determine the number of spectral channel pixel points whose edge information value is not 0 in each single spectral channel image; Determine, from the N single spectral channel images, L candidate spectral channel images whose number of pixels is greater than a number threshold, where L is a positive integer; Determine T candidate color channel images whose color wavelengths match L spectral bands from M color channel images; the M color channel images are color channel images included in the color image, and the L spectral bands are spectral bands corresponding to the L candidate spectral channel images; each candidate spectral channel image corresponds to at least one candidate color channel image, M is a positive integer, and T is a positive integer greater than or equal to L; Based on the L candidate spectral channel images, the T candidate color channel images, the rear camera intrinsic parameters, L single-channel pixel intrinsic parameters and L rear single-channel pixel extrinsic parameters, depth calculations are performed respectively to output the second intermediate depth image and the third intermediate depth image; the L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L candidate spectral channel images among the N single-channel pixel intrinsic parameters, and the L rear single-channel pixel extrinsic parameters are the rear single-channel pixel extrinsic parameters corresponding to the L candidate spectral channel images among the N rear single-channel pixel extrinsic parameters.

16. The method according to claim 15, characterized in that The method of performing depth calculation based on the L candidate spectral channel images, the T candidate color channel images, the rear camera internal parameters, the L single-channel pixel internal parameters and the L rear single-channel pixel external parameters, and outputting the second intermediate depth image and the third intermediate depth image includes: Based on each candidate spectral channel image, each candidate color channel image corresponding to each candidate spectral channel image, the rear camera internal parameter, the single channel pixel internal parameter corresponding to each candidate spectral channel image, and the rear single channel pixel extrinsic parameter corresponding to each candidate spectral channel image, depth calculation is performed respectively, and T depth images to be fused are output; Performing image fusion on the T depth images to be fused to obtain a second intermediate depth image; Perform edge detection on each candidate spectral channel image to obtain L spectral channel detection result images, and perform edge detection on each candidate color channel image to obtain T color channel detection result images; each spectral channel detection result image corresponds to at least one color channel detection result image; Based on each spectral channel detection result image, each color channel detection result image corresponding to each spectral channel detection result image, the rear camera internal parameter, the single channel pixel internal parameter corresponding to each candidate spectral channel image, and the rear single channel pixel extrinsic parameter corresponding to each candidate spectral channel image, depth calculation is performed respectively, and T depth images to be fused are output; The T depth images to be fused are fused to obtain a third intermediate depth image.

17. The method according to claim 16, characterized in that The edge detection is performed on each candidate spectral channel image to obtain L spectral channel detection result images, including: At least two detection algorithms are used respectively to perform edge detection on each candidate spectral channel image, and at least two spectral channel detection process diagrams are generated; each spectral channel detection process diagram includes at least one detection process pixel point, and each detection process pixel point corresponds to an edge information value; Mark edge pixels in the detection process pixels in the main spectral channel detection process map according to edge information values ​​corresponding to each detection process pixel in at least one auxiliary spectral channel detection process map; the main spectral channel detection process map is one of the at least two spectral channel detection process maps, and the at least one auxiliary spectral channel detection process map is a spectral channel detection process map among the at least two spectral channel detection process maps except the main spectral channel detection process map; Based on the marked main spectral channel detection process diagram, L spectral channel detection result diagrams are generated.

18. The method according to claim 16, characterized in that The step of performing edge detection on each candidate color channel image to obtain L color channel detection result images includes: Perform edge detection on each candidate color channel image respectively to obtain L color channel detection process diagrams; Based on the spectral channel detection result images of the candidate spectral channel images corresponding to each candidate color channel image, edge padding processing is performed on each color channel detection process image to obtain L color channel detection result images.

19. The method according to any one of claims 14 to 18, characterized in that The determining of the depth image according to the at least one intermediate depth image comprises: The second intermediate depth image and the third intermediate depth image are fused to obtain a depth image.

20. The method according to claim 9, characterized in that The shooting scene is a low-light scene; the at least one group of camera parameters includes a third group of camera parameters; the third group of camera parameters includes a rear camera internal parameter, a multi-channel pixel internal parameter, and a rear multi-channel pixel external parameter; the multi-channel pixel internal parameter is an internal parameter corresponding to a multi-channel pixel unit of an image sensor of the multi-spectral camera, and each rear multi-channel pixel external parameter is an external parameter of the rear camera relative to the multi-channel pixel unit; The method of using each group of camera parameters respectively, performing depth calculation according to the color image and the multispectral image, and outputting at least one intermediate depth image comprises: Based on the color image, the multispectral channel image included in the multispectral image, the rear camera intrinsic parameter, the multi-channel pixel intrinsic parameter, and the rear multi-channel pixel extrinsic parameter, a depth calculation is performed to output a fourth intermediate depth image.

21. The method according to claim 20, characterized in that The determining of the depth image according to the at least one intermediate depth image comprises: The fourth intermediate depth image is determined as the depth image.

22. The method according to any one of claims 9 to 21, characterized in that The determining of the depth image according to the at least one intermediate depth image comprises: Determining an initial intermediate depth image according to the at least one intermediate depth image; The initial intermediate depth image and a color image corresponding to the color image are input into a depth completion model, and the depth completion model performs depth completion processing on the initial intermediate depth image according to the color image to output a depth image.

23. A depth image acquisition device, characterized in that: include: An acquisition module, used to acquire a color image through a rear camera of the depth image acquisition device, and to acquire a multispectral image through a multispectral camera of the depth image acquisition device; A processing module is used to perform depth calculation based on the color image and the multispectral image collected by the collection module, and output a depth image.

24. The device according to claim 23, characterized in that The processing module is specifically used to use each set of camera parameters respectively, perform depth calculation according to the color image and the multispectral image, output at least one intermediate depth image; and determine the depth image according to the at least one intermediate depth image.

25. The device according to claim 24, characterized in that The shooting scene is a bright multi-texture detail scene; the at least one group of camera parameters includes a first group of camera parameters and a second group of camera parameters; the first group of camera parameters includes a rear camera internal parameter, a multispectral camera internal parameter and a rear multispectral external parameter; the second group of camera parameters includes a rear camera internal parameter, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters; the rear multispectral external parameter is an external parameter of the rear camera relative to the multispectral camera, each single-channel pixel internal parameter is an internal parameter corresponding to a single-channel pixel unit of the image sensor of the multispectral camera, and each rear single-channel pixel external parameter is an external parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1; The processing module is specifically used to perform depth calculation based on the color image, the multispectral image, the rear camera internal parameters, the multispectral camera internal parameters and the rear multispectral external parameters, and output a first intermediate depth image; and perform depth calculation based on the color image, N single-spectral channel images included in the multispectral image, the rear camera internal parameters, the N single-channel pixel internal parameters, and the N rear single-channel pixel external parameters, and output a second intermediate depth image.

26. The device according to claim 24, characterized in that The shooting scene is a bright scene with few texture details; the at least one group of camera parameters includes a second group of camera parameters; the second group of camera parameters includes a rear camera internal parameter, N single-channel pixel internal parameters, and N rear single-channel pixel external parameters; each single-channel pixel internal parameter is an internal parameter corresponding to a single-channel pixel unit of the image sensor of the multi-spectral camera, and each rear single-channel pixel external parameter is an external parameter of the rear camera relative to a single-channel pixel unit; N is a positive integer greater than 1; The processing module is specifically used to perform depth calculation based on the color image, the N single spectral channel images included in the multispectral image, the rear camera intrinsic parameters, the N single-channel pixel intrinsic parameters, and the N rear single-channel pixel extrinsic parameters, and output a second intermediate depth image and a third intermediate depth image.

27. The device according to claim 26, characterized in that The processing module is specifically used to determine the number of spectral channel pixels whose edge information value is not 0 in each single spectral channel image; and determine L candidate spectral channel images whose number of pixels is greater than a number threshold from the N single spectral channel images; L is a positive integer; and determine T candidate color channel images whose color wavelengths match the L spectral bands from the M color channel images; the M color channel images are the color channel images included in the color image, and the L spectral bands are the spectral bands corresponding to the L candidate spectral channel images; Each candidate spectral channel image corresponds to at least one candidate color channel image, M is a positive integer, and T is a positive integer greater than or equal to L; Furthermore, based on the L candidate spectral channel images, the T candidate color channel images, the rear camera intrinsic parameters, L single-channel pixel intrinsic parameters and L rear single-channel pixel extrinsic parameters, depth calculations are performed respectively to output the second intermediate depth image and the third intermediate depth image; the L single-channel pixel intrinsic parameters are the single-channel pixel intrinsic parameters corresponding to the L candidate spectral channel images among the N single-channel pixel intrinsic parameters, and the L rear single-channel pixel extrinsic parameters are the rear single-channel pixel extrinsic parameters corresponding to the L candidate spectral channel images among the N rear single-channel pixel extrinsic parameters.

28. The device according to claim 26, characterized in that The shooting scene is a low-light scene; the at least one group of camera parameters includes a third group of camera parameters; the third group of camera parameters includes a rear camera internal parameter, a multi-channel pixel internal parameter, and a rear multi-channel pixel external parameter; the multi-channel pixel internal parameter is an internal parameter corresponding to a multi-channel pixel unit of an image sensor of the multi-spectral camera, and each rear multi-channel pixel external parameter is an external parameter of the rear camera relative to the multi-channel pixel unit; The processing module is specifically used to perform depth calculation based on the color image, the multispectral channel image included in the multispectral image, the rear camera intrinsic parameter, the multi-channel pixel intrinsic parameter, and the rear multi-channel pixel extrinsic parameter, and output a fourth intermediate depth image.

29. The device according to claim 28, characterized in that The processing module is specifically configured to determine the fourth intermediate depth image as a depth image.

30. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the depth image acquisition method as described in any one of claims 4 to 22 are implemented.

Images