Image sensor, camera module, electronic device, and method for generating depth image

Abstract

This application discloses an image sensor, a camera module, an electronic device, and a method for generating depth images, belonging to the field of camera technology. The image sensor includes a first signal readout circuit, a first switching element, N third switching elements, and multiple pixel groups arranged in an array, where N is an integer greater than 1. Each pixel group includes N photodiodes and N photosensitive pixels, arranged in an array, with each of the N photosensitive pixels corresponding to one of N spectral patterns. The photosensitive pixels are used to sense light of a first spectral pattern, which is the spectral pattern corresponding to the photosensitive pixel. The N third switching elements are connected to the N photosensitive pixels in a one-to-one correspondence. The first connection terminal of each photosensitive pixel is electrically connected to the input terminal of the first switching element through the third switching element connected to the photosensitive pixel, and the output terminal of the first switching element is electrically connected to the input terminal of the first signal readout circuit.

Description

Technical Field

[0001] This application relates to the field of camera technology, specifically to an image sensor, camera module, electronic device, and method for generating depth images. Background Technology

[0002] In related technologies, narrowband filtered multispectral sensors are sensors capable of acquiring spectral information across multiple narrow bands. The principle involves placing specific narrowband filters on each pixel of the image sensor. These filters can be designed to allow only light within a specific wavelength range to pass through, with each wavelength range corresponding to a spectral channel of the image sensor, thereby achieving the separation and capture of spectral information at different wavelengths. Multispectral sensors can output channel images of different wavelengths. During image processing, when a composite image from two or more channels of the multispectral sensor's output is needed, a specific algorithm is typically required to synthesize the desired composite image. However, errors may be introduced during algorithm processing, leading to lower accuracy in the resulting composite image. Summary of the Invention

[0003] This application provides an image sensor, a camera module, an electronic device, and a method for generating depth images. Since the image sensor can directly output a composite image of two or more channels without needing to perform composite processing on different channels through an algorithm, it avoids errors introduced by the need for algorithm processing, thereby improving the image accuracy of the obtained composite image.

[0004] In a first aspect, an image sensor is provided, including a first signal reading circuit, a first switching element, N third switching elements, and a plurality of pixel groups arranged in an array, wherein N is an integer greater than 1;

[0005] The pixel group includes N photodiodes and N photosensitive pixels. The N photosensitive pixels are arranged in an array and are configured to correspond one-to-one with N spectral patterns. The photosensitive pixels are used to sense light of a first spectral pattern, which is the spectral pattern corresponding to the photosensitive pixel.

[0006] The N third switching devices are connected one-to-one with the N photosensitive pixels;

[0007] The first connection terminal of the photosensitive pixel is electrically connected to the input terminal of the first switch via the third switch connected to the photosensitive pixel, and the output terminal of the first switch is electrically connected to the input terminal of the first signal reading circuit.

[0008] In this embodiment, the pixel group includes N photodiodes and N photosensitive pixels, and the N photosensitive pixels are configured to correspond one-to-one with N spectral patterns. The first connection terminal of the photosensitive pixel is electrically connected to the input terminal of the first switch through a third switch connected to the photosensitive pixel, and the output terminal of the first switch is electrically connected to the input terminal of the first signal reading circuit. Thus, when it is necessary to acquire a composite image of two or more channels from N channels of images captured by N photosensitive pixels, it is only necessary to control the first switch to be turned on, control the third switch connected to the photosensitive pixel corresponding to the channel image to be synthesized to be turned on, and control the third switch connected to the photosensitive pixel corresponding to the channel image that does not need to be synthesized to be turned off. In this way, the light signals collected by the photosensitive pixels of all channels of images to be synthesized can flow to the first signal reading circuit simultaneously. At this time, the first signal reading circuit can directly read the combined signal of the light signals collected by the photosensitive pixels of all channels of images to be synthesized, and output the corresponding composite image based on this. That is, the image sensor can directly output a composite image of two or more channels without the need to synthesize different channel images through an algorithm. Therefore, it avoids the error introduced by the need for algorithm processing, thereby improving the image accuracy of the obtained composite image.

[0009] Optionally, the image sensor further includes a second signal reading circuit, a second switching element, and N fourth switching elements, wherein the N fourth switching elements are connected to the N photosensitive pixels in a one-to-one correspondence.

[0010] The second connection terminal of the photosensitive pixel is electrically connected to the input terminal of the second switch via the fourth switch connected to the photosensitive pixel, and the output terminal of the second switch is electrically connected to the input terminal of the second signal reading circuit.

[0011] In this embodiment, the image sensor further includes a second switch and N fourth switches, with each of the N fourth switches corresponding to one of the N photosensitive pixels. The second connection terminal of each photosensitive pixel is electrically connected to the input terminal of the second switch through the fourth switch connected to the photosensitive pixel, and the output terminal of the second switch is electrically connected to the input terminal of the second signal reading circuit. Thus, during signal reading, the image sensor can synchronously read signals based on the cooperation of the first and second signal reading circuits, thereby improving the signal reading speed.

[0012] Optionally, the photosensitive pixel includes a filter and a photodiode, and the photodiode is disposed opposite to the light-emitting surface of the filter;

[0013] The filter is used to transmit light of the first spectral form and to block light of the second spectral form; wherein, the second spectral form is a spectral form other than the first spectral form;

[0014] The first connection terminal of the photosensitive pixel is the negative electrode of the photosensitive diode included in the photosensitive pixel, and the second connection terminal of the photosensitive pixel is the positive electrode of the photosensitive diode included in the photosensitive pixel.

[0015] In this embodiment, the photosensitive pixel includes a filter and a photodiode, with the light-emitting surfaces of the photodiode and the filter facing each other. The filter transmits light of the first spectral type and blocks light of the second spectral type, wherein the second spectral type is a spectral type other than the first spectral type. Thus, because the filter in each photosensitive pixel can block light of the corresponding second spectral type, the photodiode in each photosensitive pixel can only receive light of the corresponding first spectral type, thereby achieving the purpose of the photosensitive pixel in sensing light of the first spectral type.

[0016] Optionally, the first signal reading circuit includes a first capacitor, a first reset switch, a high-level module, a first source follower, a fifth switch, and a first DC power supply module; the gate of the first source follower is the input terminal of the first signal reading circuit, and the gate of the first source follower is grounded through the first capacitor; the gate of the first source follower is electrically connected to the high-level module through the first reset switch; the source of the first source follower is electrically connected to the high-level module; the drain of the first source follower is grounded through the fifth switch and the first DC power supply module; the drain of the first source follower is the output terminal of the first signal reading circuit.

[0017] The second signal reading circuit includes a second capacitor, a second reset switch, a second source follower, a sixth switch, and a second DC power supply module. The gate of the second source follower is the input terminal of the second signal reading circuit, and the gate of the second source follower is grounded through the second capacitor. The gate of the second source follower is electrically connected to the high-level module through the second reset switch. The source of the second source follower is electrically connected to the high-level module. The drain of the second source follower is grounded through the sixth switch and the second DC power supply module. The drain of the second source follower is the output terminal of the second signal reading circuit.

[0018] In this embodiment, by designing the circuit structure of the image sensor, it is possible to read the signals of each spectral channel individually or to directly read the spectral data of any two or more spectral channels by controlling the aforementioned switching components.

[0019] In a second aspect, a camera module is provided, including a first camera, a second camera, and a 3D imaging device; the first camera is used to acquire RGB images, and the second camera is a multispectral camera; the multispectral camera is the image sensor described in the first aspect.

[0020] In this embodiment, since the camera module includes the image sensor described in the above embodiments, the camera module can implement all the processes of the image sensor in the above embodiments and has the same beneficial effects. To avoid repetition, it will not be described again here.

[0021] Thirdly, an electronic device is provided, including the camera module described in the second aspect.

[0022] In this embodiment, since the electronic device includes the camera module described in the above embodiments, the electronic device can implement each process of the camera module in the above embodiments and has the same beneficial effects. To avoid repetition, it will not be described again here.

[0023] Fourthly, a depth image generation method is provided, executed by the electronic device described in the third aspect, the method comprising:

[0024] A first image is captured by a first camera, a second image is captured by a second camera, and a first depth image is captured by the 3D imaging device; wherein the first image, the second image, and the first depth image are captured at the same time.

[0025] A second depth image is generated based on the first image and the second image;

[0026] A third depth image is generated based on the first depth image and the second depth image.

[0027] In this embodiment, during the depth image generation process, on the one hand, depth information of the area to be acquired is collected using a 3D imaging device to obtain a first depth image; on the other hand, a second depth image is generated based on the image information collected by the first and second cameras. Then, a third depth image is generated based on the first and second depth images. Thus, compared to the first depth image, since the third depth image is obtained by fusing the depth information from the second depth image with the first depth image, it is beneficial to improve the accuracy of the generated third depth image.

[0028] Optionally, generating a second depth image based on the first image and the second image includes:

[0029] Perform binocular stereo vision calculations on each pair of pixels in the first image and the second image to obtain a second depth image;

[0030] Wherein, a pair of pixels includes a first pixel in the first image and a second pixel in the second image, the second pixel being a pixel in the second image at a position corresponding to the first pixel, and the first pixel being any pixel in the first image.

[0031] In this embodiment, a second depth image is obtained by performing binocular stereo vision calculations on each pair of pixels in the first image and the second image, thereby realizing the generation process of the second depth image.

[0032] Optionally, the multispectral camera has N spectral channels, each corresponding to one of N spectral patterns. Each spectral channel transmits light of a first spectral pattern and cuts off light of a second spectral pattern. The first spectral pattern is the spectral pattern corresponding to the spectral channel, and the second spectral pattern is any spectral pattern other than the first spectral pattern. N is an integer greater than 1. Generating a second depth image based on the first image and the second image includes:

[0033] Based on the color channels, the first image is separated into K channel images; and based on the spectral channels, the second image is separated into N channel images corresponding one-to-one with the N spectral channels, where K is an integer greater than 1.

[0034] A third image is generated based on M channel images out of the N channel images; wherein the M channel images include a first channel image, the first channel image is the channel image with the most edge information among the N channel images, M is less than N, and M is an integer greater than or equal to 1;

[0035] A second depth image is generated based on the fourth image and the third image; wherein the fourth image is the channel image with the smallest spectral morphological distance between the wavelength and the first channel image among the K channel images.

[0036] In this embodiment, the first image is separated into K channel images based on the color channels, and the second image is separated into N channel images corresponding one-to-one with the N spectral channels. Then, a third image is generated based on M channel images from the N channel images. Since the M channel images include the first channel image, which contains the most edge information among the N channel images, the first channel image is the multispectral dominant channel among the N channel images. The fourth image is the channel image among the K channel images whose wavelength is closest to the spectral shape corresponding to the first channel image, i.e., the fourth image is the channel image whose spectral shape is closest to the multispectral dominant channel among the N channel images. Thus, generating the second depth image based on the fourth and third images can eliminate the influence of edge information from some disadvantageous scenes in the first and second image information, such as low light and monochromatic scenes, thereby improving the accuracy of the generated second depth image.

[0037] Optionally, generating a third image based on M channels of the N channel images includes:

[0038] Edge detection is performed on the N channel images respectively to obtain N edge detection information;

[0039] The channel image corresponding to the first edge detection information is determined as the first channel image; wherein, the first edge detection information is the edge detection information with the most edge information among the N edge detection information;

[0040] The second channel image and the first channel image from the N channel images are merged to obtain a third image; wherein the spectral morphology corresponding to the second channel image is adjacent to the spectral morphology corresponding to the first channel image; the N channel images include the first channel image and the second channel image.

[0041] In this embodiment, edge detection is performed on the N channel images respectively to obtain N edge detection information; the channel image corresponding to the first edge detection information is determined as the first channel image; wherein, the first edge detection information is the edge detection information with the most edge information among the N edge detection information; and the second channel image and the first channel image from the N channel images are merged to obtain a third image; wherein, the spectral morphology corresponding to the second channel image is adjacent to the spectral morphology corresponding to the first channel image; the N channel images include the first channel image and the second channel image. Since the spectral morphology of the second channel image is close to that of the first channel image, the bands of the generated third channel image can be enhanced by the second channel image, so that the generated third image has a higher signal-to-noise ratio and image quality.

[0042] Optionally, generating the second depth image based on the fourth image and the third image includes:

[0043] Perform binocular stereo vision calculations on each pair of pixels in the third and fourth images to obtain a second depth image;

[0044] Wherein, a pair of pixels includes the third pixel in the third image and the fourth pixel in the fourth image, the fourth pixel being the pixel in the fourth image that corresponds to the third pixel, and the third pixel being any pixel in the third image.

[0045] In this embodiment, a second depth image is obtained by performing binocular stereo vision calculations on each pair of pixels in the third and fourth images, thereby realizing the generation process of the second depth image.

[0046] Optionally, generating a third depth image based on the first depth image and the second depth image includes:

[0047] The first depth image, the second depth image, the first image, and the second image are input into a multimodal depth map neural network to obtain a third depth image output by the multimodal depth map neural network.

[0048] In this embodiment, by inputting the first depth image, the second depth image, the first image, and the second image into a multimodal depth map neural network, a third depth image is obtained output by the multimodal depth map neural network. Since "the third depth image integrates the depth features in the first depth image, the depth features in the second depth image, the depth features in the first image, and the depth features in the second image", the third depth image not only has a highly reliable depth source, but also incorporates deep learning to enhance robustness in multiple scenarios. It is an optimized depth image with extremely high accuracy, thus further improving the accuracy of the generated third depth image.

[0049] Fifthly, an electronic device is provided, including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, the program or instructions, when executed by the processor, implement the steps of the depth image generation method described in the fourth aspect.

[0050] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the fourth aspect.

[0051] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method described in the fourth aspect.

[0052] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method described in the fourth aspect.

[0053] In this embodiment, the pixel group includes N photodiodes and N photosensitive pixels, and the N photosensitive pixels are configured to correspond one-to-one with N spectral patterns. The first connection terminal of the photosensitive pixel is electrically connected to the input terminal of the first switch through a third switch connected to the photosensitive pixel, and the output terminal of the first switch is electrically connected to the input terminal of the first signal reading circuit. Thus, when it is necessary to acquire a composite image of two or more channels from N channels of images captured by N photosensitive pixels, it is only necessary to control the first switch to be turned on, control the third switch connected to the photosensitive pixel corresponding to the channel image to be synthesized to be turned on, and control the third switch connected to the photosensitive pixel corresponding to the channel image that does not need to be synthesized to be turned off. In this way, the light signals collected by the photosensitive pixels of all channels of images to be synthesized can flow to the first signal reading circuit simultaneously. At this time, the first signal reading circuit can directly read the combined signal of the light signals collected by the photosensitive pixels of all channels of images to be synthesized, and output the corresponding composite image based on this. That is, the image sensor can directly output a composite image of two or more channels without the need to synthesize different channel images through an algorithm. Therefore, it avoids the error introduced by the need for algorithm processing, thereby improving the image accuracy of the obtained composite image. Attached Figure Description

[0054] Figure 1 This is a schematic flowchart of a method for generating a depth image provided by some embodiments of this application;

[0055] Figure 2A This is a schematic diagram of the distribution of filters in a pixel group provided in some embodiments of this application;

[0056] Figure 2B This is a schematic diagram showing the distribution of photodiodes in a pixel group in some embodiments of this application;

[0057] Figure 3 This is a schematic diagram of pixel distribution in an image sensor provided in some embodiments of this application;

[0058] Figure 4 This is a schematic diagram of the circuit structure of the image sensor provided in some embodiments of this application;

[0059] Figure 5 These are schematic diagrams illustrating depth calculation based on binocular stereo vision methods provided in some embodiments of this application;

[0060] Figure 6 This is a schematic flowchart of a method for generating a depth image provided by some embodiments of this application;

[0061] Figure 7Schematic diagrams of the structure of electronic devices provided for some embodiments of this application;

[0062] Figure 8 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0064] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, the scope of protection for "A or B" covers at least three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. In addition, the terms "A and / or B," "at least one of A and B," and "at least one of A or B" also cover at least the above three scenarios. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0065] The following description, in conjunction with the accompanying drawings, details the method for generating depth images, electronic devices, media, and program products provided in this application through some embodiments and application scenarios.

[0066] To facilitate understanding, some terms in the embodiments of this application will be explained first:

[0067] 3D imaging refers to systems that use optical, electronic, and computer technologies to scan, reconstruct, and present three-dimensional objects. The main active 3D imaging technologies include binocular stereo vision, monocular structured light, and direct time-of-flight (DTOF) technology.

[0068] Binocular stereo vision: Binocular stereo vision is an important form of machine vision. It is a method based on the principle of parallax, acquiring three-dimensional geometric information of an object from multiple images. A typical binocular stereo vision system uses two cameras to simultaneously acquire two digital images of the object from different angles, or a single camera to acquire two digital images of the object from different angles at different times. Based on the principle of parallax, the system recovers the object's three-dimensional geometric information, reconstructing its three-dimensional contour and position. Binocular stereo vision systems have broad application prospects in the field of machine vision.

[0069] Multispectral sensors: Multispectral imaging sensors are sensors capable of simultaneously capturing spectral information across multiple wavelength ranges. Their principle involves using optical filters or beam-splitting elements to separate and capture light from different wavelength ranges, thereby enabling the acquisition and analysis of spectral information across multiple wavelengths. Traditional color cameras use three sensors to capture light in the red, green, and blue wavelength ranges respectively, while multispectral imaging sensors can capture a much wider wavelength range, typically including visible, infrared, and ultraviolet spectra. These sensors can use multiple narrowband filters or beam-splitting elements, such as prisms or gratings, to separate light of different wavelengths and then transmit them separately to corresponding pixel arrays. By simultaneously capturing spectral information across multiple wavelength ranges, multispectral imaging sensors can provide richer spectral information, thus enabling applications in many fields, such as agriculture, environmental monitoring, and geological exploration.

[0070] Please see Figure 4 , Figure 4 An image sensor provided in this application embodiment includes a first signal reading circuit 270, a first switching element 230, N third switching elements 250, and a plurality of pixel groups arranged in an array, wherein N is an integer greater than 1;

[0071] The pixel group includes N photodiodes 220 and N photosensitive pixels. The N photosensitive pixels are arranged in an array and are configured to correspond one-to-one with N spectral patterns. The photosensitive pixels are used to sense light of a first spectral pattern, which is the spectral pattern corresponding to the photosensitive pixel.

[0072] The N third switching elements 250 are connected one-to-one with the N photosensitive pixels;

[0073] The first connection terminal of the photosensitive pixel is electrically connected to the input terminal of the first switch 230 through the third switch 250 connected to the photosensitive pixel, and the output terminal of the first switch 230 is electrically connected to the input terminal of the first signal reading circuit 270.

[0074] It is understood that, since the N photosensitive pixels are configured to correspond one-to-one with the N spectral forms, the photosensitive pixel used to sense the light of the first spectral form can mean that the photosensitive pixel is used to sense the light of the first spectral form corresponding to the photosensitive pixel, wherein different photosensitive pixels correspond to different first spectral forms, and the first spectral form is the spectral form corresponding to the photosensitive pixel.

[0075] The spectral morphology may include a wavelength range. The photosensitive pixel is used to sense light within the wavelength range of the first spectral morphology corresponding to it. This can mean that the photosensitive pixel is used to sense light within the wavelength range included in the first spectral morphology. The spectral morphology can be either narrowband or broadband.

[0076] In some embodiments of this application, both the first switch 230 and the third switch 250 can be transistors or MOSFETs. The image sensor may also include a timing controller, with the control terminals of the first switch 230 and the third switch 250 electrically connected to the timing controller, thereby allowing the timing controller to control the on / off state of each switch.

[0077] In some embodiments of this application, the photosensitive pixel may include a filter 210 and a photodiode 220, with the photodiode 220 and the light-emitting surface of the filter 210 positioned opposite each other. The value of N can be set as needed; for example, N can be 4, 9, 16, etc. Please refer to [link to relevant documentation]. Figure 2A When N equals 9, this is a schematic diagram showing the distribution of the 9 filters 210 in a pixel group, where C1 to C9 represent the 9 spectral channels corresponding to the 9 filters 210. Figure 2B This is a schematic diagram showing the distribution of nine photodiodes 220 in a pixel group, where PD1 to PD9 represent the nine photodiodes 220 respectively. Please refer to [link / reference]. Figure 3 This is a schematic diagram of all pixel groups in an image sensor, where... Figure 3 The row direction has 4080 pixel groups, and the column direction has 3060 pixel values. Since 4080 * 3060 = 12,484,800, the image sensor is approximately a 12-megapixel image sensor. It can be seen that the entire image sensor target surface is formed by cyclically using 9 multispectral pixels as one pixel group. Please see... Figure 2A In some embodiments of this application, the value of N is 9, and the wavelength ranges included in the spectral morphologies corresponding to the 9 filters 210 are shown in Table 1 below:

[0078] Table 1

[0079]

[0080] In Table 1 above, numbers 1 to 9 represent nine filters, center wavelength (FWHM) represents the full width at half maximum (FWHM), and the spectral characteristics of the filters are as follows: filter number 1 corresponds to a spectral range of 400nm to 420nm; filter number 2 corresponds to a spectral range of 426nm to 454nm; filter number 3 corresponds to a spectral range of 456nm to 484nm; filter number 4 corresponds to a spectral range of 486nm to 514nm; filter number 5 corresponds to a spectral range of 516nm to 544nm; filter number 6 corresponds to a spectral range of 546nm to 574nm; filter number 7 corresponds to a spectral range of 576nm to 604nm; filter number 8 corresponds to a spectral range of 606nm to 634nm; and filter number 9 corresponds to a spectral range of 920nm to 960nm.

[0081] Please see Figure 4 In some embodiments of this application, if the signal of the photosensitive pixel to which PD11 belongs is needed, the third switch TG11_n can be turned on, the first switch 230 can be turned on, and the other third switches 250 can be turned off respectively. Only the light signal collected by the third switch TG11_n can flow to the first signal reading circuit 270. Therefore, the light signal read by the first signal reading circuit 270 is the light signal of the photosensitive pixel to which PD11 belongs, and the channel image of the photosensitive pixel to which PD11 belongs can be output. Similarly, when it is necessary to read the signal of other photosensitive pixels individually, it is only necessary to turn on the third switch TG11_n of the photosensitive pixel to be read, control the first switch 230, and control the other third switches 250 to be turned off respectively, so that the signal of each photosensitive pixel can be read individually.

[0082] Correspondingly, when it is necessary to read the combined signal of two or more photosensitive pixels to output a composite image of two or more channels, it is possible to control only the first switch 230 to be turned on, and control the third switch 250 connected to the photosensitive pixel corresponding to the channel image to be synthesized to be turned on, and control the third switch 250 connected to the photosensitive pixel corresponding to the channel image that does not need to be synthesized to be turned off. In this way, the light signals collected by the photosensitive pixels of all the channel images to be synthesized can flow to the first signal reading circuit 270 at the same time. At this time, the first signal reading circuit 270 can directly read the combined signal of the light signals collected by the photosensitive pixels of all the channel images to be synthesized, and output the corresponding composite image based on this. For example, when it is necessary to read the combined signal of PD11 and PD12, it is only necessary to control the third switch TG11_n and the third switch TG12_n to be turned on, control the first switch 230 to be turned on, and control the other third switches 250 to be turned off respectively. In this way, the optical signals collected by PD11 and PD12 can flow to the first signal reading circuit 270 at the same time. At this time, the first signal reading circuit 270 can directly read the combined signal of the optical signals collected by PD11 and PD12, and output the corresponding composite image based on it.

[0083] As can be seen, by controlling the aforementioned switching devices, it is possible to read the signals of each spectral channel individually, or to directly read the combined spectral data of any two or more spectral channels.

[0084] In this embodiment, the pixel group includes N photodiodes 220 and N photosensitive pixels, and the N photosensitive pixels are configured to correspond one-to-one with N spectral patterns. The first connection terminal of the photosensitive pixel is electrically connected to the input terminal of the first switch 230 through the third switch 250 connected to the photosensitive pixel. The output terminal of the first switch 230 is electrically connected to the input terminal of the first signal reading circuit 270. Thus, when it is necessary to acquire a composite image of two or more channels from N channels of images acquired by N photosensitive pixels, it is only necessary to control the first switch 230 to be turned on, control the third switch 250 connected to the photosensitive pixel corresponding to the channel image to be synthesized to be turned on, and control the third switch 250 connected to the photosensitive pixel corresponding to the channel image that does not need to be synthesized to be turned off. In this way, the light signals acquired by the photosensitive pixels of all the channel images to be synthesized can flow to the first signal reading circuit 270 simultaneously. At this time, the first signal reading circuit 270 can directly read the combined signal of the light signals acquired by the photosensitive pixels of all the channel images to be synthesized, and output the corresponding composite image based on this. That is, the image sensor can directly output a composite image of two or more channels without the need to synthesize different channel images through an algorithm. Therefore, it avoids the error introduced by the need for algorithm processing, thereby improving the image accuracy of the obtained composite image.

[0085] Optionally, the image sensor further includes a second switch 240 and N fourth switches 260, and the N fourth switches 260 are connected to the N photosensitive pixels in a one-to-one correspondence.

[0086] The second connection terminal of the photosensitive pixel is electrically connected to the input terminal of the second switch 240 through the fourth switch 260 connected to the photosensitive pixel, and the output terminal of the second switch 240 is electrically connected to the input terminal of the second signal reading circuit 280.

[0087] In some embodiments of this application, the first switch 230, the second switch 240, the third switch 250, and the fourth switch 260 can all be transistors or MOSFETs. The image sensor may also include a timing controller, with the control terminals of the first switch 230, the second switch 240, the third switch 250, and the fourth switch 260 electrically connected to the timing controller. Thus, the on / off state of each switch can be controlled by the timing controller.

[0088] It is understandable that, during signal reading, the image sensor can utilize the combined operation of the first signal reading circuit 270 and the second signal reading circuit 280 to simultaneously read signals, thereby improving the signal reading speed. For example, please refer to... Figure 4If you need to output the signal of PD11, the combined spectral data of PD23 and PD33, and the combined spectral data of PD22 and PD11, you can perform the following steps in sequence:

[0089] First, the first reset switch 272 is turned on to reset the image sensor. Then, the first switch 230, the second switch 240, N third switches 250, and N fourth switches 260 are turned off. The entire image sensor is exposed. After exposure, TG11_n and TG1 can be turned on, allowing the PD11 signal to flow into the first capacitor 271. Then, the fifth switch 275 in the first signal reading circuit 270 can be turned on, enabling the first signal reading circuit 270 to read the signal in the first capacitor 271, and thus output the PD11 signal through the first output circuit 290.

[0090] Then, keeping TG23_p and TG33_p disconnected, all other third switches 250 except TG23_p and TG33_p are turned on. Then, the second switch 240 is turned on, so the signals of PD23 and PD33 flow into the second capacitor 281. Then, the sixth switch 284 in the second signal reading circuit 280 can be turned on, so that the second signal reading circuit 280 can read the signal in the second capacitor 281, and thus the spectral combined data of PD23 and PD33 can be output through the second output circuit 2100.

[0091] Finally, TG22_n can be turned on, and the information of PD22 flows into the first capacitor 271. Combined with the previous signal of PD11, the combined spectral data of PD11 and PD22 can be read.

[0092] In this embodiment, the image sensor further includes a second switch 240 and N fourth switches 260, with each of the N fourth switches 260 corresponding to one of the N photosensitive pixels. The second connection terminal of each photosensitive pixel is electrically connected to the input terminal of the second switch 240 via the fourth switch 260 connected to the photosensitive pixel, and the output terminal of the second switch 240 is electrically connected to the input terminal of the second signal reading circuit 280. Thus, during signal reading, the image sensor can synchronously read signals based on the cooperation of the first signal reading circuit 270 and the second signal reading circuit 280, thereby improving the signal reading speed.

[0093] Optionally, the photosensitive pixel includes a filter 210 and a photosensitive diode 220, and the photosensitive diode 220 is disposed opposite to the light-emitting surface of the filter 210;

[0094] The filter 210 is used to transmit light of the first spectral form and to block light of the second spectral form; wherein, the second spectral form is a spectral form other than the first spectral form;

[0095] The first connection terminal of the photosensitive pixel is the negative terminal of the photosensitive diode 220 included in the photosensitive pixel, and the second connection terminal of the photosensitive pixel is the positive terminal of the photosensitive diode 220 included in the photosensitive pixel.

[0096] The filter 210, which transmits light of the first spectral form and blocks light of the second spectral form, means that the filter 210 transmits light of the first spectral form and blocks light of the corresponding second spectral form. As shown in Table 1 and Figure 2 of the specification, different filters 210 correspond to different first and second spectral forms. For example, C1 corresponds to a first spectral form of 400nm to 420nm, and its second spectral form is any other spectral form outside of 400nm to 420nm. Similarly, C3 corresponds to a first spectral form of 456nm to 484nm, and its second spectral form is any other spectral form outside of 456nm to 484nm.

[0097] It is understood that the filter 210 is used to transmit light of the first spectral type in the following way: light of the first spectral type can penetrate the filter 210 and enter the photodiode 220 below the filter 210. Correspondingly, the filter 210 is used to block light of the second spectral type in the following way: light of the second spectral type cannot penetrate the filter 210, that is, light of the second spectral type is blocked by the filter 210 and therefore cannot enter the photodiode 220 below the filter 210. Thus, since the filter 210 in each photosensitive pixel can block the corresponding second spectral type of light, the photodiode 220 in each photosensitive pixel can only receive the corresponding first spectral type of light, thereby achieving the purpose of the photosensitive pixel in sensing light of the first spectral type.

[0098] It should be noted that the one-to-one connection of the aforementioned N third switches 250 with the N photosensitive pixels means that the N third switches 250 are connected one-to-one with the N photosensitive diodes 220 included in the N photosensitive pixels. Correspondingly, the one-to-one connection of the N fourth switches 260 with the N photosensitive pixels means that the N fourth switches 260 are connected one-to-one with the N photosensitive diodes 220 included in the N photosensitive pixels. Specifically, the third switches 250 are electrically connected to the negative terminal of the photosensitive diode 220 in the corresponding photosensitive pixel, and the fourth switches 260 are electrically connected to the positive terminal of the photosensitive diode 220 in the corresponding photosensitive pixel.

[0099] In this embodiment, the photosensitive pixel includes a filter 210 and a photodiode 220, with the photodiode 220 and the light-emitting surface of the filter 210 facing each other. The filter 210 transmits light of the first spectral type and blocks light of the second spectral type, wherein the second spectral type is a spectral type other than the first spectral type. Thus, since the filter 210 in each photosensitive pixel can block the light of the corresponding second spectral type, the photodiode 220 in each photosensitive pixel can only receive light of the corresponding first spectral type, thereby achieving the purpose of the photosensitive pixel in sensing light of the first spectral type.

[0100] Optionally, the first signal reading circuit 270 includes a first capacitor 271, a first reset switch 272, a high-level module 273, a first source follower 274, a fifth switch 275, and a first DC power supply module 276; the gate of the first source follower 274 is the input terminal of the first signal reading circuit 270, and the gate of the first source follower 274 is grounded through the first capacitor 271; the gate of the first source follower 274 is electrically connected to the high-level module 273 through the first reset switch 272; the source of the first source follower 274 is electrically connected to the high-level module 273; the drain of the first source follower 274 is grounded through the fifth switch 275 and the first DC power supply module 276; the drain of the first source follower 274 is the output terminal of the first signal reading circuit 270.

[0101] The second signal reading circuit 280 includes a second capacitor 281, a second reset switch 282, a second source follower 283, a sixth switch 284, and a second DC power supply module 285. The gate of the second source follower 283 is the input terminal of the second signal reading circuit 280, and the gate of the second source follower 283 is grounded through the second capacitor 281. The gate of the second source follower 283 is electrically connected to the high-level module 273 through the second reset switch 282. The source of the second source follower 283 is electrically connected to the high-level module 273. The drain of the second source follower 283 is grounded through the sixth switch 284 and the second DC power supply module 285. The drain of the second source follower 283 is the output terminal of the second signal reading circuit 280.

[0102] The first reset switch 272, the fifth reset switch 275, the second reset switch 282, and the sixth reset switch 284 can all be transistors or MOSFETs. Furthermore, the control terminals of the first reset switch 272, the fifth reset switch 275, the second reset switch 282, and the sixth reset switch 284 are all electrically connected to the timing controller. Thus, the on / off state of each switch can be controlled by the timing controller.

[0103] Please see further. Figure 4 The image sensor further includes a first output circuit 290 and a second output circuit 2100. The first output circuit 290 includes an ADC, an ISP, and a MIPI connected in sequence. The input terminal of the ADC is the input terminal of the first output circuit 290, and the output terminal of the MIPI is the output terminal of the first output circuit 290. The output terminal of the first signal readout circuit 270 is electrically connected to the input terminal of the first output circuit 290. Thus, the signal readout structure of the first signal readout circuit 270 can be output through the first output circuit. Correspondingly, the second output circuit 2100 includes an ADC, an ISP, and a MIPI connected in sequence. The input terminal of the ADC is the input terminal of the second output circuit 2100, and the output terminal of the MIPI is the output terminal of the second output circuit 2100. The output terminal of the second signal readout circuit 280 is electrically connected to the input terminal of the second output circuit 2100. Thus, the signal readout structure of the second signal readout circuit 280 can be output through the first output circuit.

[0104] In this embodiment, by designing the circuit structure of the image sensor, it is possible to read the signals of each spectral channel individually or to directly read the spectral data of any two or more spectral channels by controlling the aforementioned switching components.

[0105] This application also provides a camera module, including a first camera, a second camera, and a 3D imaging device; the first camera is used to acquire RGB images, and the second camera is a multispectral camera; the multispectral camera is the image sensor described in the above embodiments.

[0106] In this embodiment, since the camera module includes the image sensor described in the above embodiments, the camera module can implement all the processes of the image sensor in the above embodiments and has the same beneficial effects. To avoid repetition, it will not be described again here.

[0107] This application also provides an electronic device, including the camera module described in the above embodiments.

[0108] In this embodiment, since the electronic device includes the camera module described in the above embodiments, the electronic device can implement each process of the camera module in the above embodiments and has the same beneficial effects. To avoid repetition, it will not be described again here.

[0109] In related technologies, some electronic devices include 3D imaging devices. These devices utilize optical, electronic, and computer technologies to scan, reconstruct, and render 3D objects, obtaining depth maps. However, the accuracy of the depth maps output by 3D imaging devices is typically positively correlated with the cost and power consumption of the device during operation. To balance cost and power consumption, electronic devices in related technologies often employ relatively low-precision 3D imaging devices, resulting in low-precision depth maps output by the electronic devices. Therefore, this application also provides a method for generating depth images, which is executed by the electronic device described in the above embodiments. Please refer to [link to relevant documentation]. Figure 1 The method for generating the depth image includes the following steps:

[0110] Step 101: Acquire a first image through the first camera, acquire a second image through the second camera, and acquire a first depth image through the 3D imaging device; wherein the acquisition time of the first image, the second image, and the first depth image is the same.

[0111] It is understood that the first image, the second image, and the first depth image mentioned above are images obtained by electronic devices based on the first camera, the second camera, and the 3D imaging device simultaneously capturing images of the same area or the same object.

[0112] The aforementioned first camera can be a camera device capable of outputting images, such as a main camera or a telephoto camera in an electronic device.

[0113] The aforementioned second camera may include a multispectral sensor, which can be used to realize basic functions of electronic devices such as color temperature prediction, local auto white balance (Local AWB), and color reproduction.

[0114] The aforementioned 3D imaging device can be any type of 3D imaging device in electronic devices. For example, a 3D imaging device can be a DTOF (Time-of-Flight) system, which can generate a depth map based on the time-of-flight principle. Another example is an active binocular system, which can generate a depth map based on the binocular parallax principle.

[0115] It should be noted that, in some embodiments of this application, before performing step 101 above, the intrinsic and extrinsic parameters of the first camera, second camera, and 3D imaging device in the electronic device required for 3D imaging and multi-camera alignment can be calibrated, and the calibration data can be stored in a designated location in the electronic device. The intrinsic and extrinsic parameter calibration data generally includes:

[0116] The images from the first camera, the second camera, and each channel of the second camera are calibrated using intrinsic parameters, and the intrinsic parameter data is stored in a designated location in the phone's memory.

[0117] If the RGB camera is used as the mapping target for image alignment, the external parameters of the RGB first camera and second camera, each channel of the RGB first camera and second camera, the RGB first camera and the 3D imaging device are calibrated respectively, and the external parameter data is stored in a specified location in the memory of the electronic device.

[0118] The 3D camera system calculates the necessary internal and external parameters and stores the calibration data in a designated location in the electronic device's memory.

[0119] Specifically, the intrinsic parameters of a camera typically include the following parameters:

[0120] Focal Length: Focal length refers to the focal length of a camera lens, which determines the scaling ratio of the camera image.

[0121] Principal Point: The principal point is the intersection of the imaging plane and the center of the lens, and it represents the origin of the image coordinate system.

[0122] Distortion parameters: Distortion parameters describe the radial and tangential distortions that may exist in the camera lens.

[0123] The extrinsic parameters of a camera typically include the following parameters:

[0124] Rotation matrix: describes the rotation relationship between the camera coordinate system and the world coordinate system.

[0125] Translation Vector: Describes the translation relationship between the camera coordinate system and the world coordinate system.

[0126] The meanings of these parameters are as follows:

[0127] Internal reference data describes the imaging characteristics of the camera itself, including information such as the camera's optical characteristics and imaging distortion.

[0128] The extrinsic data describes the camera's position and orientation, that is, the camera's spatial position and attitude relative to the world coordinate system.

[0129] Step 102: Generate a second depth image based on the first image and the second image.

[0130] The first and second images are obtained by capturing images of the same area or object from two different cameras within the electronic device. Since the first and second cameras are located at different positions within the electronic device, a parallax exists between them. Therefore, using a binocular stereo vision calculation method, the depth of each pixel in the first image can be calculated based on the parallax between a pair of corresponding pixels in the first and second images, thus obtaining the second depth map. The depth of a pixel in the first image refers to the distance between that pixel and the electronic device when the electronic device is capturing the first image, the second image, and the first depth image.

[0131] Step 103: Generate a third depth image based on the first depth image and the second depth image.

[0132] The first depth image and the second depth image can be fused using various image fusion methods to obtain the third depth image. The third depth image may include depth feature information from both the first and second depth images. Therefore, compared to simply acquiring and outputting the first depth image using a 3D imaging device, the third depth image incorporates depth information from the second depth image, thus including more depth information than the first depth image output by the 3D imaging device, thereby improving the accuracy of the generated third depth image.

[0133] In this embodiment, during the depth image generation process, on the one hand, depth information of the area to be acquired is collected using a 3D imaging device to obtain a first depth image; on the other hand, a second depth image is generated based on the image information collected by the first and second cameras. Then, a third depth image is generated based on the first and second depth images. Thus, compared to the first depth image, since the third depth image is obtained by fusing the depth information from the second depth image with the first depth image, it is beneficial to improve the accuracy of the generated third depth image.

[0134] Optionally, generating a second depth image based on the first image and the second image includes:

[0135] Perform binocular stereo vision calculations on each pair of pixels in the first image and the second image to obtain a second depth image;

[0136] Wherein, a pair of pixels includes a first pixel in the first image and a second pixel in the second image, the second pixel being a pixel in the second image at a position corresponding to the first pixel, and the first pixel being any pixel in the first image.

[0137] Binocular stereo vision utilizes the parallax generated by two cameras observing the same object from different positions to calculate the difference in object distance, thereby generating stereo depth perception. Specifically, for the same point on an object, after calculating the parallax between the imaging coordinates on the first and second cameras, and combining this with the pre-stored intrinsic and extrinsic parameters between the two cameras, the distance of that point on the object from the camera can be obtained.

[0138] Please see Figure 5 In some embodiments of this application, the depth of any pixel P in the area to be acquired can be calculated based on the following formula, wherein the following formula (1) is the formula for depth calculation based on the principle of binocular stereo vision:

[0139] (1)

[0140] The above formula (1) can also be converted into the following formula (2):

[0141] (2)

[0142] Where Z represents the depth of pixel P, and b represents the optical center O of the first camera. R With the optical center O of the second camera T The distance between them, where f represents the equivalent focal length and d represents the parallax, i.e., d equals X. R -X TX R This represents the coordinates of the imaging point p' of the first camera, X. T These are the coordinates of the imaging point p´´ of the second camera. For example, please see... Figure 5 In some embodiments of this application, when the value of f is 4mm, the value of d is 7mm, and the value of b is 14mm, substituting the values ​​of f, d, and b into the above formula (2) yields a depth of 8mm for pixel P. For example, in some embodiments of this application, when the value of f is 3.5mm, the value of d is 7mm, and the value of b is 12mm, substituting the values ​​of f, d, and b into the above formula (2) yields a depth of 6mm for pixel P.

[0143] It is understandable that once the depth of each pixel in the first image is calculated according to the above formula, the second depth image can be modeled based on all the pixels. The second depth image can express the depth of each pixel through visual depth.

[0144] In this embodiment, a second depth image is obtained by performing binocular stereo vision calculations on each pair of pixels in the first image and the second image, thereby realizing the generation process of the second depth image.

[0145] Optionally, the multispectral camera has N spectral channels, each corresponding to one of N spectral patterns. Each spectral channel transmits light of a first spectral pattern and cuts off light of a second spectral pattern. The first spectral pattern is the spectral pattern corresponding to the spectral channel, and the second spectral pattern is any spectral pattern other than the first spectral pattern. N is an integer greater than 1. Generating a second depth image based on the first image and the second image includes:

[0146] Based on the color channels, the first image is separated into K channel images; and based on the spectral channels, the second image is separated into N channel images corresponding one-to-one with the N spectral channels, where K is an integer greater than 1.

[0147] A third image is generated based on M channel images out of the N channel images; wherein the M channel images include a first channel image, the first channel image is the channel image with the most edge information among the N channel images, M is less than N, and M is an integer greater than or equal to 1;

[0148] A second depth image is generated based on the fourth image and the third image; wherein the fourth image is the channel image with the smallest spectral morphological distance between the wavelength and the first channel image among the K channel images.

[0149] In some embodiments of this application, the second camera includes the image sensor described in the above embodiments. The image sensor includes a plurality of pixel groups arranged in an array. Each pixel group includes N filters 210 and N photodiodes 220. The N filters 210 are arranged in an array, and each of the N filters 210 corresponds one-to-one with the N photodiodes 220. The light-emitting surfaces of the photodiodes 220 and their corresponding filters 210 are opposite each other. Each of the N filters 210 corresponds one-to-one with N spectral patterns, and the filters 210 are used to filter light signals of other spectral patterns besides their corresponding spectral patterns. Please refer to [link to relevant documentation]. Figure 2A This is a schematic diagram of the distribution of 9 filters 210 in a pixel group, where C1 to C9 represent the 9 spectral channels corresponding to the 9 filters 210. Figure 2B This is a schematic diagram showing the distribution of nine photodiodes 220 in a pixel group, where PD1 to PD9 represent the nine photodiodes 220 respectively. Please refer to [link / reference]. Figure 3 This is a schematic diagram of all pixel groups in an image sensor, where... Figure 3 The value 4080*3060=12484800 indicates that the image sensor is approximately 12 megapixels. It can be seen that the entire image sensor target surface is formed by cyclically grouping nine multispectral pixels into one pixel group. Please refer to Figure 2. In some embodiments of this application, the value of N is 9, and the spectral morphology corresponding to the nine filters 210 is shown in Table 1 above.

[0150] The first camera typically outputs a color image, which is usually composed of color information from multiple color channels. The value of K represents the number of color channels included in the first camera. For example, when the first camera is an RGB camera, it includes three color channels: R, G, and B, and correspondingly, K is 3. The above-mentioned decomposition of the first image according to color channels yields K channel images; that is, decomposing the first image according to the three color channels R, G, and B yields the R channel image, the G channel image, and the B channel image.

[0151] The second image described above can be image information composed of N channel images. Therefore, the second image can be decomposed to obtain the N channel images. Then, edge detection can be performed on each channel image to determine the first channel image among the N channel images.

[0152] The value of M can be equal to 1 or greater than 1, and can be set as needed. When the value of M is equal to 1, the M channel images are the first channel image mentioned above. When M is greater than 1, the M channel images may include the first channel image and channel images whose spectral morphology is similar to the first channel image. Alternatively, when M is greater than 1, the M channel images may include the first M channel images with more edge information among the N channel images.

[0153] The above-mentioned generation of a third image based on M channels of the N channel images can refer to directly merging the M channel images to obtain the image information. Alternatively, the image sensor can be controlled to merge the M spectral channels corresponding one-to-one with the M channel images, and the image sensor can be used to re-acquire the area to be acquired to obtain the third image. The specific settings can be configured as needed. It should be noted that it can be based on... Figure 4 The circuit structure in the image sensor is used to control the merging of M spectral channels.

[0154] The above-mentioned generation of the second depth image based on the fourth image and the third image can refer to: generating the second depth image based on the principle of binocular stereo vision using the fourth image and the third image.

[0155] Since the aforementioned image sensor only needs to control the aforementioned switching components, it can individually read the signals of each spectral channel, or directly read the spectral data combined from any two or more spectral channels. Therefore, the combined signals of M spectral channels can be directly read using the aforementioned image sensor to achieve the process of "merging M spectral channels in the image sensor". It should be noted that, compared to merging the M channel images to obtain a third image, directly reading the third image based on the image sensor's merging of the M spectral channels is more accurate because the third image can be read directly, i.e., there is no need to decompose and merge the image information acquired by the image sensor, thereby reducing errors introduced during intermediate processing.

[0156] In this embodiment, the first image is separated into K channel images based on the color channels, and the second image is separated into N channel images corresponding one-to-one with the N spectral channels. Then, a third image is generated based on M channel images from the N channel images. Since the M channel images include the first channel image, which contains the most edge information among the N channel images, the first channel image is the multispectral dominant channel among the N channel images. The fourth image is the channel image among the K channel images whose wavelength is closest to the spectral shape corresponding to the first channel image, i.e., the fourth image is the channel image whose spectral shape is closest to the multispectral dominant channel among the N channel images. Thus, generating the second depth image based on the fourth and third images can eliminate the influence of edge information from some disadvantageous scenes in the first and second image information, such as low light and monochromatic scenes, thereby improving the accuracy of the generated second depth image.

[0157] Optionally, generating a third image based on M channels of the N channel images includes:

[0158] Edge detection is performed on the N channel images respectively to obtain N edge detection information;

[0159] The channel image corresponding to the first edge detection information is determined as the first channel image; wherein, the first edge detection information is the edge detection information with the most edge information among the N edge detection information;

[0160] The second channel image and the first channel image from the N channel images are merged to obtain a third image; wherein the spectral morphology corresponding to the second channel image is adjacent to the spectral morphology corresponding to the first channel image; the N channel images include the first channel image and the second channel image.

[0161] The above-mentioned edge detection of the N channel images to obtain N edge detection information can refer to: performing edge detection on the N channel images based on an edge detection algorithm to obtain N edge detection information, wherein the edge detection algorithm can be various types of edge detection algorithms such as the Canny operator or wavelet transform. In some embodiments of this application, the value of N is 9, and edge detection can be performed on the N channel images based on S edge detection algorithms to obtain S sets of detection results, each set of detection results including N edge detection information. The S sets of detection results can be represented as: edgeC1-algo1, edgeC2-algo1, ..., edgeC9-algo1, and edgeC1-algo2, edgeC2-algo2, ..., edgeC9-algo2; ...; edgeC1-algoS, edgeC2-algoS, ..., edgeC9-algoS. Wherein, "edgeC1-algo1, edgeC2-algo1, ..., edgeC9-algo1" represents the first set of detection results. “edgeC1-algo2, edgeC2-algo2, ..., edgeC9-algo2” represent the second group of detection results. “edgeC1-algoS, edgeC2-algoS, ..., edgeC9-algoS” represent the Sth group of detection results. “edgeC1-algo1” represents the image information of the first channel image in the first group of detection results, “edgeC2-algo1” represents the image information of the second channel image in the first group of detection results, and “edgeC9-algo1” represents the image information of the ninth channel image in the first group of detection results.

[0162] To facilitate understanding, this application uses a value of 2 for S as an example to further explain the generation process of the third image. Specifically, the generation process includes the following steps: First, identify which wavelength in channels C1-C9 of each detection result possesses more edge information under both algorithms. Since the edge detection result image is a binary image, it can be determined by the number of non-zero points in the image. The wavelength with rich edge information is recorded as the feature wavelength Cn. This application uses C2 as an example of the feature wavelength for further explanation. In some embodiments of this application, the principle of color merging can refer to merging pixels to generate the channel required for the feature wavelength, resulting in a higher signal-to-noise ratio and image quality. For example, if C2 is the feature wavelength, meaning the channel image corresponding to C2 is the first channel image mentioned above, then C1 and C3 can be selected as two second channel images. Then, C1, C2, and C3 are merged and calculated to generate the third image, which has a higher signal-to-noise ratio and image quality.

[0163] In this embodiment, edge detection is performed on the N channel images respectively to obtain N edge detection information; the channel image corresponding to the first edge detection information is determined as the first channel image; wherein, the first edge detection information is the edge detection information with the most edge information among the N edge detection information; and the second channel image and the first channel image from the N channel images are merged to obtain a third image; wherein, the spectral morphology corresponding to the second channel image is adjacent to the spectral morphology corresponding to the first channel image; the N channel images include the first channel image and the second channel image. Since the spectral morphology of the second channel image is close to that of the first channel image, the bands of the generated third channel image can be enhanced by the second channel image, so that the generated third image has a higher signal-to-noise ratio and image quality.

[0164] Optionally, generating the second depth image based on the fourth image and the third image includes:

[0165] Perform binocular stereo vision calculations on each pair of pixels in the third and fourth images to obtain a second depth image;

[0166] Wherein, a pair of pixels includes the third pixel in the third image and the fourth pixel in the fourth image, the fourth pixel being the pixel in the fourth image that corresponds to the third pixel, and the third pixel being any pixel in the third image.

[0167] Binocular stereo vision utilizes the parallax generated by two cameras observing the same object from different positions to calculate the difference in object distance, thereby generating stereo depth perception. Specifically, for the same point on an object, after calculating the parallax between the imaging coordinates on the first and second cameras, and combining this with the pre-stored intrinsic and extrinsic parameters between the two cameras, the distance of that point on the object from the camera can be obtained.

[0168] Please see Figure 5 In some embodiments of this application, the depth of any pixel P in the area to be acquired can be calculated based on the following formula, wherein the following formula (3) is the formula for depth calculation based on the principle of binocular stereo vision:

[0169] (3)

[0170] The above formula (3) can also be converted into the following formula (4):

[0171] (4)

[0172] Where Z represents the depth of pixel P, and b represents the optical center O of the first camera. R With the optical center O of the second camera T The distance between them, where f represents the equivalent focal length and d represents the parallax, i.e., d equals X. R -X T X R This represents the coordinates of the imaging point p' of the first camera, X. T These are the coordinates of the imaging point p´´ of the second camera. For example, please see... Figure 5 In some embodiments of this application, when the value of f is 4mm, the value of d is 7mm, and the value of b is 14mm, substituting the values ​​of f, d, and b into the above formula (4) yields a depth of 8mm for pixel P. For example, in some embodiments of this application, when the value of f is 3.5mm, the value of d is 7mm, and the value of b is 12mm, substituting the values ​​of f, d, and b into the above formula (4) yields a depth of 6mm for pixel P.

[0173] It is understandable that once the depth of each pixel in the third image is calculated according to the above formula, the second depth image can be modeled based on all the pixels. The second depth image can express the depth of each pixel through visual depth.

[0174] In this embodiment, a second depth image is obtained by performing binocular stereo vision calculations on each pair of pixels in the third and fourth images, thereby realizing the generation process of the second depth image.

[0175] In some embodiments of this application, three depth images can be generated based on information from the second and first cameras, as well as the 3D imaging device: the first depth image, the second depth image, and the third depth image. Simultaneously, the RGB image generated by the first camera and the multispectral image generated by the second camera carry different information and have their own characteristics and advantages, as shown in Table 2 below. In practical use, combinations of several multimodal data can be used according to the needs of the actual scenario to obtain the final accurate and robust image output. For example, one or more of the first, second, and third depth images can be combined with RGB and multispectral images to adapt to optimized depth image calculations in different scenarios; the combination of several multimodal data can also be used to develop extended applications such as material recognition.

[0176] Table 2

[0177]

[0178] The above-described second depth image, based on the principle of binocular stereo vision, calculates the depth of each pixel in the first image using the first and second images. This second depth image can be called depth map A. The second depth image generated based on the fourth and third images can be called depth map B. In this embodiment, the second depth image can be either depth map A or depth map B. Specifically, the depth image generated by "RGB + multispectral binocular depth" in Table 2 is depth map A. The depth image generated by "RGB + dominant spectral channel binocular depth" is depth map B.

[0179] Optionally, generating a third depth image based on the first depth image and the second depth image includes:

[0180] The first depth image, the second depth image, the first image, and the second image are input into a multimodal depth map neural network to obtain a third depth image output by the multimodal depth map neural network.

[0181] Please see Figure 6 In this embodiment, the second depth image can be either depth map A or depth map B. The aforementioned multimodal depth map neural network can be a pre-trained depth-optimized neural network. This multimodal depth map neural network can generate a third depth image based on the first depth image, the second depth image, the first image, and the second image. The third depth image can fuse the depth features in the first depth image, the depth features in the second depth image, the depth features in the first image, and the depth features in the second image.

[0182] In some embodiments of this application, since "the third depth image integrates the depth features in the first depth image, the depth features in the second depth image, the depth features in the first image, and the depth features in the second image", the third depth image not only has a highly reliable depth source, but also incorporates deep learning to enhance robustness in multiple scenarios, making it an optimized depth image with extremely high accuracy.

[0183] In this embodiment, by inputting the first depth image, the second depth image, the first image, and the second image into a multimodal depth map neural network, a third depth image is obtained output by the multimodal depth map neural network. Since "the third depth image integrates the depth features in the first depth image, the depth features in the second depth image, the depth features in the first image, and the depth features in the second image", the third depth image not only has a highly reliable depth source, but also incorporates deep learning to enhance robustness in multiple scenarios. It is an optimized depth image with extremely high accuracy, thus further improving the accuracy of the generated third depth image.

[0184] In some embodiments, such as Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701, a memory 702, and a program or instructions stored in the memory 702 and executable on the processor 701. When the program or instructions are executed by the processor 701, they implement the various processes of the above-described depth image generation method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0185] Figure 8 A schematic diagram of the hardware structure of an electronic device according to an embodiment of this application.

[0186] The electronic device 800 includes, but is not limited to, components such as: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

[0187] The sensor 805 is used to acquire a first image through a first camera, a second image through a second camera, and a first depth image through the 3D imaging device; wherein the acquisition time of the first image, the second image, and the first depth image is the same;

[0188] The processor 810 is configured to generate a second depth image based on the first image and the second image;

[0189] The processor 810 is configured to generate a third depth image based on the first depth image and the second depth image.

[0190] Optionally, the processor 810 is configured to perform binocular stereo vision calculations on each pair of pixels in the first image and the second image to obtain a second depth image;

[0191] Wherein, a pair of pixels includes a first pixel in the first image and a second pixel in the second image, the second pixel being a pixel in the second image at a position corresponding to the first pixel, and the first pixel being any pixel in the first image.

[0192] Optionally, the multispectral camera has N spectral channels, each corresponding to one of N spectral forms. Each spectral channel transmits light of a first spectral form and cuts off light of a second spectral form. The first spectral form is the spectral form corresponding to the spectral channel, and the second spectral form is another spectral form besides the first spectral form. N is an integer greater than 1. The processor 810 is configured to perform channel separation on the first image based on the color channels to obtain K channel images, and to perform channel separation on the second image based on the spectral channels to obtain N channel images corresponding one-to-one with the N spectral channels. Here, K is an integer greater than 1.

[0193] The processor 810 is configured to generate a third image based on M channel images from the N channel images; wherein the M channel images include a first channel image, the first channel image is the channel image with the most edge information among the N channel images, M is less than N, and M is an integer greater than or equal to 1;

[0194] The processor 810 is configured to generate a second depth image based on the fourth image and the third image; wherein the fourth image is the channel image among the K channel images whose wavelength has the smallest spectral morphological distance from the first channel image.

[0195] Optionally, the processor 810 is configured to perform edge detection on the N channel images respectively to obtain N edge detection information;

[0196] The processor 810 is configured to determine the channel image corresponding to the first edge detection information as the first channel image; wherein, the first edge detection information is the edge detection information with the most edge information among the N edge detection information;

[0197] The processor 810 is used to merge the second channel image and the first channel image from the N channel images to obtain a third image; wherein the spectral morphology corresponding to the second channel image is adjacent to the spectral morphology corresponding to the first channel image; the N channel images include the first channel image and the second channel image.

[0198] Optionally, the processor 810 is configured to perform binocular stereo vision calculations on each pair of pixels in the third image and the fourth image to obtain a second depth image;

[0199] Wherein, a pair of pixels includes the third pixel in the third image and the fourth pixel in the fourth image, the fourth pixel being the pixel in the fourth image that corresponds to the third pixel, and the third pixel being any pixel in the third image.

[0200] Optionally, the processor 810 is configured to input the first depth image, the second depth image, the first image, and the second image into a multimodal depth map neural network to obtain a third depth image output by the multimodal depth map neural network.

[0201] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0202] It should be understood that, in this embodiment, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042. The GPU 8041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 807 includes a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0203] The memory 809 can be used to store software programs and various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0204] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0205] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described depth image generation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0206] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0207] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described depth image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0208] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0209] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0211] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image sensor, characterized in that, It includes a first signal reading circuit, a second signal reading circuit, a first switching element, N third switching elements, and multiple pixel groups arranged in an array, where N is an integer greater than 1; The pixel group includes N photodiodes and N photosensitive pixels. The N photosensitive pixels are arranged in an array and are configured to correspond one-to-one with N spectral patterns. The photosensitive pixels are used to sense light of a first spectral pattern, which is the spectral pattern corresponding to the photosensitive pixel. The N third switching devices are connected one-to-one with the N photosensitive pixels; The first connection terminal of the photosensitive pixel is electrically connected to the input terminal of the first switch via the third switch connected to the photosensitive pixel, and the output terminal of the first switch is electrically connected to the input terminal of the first signal reading circuit.

2. The image sensor according to claim 1, characterized in that, The image sensor further includes a second switching element and N fourth switching elements, and the N fourth switching elements are connected to the N photosensitive pixels in a one-to-one correspondence. The second connection terminal of the photosensitive pixel is electrically connected to the input terminal of the second switch via the fourth switch connected to the photosensitive pixel, and the output terminal of the second switch is electrically connected to the input terminal of the second signal reading circuit.

3. The image sensor according to claim 2, characterized in that, The photosensitive pixel includes a filter and a photodiode, and the photodiode is disposed opposite to the light-emitting surface of the filter; The filter is used to transmit light of the first spectral form and to block light of the second spectral form; wherein, the second spectral form is a spectral form other than the first spectral form; The first connection terminal of the photosensitive pixel is the negative electrode of the photosensitive diode included in the photosensitive pixel, and the second connection terminal of the photosensitive pixel is the positive electrode of the photosensitive diode included in the photosensitive pixel.

4. The image sensor according to claim 3, characterized in that, The first signal reading circuit includes a first capacitor, a first reset switch, a high-level module, a first source follower, a fifth switch, and a first DC power supply module. The gate of the first source follower is the input terminal of the first signal reading circuit, and the gate of the first source follower is grounded through the first capacitor. The gate of the first source follower is electrically connected to the high-level module through the first reset switch. The source of the first source follower is electrically connected to the high-level module. The drain of the first source follower is grounded through the fifth switch and the first DC power supply module. The drain of the first source follower is the output terminal of the first signal reading circuit. The second signal reading circuit includes a second capacitor, a second reset switch, a second source follower, a sixth switch, and a second DC power supply module. The gate of the second source follower is the input terminal of the second signal reading circuit, and the gate of the second source follower is grounded through the second capacitor. The gate of the second source follower is electrically connected to the high-level module through the second reset switch. The source of the second source follower is electrically connected to the high-level module. The drain of the second source follower is grounded through the sixth switch and the second DC power supply module. The drain of the second source follower is the output terminal of the second signal reading circuit.

5. A camera module, characterized in that, It includes a first camera, a second camera, and a 3D imaging device; the first camera is used to acquire RGB images, and the second camera is a multispectral camera; the multispectral camera is an image sensor according to any one of claims 1 to 4.

6. An electronic device, characterized in that, Includes the camera module as described in claim 5.

7. A depth image generation method, performed by the electronic device of claim 6, characterized in that, The method includes: A first image is captured by a first camera, a second image is captured by a second camera, and a first depth image is captured by the 3D imaging device; wherein the first image, the second image, and the first depth image are captured at the same time. A second depth image is generated based on the first image and the second image; A third depth image is generated based on the first depth image and the second depth image.

8. The method according to claim 7, characterized in that, The step of generating a second depth image based on the first image and the second image includes: Perform binocular stereo vision calculations on each pair of pixels in the first image and the second image to obtain a second depth image; Wherein, a pair of pixels includes a first pixel in the first image and a second pixel in the second image, the second pixel being a pixel in the second image at a position corresponding to the first pixel, and the first pixel being any pixel in the first image.

9. The method according to claim 7, characterized in that, The multispectral camera has N spectral channels, each corresponding to one of N spectral patterns. Each spectral channel transmits light of a first spectral pattern and cuts off light of a second spectral pattern. The first spectral pattern is the spectral pattern corresponding to the spectral channel, and the second spectral pattern is any spectral pattern other than the first spectral pattern. N is an integer greater than 1. Generating a second depth image based on the first image and the second image includes: Based on the color channels, the first image is separated into K channel images; and based on the spectral channels, the second image is separated into N channel images corresponding one-to-one with the N spectral channels, where K is an integer greater than 1. A third image is generated based on M channel images out of the N channel images; wherein the M channel images include a first channel image, the first channel image is the channel image with the most edge information among the N channel images, M is less than N, and M is an integer greater than or equal to 1; A second depth image is generated based on the fourth image and the third image; wherein the fourth image is the channel image with the smallest spectral morphological distance between the wavelength and the first channel image among the K channel images.

10. The method according to claim 9, characterized in that, The step of generating a third image based on M channels of the N channel images includes: Edge detection is performed on the N channel images respectively to obtain N edge detection information; The channel image corresponding to the first edge detection information is determined as the first channel image; wherein, the first edge detection information is the edge detection information with the most edge information among the N edge detection information; The second channel image and the first channel image from the N channel images are merged to obtain a third image; wherein the spectral morphology corresponding to the second channel image is adjacent to the spectral morphology corresponding to the first channel image; the N channel images include the first channel image and the second channel image.

11. The method according to claim 9, characterized in that, The step of generating the second depth image based on the fourth image and the third image includes: Perform binocular stereo vision calculations on each pair of pixels in the third and fourth images to obtain a second depth image; Wherein, a pair of pixels includes the third pixel in the third image and the fourth pixel in the fourth image, the fourth pixel being the pixel in the fourth image that corresponds to the third pixel, and the third pixel being any pixel in the third image.

12. The method according to any one of claims 7 to 11, characterized in that, The generation of a third depth image based on the first depth image and the second depth image includes: The first depth image, the second depth image, the first image, and the second image are input into a multimodal depth map neural network to obtain a third depth image output by the multimodal depth map neural network.

Images