Light sensing assembly, camera module, device and image data processing method and device

Abstract

The invention provides a light sensing assembly, a camera module, a device, an image data processing method and an image data processing device. The light sensing assembly comprises an emitter and an image sensor. The image sensor comprises a light splitting layer and a photoelectric conversion layer; the light splitting layer comprises a plurality of light splitting units, each light splitting unit comprises Q light filtering channels, at least part of the Q light filtering channels are used for acquiring spectral information of a target object, K first light filtering channels in the Q light filtering channels are used for sensing infrared light emitted by the emitter, and K second light filtering channels in the Q light filtering channels are used for sensing infrared light emitted by the emitter; the spectral transmittance curves of L second filtering channels in the Q filtering channels are different from the spectral transmittance curves of the K first filtering channels, and L + K = Q; the photoelectric conversion layer includes a pixel array, and each of the Q filter channels covers at least one pixel in the pixel array. According to the scheme, misjudgment of the image sensor in application can be reduced, and the user experience is improved.

Description

[0001] This application claims priority to Chinese Patent Application No. 202410773588.8, filed on June 14, 2024, entitled "Image Sensor, Light Sensing Component, Camera Module and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of imaging technology, and more specifically, to a light sensing component, a camera module, a device, and an image data processing method and apparatus. Background Technology

[0003] A time-of-flight (ToF) sensor is a device that measures distance using the time it takes for light to fly. Specifically, a ToF sensor emits near-infrared light toward a target object and receives the light returning from the target object. By detecting the time it takes for the light to fly, the distance between the target object and the ToF sensor is measured.

[0004] Because Time-of-Flight (ToF) sensors can measure distance information, they have wide applications in various fields, such as imaging, facial recognition, object recognition, autonomous driving, and 3D modeling. However, current ToF sensors suffer from misinterpretations when constructing scenes based on depth information, impacting the user experience.

[0005] Taking the use of Time-of-Flight (ToF) sensors for facial recognition as an example, the depth information provided by the ToF sensor can be used for anti-spoofing detection, i.e., determining whether the face is real. In some scenarios, if the depth information of a face is constructed based on a planar image, the ToF sensor may mistakenly identify the face in the constructed scene as real, thus leading to an incorrect judgment. Summary of the Invention

[0006] This application provides a light sensing component, a camera module, a device, and an image data processing method and apparatus, which can reduce misjudgments by image sensors in applications, thereby improving user experience.

[0007] In a first aspect, a light sensing component is provided, comprising: an emitter for emitting infrared light toward a target object; and an image sensor for receiving a light signal from the target object, the light signal including the infrared light emitted by the emitter; wherein the image sensor includes a beam splitting layer and a photoelectric conversion layer stacked thereon; the beam splitting layer includes a plurality of beam splitting units, each beam splitting unit including Q filter channels, at least a portion of the Q filter channels being used to acquire spectral information of the target object, the Q filter channels including K first filter channels and L second filter channels, the first filter channels being used to sense the infrared light emitted by the emitter, the spectral transmittance curve of the second filter channels being different from the spectral transmittance curve of the first filter channels, wherein 1≤K<Q, 1≤L<Q, L+K=Q, and K, L, and Q are all integers; the photoelectric conversion layer includes a pixel array, each of the Q filter channels covering at least one pixel in the pixel array, the pixel array being used to convert the light signal transmitted through the beam splitting layer into an electrical signal.

[0008] In the photosensitive component provided in this application embodiment, the image sensor has both depth detection and spectral detection functions. The spectral information reflects the spectral characteristics of the target object and can be used to determine the composition, purity, structure, etc. of the target object. The combined use of depth information and spectral information can effectively reduce misjudgments by the camera module in the application, thereby improving the user experience.

[0009] Furthermore, since the transmitter can actively emit infrared light, the photosensitive component provided in this application can perform corresponding functions even in low-light environments without the user's awareness, such as acquiring depth information and spectral information in the infrared band of a target object. Moreover, the image sensor provided in this application has a simple structure, saving cost and space.

[0010] In conjunction with the first aspect, in one possible implementation, the image sensor further includes a circuit layer for receiving electrical signals from the photoelectric conversion layer; wherein the circuit layer is disposed between the beam splitting layer and the photoelectric conversion layer, or the photoelectric conversion layer is disposed between the beam splitting layer and the circuit layer.

[0011] The circuit layer is positioned between the beam splitter layer and the photoelectric conversion layer, forming a front-illuminated structure. Front-illuminated structures have lower manufacturing costs, are suitable for mass production and application, and can meet most everyday shooting needs.

[0012] The photoelectric conversion layer is positioned between the beam splitter layer and the circuit layer, forming a back-illuminated structure. This back-illuminated structure eliminates interference caused by metal circuitry, increases light intake, and improves the sensitivity and signal-to-noise ratio of the image sensor.

[0013] In conjunction with the first aspect, in one possible implementation, the image sensor further includes a microlens layer stacked with the beam-splitting layer, wherein the beam-splitting layer is disposed between the microlens layer and the photoelectric conversion layer.

[0014] The microlens layer can focus light, helping to collect light from the target object, thereby guiding the light signal from the target object to the beam splitter layer, which can improve the light collection capability of the image sensor.

[0015] In conjunction with the first aspect, in one possible implementation, the image sensor further includes a planarization layer disposed between the microlens layer and the beam-splitting layer. The planarization layer provides a flat surface for the fabrication of the microlens layer.

[0016] In conjunction with the first aspect, in one possible implementation, the Q filter channels are arranged in a matrix, wherein in at least one row and / or at least one column of the matrix, a portion of the first filter channels are arranged adjacent to a portion of the second filter channels.

[0017] In conjunction with the first aspect, in one possible implementation, the K first filter channels are arranged at intervals in at least one row and / or at least one column of the matrix.

[0018] The first filter channels are arranged at intervals to ensure that the depth information and infrared spectral information acquired by the image sensor have sufficient resolution.

[0019] In conjunction with the first aspect, in one possible implementation, at least one second filter channel is included between two adjacent first filter channels in each row and column of the matrix.

[0020] Thus, the first filter channel occupies a high proportion in the entire beam splitting unit, which can improve the resolution of depth information and infrared spectral information acquired by the image sensor.

[0021] In conjunction with the first aspect, in one possible implementation, the spectral transmittance curves of any two of the L second filter channels are different.

[0022] In this way, each of the L second filter channels has a different spectral transmittance curve, enabling the measurement of a wider spectral range.

[0023] In conjunction with the first aspect, in one possible implementation, in the row direction of the matrix, the filter channel adjacent to the first filter channel has a first spectral transmittance curve; in the column direction of the matrix, the filter channel adjacent to the first filter channel has a second spectral transmittance curve.

[0024] In this way, the environment around each first filter channel is consistent, which can ensure that the response of the first filter channel is as consistent as possible and prevent crosstalk.

[0025] In conjunction with the first aspect, in one possible implementation, the first spectral transmittance curve is the same as the second spectral transmittance curve.

[0026] In conjunction with the first aspect, in one possible implementation, the filter channels surrounding the first filter channel have different spectral transmittance curves.

[0027] In this way, the L second filter channels occupy a high proportion in the entire beam splitting unit, which can expand the spectral range acquired by the image sensor and improve the resolution of spectral information.

[0028] In conjunction with the first aspect, in one possible implementation, the optical signal also includes ambient light, and L second filter channels are used to sense the ambient light.

[0029] Thus, L second filter channels can be used to acquire spectral information of the target object based on ambient light.

[0030] In conjunction with the first aspect, in one possible implementation, the ambient light includes visible light.

[0031] In conjunction with the first aspect, in one possible implementation, the spectral information of the target object includes the spectral information of the target object based on ambient light and the spectral information of the target object based on infrared light emitted by the transmitter.

[0032] In conjunction with the first aspect, in one possible implementation, the spectral transmittance curves of the K first filter channels are identical.

[0033] That is, the infrared light sensed by the K first filter channels has the same wavelength.

[0034] In conjunction with the first aspect, in one possible implementation, the transmitter is used to emit first infrared light and second infrared light, a portion of the K first filter channels is used to sense the first infrared light, and a portion of the K first filter channels is used to sense the second infrared light.

[0035] The center wavelength of the first infrared light is different from that of the second infrared light. That is, the K first filter channels can sense infrared light of different wavelengths. Based on the infrared light of different wavelengths emitted by the transmitter, the spectral information of the target object based on infrared light can be obtained.

[0036] In conjunction with the first aspect, in one possible implementation, the transmitter is used to emit first infrared light and second infrared light, K first filter channels are used to sense the first infrared light, and at least a portion of L second filter channels are used to sense the second infrared light.

[0037] The center wavelength of the first infrared light is different from that of the second infrared light. That is, the Q filter channels can sense infrared light of different wavelengths. Based on the infrared light of different wavelengths emitted by the transmitter, the spectral information of the target object based on infrared light can be obtained.

[0038] In conjunction with the first aspect, in one possible implementation, the image sensor includes a first operating mode and a second operating mode, wherein when the image sensor is in the first operating mode, the image sensor is used to acquire a first image frame when the transmitter is off and / or acquire a second image frame when the transmitter is on, the first image frame and / or the second image frame being used to acquire spectral information of the target object; when the image sensor is in the second operating mode, the image sensor is used to continuously acquire at least one third image frame when the transmitter is on, the at least one third image frame being used to acquire depth information of the target object, wherein the exposure time of the at least one third image frame is the same.

[0039] The same image sensor can be used to acquire depth and spectral information of a target object in different operating modes. There is no switching waiting time when the image sensor switches between different operating modes, which can improve efficiency.

[0040] In conjunction with the first aspect, in one possible implementation, when the image sensor is in the second operating mode, the image sensor is also used to acquire a fourth image frame when the transmitter is off. The fourth image frame is used to acquire environmental interference information, and the exposure time of the fourth image frame is the same as the exposure time of the third image frame.

[0041] The fourth image frame is used to calculate environmental interference information, which can be used to eliminate the influence of environmental factors when acquiring depth information.

[0042] In conjunction with the first aspect, in one possible implementation, when the image sensor is in the first operating mode, the image sensor is also used to acquire a fifth image frame when the transmitter is in the off state. The fifth image frame is used to acquire environmental interference information, and the exposure time of the fifth image frame is the same as the exposure time of the second image frame.

[0043] The fifth image frame can be used to calculate environmental interference information, thereby performing differential calculation with the second image frame obtained by the image sensor when the transmitter is turned on to obtain differential infrared spectral information and eliminate the influence of environmental factors.

[0044] In conjunction with the first aspect, in one possible implementation, the photosensitive component further includes a processing unit for acquiring depth information and spectral information of the target object based on the electrical signal.

[0045] In conjunction with the first aspect, in one possible implementation, the spectral information of the target object includes the spectral information of the target object based on ambient light and the spectral information of the target object based on infrared light emitted by the transmitter; the processing unit is used to perform channel splitting on the first image frame to obtain data for each of the Q filter channels, the data of each of the Q filter channels being used to obtain the spectral information of the target object based on ambient light, wherein the first image frame is acquired by the image sensor when the transmitter is off; the processing unit is further used to extract data from the third filter channel from the second and fifth image frames respectively, the data of the third filter channel being used to obtain the spectral information of the target object based on infrared light emitted by the transmitter, the third filter channel being one of the Q filter channels. The processing unit is further configured to extract data from the third filter channel for each of the at least one third image frame and the fourth image frame, which are used to obtain depth information of the target object. The at least one third image frame is continuously acquired by the image sensor when the transmitter is on, and the fourth image frame is acquired by the image sensor when the transmitter is off. The exposure time of the second image frame is the same as that of the fifth image frame.

[0046] Image sensors can acquire full-size images, and the processing unit can then extract the necessary data to calculate the depth and spectral information of the target object. Image sensors have simple operating modes, and there is no need to wait for switching between different operating modes.

[0047] In conjunction with the first aspect, in one possible implementation, the processing unit is used to acquire depth information of the target object based on direct or indirect time of flight.

[0048] In a second aspect, an image sensor is provided for use in a light sensing component. The light sensing component further includes an emitter for emitting infrared light toward a target object. The image sensor is used to receive light signals from the target object, the light signals including infrared light emitted by the emitter. The image sensor includes a stacked beam-splitting layer and a photoelectric conversion layer. The beam-splitting layer includes a plurality of beam-splitting units, each beam-splitting unit including Q filter channels. At least a portion of the Q filter channels are used to acquire spectral information of the target object. The Q filter channels include K first filter channels and L second filter channels. The first filter channels are used to sense the infrared light emitted by the emitter. The spectral transmittance curve of the second filter channel is different from that of the first filter channel, where 1 ≤ K < Q, 1 ≤ L < Q, L + K = Q, and K, L, and Q are all integers. The photoelectric conversion layer includes a pixel array, each of the Q filter channels covering at least one pixel in the pixel array. The pixel array is used to convert the light signals transmitted through the beam-splitting layer into electrical signals.

[0049] In conjunction with the second aspect, in one possible implementation, the image sensor further includes a circuit layer for receiving electrical signals from the photoelectric conversion layer; wherein the circuit layer is disposed between the beam splitting layer and the photoelectric conversion layer, or the photoelectric conversion layer is disposed between the beam splitting layer and the circuit layer.

[0050] In conjunction with the second aspect, in one possible implementation, the image sensor further includes a microlens layer stacked with the beam-splitting layer, wherein the beam-splitting layer is disposed between the microlens layer and the photoelectric conversion layer.

[0051] In conjunction with the second aspect, in one possible implementation, the image sensor further includes a planarization layer disposed between the microlens layer and the beam-splitting layer.

[0052] In conjunction with the second aspect, in one possible implementation, the Q filter channels are arranged in a matrix, wherein in at least one row and / or at least one column of the matrix, a portion of the first filter channels are arranged adjacent to a portion of the second filter channels.

[0053] In conjunction with the first aspect, in one possible implementation, the K first filter channels are arranged at intervals in at least one row and / or at least one column of the matrix.

[0054] In conjunction with the second aspect, in one possible implementation, at least one second filter channel is included between two adjacent first filter channels in each row and column of the matrix.

[0055] In conjunction with the second aspect, in one possible implementation, the spectral transmittance curves of any two of the L second filter channels are different.

[0056] In conjunction with the second aspect, in one possible implementation, in the row direction of the matrix, the filter channel adjacent to the first filter channel has a first spectral transmittance curve; in the column direction of the matrix, the filter channel adjacent to the first filter channel has a second spectral transmittance curve.

[0057] In conjunction with the second aspect, in one possible implementation, the first spectral transmittance curve is the same as the second spectral transmittance curve.

[0058] In conjunction with the second aspect, in one possible implementation, the filter channels surrounding the first filter channel have different spectral transmittance curves.

[0059] In conjunction with the second aspect, in one possible implementation, the optical signal also includes ambient light, with L second filter channels used to sense the ambient light.

[0060] In conjunction with the second aspect, in one possible implementation, the ambient light includes visible light.

[0061] In conjunction with the second aspect, in one possible implementation, the spectral information of the target object includes the spectral information of the target object based on ambient light and the spectral information of the target object based on infrared light emitted by the transmitter.

[0062] In conjunction with the second aspect, in one possible implementation, the spectral transmittance curves of the K first filter channels are identical.

[0063] In conjunction with the second aspect, in one possible implementation, the transmitter is used to emit first infrared light and second infrared light, a portion of the K first filter channels is used to sense the first infrared light, and a portion of the K first filter channels is used to sense the second infrared light.

[0064] In conjunction with the second aspect, in one possible implementation, the transmitter is used to emit first infrared light and second infrared light, K first filter channels are used to sense the first infrared light, and at least a portion of L second filter channels are used to sense the second infrared light.

[0065] In conjunction with the second aspect, in one possible implementation, the image sensor includes a first operating mode and a second operating mode. When the image sensor is in the first operating mode, it acquires a first image frame when the transmitter is off and / or acquires a second image frame when the transmitter is on, the first and / or second image frames being used to acquire spectral information of the target object. When the image sensor is in the second operating mode, it continuously acquires at least one third image frame when the transmitter is on, the at least one third image frame being used to acquire depth information of the target object, wherein the exposure time of the at least one third image frame is the same.

[0066] In conjunction with the second aspect, in one possible implementation, when the image sensor is in the second operating mode, the image sensor is also used to acquire a fourth image frame when the transmitter is off. The fourth image frame is used to acquire environmental interference information, and the exposure time of the fourth image frame is the same as the exposure time of the third image frame.

[0067] In conjunction with the second aspect, in one possible implementation, when the image sensor is in the first operating mode, the image sensor is also used to acquire a fifth image frame when the transmitter is in the off state. The fifth image frame is used to acquire environmental interference information, and the exposure time of the fifth image frame is the same as the exposure time of the second image frame.

[0068] In conjunction with the second aspect, in one possible implementation, the image sensor further includes a processing unit for acquiring depth information and spectral information of the target object based on the electrical signal.

[0069] In conjunction with the second aspect, in one possible implementation, the spectral information of the target object includes the spectral information of the target object based on ambient light and the spectral information of the target object based on infrared light emitted by the transmitter; the processing unit is used to perform channel splitting on the first image frame to obtain data for each of the Q filter channels, the data for each of the Q filter channels being used to obtain the spectral information of the target object based on ambient light, wherein the first image frame is acquired by the image sensor when the transmitter is off; the processing unit is further used to extract data from the third filter channel from the second and fifth image frames respectively, the data from the third filter channel being used to obtain the spectral information of the target object based on infrared light emitted by the transmitter, the third filter channel being Q filter channels. The processing unit is configured to use a filter channel for sensing infrared light emitted by the transmitter. The second image frame is acquired by the image sensor when the transmitter is in the on state, and the fifth image frame is acquired by the image sensor when the transmitter is in the off state. The exposure time of the second image frame is the same as that of the fifth image frame. The processing unit is also configured to extract data from the third filter channel for each of the at least one third image frame and the fourth image frame. The data from the third filter channel is used to obtain the depth information of the target object. The at least one third image frame is continuously acquired by the image sensor when the transmitter is in the on state, and the fourth image frame is acquired by the image sensor when the transmitter is in the off state. The exposure time of the third image frame is the same as that of the fourth image frame.

[0070] In conjunction with the second aspect, in one possible implementation, the processing unit is used to acquire depth information of the target object based on direct or indirect flight time.

[0071] Thirdly, a camera module is provided, including a lens assembly and a light sensing component as described in the first aspect and any implementation thereof, wherein a light signal from the target object is imaged on an image sensor in the light sensing component after passing through the lens assembly.

[0072] Fourthly, an electronic device is provided, including the light sensing component of the first aspect, or the image sensor of the second aspect, or the camera module of the third aspect.

[0073] In conjunction with the fourth aspect, in one possible implementation, the electronic device further includes an image processor that is communicatively connected to an image sensor and is used to receive and process image data from the image processor.

[0074] Fifthly, an image data processing method is provided, applied to an electronic device, the electronic device including the light sensing component of the first aspect and any implementation thereof, the method comprising: acquiring image data based on the light sensing component, the image data including depth information and spectral information of a target object; and outputting a processing result based on the depth information and spectral information of the target object.

[0075] The light sensing component has both depth detection and spectral detection functions. The combined use of depth and spectral information can effectively reduce misjudgments by the camera module in applications, thereby improving the user experience.

[0076] In conjunction with the fifth aspect, in one possible implementation, the spectral information of the target object includes spectral information based on ambient light and spectral information based on infrared light emitted by the transmitter.

[0077] In conjunction with the fifth aspect, in one possible implementation, the image data is used for facial recognition of the target object, and a processing result is output based on the depth information and spectral information of the target object, including: obtaining a first recognition result based on the depth information of the target object, the first recognition result including information on whether the target object is a real person and / or information on whether the depth information of the target object matches a pre-stored first facial feature; obtaining a second recognition result based on the spectral information of the target object, the second recognition result including information on whether the target object is a real person and / or information on whether the spectral information of the target object matches a pre-stored second facial feature; and outputting a final recognition result based on the first recognition result and the second recognition result.

[0078] By utilizing the acquired depth and spectral information, liveness detection can be performed, which can enhance anti-counterfeiting security and identification accuracy.

[0079] In conjunction with the fifth aspect, in one possible implementation, the image data is used for material identification of a target object, and a processing result is output based on the depth information and spectral information of the target object, including: obtaining a first identification result based on the depth information of the target object; obtaining a second identification result based on the spectral information of the target object; and outputting a final identification result based on the first identification result and the second identification result.

[0080] Material identification can be performed using the acquired depth and spectral information, which can enhance the accuracy of material recognition.

[0081] In conjunction with the fifth aspect, in one possible implementation, the image data is used for 3D modeling, and the processing result is output based on the depth information and spectral information of the target object, including: obtaining a depth model of the target object based on the depth information of the target object; obtaining the color information of the target object based on the spectral information based on ambient light; obtaining the grayscale image information of the target object based on the spectral information based on the infrared light emitted by the emitter; and outputting a 3D model of the target object based on the depth model, the color information, and the grayscale image information of the target object.

[0082] By fusing acquired environmental spectral information, depth information, and infrared spectral information to create a 3D model of the target object, the efficiency of model reconstruction, as well as the quality and realism of the model, can be improved.

[0083] A sixth aspect provides an electronic device comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the methods of the fifth aspect and any implementation thereof.

[0084] In a seventh aspect, a computer program product is provided, comprising: computer program code or instructions that, when executed on a computer, cause the computer to perform the methods described in the fifth aspect and any implementation thereof.

[0085] It should be noted that the aforementioned computer program code or instructions may be stored, in whole or in part, on a storage medium, which may be packaged together with the processor or packaged separately from the processor. This application does not specifically limit this.

[0086] Eighthly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the fifth aspect and any implementation thereof.

[0087] Ninthly, a chip system is provided, the chip system including a processor for calling a computer program or computer instructions stored in a memory to cause the processor to execute the methods in the fifth aspect and any implementation thereof.

[0088] In conjunction with aspect nine, in one possible implementation, the processor is coupled to memory via an interface. The processor reads instructions stored in memory through this interface.

[0089] In conjunction with the ninth aspect, in one possible implementation, the chip system further includes a memory in which computer programs or computer instructions are stored.

[0090] In a tenth aspect, a chip is provided, the chip system including circuitry for performing the methods of the fifth aspect and any implementation thereof. Attached Figure Description

[0091] Figure 1 This is a schematic structural diagram of an electronic device to which this application embodiment applies.

[0092] Figure 2 This is a schematic exploded view of a camera module provided in an embodiment of this application.

[0093] Figure 3 This is a schematic cross-sectional view of a camera module provided in an embodiment of this application.

[0094] Figure 4 This is a schematic block diagram of a photosensitive component provided in an embodiment of this application.

[0095] Figure 5 yes Figure 4 A partial schematic diagram of the image sensor in the light sensing component shown.

[0096] Figure 6 This is a cross-sectional schematic diagram of a photosensitive component provided in an embodiment of this application.

[0097] Figure 7-12 This is a schematic diagram of several arrangements of the filter channels in the beam splitter layer of the image sensor provided in the embodiments of this application.

[0098] Figure 13 This is a schematic diagram of a frame structure of an image sensor provided in an embodiment of this application.

[0099] Figure 14 This is a schematic diagram illustrating the working principle of the image sensor provided in the embodiments of this application.

[0100] Figure 15This is a schematic diagram illustrating the processing of data acquired by an image sensor according to an embodiment of this application.

[0101] Figure 16-18 This is a schematic diagram of the working timing of the image sensor provided in the embodiments of this application.

[0102] Figure 19 This is a schematic diagram of another frame structure of the image sensor provided in the embodiments of this application.

[0103] Figure 20 This is a schematic block diagram of an image data processing method provided in an embodiment of this application.

[0104] Figure 21 This is a schematic flowchart of a facial recognition method provided in an embodiment of this application.

[0105] Figure 22 This is a schematic flowchart of a material identification method provided in an embodiment of this application.

[0106] Figure 23 This is a schematic flowchart of a 3D modeling method provided in an embodiment of this application. Detailed Implementation

[0107] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0108] It should be noted that, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this article is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0109] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more, and "at least one" and "one or more" refer to one, two, or more than two. The singular expressions "a," "an," "the," "the," "this," and "this" are intended to also include expressions such as "one or more," unless the context explicitly indicates otherwise.

[0110] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0111] In the description of the embodiments of this application, the terms "upper," "lower," "inner," "outer," "vertical," and "horizontal," etc., indicate orientations or positional relationships relative to the indicated placement of components in the accompanying drawings. It should be understood that these directional terms are relative concepts, used for relative description and clarification, and not to indicate or imply a specific orientation that the device or component must have, or its construction and operation in a specific orientation. They can change accordingly depending on the orientation of the components in the accompanying drawings, and therefore should not be construed as limiting this application. Furthermore, "vertical" in this application is not strictly vertical, but within the allowable error range. "Parallel" is not strictly parallel, but within the allowable error range.

[0112] In the embodiments of this application, the same reference numerals are used to denote the same component or part. Furthermore, the parts in the drawings are not drawn to scale, and the dimensions and sizes of the parts shown are merely exemplary and should not be construed as limiting this application.

[0113] To facilitate understanding, the technical terms used in this application will be explained and described below.

[0114] The optical axis is an imaginary line in an optical system, which can be understood as the direction in which light rays travel through the system. For a symmetrical transmission system, the optical axis generally coincides with the rotation center line of the optical system. If a ray of light coincides with the optical axis, it will travel along the optical axis within the optical system.

[0115] Auto focus (AF) works by using the principle of light reflection from the subject. The light reflected from the subject passes through the lens and is imaged and received on the image sensor. After being processed by a computer, the image sensor drives the focusing device to focus.

[0116] Optical image stabilization (OIS) refers to the use of optical components in imaging instruments such as mobile phones or cameras to avoid or reduce camera shake during the capture of optical signals, thereby improving image quality. A common approach is to use a gyroscope for shake detection, and then use an OIS motor to translate or rotate the entire lens in the opposite direction to compensate for image blur caused by camera shake during exposure.

[0117] An image sensor is a functional device that uses the photoelectric conversion function of an optoelectronic device to convert an optical image on a photosensitive surface into an electrical signal that is proportional to the optical image.

[0118] The exposure time of an image sensor refers to the length of time that the image sensor is exposed to light while capturing an image.

[0119] Front-side illumination (FSI) is a common design structure in image sensors. Its characteristic is that light needs to pass through a transparent circuit layer on the sensor surface to reach the photosensitive element, and then be converted into an electrical signal.

[0120] Back-side illuminated (BSI) is a common design structure in image sensors. Its characteristic is that the photosensitive element is located on the back of the sensor, and light shines directly onto the photosensitive element and is converted into an electrical signal.

[0121] A photodiode (PD) is a semiconductor device consisting of a PN junction. It has unidirectional conductivity and can convert light signals into electrical signals.

[0122] A single-photon avalanche diode (SPAD) is a photodiode that operates in Geiger mode (with a reverse bias voltage greater than its avalanche breakdown voltage) and utilizes avalanche breakdown to achieve single-photon detection. When a photon enters the SPAD, it is converted into an electron. This electron is accelerated in the depletion layer and collides with other electrons to produce more electrons, causing the diode to avalanche break down and amplifying the output current of the incident light signal.

[0123] A time-to-digital converter (TDC) is a circuit used to measure time. It converts continuous time signals into digital signals, thereby digitizing time measurement. A TDC can accurately measure and record the time intervals between signals.

[0124] A time-of-flight (ToF) sensor is a device that measures distance using the time-of-flight of light. Specifically, a ToF sensor emits near-infrared light toward a target object and receives the light reflected back from the target object, using the time-of-flight information of the light to measure the distance to the target object.

[0125] Based on their different implementation principles, Time-of-Flight (ToF) sensors are divided into two types: direct time-of-flight (dToF) sensors and indirect time-of-flight (iToF) sensors. The main difference between dToF and iToF sensors lies in their method of measuring the time of flight of light. dToF sensors calculate the time of flight by directly measuring the time interval between the transmitted and received signals. iToF sensors calculate the time of flight by measuring the phase difference between the transmitted and received signals.

[0126] A vertical-cavity surface-emitting laser (VCSEL), also known as a vertical resonant cavity surface-emitting laser, is a semiconductor laser in which the laser beam is emitted perpendicular to the top surface, making the output beam easier to control and focus, and also allowing for high integration.

[0127] The spectrum, or optical spectrum, is the pattern of monochromatic light separated into wavelengths (or frequencies) after polychromatic light is dispersed by a dispersive system (such as a prism or grating). This pattern is then projected onto a detector by an imaging system.

[0128] Light waves have different names depending on their wavelength. Generally, light waves with wavelengths between 380nm and 750nm are called visible light; light waves with wavelengths shorter than 380nm (approximately between 10nm and 380nm) are called ultraviolet light; and light waves with wavelengths longer than 750nm (approximately between 750nm and 1000μm) are called infrared light. Infrared light can be further divided into near-infrared, mid-infrared, far-infrared, etc. For example, the wavelength range of near-infrared light is typically from 750nm to 1400nm. Infrared light is invisible to the human eye but can be captured by infrared sensors.

[0129] An infrared (IR) image is an image created by capturing the intensity of infrared light radiated or reflected by a target object. IR images are single-channel images, where information is primarily represented by grayscale levels—that is, by different brightness levels representing the temperature or thermal radiation intensity of different areas of the object's surface—rather than by color. Therefore, IR images are grayscale images.

[0130] Multispectral technology refers to a spectral detection technique that can simultaneously acquire multiple optical spectrum bands (usually three or more), extending beyond visible light into infrared and ultraviolet light. When combined with imaging hardware, multispectral technology can present multispectral information in image form.

[0131] It should be noted that the above-described terms and concepts are for illustrative purposes only and should not be construed as limiting the embodiments of this application.

[0132] Figure 1 A schematic structural diagram of an electronic device to which embodiments of this application are applicable is shown.

[0133] In this application, the electronic device involved is an electronic device with imaging capabilities, such as a mobile phone, personal digital assistant (PDA), tablet computer, laptop computer, camera, video recorder, smartwatch, smart bracelet, point of sale (POS) terminal, in-vehicle system, television (e.g., smart screen), wearable device, etc. This application does not impose any special limitations on the specific form of the electronic device. For ease of explanation and understanding, the following description uses a mobile phone as an example.

[0134] For example, Figure 1 Images (a) and (b) schematically show the front and back of the electronic device 100, respectively. Figure 1 As shown, the electronic device 100 may include a housing 101, a display panel (DP) 102, and a camera compact module (CCM) 103.

[0135] The housing 101 has a receiving space for accommodating the components of the electronic device 100. The housing 101 also serves to protect the electronic device 100 and support the entire device. The display screen 102 and the camera module 103 are disposed within the receiving space of the housing 101 and connected to the housing 101. In some embodiments, the housing 101 may include a back cover opposite to the display screen 102 and a mid-frame disposed between the back cover and the display screen 102; the display screen 102 and the camera module 103 may be fixed to the mid-frame. The housing 101 may be made of metal, plastic, ceramic, or glass, etc.

[0136] The display screen 102 is used to display images, such as images captured by the camera module 103. The display screen 102 can be a liquid crystal display (LCD) screen, an organic light-emitting diode (OLED) screen, etc. The display screen 102 can be a regular screen, or an irregularly shaped screen, a foldable screen, etc. The display screen 102 can be located on the front and / or back of the electronic device 100. Here, the front of the electronic device 100 can be understood as the side facing the user when using the electronic device 100, and the back of the electronic device 100 can be understood as the side facing away from the user when using the electronic device 100.

[0137] The camera module 103 is used to capture still images or videos. The camera module 103 can be located on the front and / or back of the electronic device 100. A camera module 103 located on the front of the electronic device 100 can also be called a front-facing camera, and a camera module 103 located on the back of the electronic device 100 can also be called a rear-facing camera. During shooting, the user can select the appropriate camera module according to their shooting needs. In some embodiments, when the display screen 102 can be folded, the camera module 103 can function as either a front-facing camera or a rear-facing camera as the display screen 102 folds. It is understood that the location of the camera module 103 can be determined according to actual needs. Figure 1 The installation locations shown are merely illustrative.

[0138] In some embodiments, the camera module 103 can be a vertical module or a folding module (or periscope camera module). A vertical camera module can be understood as light entering the camera module directly hitting the image sensor without bending the light path. A folding camera module can be understood as light entering the camera module needing to pass through optical elements such as reflectors, lenses, and prisms before hitting the image sensor, resulting in a folded light path.

[0139] In some embodiments, the camera module 103 may be a telephoto module, a wide-angle module, an ultra-wide-angle module, or a depth-of-field module.

[0140] This application embodiment does not limit the number of camera modules 103; it can be one, two, four, or even more. For example, one or more camera modules 103 can be set on the front of the electronic device 100, and / or one or more camera modules 103 can be set on the back of the electronic device 100. When multiple camera modules 103 are set, they can be identical or different. For example, the multiple camera modules 103 may have different lens optical parameters, different lens placement positions, or different lens shapes. This application embodiment also does not limit the relative positions of the multiple camera modules. For example, one or more of the multiple camera modules 103 can serve as the main camera module. Typically, the main camera module is responsible for the main shooting task, usually has the highest pixel count, and can provide higher resolution and a stronger sensor, thereby meeting the user's photography needs in different scenarios.

[0141] In some embodiments, the electronic device 100 may further include a protective lens 104 for protecting the camera module 103. The protective lens 104 is disposed on the housing 101 and covers the camera module 103. In some embodiments, the protective lens 104 may also cover the display screen 102 of the electronic device 100 or the back of the electronic device 100.

[0142] In some embodiments, the protective lens 104 may be made of glass, sapphire, ceramic, etc., and this application does not impose any special limitations on it. For example, the protective lens 104 is transparent, and light from outside the electronic device 100 can enter the camera module 103 through the protective lens 104.

[0143] In some embodiments, the electronic device 100 may further include a circuit board and an image processor, which are located within a receiving space formed by the housing 101. The image processor is fixed to and electrically connected to the circuit board. The image processor is communicatively connected to the camera module 103. The image processor is used to acquire image data from the camera module 103 and process the image data. The communication connection between the camera module 103 and the image sensor may include data transmission via electrical connections such as wiring, or data transmission via coupling or other methods.

[0144] In some embodiments, the electronic device 100 may further include an analog-to-digital converter (A / D converter). The A / D converter is connected between the camera module 103 and the image processor. The A / D converter is used to convert the signal generated by the camera module 103 into a digital signal and transmit it to the image sensor. After being processed by the image sensor, the digital image signal can be transmitted to the display module, and finally displayed as an image or video on the display screen 102.

[0145] In some embodiments, the electronic device 100 may further include a memory communicatively connected to the image sensor. The image processor processes the digital image signal before transmitting the image to the memory, so that the image can be retrieved from the memory and displayed on the display screen 102 whenever it is needed for subsequent viewing. In some embodiments, the image processor may also compress the processed digital image signal before storing it in the memory to save memory space.

[0146] It should be understood that Figure 1 The structure shown in the diagram does not constitute a specific limitation on the electronic device 100. The electronic device 100 may include more or fewer components than shown in the diagram. For example, the electronic device 100 may also include one or more of the following components: battery, flash, earpiece, buttons, sensors, etc. The electronic device 100 may also have a different component arrangement than shown in the diagram.

[0147] Figure 2 and Figure 3 A schematic diagram of the structure of a camera module provided in an embodiment of this application is shown. Figure 2 This is a schematic exploded view of camera module 200. Figure 3 This is a schematic cross-sectional view of the camera module 200. Figure 2 The camera module 200 in the middle can be Figure 1 An exemplary structure of the camera module 103 is shown below. (The following is in conjunction with...) Figure 2 and Figure 3 A brief introduction to the structure of camera module 200.

[0148] For ease of description, the optical axis direction of the camera module 200 is defined as the Z direction, and the two directions perpendicular to the optical axis are the X and Y directions, with the X direction perpendicular to the Y direction. In the Z direction, the side facing the object being photographed is the front side, and the side facing away from the object is the rear side. In the X and Y directions, the direction closer to the optical axis is the inner side, and the direction facing away from the optical axis is the outer side. In this embodiment, the optical axis direction is the direction in which the optical system transmits light.

[0149] Here, the definitions of X, Y, Z directions and front, back, inside, and outside also apply to the various figures described below. It should be noted that the above definitions of X, Y, Z directions and front, back, inside, and outside are merely for the convenience of describing the positional, connection, or motion relationships between the components in the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0150] like Figure 2 and Figure 3 As shown, the camera module 200 may include a housing 210, a lens assembly 220, a lens actuator 230, and a light sensing assembly 240.

[0151] The housing 210 has a receiving space for accommodating the lens assembly 220, lens actuator 230, light sensor assembly 240, etc. Additionally, the housing 210 also serves a protective and support function. It is understandable that... Figure 2 and Figure 3 The structure of the housing 210 shown is merely exemplary and does not limit the scope of this application. Those skilled in the art can design the shape of the housing 210 according to actual needs.

[0152] The lens assembly 220 mainly includes a lens group 221 and a lens barrel 222, wherein the lens group 221 is housed within the receiving space formed by the lens barrel 222. The lens assembly 220 is used to image the scene on the object side onto the image plane on the image side. In some embodiments, the lens assembly 220 can also perform certain processing on the received imaging beam, such as aberration correction and chromatic aberration elimination. Here, the imaging beam refers to the beam formed by the light incident on the camera module 200.

[0153] Lens group 221 may include at least one lens. The at least one lens may be different or at least partially the same. This application embodiment does not specifically limit the number of lenses included in lens group 221. Those skilled in the art can set the number of lenses according to actual needs, such as 1, 2, 3, 5, 8 or more.

[0154] The focal length of lens group 221 can be fixed, and correspondingly, lens assembly 220 is a prime lens. Alternatively, the focal length of lens group 221 can be adjusted, and correspondingly, lens assembly 220 is a zoom lens. For example, the focal length of lens group 221 can be adjusted by changing the relative positions of the lenses within it.

[0155] The lens barrel 222 has a receiving space, primarily for accommodating the lens assembly 221. In some embodiments, the lens barrel 222 can be a single unit, with the lens assembly 221 housed within this single unit. In other embodiments, the lens barrel 222 may also comprise multiple lens barrel sections, with the lens groups of the lens assembly 221 disposed within these multiple lens barrel sections, wherein each lens barrel section and the lenses housed therein can be referred to as a lens group. Exemplarily, the relative positions between these multiple lens barrel sections can be adjusted, enabling optical zoom by adjusting the relative positions of the lenses.

[0156] Understandable. Figure 2 and Figure 3 The structure of the lens barrel 222 and the connection method between the lens group 221 and the lens barrel 222 are merely exemplary and do not limit the embodiments of this application.

[0157] The lens actuator 230 is used to move the lens assembly 220 to achieve autofocus and / or optical image stabilization. In some embodiments, the lens actuator 230 may also be referred to as a lens motor, or simply a motor.

[0158] like Figure 3 As shown, the lens assembly actuator 230 may include a motor (hereinafter referred to as the AF motor) 231 for moving the lens assembly 220 for AF (autofocus) and / or a motor (hereinafter referred to as the OIS motor) 232 for moving the lens assembly 220 for OIS (optical image stabilization). Specifically, the AF motor 231 is used to move the lens assembly 220 for autofocus in the Z direction, and the OIS motor 232 is used to move the lens assembly 220 for optical image stabilization in the X and / or Y directions. In some embodiments, the AF motor 231 and the OIS motor 232 may be two independent components, each independently driving the lens assembly 220 for AF and OIS. Alternatively, the AF motor 231 and the OIS motor 232 may be integrated into one unit, with a single motor driving the lens assembly 220 for AF and OIS. Figure 3 The exemplary embodiment shows that the mirror assembly actuator 230 includes separate AF motor 231 and OIS motor 232, but it should be understood that the embodiments of this application are not limited thereto.

[0159] In some embodiments, the AF motor 231 or the OIS motor 232 can be used to move the entire lens assembly 220, or to move a portion of the lens assembly 220. For example, if one part of the lens assembly 220 is relatively fixed and another part is movable, the AF motor 231 or the OIS motor 232 can drive the movable part to move, thereby changing the optical path to achieve the desired function.

[0160] In some embodiments, the AF motor 231 or the OIS motor 232 may be a voice coil motor (VCM), a shape memory alloy (SMA) motor, a stepping motor, a piezoelectric motor, etc. It should be understood that the specific structure of the AF motor 231 or the OIS motor 232 may be designed and selected according to the selected driving method, and the embodiments of this application do not limit this.

[0161] The light sensing component 240 can sense and detect light incident on the lens for imaging. For example, the light sensing component 240 may include a filter, an image sensor, a circuit board, etc., located behind the lens assembly 220, wherein the filter is disposed between the lens assembly 220 and the image sensor.

[0162] Optical filters can eliminate unwanted light projected onto an image sensor, preventing problems such as ghosting, stray light, and color cast during image formation. For example, an infrared cutoff filter or a filter that blocks other light wavelengths can be used.

[0163] An image sensor is a semiconductor chip containing hundreds of thousands to millions of photodetectors on its surface. When illuminated, these photodetectors generate electrical charges, converting light signals into electrical signals. Image sensors can be charge-coupled devices (CCDs), complementary metal-oxide-semiconductor (CMOS) devices, or single-photon avalanche diodes (SPADs).

[0164] The circuit board is used to transmit electrical signals; it can be a flexible printed circuit (FPC) or a printed circuit board (PCB). The image sensor can be electrically connected to the circuit board via wires to extract the signal.

[0165] In some embodiments, the photosensing component 240 may further include a microelectromechanical system (MEMS) actuator for driving the image sensor to move along the optical axis and / or perpendicular to the optical axis, thereby achieving autofocus and / or optical image stabilization. The MEMS actuator can be driven by electrostatic force, magnetoelectric force, piezoelectric force, thermoelectric force, etc. It should be understood that the specific structure of the MEMS actuator can be designed and selected according to the chosen driving method, and this application does not limit this.

[0166] It should be understood that Figure 2 and Figure 3 The structure shown in the diagram does not constitute a specific limitation on the camera module 200. The camera module 200 may include more or fewer components than shown. For example, the camera module 200 may also include connectors and peripheral electronic components, or the camera module 200 may not include the lens actuator 230, which will not be described in detail here.

[0167] Time-of-Flight (ToF) sensors are devices that measure distance using the time-of-flight of light. Specifically, a ToF sensor emits near-infrared light towards a target object and receives the light returning from the object. By detecting the time of flight of the light, the distance between the target object and the ToF sensor is measured. Because ToF sensors can measure distance information, they have wide applications in various fields, such as imaging, facial recognition, object recognition, autonomous driving, and 3D modeling. However, current ToF sensors suffer from misjudgments when constructing scenes based on depth information, affecting the user experience. Taking facial recognition as an example, the depth information provided by a ToF sensor can be used for anti-spoofing detection, i.e., determining whether someone is a real person. In some scenarios, if the depth information of a face is constructed based on a planar image, the ToF sensor may mistakenly identify the face in the constructed scene as real, leading to incorrect judgments and insufficient anti-spoofing capabilities.

[0168] In view of this, this application will provide an image sensor and a light sensing component based on the image sensor, which can reduce misjudgments of the camera module in the application, thereby improving the user experience.

[0169] Figure 4 A schematic block diagram of a light sensing component provided in an embodiment of this application is shown. Figure 4 The light sensing component 300 shown can be applied to Figure 1 In the illustrated electronic device 100, for example, it is applied to Figure 2 The camera module 200 shown is an example. Figure 4 The light sensing component 300 shown can be Figure 2 A specific example of the light sensing component 240 shown.

[0170] like Figure 4 As shown, the light sensing component 300 includes a transmitting module 310 and a receiving module 320. The transmitting module 310 is used to emit infrared light towards a target object, and the receiving module 320 is used to receive light signals from the target object and convert them into electrical signals. The light signals received by the receiving module 320 include infrared light emitted by the transmitting module 310 to the target object and reflected by the target object, as well as ambient light (e.g., visible light) radiated and / or reflected by the target object. In this embodiment, the receiving module 320 is an image sensor.

[0171] The emitting module 310 includes an emitter 311 for emitting infrared light. For example, the emitter 311 can emit near-infrared light toward a target object, meaning the wavelength of the light source emitted by the emitter 311 is in the range of 750nm-1400nm. For instance, the emitter 311 emits a beam of light with a single center wavelength (e.g., 940nm, 810nm, or 850nm).

[0172] In some embodiments, the emitter 311 can be a laser emitter, such as a vertical-cavity surface-emitting laser. Laser emitters emit lasers with good monochromaticity, high power, and can be modulated at high frequencies, exhibiting the ability to operate stably for extended periods.

[0173] In this embodiment of the application, the transmitting module 310 may include one or more transmitters 311. If the transmitting module 310 includes multiple transmitters 311, the center wavelength of the infrared light emitted by each transmitter 311 is different.

[0174] For example, if the transmitting module 310 includes a transmitter 311, the transmitter 311 can emit infrared light with a center wavelength of 940nm. Because 940nm wavelength light emitted by the sun is easily absorbed by moisture in the atmosphere, the 940nm light waves in the atmosphere have less interference with the photosensing component 300 and less interference with the human eye. The corresponding photosensing component 300 can be used outdoors.

[0175] For example, if the emitting module 310 includes an emitter 311, the emitter 311 can emit infrared light with a center wavelength of 850nm. Because silicon-based image sensors have high quantum efficiency at 850nm wavelength, they have the highest responsivity for this spectrum and can achieve the best signal-to-noise ratio. The corresponding photosensing component 300 can be used indoors.

[0176] For example, if the transmitting module 310 includes multiple transmitters 311, any two transmitters 311 may emit infrared light with different center wavelengths. As an example, and not a limitation, the transmitting module 310 includes a first transmitter and a second transmitter, wherein the first transmitter emits infrared light with a center wavelength of a first wavelength (e.g., 940nm), and the second transmitter emits infrared light with a center wavelength of a second wavelength (e.g., 850nm), which is different from the first wavelength. This allows for the selection of a suitable transmitter based on the specific application scenario (e.g., indoor or outdoor), or, if all transmitters are operating, the appropriate data can be selected for subsequent applications.

[0177] It should be noted that the center wavelength of the infrared light emitted by transmitter 311 can be determined according to actual needs, and the above values ​​are only illustrative examples.

[0178] In some embodiments, if the emitting module 310 includes a plurality of emitters 311, the light emission mode of the plurality of emitters 311 may be synchronous light emission, sequential light emission, or alternating light emission, etc.

[0179] In some embodiments, the emitting module 310 may further include an optical component 312, which is used to ensure the performance and application effect of the laser emitted by the transmitter 311.

[0180] For example, optical component 312 may include a diffuser. A diffuser, also known as a beam shaper or beam homogenizer, is a beam shaping element that can form a specific shape and angle. For instance, the diffuser can convert the point laser emitted by emitter 311 into a surface laser, achieving a uniform distribution of light.

[0181] For example, the optical component 312 may include a lens that can be used to adjust and shape the beam output by the emitter 311.

[0182] For example, optical component 312 may include a waveguide for guiding the laser beam emitted by transmitter 311 to other optical elements.

[0183] It is understood that the above description of the optical elements included in the optical assembly 312 is merely exemplary. In practical applications, those skilled in the art can select appropriate optical elements to constitute the optical assembly 312 as needed.

[0184] In some embodiments, the transmitting module 310 may include a driving component 313 for controlling the operation of the transmitter 311. For example, the driving component 313 may control the operating state of at least one transmitter 311 included in the transmitting module 310. For instance, the driving component 313 may control a transmitter 311 to be in an on state (i.e., emitting light) at time A and to be in an off state (i.e., not emitting light) at time B. By way of example and not limitation, the driving component 313 may also control the output power of the corresponding transmitter 311.

[0185] For example, the driving component 313 may include a driving chip for directly controlling the operation of the transmitter 311. For instance, if the transmitting module 310 includes multiple transmitters 311, the driving component 313 may include multiple driving chips, each corresponding to one of the multiple transmitters 311.

[0186] For example, the driving component 313 may include a driving circuit for providing the current and voltage signals required by the driving chip.

[0187] For example, the drive component 313 may include a temperature control system for detecting the operating temperature of the drive chip to ensure its stability and reliability.

[0188] It is understood that the above description of the components included in the drive component 313 is merely exemplary. In practical applications, those skilled in the art can design the configuration of the drive component 313 as needed.

[0189] Continue to refer to Figure 4The receiving module 320 (i.e., the image sensor) includes a substrate 321 and a beam-splitting layer 322 stacked together. The beam-splitting layer 322 is disposed on the side of the substrate 321 closer to the target object. The beam-splitting layer 322 is used to split the incident light, and the substrate 321 is used to convert the split light signal into an electrical signal.

[0190] refer to Figure 5 The beam-splitting layer 322 includes a plurality of repeatedly arranged beam-splitting units 322a, each beam-splitting unit 322a including Q filter channels, at least a portion of which are used to acquire spectral information of the target object, where Q ≥ 2 and Q is an integer. For example, Q is an integer greater than or equal to 4.

[0191] The Q filter channels include K first filter channels and L second filter channels. The first filter channels are used to sense the infrared light emitted by the transmitter 311. The spectral transmittance curve of the second filter channel is different from that of the first filter channel, where 1≤K<Q, 1≤L<Q, L+K=Q, and K, L and Q are all integers.

[0192] This application describes a beam splitter layer 322 that can sense infrared light emitted by the emitter 311 (even if the infrared light emitted by the emitter 311 passes through), and can also sense light of different wavelengths. Thus, depth detection can be performed using the former, and spectral detection can be performed using the latter. Therefore, the image sensor provided by this application can simultaneously possess both depth detection and spectral detection capabilities. By combining the depth and spectral information acquired by the image sensor, misjudgments by the camera module in applications can be effectively reduced, thereby improving the user experience.

[0193] The "spectral transmittance curve" involved in this application is a curve showing the relationship between transmittance and wavelength, and can also be simply referred to as the "transmittance curve". The infrared light emitted by the first filter channel sensing emitter 311 involved in this application refers to the infrared light emitted by the emitter 311 from the target object sensed by the first filter channel.

[0194] In some embodiments, K first filter channels are used to sense infrared light emitted by the transmitter 311, and L second filter channels are used to sense ambient light. That is, the optical signal received by the receiving module 320 includes ambient light and infrared light emitted by the transmitter 311, wherein the K first filter channels can sense the infrared light emitted by the transmitter 311, and the L second filter channels can sense the ambient light. For ease of description, this application refers to this implementation as "Case 1". For example, ambient light may include visible light and / or infrared light (e.g., near-infrared light). Exemplarily, when the ambient light includes visible light, L may be an integer greater than or equal to 3.

[0195] It is understandable that if the ambient light includes infrared light, and the infrared light in the environment has the same wavelength as the infrared light emitted by the transmitter 311, then the K first filter channels can also sense the infrared light of the corresponding wavelength in the ambient light.

[0196] In one example, if the emitting module 310 emits only one wavelength of infrared light, for example, if the emitting module 310 includes an emitter 311, the spectral transmittance curves of the K first filter channels can be the same. For example, the center wavelength of the infrared light sensed by the first filter channel corresponds to the center wavelength of the infrared light emitted by the emitter 311.

[0197] In another example, if the emitting module 310 is capable of emitting infrared light of multiple wavelengths, for example, if the emitting module 310 includes multiple emitters 311, the K first filter channels can be divided into multiple parts, with each part's first filter channel used to sense one wavelength of infrared light. Taking the emitting module 310 including a first emitter and a second emitter as an example, the first emitter can emit first infrared light, and the second emitter can emit second infrared light, with the first and second infrared lights having different wavelengths. Then, a portion of the K first filter channels is used to sense the first infrared light, and another portion of the K first filter channels is used to sense the second infrared light. It can be understood that the center wavelength of any first filter channel is the same as the center wavelength of the infrared light emitted by one of the emitters 311.

[0198] At least a portion of the Q filter channels are used to acquire spectral information of the target object. For example, if a second filter channel is used to sense ambient light, then the L second filter channels are used to acquire spectral information of the target object based on ambient light. Alternatively, the K first filter channels can be used to acquire spectral information of the target object based on infrared light emitted by the emitter (hereinafter referred to as emitted light). Exemplarily, the spectral information acquired by the K first filter channels can form an infrared image (i.e., a grayscale image) of the target object. Exemplarily, when the ambient light includes visible light, the spectral information acquired by the L second filter channels can form a color image of the target object.

[0199] In some embodiments, the spectral information of the target object may include spectral information based on ambient light and spectral information based on emitted light.

[0200] In some embodiments, K first filter channels and L second filter channels are all used to sense infrared light emitted by the transmitter, wherein the wavelengths of the infrared light sensed by the first filter channels and the second filter channels are different. For ease of description, this application refers to this implementation as "Case 2". Taking the transmitting module 310 including a first transmitter and a second transmitter as an example, the first transmitter can emit first infrared light, and the second transmitter can emit second infrared light, the wavelengths of the first infrared light and the second infrared light are different. Then, all K first filter channels are used to sense the first infrared light, and all L second filter channels are used to sense the second infrared light.

[0201] Thus, the spectral information of the target object can include spectral information obtained based on infrared light of multiple wavelengths emitted by the transmitter.

[0202] It is understandable that if the ambient light includes infrared light, and the infrared light in the environment has the same wavelength as the infrared light emitted by the transmitter 311, then the K first filter channels and L second filter channels can also sense the infrared light of the corresponding wavelength in the ambient light.

[0203] In some embodiments, the Q filter channels in the beam splitter 322a can be arranged in a matrix. This matrix can be an M-row × N-column matrix, where M ≥ 1, N ≥ 1, and both M and N are integers. For example, M ≥ 2 and N ≥ 2, such as M being an integer greater than or equal to 3 (e.g., 4, 5, 6, 8, 9, 12, 16, or larger), and / or N being an integer greater than or equal to 3 (e.g., 4, 5, 6, 8, 9, 12, 16, or larger).

[0204] In some embodiments, the Q filter channels in the beam-splitting unit 322a can be formed by means of dyes, interference films, or metasurfaces. Accordingly, the image sensor is an on-chip multispectral image sensor.

[0205] The beam splitter unit 322a will be described in more detail below with reference to the accompanying drawings, and will not be elaborated on here.

[0206] Return to reference Figure 4 The substrate 321 may include a photoelectric conversion layer 3211 and a circuit layer 3212 stacked together. The photoelectric conversion layer 3211 is used to convert the light signal transmitted through the beam splitter 322 into an electrical signal, and the circuit layer 3212 is used to receive the electrical signal from the photoelectric conversion layer 3211. For example, the circuit layer 3212 can process and analyze the received electrical signal.

[0207] The photoelectric conversion layer 3211 includes a pixel array. Each of the Q filter channels in the beam splitting unit 322a covers at least one pixel in the pixel array. Specifically, the pixel array is used to convert the light signal transmitted through the beam splitting layer 322 into an electrical signal. For example, if the pixel array includes 1600*1200 pixels and the designed image resolution is 400*300, then each beam splitting unit 322a needs to cover 4*4 pixels.

[0208] In some embodiments, the circuit layer 3212 may be disposed between the beam splitting layer 322 and the photoelectric conversion layer 3211 to form a front-illuminated structure. The front-illuminated structure has a low manufacturing cost, is suitable for mass production and application, and can meet most daily shooting needs.

[0209] In other embodiments, the photoelectric conversion layer 3211 may be disposed between the beam splitting layer 322 and the circuit layer 3212 to form a back-illuminated structure. The back-illuminated structure can eliminate interference caused by metal circuits, increase the amount of light entering the sensor, and improve the sensitivity and signal-to-noise ratio of the image sensor.

[0210] In some embodiments, the structure of substrate 321 is the same as that of existing iToF or dToF sensor substrates, but the operating mode is different. For example, substrate 321 can be an iToF sensor chip or a dToF sensor chip. Accordingly, Figure 4 The receiving module 320 shown can be understood as being formed by fabricating a beam-splitting layer 322 on the basis of an iToF sensor chip or a dToF sensor chip. The operating mode of the image sensor provided in this application embodiment will be described in detail below with reference to the accompanying drawings, and will not be elaborated upon here.

[0211] In some embodiments, if the pixels of the photoelectric conversion layer 3211 are CMOS pixels, the photoelectric conversion layer 3211 can output the electrical signal corresponding to the photons accumulated during the working time of each pixel to the circuit layer 3212. If the pixels of the photoelectric conversion layer 3211 are SPAD pixels, the photoelectric conversion layer 3211 can output the electrical signal generated by each SPAD pixel being triggered by photons to the circuit layer 3212.

[0212] In some embodiments, circuit layer 3212 can also be used to control the operation of the sensor, transmit the processed data to other systems or devices, etc.

[0213] In some embodiments, the circuit layer 3212 can output digital signals corresponding to the electrical signal intensity of each pixel under different working modes. This data can be transmitted to the processing module 330 and processed into image intensity information of the target object. Then, according to the working mode of the device at the time of acquisition, the grayscale image, depth image, or spectral image of the target object can be obtained.

[0214] In some embodiments, the substrate 321 may be made of silicon.

[0215] In practical applications, the material of the substrate 321 can be determined based on the wavelength of the infrared light emitted by the emitting module 310.

[0216] In some embodiments, the receiving module 320 may further include a microlens layer 324 stacked with the beam-splitting layer 322, wherein the beam-splitting layer 322 is disposed between the microlens layer 324 and the substrate 321. The microlens layer 324 can focus light, helping to collect light from the target object, thereby guiding the light signal from the target object to the beam-splitting layer 322, which can improve the light-gathering capability of the image sensor. In addition, the microlens layer 324 can also reduce light scattering around the image sensor, avoiding light loss and interference, which is beneficial to improving the accuracy and stability of the measurement.

[0217] In some embodiments, the receiving module 320 may further include a planarization layer 323 disposed between the microlens layer 324 and the beam-splitting layer 322, the planarization layer 323 covering the beam-splitting layer 322. The surface of the beam-splitting layer 322 facing the microlens layer 324 may be uneven, and the planarization layer 323 serves to provide a flat surface for the fabrication of the microlens layer 324. Additionally, the planarization layer 323 may also provide protection for the beam-splitting layer 322, such as isolating it from moisture.

[0218] In some embodiments, if the receiving module 320 does not include the microlens layer 324, the planarization layer 323 may only serve to protect the beam splitting layer 322, and in some cases may also be referred to as a protective layer.

[0219] For example, the material of the planarization layer / protective layer 323 may be silicon dioxide or resin.

[0220] In some embodiments, the light sensing component 300 may further include a processing module 330, which is used to further process the digital signal acquired by the circuit layer 3212 in the receiving module 320. For example, the processing module 330 may acquire image intensity information, depth information, or spectral information of the target object based on the electrical signal converted from the light signal. The spectral information or image intensity information can be used to acquire a color image or infrared image of the target object, and the depth information can be used to acquire a depth image of the target object.

[0221] The processing module 330 and the receiving module 320 can be separate devices, each performing its own function. Alternatively, the processing module 330 and the receiving module 320 can be integrated into a single device, with that device performing the corresponding function. This application does not limit the scope of this embodiment.

[0222] In some embodiments, the processing module 330 may include a signal processing module and an application processing module. The signal processing module may be used to further process the electrical signals acquired by the circuit layer 3212 in the receiving module 320, such as acquiring image intensity / brightness information, depth information, or spectral information of the target object. The application processing module processes the image intensity / brightness information, depth information, or spectral information of the target object and outputs application results to be applied to specific scenarios, such as outputting face recognition results, material recognition results, or 3D modeling results.

[0223] The signal processing module and the application processing module can be separate devices, each performing its own function. For example, the signal processing module can be an image signal processing (ISP) processor, and the application processing module can be an application processor (AP). Alternatively, the signal processing module and the application processing module can be integrated into a single device, with that device performing the corresponding functions. This application does not limit this approach.

[0224] In some embodiments, the transmitter 311 can emit a modulated infrared light signal of a specific frequency toward the target object. Correspondingly, the receiving module 320 can acquire the depth information of the target object based on an indirectly measured time-of-flight of light. Specifically, the image sensor can receive the infrared light signal reflected back from the target object, and indirectly measure the time-of-flight of light by calculating the phase shift between the received and emitted signals based on the accumulated charge during the exposure (integration) time, thereby acquiring the depth of the target object. The depth d of the target object can be calculated based on the following formula (1):

[0225]

[0226] in, denoted as , where c is the phase shift (i.e., phase difference) between the received and transmitted signals; c is the speed of light; and f is the frequency of the modulation signal.

[0227] For example, based on this indirect measurement of light time of flight, circuit layer 3212 may include timing control circuitry for controlling the order and timing of data sampling, signal processing, and data transmission from the sensor.

[0228] In other embodiments, transmitter 311 can emit infrared pulse waves toward the target object. Accordingly, receiving module 320 can obtain the depth information of the target object based on directly measured time-of-flight of light. Specifically, the photoelectric conversion layer 3211 of the image sensor includes a single-photon avalanche diode (SPAD) array, and the circuit layer 3212 includes a time-to-digital converter (TDC). The SPAD can receive pulse waves emitted back from the target object, and the TDC can record the time of flight of each received light signal, that is, the time interval between the emitted pulse and the received pulse. Transmitter 311 emits N light signals within a single frame measurement time, and the image sensor receives N light signals accordingly within a single frame measurement time. By performing histogram statistics on the recorded N signals and time intervals (e.g., performed by processing module 330), the time difference with the highest frequency is the time of flight t, which is used to calculate the depth of the target object. The depth of the target object can be calculated based on the following formula (2):

[0229]

[0230] Where c is the speed of light; t is the time of flight of light obtained by directly measuring the time interval between the emitted pulse and the received pulse.

[0231] For ease of understanding, Figure 6 A schematic diagram of the optical path of the photosensing component 300 is shown, where the dashed lines with arrows represent light rays. Figure 6 As shown, the emitting module 310 can emit infrared light towards the target object, and correspondingly, the target object can reflect the infrared light. In some embodiments, it can also radiate or reflect light of other wavelengths (such as the visible light band). The light from the target object passes through the microlens layer 324 (if present) and reaches the beam splitting layer 322. After being split by the Q filter channels in the beam splitting layer 322, it reaches the substrate 321. The photoelectric conversion layer 3211 in the substrate 321 ( Figure 6 (Not shown in the image) converts the received optical signal into an electrical signal. The circuit layer 3212 in the substrate 321 (…) Figure 6 (Not shown) The receiving module receives electrical signals from the photoelectric conversion layer 3211 and transmits them to the processing module 330. The processing module 330 can process the received electrical signals to obtain image grayscale information, depth information, or spectral information of the target object.

[0232] The image sensor provided in this application is equivalent to fabricating a spectral structure on a ToF sensor substrate, thereby forming an on-chip multispectral ToF sensor chip. Thus, the image sensor provided in this application can simultaneously possess depth detection and spectral detection functions. The spectral information reflects the spectral characteristics of the target object and can be used to determine the target object's composition, purity, structure, etc. The combined use of depth and spectral information can effectively reduce misjudgments by the camera module in applications, expand application scenarios, and thus improve user experience. For example, the spectral information acquired by the image sensor can be used for biometric anti-counterfeiting, thereby compensating for the anti-counterfeiting deficiencies of existing ToF sensors and improving anti-counterfeiting capabilities. The image sensor provided in this application can also be used for applications such as material recognition and color 3D modeling. Furthermore, since the transmitter can actively emit infrared light, the photosensitive component provided in this application can also perform corresponding functions in low-light environments without the user's awareness, such as acquiring depth information and spectral information in the infrared band of the target object. Moreover, the image sensor provided in this application has a simple structure, saving cost and space.

[0233] In this application embodiment, there are multiple ways to configure the K first filter channels in the beam splitting unit 322a. For ease of understanding, case 1 is described below as an example. However, it is understood that the following embodiment is also applicable to case 2.

[0234] In some embodiments, the Q filter channels are arranged in a matrix, wherein in at least one row and / or at least one column of the matrix, a portion of the first filter channel and a portion of the second filter channel are arranged adjacent to each other.

[0235] As an example, see reference Figure 7 Black-filled squares represent the first filter channel, and pattern-filled squares represent the second filter channel. For example... Figure 7 As shown, the Q filter channels are arranged in an M x N matrix, wherein there are consecutive first filter channels in the row and / or column directions of the matrix. Specifically, there are consecutive first filter channels in one or more rows of the matrix; or, there are consecutive first filter channels in one or more columns of the matrix; or, there are consecutive first filter channels in at least one row and at least one column of the matrix. For example, the K first filter channels can be distributed in a cross shape in one row and one column of the matrix.

[0236] The continuous arrangement of the first filter channels is beneficial to the preparation of the first filter channels.

[0237] As another example, see Figure 8 Black-filled squares represent the first filter channel, and pattern-filled squares represent the second filter channel. For example... Figure 8As shown, the Q filter channels are arranged in an M x N matrix, wherein K first filter channels are spaced apart along the row and / or column directions of the matrix. Specifically, the K first filter channels are spaced apart (i.e., non-contiguous) in at least one row and / or at least one column of the matrix. In other words, at least one second filter channel is included between any two adjacent first filter channels in the row and / or column directions of the matrix.

[0238] The K first filter channels are arranged at intervals to ensure that the depth information and infrared spectral information acquired by the image sensor have sufficient resolution.

[0239] For example, refer to Figure 8 K first filter channels are distributed in each row and each column of the matrix, with the first filter channels in each row and each column arranged at intervals. For example, the K first filter channels are arranged in a one-to-one interval pattern in each row and each column of the matrix, that is, in each row of the matrix, a second filter channel is provided between two adjacent first filter channels, and in each column of the matrix, a second filter channel is provided between two adjacent first filter channels.

[0240] Thus, the first filter channel occupies a high proportion in the entire beam splitting unit, which can improve the resolution of depth information and infrared spectral information acquired by the image sensor.

[0241] For example, refer to Figure 9 K first filter channels are distributed across multiple rows or columns of a matrix, with the first filter channels spaced apart in each row or column. For example, the K first filter channels are arranged with alternating intervals in each row or column. That is, in each row or column, a second filter channel is positioned between two adjacent first filter channels.

[0242] Thus, the L second filter channels occupy a high proportion in the entire beam splitting unit, which can expand the spectral range acquired by the image sensor and improve the resolution of spectral information.

[0243] As another example, see [reference] Figure 10 The number of first filter channels, K, is 1, and the first filter channel can be located at the center of the beam splitting unit. Figure 10 The arrangement of the first filter channel shown can be considered as a special way of arranging the first filter channels at intervals.

[0244] In this application embodiment, there are multiple ways to configure the L second filter channels in the beam splitting unit 322a. For ease of understanding, case 1 is described below as an example. However, it is understood that the following embodiment is also applicable to case 2.

[0245] As an example, see reference Figure 7 or Figure 8 Among the aforementioned L second filter channels, the spectral transmittance curves of any two second filter channels are different. For example, when the second filter channel is a narrowband filter, the center wavelengths of the light transmitted through any two second filter channels are different.

[0246] In this way, each of the aforementioned L second filter channels has a different transmittance curve, enabling the measurement of a wider spectral range.

[0247] As another example, see Figure 9 Any two first filter channels among the K first filter channels have the same adjacent filter channels. Here, "adjacent filter channels have the same" means that adjacent filter channels have the same type of spectral transmittance curve.

[0248] When each first filter channel has the same adjacent filter channels, the environment around each first filter channel is consistent, which can ensure that the response of the first filter channel is as consistent as possible and prevent crosstalk.

[0249] It should be noted that the adjacent filter channels of a certain first filter channel include the filter channels adjacent to the first filter channel in the row direction of the matrix and the filter channels adjacent to the first filter channel in the column direction of the matrix. For example, if K first filter channels include filter channel #1 and filter channel #2, and the filter channels adjacent to filter channel #1 and filter channel #2 both include filter channels of the first spectral transmittance curve and filter channels of the second spectral transmittance curve, then filter channel #1 and filter channel #2 are considered to have the same adjacent filter channels.

[0250] For example, and not as a limitation, see reference. Figure 9 In the K first filter channels, in the row direction of the matrix, the filter channel adjacent to each first filter channel has a first spectral transmittance curve; in the column direction of the matrix, the filter channel adjacent to each first filter channel has a second spectral transmittance curve. In this way, the first filter channel located at the edge of the beam splitting unit and the first filter channel located in the middle of the beam splitting unit can have the same surrounding environment.

[0251] In some embodiments, the first spectral transmittance curve is the same as the second spectral transmittance curve. That is, the filter channels adjacent to each first filter channel have the same spectral transmittance curve.

[0252] As yet another example, see reference Figure 10 The filter channels surrounding the first filter channel have different spectral transmittance curves. That is, for any given first filter channel, the spectral transmittance curves of the filter channels surrounding it are different. The filter channels surrounding the first filter channel mentioned here refer to filter channels that have overlapping edges or points with the first filter channel.

[0253] To facilitate understanding, the following examples, which combine the configuration of K first filter channels and L second filter channels, provide some more specific and non-limiting examples.

[0254] refer to Figure 7 The 16 filter channels (Q = 16) are arranged in a 4x4 matrix, including 7 first filter channels (K = 7) and 9 second filter channels (L = 9). The 7 first filter channels are distributed in a cross shape in one row and one column of the matrix. The spectral transmittance curves of the 7 first filter channels are all the same, while the spectral transmittance curves of the 9 second filter channels are all different. Therefore, in case 1, if the second filter channels are used to sense visible light, the beam splitter includes one infrared filter channel and nine different visible light filter channels, with the infrared filter channel accounting for 7 / 16. In case 2, the beam splitter includes 10 different infrared filter channels.

[0255] refer to Figure 8 The 16 filter channels (Q = 16) are arranged in a 4x4 matrix, including 8 first filter channels (K = 8) and 8 second filter channels (L = 8). The 8 first filter channels are distributed across each row and column of the matrix and are spaced apart. All 8 first filter channels have the same spectral transmittance curve, while all 8 second filter channels have different spectral transmittance curves. Therefore, in case 1, if the second filter channels are used to sense visible light, the beam splitter includes one infrared filter channel and eight different visible light filter channels, with the infrared filter channel accounting for half. In case 2, the beam splitter includes nine different infrared filter channels.

[0256] refer to Figure 9The 16 filter channels (Q = 16) are arranged in a 4x4 matrix, including 4 first filter channels (K = 4) and 12 second filter channels (L = 12). The first filter channels are used to sense emitted light, and the second filter channels are used to sense visible light. The 4 first filter channels are distributed across two rows of the matrix and spaced apart in their respective columns. All 4 first filter channels have identical spectral transmittance curves, and each first filter channel is adjacent to a second filter channel with a first spectral transmittance curve in the row direction and to a second filter channel with a second spectral transmittance curve in the column direction. Correspondingly, among the 12 second filter channels, 4 have first spectral transmittance curves, 4 have second spectral transmittance curves, and the remaining 4 have different spectral transmittance curves. Therefore, in case 1, if the second filter channels are used to sense visible light, the beam splitter includes one infrared filter channel and six different visible light filter channels, with the infrared filter channel accounting for 1 / 4. In case 2, the beam splitter includes seven different infrared filter channels.

[0257] refer to Figure 10 The Q = 9 filter channels are arranged in a 3x3 matrix, including K = 1 first filter channel and L = 8 second filter channels. The first filter channel is used to sense emitted light, and the second filter channels are used to sense visible light. The 1 first filter channel is located at the center of the matrix. The spectral transmittance curves of the 8 second filter channels are all different. Therefore, in case 1, if the second filter channel is used to sense visible light, the beam splitter includes 1 infrared filter channel and 8 different visible light filter channels, with the infrared filter channel accounting for 1 / 9. In case 2, the beam splitter includes 9 different infrared filter channels.

[0258] It is understood that the values ​​of M and N mentioned above are merely exemplary. In other embodiments, M and N may not be equal, or M and N may take other values. This application does not limit this.

[0259] It should also be understood that the proportion of the K first filter channels in the beam splitting unit, the proportion of the L second filter channels in the beam splitting unit, and the setting of the spectral transmittance curves of the L second filter channels can all be determined according to actual needs. The above is only an illustrative example and this application is not limited thereto.

[0260] Figures 7 to 10 The explanation is based on the example that the spectral transmittance curves of the K first filter channels are all the same. For example, in case 1, the emitting module 310 can emit only one wavelength of infrared light, for example, the emitting module 310 includes an emitter 311.

[0261] In other embodiments, if the emitting module 310 can emit infrared light of T wavelengths, for example, if the emitting module 310 includes T emitters and the center wavelengths of the infrared light emitted by the T emitters are different, then the K first filter channels in the beam splitting unit 322a can be divided into T types, each type of filter channel being able to transmit different amounts of infrared light. The spectral transmittance curves of the T types of filter channels correspond one-to-one with the center wavelengths of the infrared light emitted by the T emitters, where T ≥ 2 and T is an integer.

[0262] For example, refer to Figure 11 Taking the transmitting module 310, which includes a first transmitter and a second transmitter, as an example, the first transmitter emits first infrared light, and the second transmitter emits second infrared light. The center wavelengths of the first and second infrared lights are different. The beam splitting unit 322a includes K first filter channels, which consist of K1 first-type filter channels and K2 second-type filter channels. The K1 first-type filter channels are used for the transmission of first infrared light, and the K2 second-type filter channels are used for the transmission of second infrared light, where K1 + K2 = K, K1 ≥ 1, K2 ≥ 1, and K1 and K2 are both integers. That is, the K first filter channels are divided into two types: one type of filter channel is used for the transmission of first infrared light, and its quantity is K1; the other type of filter channel is used for the transmission of second infrared light, and its quantity is K2.

[0263] The above description mainly uses Case 1 as an example to illustrate how to set the filter channel. The setting method for the filter channel in Case 2 is similar, and will only be illustrated below.

[0264] If the emitting module 310 can emit infrared light of T wavelengths, for example, if the emitting module 310 includes T emitters, and the center wavelengths of the infrared light emitted by the T emitters are different, then the Q filter channels can be divided into T types, and each type of filter channel can transmit different amounts of infrared light. The spectral transmittance curves of the T types of filter channels correspond one-to-one with the center wavelengths of the infrared light emitted by the T emitters, where T ≥ 2 and T is an integer.

[0265] For example, refer to Figure 12 Taking the transmitting module 310, which includes a first transmitter and a second transmitter, as an example, the first transmitter emits first infrared light, and the second transmitter emits second infrared light. The center wavelengths of the first and second infrared lights are different. The beam splitter 322a includes K first filter channels for the transmission of first infrared light and L second filter channels for the transmission of second infrared light, where K + L = Q. That is, the Q filter channels of the beam splitter 322a are divided into two types: one type of filter channel (K channels) for the transmission of first infrared light and the other type of filter channel (L channels) for the transmission of second infrared light.

[0266] The above combination Figures 7 to 12 The main focus is on the detailed introduction of the beam splitting unit. The following section will combine... Figures 13 to 19 The frame structure and timing of the image sensor are explained.

[0267] Figure 13 A schematic diagram of a frame structure of an image sensor provided in an embodiment of this application is shown. Figure 13 In the image sensor shown, K first filter channels are used to sense emitted light, and L second filter channels are used to sense ambient light. Correspondingly, the emitting module 310 emits infrared light of one or more wavelengths; for example, the emitting module 310 includes one or more emitters 311.

[0268] like Figure 13 As shown, the image sensor implements a complete workflow within a large frame, that is, simultaneously acquiring depth and spectral information.

[0269] The large frame includes a depth frame group, an environmental spectrum frame group, and an infrared spectrum frame group. The depth frame group is used to obtain the depth information of the target object, the environmental spectrum frame group is used to obtain the spectral information of the target object based on ambient light, and the infrared spectrum frame group is used to obtain the spectral information of the target object based on emitted light.

[0270] The ambient spectral frame group includes at least one first frame, which has the same exposure time. The transmitter 311 is in an OFF state within the first frame. Images acquired by the image sensor within each first frame can be used to acquire spectral information (e.g., visible light spectral information) of the target object based on ambient light. For example, the ambient spectral frame group includes one first frame.

[0271] In some embodiments, if the environmental spectral frame group includes multiple first frames, the image with the best image quality (e.g., the highest signal-to-noise ratio) can be selected from the images acquired by the multiple first frames respectively, or the multiple first frames can be fused to obtain an image with a higher signal-to-noise ratio / dynamic range, so as to obtain the spectral information of the target object based on ambient light.

[0272] The infrared spectral frame group includes at least one second frame, which has the same exposure time. The emitter 311 is in an ON state within each second frame. Images acquired by the image sensor within each second frame can be used to obtain infrared spectral information of the target object based on the emitted light. For example, the infrared spectral frame group includes one second frame.

[0273] In some embodiments, if the infrared spectral frame group includes multiple second frames, the image with the best image quality (e.g., the highest signal-to-noise ratio) can be selected from the images acquired by the multiple second frames respectively, or the multiple second frames can be fused to obtain an image with a higher signal-to-noise ratio / dynamic range, so as to obtain the infrared spectral information of the target object based on the emitted light.

[0274] In some embodiments, the infrared spectral frame group further includes at least one first environmental frame, the exposure time of which is the same as that of the second frame. The transmitter 311 is in an off state during the first environmental frame. Image data acquired by the image sensor in the first environmental frame can be used to calculate environmental interference information, thereby removing the influence of environmental factors when acquiring infrared spectral information based on emitted light.

[0275] Here, the spectral information of the target object can include the spectral information of the target object based on ambient light and the spectral information of the target object based on emitted light.

[0276] The depth frame group includes at least one third frame, which has the same exposure time. The transmitter 311 is in an ON state during each of the at least one third frame. Images acquired by the image sensor during each of the at least one third frame can be used to obtain depth information of the target object.

[0277] In some embodiments, if depth information is obtained based on direct time-of-flight, the images acquired by the image sensor within each third frame can be used to obtain the depth information of the target object. Taking a group of depth frames including a third frame as an example, the time difference t between the reflected light and the emitted light can be obtained from the image acquired through the third frame, and the depth information of the target object can be calculated based on the above formula (2). The "reflected light" involved in this application refers to the light reflected by the target object after the emitted light is incident on it.

[0278] For example, if the depth frame group includes multiple third frames, the image with the best image quality can be selected from the images acquired by the multiple third frames to obtain the depth information of the target object.

[0279] In other embodiments, if depth information is obtained based on indirect time-of-flight, the images acquired by the at least one third small frame are used to obtain the phase difference between the reflected light and the emitted light. The depth information of the target object can be calculated based on the above formula (1).

[0280] To facilitate understanding, the following will be combined with Figure 14 A brief description of the circuit structure diagram of the image sensor shown is provided below. For example... Figure 14As shown in (a), when the photosensitive component 300 is working, the transmitting module 310 emits a modulated IR light waveform, which is reflected by the target object and received by the receiving module 320 (i.e., the image sensor). Since light propagation takes time, there is a phase difference between the light waveform received by the receiving module 320 and the light waveform emitted by the transmitting module 310. This phase difference is related to the distance to the target object and the modulation frequency of the emitted light.

[0281] like Figure 14 As shown in (b), a beam-splitting layer 322 is fabricated on the substrate of the image sensor. The beam-splitting layer 322 includes a periodic structure composed of different filter materials, namely beam-splitting units 322a. Figure 14 As shown in (c), the beam splitter 322a includes Q filter channels, each filter channel covering one or more pixels. A simplified schematic diagram of the circuit structure of each pixel is shown below. Figure 14 As shown in (d), each pixel includes two charge storage structures, called TapA and TapB, which are respectively controlled by circuit switches TX. A and TX B It determines whether or not it works. The photodiode (PD) in the middle is used to convert the received photons into electrons.

[0282] Within the third sub-frame, TapA and TapB work alternately. When TapA is working, TX... A Closed, TX B When TapA is disconnected, it accumulates the received light signal. For pixels covered by the first filter channel, TapA accumulates the received ambient light signal and IR light signal (i.e., the light signal after the infrared light emitted by the transmitter is reflected by the target object). TapA accumulates the ambient light signal reflected by the target object throughout the entire working time, and accumulates the IR light signal when it is received. Similarly, when TapB is working, TX... B Closed, TX A When the filter is disconnected, TapB accumulates the received optical signal. For pixels covered by the first filter channel, TapB accumulates both the received ambient light signal and the IR light signal. Specifically, TapB accumulates the ambient light signal reflected from the target object throughout its operating time, and accumulates the IR light signal when it receives it. For pixels covered by the first filter channel, within one operating cycle of TapA and TapB (i.e., the sum of the single operating time of TapA and the single operating time of TapB), the sum of the accumulation time of the IR light signal by TapA and TapB during their operating time equals one high-level time of the emitted light. The phase difference between the reflected and emitted light can be calculated using the optical signals accumulated by TapA and TapB.

[0283] For example, without considering noise at all, the depth frame group may include a third frame, and the depth information of the target object can be calculated based on the image obtained from this third frame. Specifically, the depth information is calculated by the following formula (3) or formula (4). Substituting the value of into formula (1) will allow us to calculate the depth of the target object.

[0284]

[0285] Where Q0 is the cumulative value of TapA for the IR optical signal within one working cycle, Q π This represents the cumulative value of TapB for the IR optical signal within one working cycle.

[0286] For example, considering noise (such as circuit noise, ambient light noise, etc.), the depth frame group may include multiple third frames. The depth information of the target object can be calculated based on the image obtained from these multiple third frames, while removing the influence of noise, thus resulting in a more accurate depth calculation. Specifically, there is a first phase difference between two adjacent third frames in these multiple third frames.

[0287] For example, a depth frame group includes four third frames with a first phase difference of 90°. Correspondingly, the phase delays of these four third frames relative to the emitted light can be 0°, 90°, 180°, and 270°, respectively. Specifically, this is calculated using the following formula (5). Substituting the value of into formula (1) will allow us to calculate the depth of the target object.

[0288]

[0289] Where I = subframe4'-subframe2', Q = subframe1'-subframe3', subframe1' is the pixel data corresponding to the first filter channel extracted from the image obtained from the third subframe with a phase delay of 0°, subframe2' is the pixel data corresponding to the first filter channel extracted from the image obtained from the third subframe with a phase delay of 90°, subframe3' is the pixel data corresponding to the first filter channel extracted from the image obtained from the third subframe with a phase delay of 180°, and subframe4' is the pixel data corresponding to the first filter channel extracted from the image obtained from the third subframe with a phase delay of 270°.

[0290] Differential denoising can be performed using formula (5) to obtain more accurate results. This allows for the calculation of a more accurate depth.

[0291] It is understood that the number of third frames mentioned above is merely exemplary. In other embodiments, other numbers of third frames may be used to remove noise effects, and this application does not limit this.

[0292] In some embodiments, the depth frame group further includes at least one second environmental frame, the exposure time of which is the same as that of the third frame. The transmitter 311 is in an off state in this second environmental frame. Image data acquired by the image sensor in the second environmental frame can be used to assess the influence of ambient light when acquiring depth information, such as assessing the intensity of ambient light. For example, excessively strong ambient light may cause the proportion of reflected light signal in the signal value to be too low, affecting the reliability of the depth results. Exemplarily, the depth frame group includes one second environmental frame.

[0293] In this embodiment of the application, each frame in the environmental spectrum frame group is acquired continuously, each frame in the depth frame group is acquired continuously, and each frame in the infrared spectrum frame group is acquired continuously.

[0294] Understandable. Figure 13 In the frame structure shown, the positions of the environmental spectrum subframe group, the depth subframe group, and the infrared spectrum subframe group are merely exemplary. In some other embodiments, the relative positions of the above three subframe groups in the larger frame can be interchanged.

[0295] In the embodiments of this application, the exposure times of the first small frame, the second small frame, and the second small frame can be the same or different. The specific design can be made according to actual needs, and no special limitation is made here.

[0296] Correspondingly, the image sensor has operating mode #1, operating mode #2 and operating mode #3.

[0297] In operating mode #1, the transmitter 311 is off. The beam splitter 322 receives ambient light signals radiated or reflected from the target object and splits the ambient light signals. The substrate 321 converts the split light signals into electrical signals to acquire image data within the first small frame. The image data acquired by the image sensor within the first small frame includes the ambient light signal, which can be used to obtain the spectral information of the target object based on ambient light. For example, the processing module 330 can process the image data acquired by the image sensor within the first small frame, such as splitting the image data within the first small frame into channels to obtain low-resolution image data corresponding to Q different filter channels, or extracting image data corresponding to L second filter channels from the image data within the first small frame, thereby calculating the spectral information of the target object based on ambient light.

[0298] In operating mode #3, the transmitter 311 is in the on state during the second small frame. The beam splitter 322 receives ambient light signals radiated or reflected from the target object and emitted light signals reflected by the target object, and splits the ambient light signals and emitted light signals. The substrate 321 converts the split light signals into electrical signals, acquiring image data during the second small frame. The image data acquired by the image sensor during the second small frame includes ambient light signals and emitted light signals, which can be used to acquire spectral information of the target object based on emitted light. For example, if the spectral transmittance curves of the K first filter channels are all the same, the electrical signals converted from the light signals passing through the first filter channels in the image data corresponding to the second small frame are used to calculate the spectral information of the target object based on emitted light. Furthermore, if the spectral transmittance curves of the K first filter channels are different, such as if the K first filter channels are divided into multiple types, each type of filter channel being used to sense one wavelength of emitted light, then the electrical signals collected by each type of filter channel in the image data corresponding to the second small frame are used to calculate the spectral information of the emitted light at the corresponding wavelength.

[0299] In some embodiments, in operating mode #3, the transmitter 311 is in a turned-off state within the first environmental frame. The beam splitter 322 receives ambient light signals radiated or reflected from the target object and splits the ambient light signals. The substrate 321 converts the split light signals into electrical signals to acquire image data within the first environmental frame. The image data acquired by the image sensor within the first environmental frame includes ambient light signals, which can be used to calculate environmental interference information to remove the influence of ambient light factors when acquiring spectral information based on emitted light. For example, if the spectral transmittance curves of the K first filter channels are all the same, the electrical signals converted from the light signals passing through the first filter channels in the image data corresponding to the first environmental frame are used to calculate environmental interference information. Furthermore, if the K first filter channels are divided into multiple types, the electrical signals acquired by each type of filter channel in the image data corresponding to the first environmental frame are used to calculate environmental interference information for the corresponding wavelength.

[0300] In operating mode #2, the transmitter 311 is in the on state during the third small frame. The beam splitter 322 receives ambient light signals radiated or reflected from the target object and emitted light signals reflected from the target object, and splits the ambient light signals and emitted light signals. The substrate 321 converts the split light signals into electrical signals, acquiring image data within the third small frame. The image data acquired by the image sensor in the third small frame includes ambient light signals and emitted light signals, which can be used to obtain depth information of the target object. For example, if the spectral transmittance curves of the K first filter channels are all the same, the electrical signals converted from the light signals passing through the first filter channels in the image data corresponding to the third small frame are used to calculate the depth information of the target object. Furthermore, if the K first filter channels are divided into multiple types, and each type of filter channel is used to sense one wavelength of emitted light, then the electrical signals collected by each type of filter channel in the image data corresponding to the third small frame are used to calculate the depth information detected by the emitted light of the corresponding wavelength.

[0301] In some embodiments, in operating mode #2, the transmitter 311 is in a turned-off state within the second environmental frame. The beam splitter 322 receives ambient light signals radiated or reflected from the target object and splits the ambient light signals. The substrate 321 converts the split light signals into electrical signals to acquire image data within the second environmental frame. The image data acquired by the image sensor within the second environmental frame includes ambient light signals, which can be used to calculate environmental interference information to evaluate the confidence level of ambient light in the depth results when acquiring depth information. For example, if the spectral transmittance curves of the K first filter channels are all the same, the electrical signals converted from the light signals passing through the first filter channels in the image data corresponding to the second environmental frame are used to calculate environmental interference information. Furthermore, if the K first filter channels are divided into multiple types, the electrical signals acquired by each type of filter channel in the image data corresponding to the second environmental frame are used to calculate environmental interference information for the corresponding wavelength.

[0302] It should be noted that the spectral information of the target object based on emitted light mentioned above includes the brightness information (or intensity information) of the target object based on emitted light. The spectral information of the target object based on ambient light mentioned above also includes the brightness information (or intensity information) of the target object based on ambient light.

[0303] In operating modes #1 and #3, the image sensor is in intensity information output mode to acquire the spectral information of the target object; this is referred to as the first operating mode in this embodiment. In operating mode #2, the image sensor is in depth information output mode to acquire the depth information of the target object; this is referred to as the second operating mode in this embodiment.

[0304] In this embodiment of the application, working mode #3 is optional.

[0305] In some embodiments, if depth information is acquired based on direct time-of-flight, the image sensor can operate in working mode #3, simultaneously acquiring the depth information of the target object and the spectral information of the target object based on emitted light.

[0306] It should be noted that the reference Figure 15 The image sensor acquires a full-size image of the entire pixel array in each small frame. However, the calculation units involved in formulas (1) to (5) mentioned above are pixels, especially the pixels corresponding to the first filter channel. This is because the electrical signals collected by the pixels corresponding to the first filter channel are the signals actually used to calculate the depth information of the target object or the spectral information of the target object based on emitted light. In other words, the image sensor acquires a full-size image in each small frame, which includes the data of the first filter channel and the data of the second filter channel. However, when calculating the depth information of the target object and the spectral information of the target object based on emitted light, this complete image data is processed. Specifically, the data of the first filter channel can be extracted to form an IR image, which can be used to calculate the depth information or the spectral information based on emitted light.

[0307] For example, when calculating the depth information of a target object, the images acquired in the third frame and the second environment frame (if any) can be processed to extract the data of the first filter channel to obtain their respective IR images. The depth information can be calculated based on the IR image corresponding to the image acquired in the third frame and the above formulas (1) to (5). The environmental interference information can be calculated based on the IR image corresponding to the image acquired in the second environment frame. In some embodiments, if K first filter channels are used to sense different wavelengths of emitted light, when calculating the depth information, the data of each type of filter channel can also be extracted from the images acquired in the third frame and the second environment frame (if any) to calculate the depth information detected by each wavelength of emitted light.

[0308] For example, when calculating the spectral information of a target object based on emitted light, the images acquired in the second frame and the first environment frame (if any) can be processed to extract the data of the first filter channels to obtain their respective IR images. The intensity information of the target object based on emitted light can be calculated from the IR image corresponding to the image acquired in the second frame. Environmental interference information can be calculated from the IR image corresponding to the image acquired in the first environment frame. In some embodiments, if K first filter channels are used to sense emitted light of different wavelengths, when calculating the spectral information based on emitted light, the data of each type of filter channel can be extracted from the images acquired in the second frame and the first environment frame (if any) to calculate the spectral information detected by each wavelength of emitted light.

[0309] It is understandable that when calculating the spectral information of a target object based on ambient light, the image acquired in the first small frame can be processed to extract data from the second filter channel, and then the spectral information of the target object based on ambient light can be calculated based on the processed image. Alternatively, the above extraction processing can be omitted, and the spectral information of the target object based on ambient light can be directly obtained from the complete image data acquired by the image sensor in the first small frame.

[0310] In this embodiment, the infrared spectral frame group is optional. In other embodiments, the large frame of the image sensor may not include the infrared spectral frame group. Accordingly, the spectral information of the target object acquired by the processing module 330 only includes the spectral information of the target object based on ambient light, and does not include the spectral information of the target object based on emitted light.

[0311] In some embodiments, if depth information is acquired based on direct time-of-flight (TOF), for example, if the substrate 321 of the image sensor is a dToF sensor chip substrate structure, the large frame of the image sensor may not include the infrared spectral small frame group. However, the processing module 330 can calculate the spectral information of the target object based on emitted light, such as the intensity information of the target object based on emitted light, based on the image acquired within the depth small frame group. In other words, the image acquired within the depth small frame group can be used to calculate the time difference t between emitted and reflected light to obtain the depth information of the target object, or it can be used to obtain the intensity information of the target object based on emitted light.

[0312] Corresponding to Figure 13 With the frame structure shown, the timing sequence of the image sensor can be as follows:

[0313] When the image sensor is in working mode #1, the image sensor acquires the corresponding first image frame in each first small frame, wherein the transmitter 311 is in the off state in the first small frame, and the first image frame is used to acquire the spectral information of the target object in the ambient light.

[0314] When the image sensor is in operating mode #3, it acquires a corresponding second image frame within each second sub-frame, wherein the transmitter 311 is in the on state within the second sub-frame. At least one consecutively acquired second image frame is used to obtain spectral information of the target object based on the emitted light.

[0315] Optionally, when the image sensor is in operating mode #3, the image sensor acquires a corresponding fifth image frame within each first environmental frame, wherein the transmitter 311 is in a turned-off state within the first environmental frame. The fifth image frame acquired by the image sensor with the transmitter off can be used to calculate environmental interference information, thereby performing differential analysis with the second image frame acquired by the image sensor with the transmitter on to obtain differential infrared spectral information, in order to eliminate the influence of environmental factors. The exposure time of the fifth image frame is the same as the exposure time of the second image frame.

[0316] When the image sensor is in operating mode #2, it acquires a corresponding third image frame within each third sub-frame, wherein the transmitter 311 is in the on state (i.e., the transmitting state) within the third sub-frame. Depth information of the target object can be obtained by acquiring at least one consecutive third image frame. The exposure time of at least one third image frame is the same.

[0317] In one example, at least one third image frame obtained by the image sensor with the transmitter turned on can be used to obtain the time difference between the emitted light and the reflected light, which can be substituted into formula (2) to calculate the depth information of the target object.

[0318] In another example, at least one third image frame acquired by the image sensor when the transmitter is on can be used to obtain the phase difference between the emitted light and the emitted light. This phase difference can be substituted into formula (1) to calculate the depth information of the target object. For example, the at least one third image frame includes multiple image frames acquired when the transmitter 311 is on, and there is a first phase difference between two adjacent image frames in the multiple image frames. Specifically, the multiple image frames are used to differentially calculate the noise-reduced phase difference, thereby obtaining more accurate depth information. Taking a first phase difference of 90° as an example, the at least one third image frame can include four image frames acquired when the transmitter 311 is on, and the phase delays of the four image frames relative to the emitted light are 0°, 90°, 180°, and 270°, respectively.

[0319] Optionally, when the image sensor is in operating mode #2, the image sensor acquires a corresponding fourth image frame within each second environmental frame, wherein the transmitter 311 is in a turned-off state within the second environmental frame. The fourth image frame acquired by the image sensor with the transmitter 311 turned off is used to calculate environmental interference information to eliminate the influence of environmental factors when acquiring depth information. The exposure time of the fourth image frame is the same as the exposure time of the third image frame.

[0320] In some embodiments, if the depth information of the target object is calculated based on the direct time of flight, the image sensor may only have the above-described operating modes #1 and #2, wherein at least one third image frame and a fourth image frame (if any) acquired in operating mode #2 are used to acquire both the depth information of the target object and the spectral information of the target object based on the emitted light.

[0321] It should be noted that the infrared spectra involved in the embodiments of this application all refer to the spectral information formed based on the infrared light emitted by the transmitter.

[0322] In some embodiments, the circuit layer 3212 of the image sensor may also include control circuitry for controlling the operating state (i.e., ON or OFF) of the transmitter 311.

[0323] To further understand this, the following description will use the calculation of the target object's depth information based on indirect time-of-flight as an example to illustrate the working process of an image sensor.

[0324] Figure 16 This diagram illustrates the working principle of the image sensor in operating mode #2. Figure 16 As shown, in this mode, the transmitting module 310 has two working states: ON and OFF. When the transmitting module 310 is in the ON state, it outputs four subframes according to the different working configurations of TapA and TapB: subframe 1, subframe 2, subframe 3, and subframe 4. When the transmitting module 310 is in the OFF state, it outputs subframe 5. Subframes 1 to 4 are all full-size images. After extracting the image data of the filter channel (i.e., extracting the signal value of the first filter channel separately), the low-resolution subframes 1', 2', 3', and 4' in the aforementioned formula (5) are obtained respectively.

[0325] Specifically, Figure 16 In the waveform of TapA, the high-order bit represents the switch TX. A When closed, TapA is in the working state; the low-order waveform represents switch TX. A When TapA is open, it is in a non-operating state. Similarly, the high-order bits of the TapB waveform represent the TX switch. B When closed, TapB is in the working state; the low bit of the waveform represents switch TX. B When opened, TapB is not working.

[0326] When the transmitter is ON, in the first frame of the depth frame group, the start time of TapA is perfectly aligned with the first high-level time of the emitted light, with no phase difference between them. TapB, on the other hand, starts working immediately after the first high-level time, with a phase difference of exactly 180° from TapA. In each subsequent frame, the working time of TapB is 180° different from that of TapA. Figure 16 The right-hand diagonal stripe represents the reflected light signal accumulated in Tap A, the left-hand diagonal stripe represents the reflected light signal accumulated in Tap B, and the dotted areas represent the ambient light signals accumulated in both Taps. When the reflected light energy reaches the image sensor, Tap A or Tap B is in an active state and will accumulate the reflected light signal. As long as a Tap (e.g., Tap A or Tap B) is in an active state, it will accumulate the ambient light signal.

[0327] For example, in the first small frame that is ultimately output as small frame 1, the signal value of each image pixel is the difference between TapA and TapB (TapA-TapB) for each pixel. This can eliminate the influence of ambient light. In this embodiment, the frame is referred to as the phase difference between TapA and the emitted light. The first small frame is the phase delay 0° (phase0), corresponding to the two Taps TapA(0) and TapB(0). Similarly, in the second small frame, the TapA of the emitted light is 90° out of phase, called the phase delay 90° (phase90), and the two Taps are TapA(90) and TapB(90). In the third small frame, the TapA of the emitted light is 180° out of phase, called the phase delay 180° (phase180), and the two Taps are TapA(180) and TapB(180). The fourth subframe, TapA, is 270° out of phase with the emitted light, which is called phase delay 270° (phase270). The two Taps are TapA (270) and TapB (270). Four subframes with different phase delays are used to differentially subtract the noise and signal conversion differences caused by process variations between TapA and TapB. For example, the reflected light signal and ambient light received by TapA (0) and TapB (180) are the same, but the noise introduced by the circuit is different. By using subframe1-subframe3, i.e., TapA (0) + TapB (180) - (TapB (0) + TapA (180)), both ambient light and circuit noise can be removed simultaneously.

[0328] When the transmitter is OFF, only TapA is on, while TapB remains off. The output subframe 5 image signal value is approximately the same as the signal value of TapA. Subframe 5 is used to evaluate the intensity of ambient light during the depth test. Excessive ambient light will result in a low proportion of reflected light signal in the signal value, affecting the reliability of the depth results.

[0329] Similarly, the small frame 5 outputs a full-size image. After extracting the image data of the filter channel (that is, taking out the signal value of the first filter channel separately), a low-resolution image can be obtained, which can be used to evaluate the reliability of the results when calculating depth.

[0330] In this embodiment, small frames 1, 2, 3, and 4 are specific examples of the aforementioned third small frame. Small frame 5 is a specific example of the aforementioned second environment small frame.

[0331] Figure 17 This diagram illustrates the working principle of the image sensor in operating mode #3. Figure 17 As shown, in this mode, the transmitting module 310 has two operating states: ON and OFF. When the transmitting module 310 is in the ON state, it outputs subframe 6. When the transmitting module 310 is in the OFF state, it outputs subframe 7. Both subframe 6 and subframe 7 output full-size images. After extracting the image data of the filter channel (i.e., extracting the signal value of the first filter channel separately), a low-resolution image can be obtained for differential acquisition of the spectral information of the target object based on the emitted light.

[0332] Specifically, when the transmitting module is ON, only TapA is turned on with a phase delay of 0°, while TapB remains closed. The output image signal consists of ambient light and reflected light signals. When the transmitting module is OFF, only TapA is turned on, while TapB remains closed. The output image signal is the ambient light signal, and subframe6-subframe7 represents the reflected light signal value.

[0333] In this embodiment, frame 6 is a specific example of the aforementioned second frame. Frame 7 is a specific example of the aforementioned first environment frame.

[0334] Figure 18 This diagram illustrates the working principle of the image sensor in operating mode #1. Figure 18 As shown, in this mode, the emission module 310 has an OFF state, and the image sensor outputs subframe 8. Subframe 8 outputs a full-size image, which can then be split into channels (i.e., extracting the signal values ​​of each filter channel or the signal values ​​of the second filter channel) to obtain M×N (or L) low-resolution sub-images with different channels, which are used to calculate the spectral information of the target object based on ambient light.

[0335] Specifically, when the transmitter is OFF, only TapA is turned on, while TapB remains closed, allowing only ambient light to reach the image sensor. Consequently, the image signal output by the image sensor is entirely an ambient light signal.

[0336] In this embodiment, small frame 8 is a specific example of the aforementioned first small frame.

[0337] In summary, in this embodiment, an on-chip multispectral image sensor is formed by fabricating a beam-splitting layer on top of the image sensor chip. The differences between the various small frame groups in its frame structure are only the ON / OFF settings of the transmission module and the operating states of TapA / TapB. These configurations can be set through different chip registers, and after configuration, a set of parameters can be sent to the chip. After power-on, different small frame groups work sequentially according to the configured order. There is no switching waiting time when the image sensor switches between different operating modes, which improves efficiency.

[0338] In some embodiments, the receiving module 320 outputs a full-size image, and the processing module 330 performs post-processing on the image output by the receiving module 320, such as the aforementioned processing of extracting the first filter channel signal, the processing of extracting the second filter channel signal, the processing of splitting Q filter channel signals, and calculating the depth information of the target object, the spectral information based on ambient light, the spectral information based on emitted light, etc.

[0339] Figure 19 This illustration shows another frame structure diagram of the image sensor provided in an embodiment of this application. Figure 19 In the image sensor shown, K first filter channels are used to sense emitted light, and the spectral transmittance curves of the K first filter channels are different. L second filter channels are used to sense ambient light. Specifically, each beam-splitting unit of the beam-splitting layer has multiple types of first filter channels, and the spectral transmittance curves of different types of first filter channels are different. For example, the emission module 310 includes T emitters 311, which are used to emit T types of infrared light. Correspondingly, the beam-splitting unit includes T types of first filter channels, and the center wavelength of the infrared light emitted by the T emitters 311 corresponds one-to-one with the spectral transmittance curves of the T types of first filter channels.

[0340] and Figure 13 The difference lies in the fact that, corresponding to type T infrared light, Figure 19The large frame shown includes T groups of depth frames (e.g., depth frame group_1, depth frame group_2, depth frame group_3, ..., depth frame group_T), an environmental spectrum frame group, and T groups of infrared spectrum frames (e.g., infrared spectrum frame group_1, infrared spectrum frame group_2, infrared spectrum frame group_3, ..., infrared spectrum frame group_T). The T groups of depth frames are used to acquire the depth information of the target object detected by T types of infrared light, the environmental spectrum frame group is used to acquire the spectral information of the target object in ambient light, and the T groups of infrared spectrum frames are used to acquire the infrared spectral information of the target object detected by T types of infrared light.

[0341] The T depth frame groups correspond one-to-one with the T types of infrared light. Each depth frame group in the T depth frame groups is used to obtain the depth information of the target object detected by the corresponding infrared light.

[0342] Each depth frame group includes at least one third frame, all of which have the same exposure time. The transmitter corresponding to the depth frame group is in the on state during the at least one third frame (other transmitters may be on or off). Images acquired by the image sensor in each of the at least one third frame can be used to obtain depth information of the target object detected by infrared light.

[0343] The method by which each depth frame group acquires the depth information from the corresponding infrared light detection is the same as... Figure 13 The method for obtaining depth information for medium-depth small frame groups is similar; please refer to the relevant description above for details. For the sake of brevity, it will not be repeated here.

[0344] In some embodiments, each depth frame group further includes at least one second environment frame, the exposure time of which is the same as that of the third frame. The transmitter corresponding to the depth frame group is in a turned-off state in the second environment frame (other transmitters may be turned on or off). Image data acquired by the image sensor in the second environment frame can be used to assess the influence of the environment when acquiring depth information from corresponding infrared light detection. For example, each depth frame group includes one second environment frame.

[0345] The environmental spectral frame group includes at least one first frame, and all T transmitters 311 are in a turned-off state within the first frame. The image acquired by the image sensor within each first frame can be used to obtain the spectral information (e.g., visible light spectral information) of the target object in ambient light. For example, the environmental spectral frame group includes one first frame.

[0346] Figure 19 The environmental spectral small frame group shown and Figure 13 The structures of the environmental spectral frames shown are similar, but the differences are... Figure 19In the frame structure shown, when the image sensor acquires the environmental spectral information of the target object, all T transmitters are in the off state in the first sub-frame.

[0347] The T infrared spectral frames correspond one-to-one with the T types of infrared light. Each of the T infrared spectral frames is used to acquire the infrared spectral information of the target object detected by the corresponding infrared light.

[0348] Each infrared spectral frame group includes at least one second frame, in which the corresponding transmitter is in an on state (other transmitters may be on or off). The image acquired by the image sensor within each second frame can be used to obtain the infrared spectral information of the target object detected by the infrared light emitted by the corresponding transmitter. For example, each infrared spectral frame group includes one second frame.

[0349] In some embodiments, each infrared spectral frame group further includes at least one first environmental frame, the exposure time of which is the same as that of the second frame. The corresponding transmitter is in a switched-off state in the first environmental frame (other transmitters may be switched on or off). Image data acquired by the image sensor in the first environmental frame can be used to remove the influence of environmental factors when acquiring spectral information based on the corresponding infrared light.

[0350] The method by which each infrared spectral frame group acquires the infrared spectral information corresponding to the infrared light detection is the same as... Figure 13 The method for obtaining spectral information based on emitted light using mid-infrared spectral small frames is similar; please refer to the relevant description above for details. For the sake of brevity, it will not be repeated here.

[0351] In this embodiment, each frame in the environmental spectrum frame group is acquired continuously, each frame in the depth frame group is acquired continuously, and each frame in the infrared spectrum frame group is acquired continuously.

[0352] Understandable. Figure 19 In the frame structure shown, the positions of the environmental spectral subframe group, the depth subframe group, and the infrared spectral subframe group are merely exemplary. In other embodiments, the relative positions of each of these subframe groups within the larger frame can be interchanged. For example, the T depth subframe groups or the T infrared spectral subframe groups can be interchanged as a whole within the larger frame, and each depth subframe group or each infrared spectral subframe group within the T depth subframe groups can also be interchanged individually within the larger frame.

[0353] In this embodiment, at least some of the infrared spectral frame groups in the T infrared spectral frame groups are optional. That is, in some other embodiments, the large frame of the image sensor may not include infrared spectral frame groups or may include some infrared spectral frame groups.

[0354] Correspondingly, in the process of acquiring the spectral information of the target object based on ambient light, the image sensor operates in... Figure 13 Under the described operating mode #1, the image sensor operates in the process of acquiring depth information of the target object detected by each type of infrared light. Figure 13 Under the described operating mode #2, the image sensor operates in the process of acquiring spectral information of the target object detected by each infrared light based on the emitted light. Figure 13 The described working mode #3.

[0355] The above combination Figures 13 to 19 This section primarily describes the frame structure of the image sensor in Case 1. The frame structure for the image sensor in Case 2 is similar. For a detailed description, please refer to [link / reference]. Figures 13 to 19 The following description only explains the differences.

[0356] In scenario 2, all K first filter channels and L second filter channels in the beam-splitting unit are used to sense emitted light. For example, the emitting module 310 emits infrared light of T wavelengths. If the emitting module includes T emitters 311, different emitters emit infrared light of different wavelengths toward the target object, where T is an integer greater than or equal to 2. Accordingly, the beam-splitting unit includes T types of filter channels. Different types of filter channels have different spectral transmittance curves, and each type of filter channel is used to sense one of the T wavelengths of infrared light. Each type of filter channel includes at least one first filter channel or at least one second filter channel.

[0357] Considering that the ambient light may not contain the infrared light emitted by the transmitter, or may only contain a portion of it, the ambient spectral subframe group is optional in the frame structure of the image sensor in Case 2. That is, in some embodiments, the large frames in the image sensor may not include the ambient spectral subframe group. Accordingly, the spectral information of the target object only includes the spectral information of the target object based on the emitted light.

[0358] In case 1, the number of filter channels in the beam splitter for sensing emitted light is less than Q.

[0359] In case 2, the number of filter channels in the beam splitter unit for sensing emitted light is equal to Q. For example, if the emission module 310 includes T emitters (T≥2), all Q filter channels can be filter channels for sensing emitted light, where the Q filter channels include T types, corresponding to T types of emitted light. The data collected by these T types of filter channels can be used to obtain depth information and infrared spectral information for each type of emitted light. Accordingly, the spectral information of the target object only includes IR band information provided by the T types of filter channels.

[0360] Correspondingly, for biometric anti-counterfeiting applications, this solution can, on the basis of using depth information and IR image information for anti-counterfeiting, use spectral information obtained by supplementing light in T different IR bands to make additional judgments on living material, which can make up for the problem that the visible light spectrum cannot play a spectral anti-counterfeiting function in low light or no ambient light.

[0361] In summary, the image sensor provided in this application embodiment may include a first operating mode and a second operating mode.

[0362] When the image sensor is in a first operating mode (e.g., the aforementioned operating modes #1 and / or #3), the image sensor is used to acquire a first image frame (e.g., an image acquired within the aforementioned first sub-frame) when the transmitter is off and / or to acquire a second image frame (e.g., an image acquired within the aforementioned second sub-frame) when the transmitter is on. The first image frame and / or the second image frame are used to acquire the spectral information of the target object. For example, the first image frame is used to acquire the spectral information of the target object based on ambient light, and the second image frame is used to acquire the spectral information of the target object based on emitted light.

[0363] When the image sensor is in the second operating mode (e.g., the aforementioned operating mode #2), the image sensor is used to continuously acquire at least one third image frame (e.g., an image acquired within the aforementioned third small frame) while the transmitter is in the on state. The at least one third image frame is used to acquire depth information of the target object, wherein the exposure time of the at least one third image frame is the same.

[0364] In some embodiments, when the image sensor is in the second operating mode, the image sensor is also used to acquire a fourth image frame (such as an image acquired within the aforementioned second environmental frame) when the transmitter is in the off state. The fourth image frame is used to acquire environmental interference information, and the exposure time of the fourth image frame is the same as the exposure time of the third image frame.

[0365] In some embodiments, when the image sensor is in a first operating mode, the image sensor is also used to acquire a fifth image frame (such as an image acquired within the aforementioned first environmental frame) while the transmitter is in a turned-off state. The fifth image frame is used to acquire environmental interference information, and the exposure time of the fifth image frame is the same as the exposure time of the second image frame.

[0366] In some embodiments, the spectral information of the target object includes the spectral information of the target object based on ambient light and the spectral information of the target object based on emitted light.

[0367] In some embodiments, the processing module 330 is used to perform channel splitting on the first image frame to obtain data for each of the Q filter channels. The data for each of the Q filter channels is used to obtain spectral information of the target object based on ambient light.

[0368] In some embodiments, the processing module 330 is further configured to extract data from the third filter channel from the second image frame and the fifth image frame respectively. The data from the third filter channel is used to obtain spectral information of the target object based on the emitted light. The third filter channel is the filter channel used to sense the emitted light among the Q filter channels.

[0369] In some embodiments, the processing module 330 is further configured to extract data of a third filter channel from each of the at least one third image frame and the fourth image frame, respectively. The data of the third filter channel is used to obtain depth information of the target object. The third filter channel is a filter channel among Q filter channels used to sense emitted light.

[0370] In some embodiments, the processing module 330 is used to acquire depth information of a target object based on direct time-of-flight or indirect time-of-flight. Based on the image sensor and light-sensing component provided in the above embodiments, depth and spectral information of the target object can be acquired using the same device. Furthermore, the depth and spectral information of the target object can be transmitted to the application processing module for processing to output application results.

[0371] Figure 20 A schematic flowchart of an image data processing method provided in an embodiment of this application is shown. Figure 20 The method 400 shown is applied to an electronic device including the light sensing component 300 provided in the foregoing embodiments. The method 400 includes steps S410 to S420.

[0372] S410: Image data of the target object is acquired based on the light sensing component. The image data includes the depth information and spectral information of the target object.

[0373] In the light sensing component provided in this application embodiment, the image sensor can simultaneously acquire the depth information and spectral information of the target object.

[0374] In some embodiments, the spectral information of the target object may include ambient light-based spectral information, that is, the spectral information of the target object in ambient light (e.g., the visible light band). For example, visible light-based spectral information can be used to obtain the color information of the target object.

[0375] In some embodiments, the spectral information may include spectral information based on the infrared light emitted by the transmitter, i.e., the spectral information of the target object in the emitted light band. The spectral information based on the emitted light can be used to acquire an infrared image of the target object.

[0376] For a detailed description of image data acquisition, please refer to the above introduction on the timing and frame structure of image sensors.

[0377] S420 outputs the processing results based on the depth information and / or spectral information of the target object.

[0378] For example, if the first condition is met, the electronic device can output the processing result based on the depth information of the target object; if the second condition is met, the electronic device can output the processing result based on the spectral information of the target object; if the third condition is met, the electronic device can output the processing result based on both the depth information and the spectral information of the target object.

[0379] For example, an electronic device can obtain a first processing result based on the depth information of the target object, obtain a second processing result based on the spectral information of the target object, and then output the final processing result based on the first and second processing results.

[0380] In some embodiments, if the spectral information of the target object includes spectral information based on ambient light and spectral information based on emitted light, the electronic device obtains a second processing result based on the spectral information of the target object by: obtaining a third processing result based on the spectral information based on ambient light, obtaining a fourth processing result based on the spectral information based on emitted light, and obtaining a second processing result based on the third processing result and the fourth processing result.

[0381] For example, the second processing result includes the third processing result and / or the fourth processing result.

[0382] The above step S420 will be described in detail below with reference to specific application scenarios.

[0383] In one example, the image data acquired in step S410 is used for facial recognition of the target object. Accordingly, step S420 can output the facial recognition result based on the depth information and / or spectral information of the target object. Step S420 may specifically include:

[0384] The first recognition result is obtained based on the depth information of the target object. The first recognition result includes information on whether the target object is a real person and / or information on whether the depth information of the target object matches the pre-stored first facial features.

[0385] The second recognition result is obtained based on the spectral information of the target object. The second recognition result includes information on whether the target object is a real person and / or information on whether the spectral information of the target object matches the pre-stored second facial features.

[0386] The final recognition result is output based on the first and second recognition results.

[0387] For example, the spectral information of the target object includes spectral information based on ambient light and spectral information based on emitted light. Step S420 may specifically include:

[0388] A first recognition result is obtained based on the depth information of the target object, and the first recognition result includes information on whether the target object is a real person;

[0389] A third identification result is obtained based on the spectral information of ambient light, which includes information on whether the target object is a real person;

[0390] A fourth identification result is obtained based on the spectral information of the emitted light, which includes information on whether the target object is a real person;

[0391] The final recognition result is output based on the first recognition result, the third recognition result, and the fourth recognition result.

[0392] Here, the second identification result may include the third identification result and / or the fourth identification result, or the second identification result may be determined based on the third identification result and the fourth identification result.

[0393] For example, if any one of the first, third, and fourth recognition results indicates that the target object is not a real person, then the result of face recognition failure is output. If the first, third, and fourth recognition results all indicate that the target object is a real person, and the facial features of the target object match the pre-stored facial features, then the result of face recognition success is output.

[0394] In some embodiments, at least one of the first recognition result, the third recognition result, and the fourth recognition result includes information on whether the facial features of the target object match the pre-stored facial features.

[0395] In some embodiments, the pre-stored facial features may be in the form of an infrared image of the face, or spectral information of the face, or depth information of the face.

[0396] Of course, in some other embodiments, the electronic device may perform facial recognition based on preset conditions, according to at least one of the depth information of the target object, the spectral information based on ambient light, and the spectral information based on emitted light, in order to output the final recognition result.

[0397] For example, the electronic device obtains a third recognition result based on spectral information based on ambient light, which includes information on whether the target object is a real person; when the third recognition result indicates that the target object is a real person, the electronic device then obtains a first recognition result based on the depth information of the target object and / or obtains a fourth recognition result based on spectral information based on emitted light, wherein the first recognition result or the fourth recognition result includes information on whether the facial features of the target object match the pre-stored facial features.

[0398] In another example, the image data acquired in step S410 is used for material identification of the target object. Accordingly, step S420 can output the material identification result based on the depth information and / or spectral information of the target object. Specifically, step S420 may include:

[0399] The first recognition result is obtained based on the depth information of the target object;

[0400] The second recognition result is obtained based on the spectral information of the target object;

[0401] The final recognition result is output based on the first and second recognition results.

[0402] For example, the spectral information of the target object includes spectral information based on ambient light and spectral information based on emitted light. Step S420 may specifically include:

[0403] The first recognition result is obtained based on the depth information of the target object;

[0404] The third identification result is obtained based on the spectral information of ambient light;

[0405] The fourth identification result is obtained based on the spectral information of the emitted light;

[0406] The final recognition result is output based on the first recognition result, the third recognition result, and the fourth recognition result.

[0407] Here, the second identification result may include the third identification result and / or the fourth identification result, or the second identification result may be determined based on the third identification result and the fourth identification result.

[0408] For example, if the first, third, and fourth identification results are all the same, then any material identification result can be output. If at least two of the first, third, and fourth identification results are different, then a result indicating material identification failure can be output.

[0409] For example, if at least two of the first, third, and fourth identification results are the same, the material identification result and the corresponding confidence level can be output. If the first, third, and fourth identification results are all different, the result of material identification failure can be output.

[0410] In some embodiments, the first identification result, the third identification result, and the fourth identification result may have different weights or priorities. When at least two of the first identification result, the third identification result, and the fourth identification result are different, the final identification result may be output based on the identification result with the highest weight or priority.

[0411] In yet another example, the image data acquired in step S410 is used for 3D modeling, and correspondingly, step S420 can output the modeling result based on the depth information and / or spectral information of the target object.

[0412] For example, step S420 may specifically include:

[0413] Obtain the depth model of the target object based on its depth information;

[0414] Obtain the color information and / or grayscale image information of the target object based on its spectral information;

[0415] Output a 3D model of the target object based on its depth model, color information, and / or grayscale image information.

[0416] In some embodiments, the spectral information of the target object includes spectral information based on ambient light and spectral information based on emitted light. Step S420 may specifically include:

[0417] Obtain the depth model of the target object based on its depth information;

[0418] Obtain the color information of the target object based on the spectral information of ambient light;

[0419] The grayscale image information of the target object is obtained based on the spectral information of the emitted light;

[0420] Output a 3D model of the target object based on its depth model, color information, and grayscale image information.

[0421] Grayscale image information has higher resolution and can be combined with color information to obtain a higher resolution color image. This grayscale image information includes brightness information, which can be obtained through data collected by a filter channel that can sense emitted light.

[0422] To facilitate understanding, the specific application of the image data processing method provided in this application is described below with specific, non-limiting examples.

[0423] Figure 21 A schematic flowchart of a facial recognition method provided in an embodiment of this application is shown. In this example, the spectral information of the target object includes visible light-based spectral information (hereinafter referred to as visible light spectral information for ease of description) and emitted light-based spectral information (hereinafter referred to as infrared spectral information for ease of description). When the photosensitive component starts working, Figure 21 The method 500 shown includes the following steps:

[0424] S511, the electronic device acquires infrared spectral information at a first resolution.

[0425] For example, the first resolution can be X1*Y1, and the size of the first resolution is related to the proportion of the filter channel in the beam splitter used to sense the infrared light emitted by the transmitter.

[0426] For example, the infrared spectral information of the first resolution can be an infrared image of the target object.

[0427] S512, the electronic device compares infrared spectral information with stored facial image feature information to determine whether they match, and / or determines whether the person is a real person based on the infrared spectral information.

[0428] The facial image feature information involved in step S512 can be an infrared image of the face.

[0429] S521, the electronic device acquires visible light spectral information at a second resolution.

[0430] For example, the second resolution can be X2*Y2, and the size of the second resolution is related to the proportion of the second filter channel in the beam splitting unit.

[0431] S522, the electronic device compares the visible light spectral information with the stored facial image feature information to determine whether they match, and / or determines whether the person is a real person based on the visible light spectral information of the entire facial region.

[0432] The facial image feature information involved in step S522 can be a color image of a face.

[0433] S531, the electronic device acquires depth information at a third resolution.

[0434] For example, the third resolution can be X3*Y3, and the size of the third resolution is related to the proportion of the filter channel in the beam splitter used to sense the infrared light emitted by the transmitter.

[0435] S532, the electronic device compares the depth information with the stored facial image feature information to determine whether they match, and / or determines whether the person is a real person based on the depth information of the entire facial region.

[0436] The facial image feature information involved in step S532 can be a depth image of the face.

[0437] S520, the electronic device outputs the facial recognition result based on the judgment results of steps S512, S522 and S532.

[0438] In this method, at least one of the acquired visible light spectral information, infrared spectral information, and depth information can be used to determine the material of a living object, which can enhance anti-counterfeiting security.

[0439] In this embodiment, the first resolution, the second resolution, and the third resolution may be the same or different, and the value of each resolution may depend on the proportion of different filter channels in the beam splitting unit.

[0440] In this embodiment, steps S511 and S512 constitute the first branch process, steps S521 and S522 constitute the second branch process, and steps S531 and S532 constitute the third branch process. The first branch process, the second branch process, and the third branch process can be executed independently or sequentially, and this application does not limit this.

[0441] Figure 22 A schematic flowchart illustrating a material identification method provided in an embodiment of this application is shown. In this example, the spectral information of the target object includes visible light spectral information and infrared spectral information. When the light sensing component starts working, Figure 22 The method 600 shown includes the following steps:

[0442] S611, the electronic device acquires infrared spectral information at a first resolution.

[0443] For example, the infrared spectral information of the first resolution can be an infrared image of the target object.

[0444] S612, electronic devices match material libraries based on infrared spectral information.

[0445] For example, electronic devices can acquire 2D images based on infrared spectral information, which are also infrared images, and match material libraries based on the infrared images.

[0446] S621, the electronic device acquires visible light spectral information at a second resolution.

[0447] S622, electronic devices match material libraries based on visible light spectrum information.

[0448] S631, the electronic device acquires depth information at a third resolution.

[0449] S632, the electronic device matches the material library based on depth information.

[0450] For example, electronic devices can obtain the material shape of a target object based on depth information and match it to a material library.

[0451] S620, the electronic device outputs the material identification result based on the matching results of steps S612, S622 and S632.

[0452] This method utilizes at least one of the acquired visible light spectral information, infrared spectral information, and depth information to determine the material, thereby enhancing the accuracy of material identification.

[0453] In this embodiment, steps S611 and S612 constitute the first branch process, steps S621 and S622 constitute the second branch process, and steps S631 and S632 constitute the third branch process. The first branch process, the second branch process, and the third branch process can be executed independently or sequentially, and this application does not limit this.

[0454] Figure 23 A schematic flowchart of a 3D modeling method provided in an embodiment of this application is shown. In this example, the spectral information of the target object includes visible light spectral information and infrared spectral information. When the light sensing component starts working, Figure 23 The method 700 shown includes the following steps:

[0455] S711, the electronic device acquires infrared spectral information at a first resolution.

[0456] S712, the electronic device acquires grayscale image information of the target object based on infrared spectral information.

[0457] Here, the 2D grayscale image information has higher precision, and its intensity signal can provide more details, thus preserving resolution.

[0458] S721, the electronic device acquires visible light spectral information at a second resolution.

[0459] S722, the electronic device obtains the color information of the target object based on the visible light spectrum information.

[0460] S731, the electronic device acquires depth information at a third resolution.

[0461] S732, an electronic device reconstructs a depth model of a target object based on depth information.

[0462] S720, the electronic device establishes a 3D model by fusing the results of steps S712, S722 and S732.

[0463] This method utilizes the acquired visible light spectral information, depth information, and 2D grayscale image information to fuse and establish a 3D model of the target object, which can improve the efficiency of model reconstruction as well as the quality and realism of the model.

[0464] In this embodiment, steps S711 and S712 constitute the first branch process, steps S721 and S722 constitute the second branch process, and steps S731 and S732 constitute the third branch process. The first branch process, the second branch process, and the third branch process can be executed independently or sequentially, and this application does not limit this.

[0465] In some embodiments, the first branch process is optional.

[0466] This application provides a camera module, which includes a lens group and a light sensing component 300 in the foregoing embodiments, wherein the light signal from the target object is imaged on the image sensor in the light sensing component after passing through the lens group.

[0467] This application provides an electronic device that includes the light sensing component 300 from the foregoing embodiments, or the image sensor from the foregoing embodiments.

[0468] This application provides an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions, which, when executed by one or more processors, cause the electronic device to perform the aforementioned image data processing method.

[0469] This application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the aforementioned image data processing method.

[0470] This application provides a chip that includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the aforementioned image data processing method.

[0471] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0472] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0473] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0474] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0475] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0476] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0477] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A photosensitive component, characterized in that, include: A transmitter used to emit infrared light toward a target object; An image sensor for receiving light signals from the target object, the light signals including infrared light emitted by the transmitter; The image sensor includes a beam splitting layer and a photoelectric conversion layer stacked together. The beam-splitting layer includes multiple beam-splitting units, each beam-splitting unit including Q filter channels. At least a portion of the Q filter channels are used to acquire the spectral information of the target object. The Q filter channels include K first filter channels and L second filter channels. The first filter channels are used to sense the infrared light emitted by the emitter. The spectral transmittance curve of the second filter channel is different from that of the first filter channel, where 1≤K<Q, 1≤L<Q, L+K=Q, and K, L, and Q are all integers. The photoelectric conversion layer includes a pixel array, and each of the Q filter channels covers at least one pixel in the pixel array. The pixel array is used to convert the light signal transmitted through the beam splitter into an electrical signal.

2. The photosensitive component according to claim 1, characterized in that, The image sensor further includes a circuit layer for receiving electrical signals from the photoelectric conversion layer; wherein... The circuit layer is disposed between the beam splitting layer and the photoelectric conversion layer, or the photoelectric conversion layer is disposed between the beam splitting layer and the circuit layer.

3. The photosensitive component according to claim 1 or 2, characterized in that, The image sensor further includes a microlens layer stacked with the beam splitter layer, wherein the beam splitter layer is disposed between the microlens layer and the photoelectric conversion layer.

4. The photosensitive component according to claim 3, characterized in that, The image sensor further includes a planarization layer disposed between the microlens layer and the beam splitting layer.

5. The photosensitive component according to any one of claims 1 to 4, characterized in that, The Q filter channels are arranged in a matrix, wherein in at least one row and / or at least one column of the matrix, some of the first filter channels are arranged adjacent to some of the second filter channels.

6. The photosensitive component according to claim 5, characterized in that, K of the first filter channels are arranged at intervals in at least one row direction / or at least one column of the matrix.

7. The photosensitive component according to claim 5 or 6, characterized in that, In each row and column of the matrix, at least one second filter channel is included between two adjacent first filter channels.

8. The photosensitive component according to any one of claims 5 to 7, characterized in that, The spectral transmittance curves of any two of the L second filter channels are different.

9. The photosensitive component according to any one of claims 5 to 7, characterized in that, In the row direction of the matrix, the filter channel adjacent to the first filter channel has a first spectral transmittance curve; In the column direction of the matrix, the filter channel adjacent to the first filter channel has a second spectral transmittance curve.

10. The photosensitive component according to claim 9, characterized in that, The first spectral transmittance curve is the same as the second spectral transmittance curve.

11. The photosensitive component according to any one of claims 5 to 8, characterized in that, The filter channels surrounding the first filter channel have different spectral transmittance curves.

12. The photosensitive component according to any one of claims 1 to 11, characterized in that, The optical signal also includes ambient light, and L second filter channels are used to sense the ambient light.

13. The photosensitive component according to claim 12, characterized in that, The spectral transmittance curves of the K first filter channels are the same.

14. The photosensitive component according to claim 12, characterized in that, The transmitter is used to emit a first infrared light and a second infrared light, and a portion of the K first filter channels is used to sense the first infrared light, and a portion of the K first filter channels is used to sense the second infrared light.

15. The photosensing component according to any one of claims 1 to 11, characterized in that, The transmitter is used to emit a first infrared light and a second infrared light, K first filter channels are used to sense the first infrared light, and at least a portion of L second filter channels are used to sense the second infrared light.

16. The photosensitive component according to any one of claims 1 to 15, characterized in that, The image sensor includes a first operating mode and a second operating mode, wherein... When the image sensor is in the first working mode, the image sensor is used to acquire a first image frame when the transmitter is in the off state and / or acquire a second image frame when the transmitter is in the on state, and the first image frame and / or the second image frame is used to acquire the spectral information of the target object; When the image sensor is in the second working mode, the image sensor is used to continuously acquire at least one third image frame when the transmitter is in the on state. The at least one third image frame is used to acquire the depth information of the target object, wherein the exposure time of the at least one third image frame is the same.

17. The photosensitive component according to claim 16, characterized in that, When the image sensor is in the second working mode, the image sensor is also used to acquire a fourth image frame when the transmitter is in the off state. The fourth image frame is used to acquire environmental interference information, and the exposure time of the fourth image frame is the same as the exposure time of the third image frame. and / or When the image sensor is in the first working mode, the image sensor is also used to acquire a fifth image frame when the transmitter is in the off state. The fifth image frame is used to acquire environmental interference information, and the exposure time of the fifth image frame is the same as the exposure time of the second image frame.

18. The photosensitive component according to any one of claims 1 to 17, characterized in that, The light sensing component also includes: The processing unit is used to acquire the depth information and spectral information of the target object based on the electrical signal.

19. The photosensitive component according to claim 18, characterized in that, The spectral information of the target object includes the spectral information of the target object based on ambient light and the spectral information of the target object based on infrared light emitted by the transmitter; The processing unit is used to perform channel splitting on the first image frame to obtain data for each of the Q filter channels. The data for each of the Q filter channels is used to obtain the spectral information of the target object based on ambient light. The first image frame is acquired by the image sensor when the transmitter is in the off state. The processing unit is further configured to extract data from the third filter channel from the second image frame and the fifth image frame respectively. The data from the third filter channel is used to obtain the spectral information of the infrared light emitted by the target object based on the emitter. The third filter channel is the filter channel among the Q filter channels used to sense the infrared light emitted by the emitter. The second image frame is acquired by the image sensor when the emitter is in the open state, and the fifth image frame is acquired by the image sensor when the emitter is in the closed state. The exposure time of the second image frame is the same as the exposure time of the fifth image frame. The processing unit is further configured to extract data from the third filter channel for each of the at least one third image frame and the fourth image frame, respectively. The data from the third filter channel is used to obtain the depth information of the target object. The at least one third image frame is continuously acquired by the image sensor when the transmitter is in the on state, and the fourth image frame is acquired by the image sensor when the transmitter is in the off state. The exposure time of the third image frame is the same as the exposure time of the fourth image frame.

20. The photosensitive component according to claim 18 or 19, characterized in that, The processing unit is used to obtain the depth information of the target object based on the direct flight time or the indirect flight time.

21. An image sensor, characterized in that, The image sensor is used in a light sensing component, which further includes an emitter for emitting infrared light toward a target object. The image sensor is used to receive light signals from the target object, the light signals including infrared light emitted by the emitter. The image sensor includes a beam splitter layer and a photoelectric conversion layer stacked together. The beam-splitting layer includes multiple beam-splitting units, each beam-splitting unit including Q filter channels. At least a portion of the Q filter channels are used to acquire the spectral information of the target object. The Q filter channels include K first filter channels and L second filter channels. The first filter channels are used to sense the infrared light emitted by the emitter. The spectral transmittance curve of the second filter channel is different from that of the first filter channel, where 1≤K<Q, 1≤L<Q, L+K=Q, and K, L, and Q are all integers. The photoelectric conversion layer includes a pixel array, and each of the Q filter channels covers at least one pixel in the pixel array. The pixel array is used to convert the light signal transmitted through the beam splitter into an electrical signal.

22. The image sensor according to claim 21, characterized in that, The image sensor further includes a circuit layer for receiving electrical signals from the photoelectric conversion layer; wherein... The circuit layer is disposed between the beam splitting layer and the photoelectric conversion layer, or the photoelectric conversion layer is disposed between the beam splitting layer and the circuit layer.

23. The image sensor according to claim 21 or 22, characterized in that, The image sensor further includes a microlens layer stacked with the beam splitter layer, wherein the beam splitter layer is disposed between the microlens layer and the photoelectric conversion layer.

24. The image sensor according to claim 23, characterized in that, The image sensor further includes a planarization layer disposed between the microlens layer and the beam splitting layer.

25. The image sensor according to any one of claims 21 to 24, characterized in that, The Q filter channels are arranged in a matrix, wherein in at least one row and / or at least one column of the matrix, some of the first filter channels are arranged adjacent to some of the second filter channels.

26. The image sensor according to claim 25, characterized in that, K of the first filter channels are arranged at intervals in at least one row and / or at least one column of the matrix.

27. The image sensor according to claim 25 or 26, characterized in that, In each row and column of the matrix, at least one second filter channel is included between two adjacent first filter channels.

28. The image sensor according to any one of claims 25 to 27, characterized in that, The spectral transmittance curves of any two of the L second filter channels are different.

29. The image sensor according to any one of claims 25 to 27, characterized in that, In the row direction of the matrix, the filter channel adjacent to the first filter channel has a first spectral transmittance curve; In the column direction of the matrix, the filter channel adjacent to the first filter channel has a second spectral transmittance curve.

30. The image sensor according to claim 29, characterized in that, The first spectral transmittance curve is the same as the second spectral transmittance curve.

31. The image sensor according to any one of claims 25 to 28, characterized in that, The filter channels surrounding the first filter channel have different spectral transmittance curves.

32. The image sensor according to any one of claims 21 to 31, characterized in that, The optical signal also includes ambient light, and L second filter channels are used to sense the ambient light.

33. The image sensor according to claim 32, characterized in that, The spectral transmittance curves of the K first filter channels are the same.

34. The image sensor according to claim 32, characterized in that, The transmitter is used to emit a first infrared light and a second infrared light, and a portion of the K first filter channels is used to sense the first infrared light, and a portion of the K first filter channels is used to sense the second infrared light.

35. The image sensor according to any one of claims 21 to 31, characterized in that, The transmitter is used to emit a first infrared light and a second infrared light, K first filter channels are used to sense the first infrared light, and at least a portion of L second filter channels are used to sense the second infrared light.

36. The image sensor according to any one of claims 21 to 35, characterized in that, The image sensor includes a first operating mode and a second operating mode, wherein... When the image sensor is in the first working mode, the image sensor is used to acquire a first image frame when the transmitter is in the off state and / or acquire a second image frame when the transmitter is in the on state, and the first image frame and / or the second image frame is used to acquire the spectral information of the target object; When the image sensor is in the second working mode, the image sensor is used to continuously acquire at least one third image frame when the transmitter is in the on state. The at least one third image frame is used to acquire the depth information of the target object, wherein the exposure time of the at least one third image frame is the same.

37. The image sensor according to claim 36, characterized in that, When the image sensor is in the second working mode, the image sensor is also used to acquire a fourth image frame when the transmitter is in the off state. The fourth image frame is used to acquire environmental interference information, and the exposure time of the fourth image frame is the same as the exposure time of the third image frame. and / or When the image sensor is in the first working mode, the image sensor is also used to acquire a fifth image frame when the transmitter is in the off state. The fifth image frame is used to acquire environmental interference information, and the exposure time of the fifth image frame is the same as the exposure time of the second image frame.

38. The image sensor according to any one of claims 21 to 37, characterized in that, The image sensor also includes: The processing unit is used to acquire the depth information and spectral information of the target object based on the electrical signal.

39. The image sensor according to claim 38, characterized in that, The spectral information of the target object includes the spectral information of the target object based on ambient light and the spectral information of the target object based on infrared light emitted by the transmitter; The processing unit is used to perform channel splitting on the first image frame to obtain data for each of the Q filter channels. The data for each of the Q filter channels is used to obtain the spectral information of the target object based on ambient light. The first image frame is acquired by the image sensor when the transmitter is in the off state. The processing unit is further configured to extract data from the third filter channel from the second image frame and the fifth image frame respectively. The data from the third filter channel is used to obtain the spectral information of the infrared light emitted by the target object based on the emitter. The third filter channel is the filter channel among the Q filter channels used to sense the infrared light emitted by the emitter. The second image frame is acquired by the image sensor when the emitter is in the open state, and the fifth image frame is acquired by the image sensor when the emitter is in the closed state. The exposure time of the second image frame is the same as the exposure time of the fifth image frame. The processing unit is further configured to extract data from the third filter channel for each of the at least one third image frame and the fourth image frame, respectively. The data from the third filter channel is used to obtain the depth information of the target object. The at least one third image frame is continuously acquired by the image sensor when the transmitter is in the on state, and the fourth image frame is acquired by the image sensor when the transmitter is in the off state. The exposure time of the third image frame is the same as the exposure time of the fourth image frame.

40. The image sensor according to claim 38 or 39, characterized in that, The processing unit is used to obtain the depth information of the target object based on the direct flight time or the indirect flight time.

41. A camera module, characterized in that, The device includes a lens assembly and a light-sensing component as described in any one of claims 1 to 20, wherein a light signal from the target object is imaged on an image sensor in the light-sensing component after passing through the lens assembly.

42. An electronic device, characterized in that, It includes a light sensing component as described in any one of claims 1 to 20, or an image sensor as described in any one of claims 21 to 40, or a camera module as described in claim 41.

43. An image data processing method, characterized in that, Applied to an electronic device, the electronic device including a light-sensing component as described in any one of claims 1 to 20, the method includes: Image data is acquired based on the light sensing component, and the image data includes the depth information and spectral information of the target object; The processing result is output based on the depth information and spectral information of the target object.

44. The method according to claim 43, characterized in that, The spectral information of the target object includes spectral information based on ambient light and spectral information based on infrared light emitted by the transmitter.

45. The method according to claim 43 or 44, characterized in that, The image data is used for facial recognition of the target object, and the step of outputting processing results based on the depth information and spectral information of the target object includes: A first recognition result is obtained based on the depth information of the target object. The first recognition result includes information on whether the target object is a real person and / or information on whether the depth information of the target object matches a pre-stored first facial feature. A second recognition result is obtained based on the spectral information of the target object. The second recognition result includes information on whether the target object is a real person and / or information on whether the spectral information of the target object matches a pre-stored second facial feature. The final recognition result is output based on the first recognition result and the second recognition result.

46. ​​The method according to claim 43 or 44, characterized in that, The image data is used for material identification of the target object, and the step of outputting processing results based on the depth information and spectral information of the target object includes: A first recognition result is obtained based on the depth information of the target object; A second recognition result is obtained based on the spectral information of the target object; The final recognition result is output based on the first recognition result and the second recognition result.

47. The method according to claim 44, characterized in that, The image data is used for 3D modeling, and the step of outputting processing results based on the depth information and spectral information of the target object includes: The depth model of the target object is obtained based on the depth information of the target object; The color information of the target object is obtained based on the spectral information of ambient light; The grayscale image information of the target object is obtained based on the spectral information of the infrared light emitted by the transmitter; Output a 3D model of the target object based on the depth model, color information, and grayscale image information of the target object.

48. An electronic device, characterized in that, include: One or more processors; One or more memory units; The one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 43 to 47.

49. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 43 to 47.

Images