A multispectral depth camera, spectral and depth measurement method

Abstract

The present application belongs to the field of optical measurement, and particularly relates to a multispectral depth camera, a spectral and depth measurement method. The multispectral depth camera comprises a transmitting module, a collecting module and a control processor. The transmitting module comprises a wide-spectrum light source and a pulsed light source. The wide-spectrum light source is used for continuously transmitting a wide-spectrum light beam towards a target, and the pulsed light source is used for transmitting a pulsed light beam of a predetermined wavelength towards the target. The collecting module comprises an image sensor, which comprises a plurality of pixels. The plurality of pixels are used for respectively receiving a plurality of different sub-beams of different predetermined wavebands in the wide-spectrum light beam reflected by the target, and for receiving the pulsed light beam reflected by the target. The control processor is used for calculating depth data and spectral data of the target according to charge signals generated by the pixels. The present application can simultaneously measure spectral information and depth information of the target.

Description

Technical Field

[0001] This invention belongs to the field of optical measurement technology, and particularly relates to a multispectral depth camera and a method for measuring spectrum and depth. Background Technology

[0002] Depth information enables the analysis of a target's three-dimensional information, while spectral information reveals the target's response to various spectra, facilitating qualitative and even quantitative analysis. However, current cameras have limited functionality; depth cameras can only measure the target's depth, and multispectral cameras can only measure its spectral information. Therefore, a terminal must integrate both depth and multispectral cameras to measure both depth and spectral information of the target. Summary of the Invention

[0003] The purpose of this application is to provide a multispectral depth camera and a method for measuring spectrum and depth, aiming to solve the problem that the terminal must integrate a depth camera and a multispectral camera to measure the depth and spectral information of a target.

[0004] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0005] In a first aspect, a multispectral depth camera is provided, comprising a transmission module, an acquisition module, and a control processor. The transmission module includes a broadband light source and a pulsed light source. The broadband light source is used to continuously emit a broadband beam toward the target, and the pulsed light source is used to emit a pulsed beam of a predetermined wavelength toward the target. The acquisition module includes an image sensor, which includes multiple pixels. The multiple pixels are used to receive multiple sub-beams of different predetermined wavelengths from the broadband beam reflected back from the target, and to receive the reflected pulsed beam. The control processor is used to receive the charge signals generated by each pixel and to calculate the depth data and spectral data of the target based on the charge signals.

[0006] In some embodiments, the wavelength range of the broadband light beam is 400 nm to 1000 nm, and the wavelength of the pulsed light beam is 850 nm or 940 nm. In some embodiments, a pixel includes a filter and a photosensitive chip. The filter allows sub-beams of a predetermined wavelength band to pass through. The filter curves of the filters of different pixels are different, and all of them allow the pulsed light beam to pass through. The photosensitive chip is used to receive the light beam filtered by the filter.

[0007] In some embodiments, the photosensitive chip includes a photoelectric conversion element and three taps. The photoelectric conversion element receives the charge generated by the reflected sub-beam and pulsed beam. The three taps sequentially and staggeredly acquire the charge to obtain a first charge, a second charge, and a third charge. The control processor calculates the depth value and spectral response of the corresponding pixel based on the first charge, the second charge, and the third charge. The control processor calculates the depth value of the corresponding pixel according to the following formula: Where Q1 is the charge collected by the first tap that collects the charge generated by the pulse beam, Q2 is the charge collected by the second tap that collects the charge generated by the pulse beam, Q0 is the charge collected by the tap that collects only the charge generated by the sub-beam, m = n-1, where n is the tap number corresponding to Q1, T h This refers to the pulse width of the transmitting module and its three taps. In some embodiments, the control processor uses Q0 as the spectral response of the corresponding pixel, and the target's spectral data includes the spectral responses of each pixel.

[0008] Secondly, embodiments of this application also provide a method for measuring spectrum and depth, including: controlling a broadband light source to continuously emit a broadband light beam toward a target, and a pulse light source to emit a pulse light beam toward the target; controlling an image sensor to receive reflected sub-beams and pulse beams, the image sensor including multiple pixels, the multiple pixels being used to receive multiple sub-beams of different predetermined bands in the broadband light beam emitted by the target, and all receiving reflected pulse beams; receiving charge signals generated by each pixel, and calculating the spectral data and depth data of the target based on the charge signals.

[0009] In some embodiments, a pixel includes a filter and a photosensitive chip. The filter allows sub-beams of a predetermined wavelength to pass through. The filter curves of filters for different pixels are different, and all allow pulsed beams to pass through. The photosensitive chip is used to receive the beam filtered by the filter. The photosensitive chip includes a photoelectric conversion element and three taps. The photoelectric conversion element receives the reflected sub-beams and pulsed beams to generate charge. The three taps sequentially and staggeredly collect the charge to obtain a first charge, a second charge, and a third charge. The spectral data and depth data of the target are calculated based on the charge signals, including: calculating the depth value and spectral response of the corresponding pixel based on the first charge, the second charge, and the third charge.

[0010] In some embodiments, the depth value is calculated according to the following formula: Where Q1 is the charge collected by the first tap that collects the charge generated by the pulse beam, Q2 is the charge collected by the second tap that collects the charge generated by the pulse beam, Q0 is the charge collected by the tap that collects only the charge generated by the sub-beam, m = n-1, where n is the tap number corresponding to Q1, T h This refers to the pulse width of the transmitting module and the three taps. In some embodiments, the spectral response of a pixel is Q0, and the spectral data of the target includes the spectral response of each pixel.

[0011] The beneficial effects of this application are as follows: By designing the image sensor in the emission module and the acquisition module, when measuring the target, the broadband light source continuously emits multiple sub-beams of different wavelengths towards the target, and the pulse light source emits pulse beams towards the target. Multiple pixels respectively acquire the reflected sub-beams and pulse beams of multiple predetermined wavelengths to generate corresponding charge signals. The control processor analyzes and processes the charge signals generated by each pixel to calculate the spectral data and depth data of the target, thereby realizing the spectral measurement and depth measurement of the target. Thus, the multispectral depth camera of this application has two functions: spectral measurement and depth measurement, and its application range is wider. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or exemplary technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating the structural principle of the multispectral depth camera provided in the embodiments of this application;

[0014] Figure 2 This is a partial structural schematic diagram of the image sensor provided in an embodiment of this application;

[0015] Figure 3 This is a schematic diagram of the pixel structure provided in the embodiments of this application;

[0016] Figure 4 This is a schematic diagram of the filter curve of a nine-pixel filter provided in an embodiment of this application;

[0017] Figure 5 This is a schematic diagram of the optical signal emission and acquisition of the multispectral depth camera provided in the embodiments of this application;

[0018] Figure 6 This is a schematic diagram of the steps of a spectral and depth measurement method in another embodiment of this application.

[0019] The following are the labeling elements in the figure:

[0020] 100. Multispectral depth camera; 10. Emission module; 11. Broadband light source; 12. Pulsed light source; 13. Diffuser; 14. Housing; 20. Acquisition module; 21. Lens; 22. Image sensor; 23. Lens barrel; 30. Control processor; 220. Pixel unit; 221. First pixel; 222. Second pixel; 223. Third pixel; 224. Fourth pixel; 225. Fifth pixel; 226. Sixth pixel; 227. Seventh pixel; 228. Eighth pixel; 229. Ninth pixel; 2211. Filter; 2212. Photosensitive chip; 2213. Microlens. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of this application.

[0022] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly or indirectly attached to that other component. When a component is referred to as "connected to" another component, it can be directly or indirectly connected to that other component. The terms "upper," "lower," "left," "right," etc., indicate orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, and are for ease of description only, not to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features. "A plurality" means two or more, unless otherwise explicitly defined.

[0023] Please see Figure 1 This application provides a multispectral depth camera 100, which has both spectral measurement and depth measurement functions. The camera has higher functional integration and a wider range of applications. The multispectral depth camera 100 includes a transmission module 10, a data acquisition module 20, and a control processor 30.

[0024] The emission module 10 includes a broadband light source 11 and a pulsed light source 12. The broadband light source 11 continuously emits a broadband beam towards the target. The wavelength range of the broadband beam is between 400 nm and 1000 nm, meaning the emission spectrum of the broadband light source 11 covers the visible and near-infrared bands, enriching the spectral information in the subsequently generated multispectral image. The broadband beam consists of sub-beams with wavelengths between 400 nm and 1000 nm. The broadband light source 11 can be a broadband LED lamp, a halogen lamp, etc., which are not listed here. The pulsed light source 12 emits a pulsed beam of a specific wavelength towards the target. The pulsed light source 12 is turned on and off according to a specific timing sequence to emit the pulsed beam. The wavelength emitted by the pulsed light source 12 is an infrared wavelength, such as 850 nm or 940 nm. The pulsed light source 12 can be a vertical-cavity surface-emitting laser (VCSEL) or other laser source.

[0025] Please see Figure 1 The emission module 10 also includes a diffuser 13 and a housing 14. The broadband light source 11 and the pulsed light source 12 are both installed inside the housing 14. The diffuser 13 is installed in the housing 14 and is positioned relative to the broadband light source 11 and the pulsed light source 12 to diffuse the light beams emitted by the broadband light source 11 and the pulsed light source 12 and project uniform surface light.

[0026] Please see Figure 1 The acquisition module 20 includes an image sensor 22, which includes at least one pixel unit 220, and the pixel unit 220 includes multiple pixels (e.g., ...). Figure 2 In the diagram, 221 to 229 represent nine different pixels. These pixels are used to receive sub-beams of different wavelengths from the reflected broadband beam, and also to receive pulsed beams, thereby obtaining spectral data of multiple different wavelengths and phase information of the pulsed beams. The multiple pixels operate at the same timing to avoid pixel data asynchrony. In one embodiment, the multiple different wavelengths include multiple different visible light bands and multiple different infrared light bands to enhance spectral diversity.

[0027] like Figure 3 and Figure 4As shown, each pixel includes a filter 2201 and a photosensitive chip 2202. The filter curves of the filters 2201 of different pixels are different, filtering out non-target sub-beams in the broadband beam and allowing the corresponding target sub-beams (i.e., sub-beams of the corresponding wavelength band) to pass through. Thus, different pixels receive sub-beams of different wavelengths to facilitate the measurement of the target's spectral information. However, the filters 2201 of different pixels all allow pulse beams to pass through, so as to calculate the phase shift based on the received pulse beams and then calculate the time of flight and depth data. For example, some filters 2201 allow green light to pass through, some filters 2201 allow red light to pass through, some filters 2201 allow blue light to pass through, some filters 2201 allow yellow light to pass through, and some filters 2201 allow infrared light to pass through. The filters 2201 of multiple pixels can be selected according to the application scenario. In one embodiment, the sub-beams received by multiple pixels include visible light beams (green, red, blue, yellow, etc.) and near-infrared beams (e.g., 760nm, 850nm, 940nm, etc.), thereby obtaining visible light spectrum and near-infrared light spectrum information.

[0028] The photosensitive chip 2202 is used to receive the light beam (including the reflected target sub-beam and pulsed beam) filtered by the filter 2201 and generate a charge. The photosensitive chip 2202 includes a photoelectric conversion element, a first tap, a second tap, and a third tap. The photoelectric conversion element receives the light beam (including the target sub-beam and pulsed beam) and generates a charge. The first tap, second tap, and third tap are used to sequentially and staggeredly acquire the charge generated by the photoelectric conversion element to obtain a first charge, a second charge, and a third charge, respectively. The first tap, second tap, and third tap have different operating timings but the same pulse width.

[0029] The control processor 30 is connected to both the transmitting module 10 and the acquisition module 20. It controls the activation and deactivation of the transmitting module 10 and the acquisition module 20. The control processor 30 receives the charge signals generated by each pixel and calculates the spectral response and depth value of each pixel to obtain a depth image and a multispectral image of the target. For example, the charge signal includes a first charge, a second charge, and a third charge acquired by three taps. The control processor 30 receives, analyzes, and processes the first, second, and third charge values, calculates the spectral response and depth value corresponding to each pixel, generates a depth image based on the depth values ​​of all pixels, and generates a multispectral image based on the spectral data of all pixels.

[0030] The control processor 30 calculates the depth data d according to the following formula (1) and the spectral response S according to the following formula (2).

[0031]

[0032] S = min(Q0, Q1, Q2); (2)

[0033] In equations (1) and (2) above, Q1 is the charge collected by the first tap that collects the charge generated by the pulsed beam, Q2 is the charge collected by the second tap that collects the charge generated by the pulsed beam, Q0 is the charge collected by the tap that collects the charge generated only by receiving ambient light, m ​​= n-1, n is the number of the tap corresponding to Q1, and T h The pulse width of the transmission module 10 and its three taps is given by C, which represents the speed of light. The numbers of the first tap, the second tap, and the third tap are 1, 2, and 3, respectively. min(Q0, Q1, Q2) refers to the minimum value among Q0, Q1, and Q2.

[0034] The multispectral depth camera provided in this embodiment, through the design of the image sensor in the emission module 10 and the acquisition module 20, when measuring the target, continuously emits multiple sub-beams of different wavelengths towards the target from a broadband light source, and emits pulse beams towards the target from a pulse light source. Multiple pixels respectively acquire the reflected sub-beams and pulse beams of multiple predetermined wavelengths, generating corresponding charge signals. The control processor 30 analyzes and processes the charge signals generated by each pixel, and calculates the spectral data and depth data of the target respectively, thereby realizing the simultaneous measurement of the spectral information and depth information of the target, with high measurement efficiency and a wider range of applications.

[0035] like Figure 2 As shown, to more clearly illustrate the technical solution of this application, this application uses a pixel unit 220 comprising nine pixels as an example, with the nine pixels arranged in a 3*3 format, as shown. Figure 4 As shown, Figure 4 (a) to Figure 4 (i) The filter curves of the nine pixels are shown respectively. Each filter is different so that a single pixel unit 220 can acquire nine spectral responses, which helps to improve the accuracy of subsequent applications such as face recognition and liveness detection based on multispectral images. The nine pixels are the first pixel 221, the second pixel 222, the third pixel 223, the fourth pixel 224, the fifth pixel 225, the sixth pixel 226, the seventh pixel 227, the eighth pixel 228, and the ninth pixel 229.

[0036] Taking the first pixel 221 as an example, let's explain the specific details of how the control processor 30 calculates the depth value d1 and spectral response S1 of the first pixel 221. Figure 6 As shown, Figure 6In the diagram, L1 represents the operating timing of the broadband light source 11, L2 represents the operating timing of the pulsed light source 12, R1 represents the reflected pulsed beam signal, and A1, B1, and C1 represent the operating timing of the first tap, second tap, and third tap of the first pixel 221, respectively. A high level in L1, L2, A1, B1, and C1 indicates the light source is on, and a low level indicates it is off. During the measurement process, the broadband light source 11 remains continuously on, and both the first tap and the pulsed light source 12 are at T... h1 The time period begins, and the second tap occurs at T. h2 The time period begins, and the third tap occurs at T. h3 When the time period begins, the first photoelectric conversion element is in T h1 T h2 T h3 The receiving beam is activated and a charge is generated during all three time periods. There is a certain time difference between the transmission and reception of the pulse beam, which is the flight time t. This application obtains the flight time t by calculating the phase deviation between the transmitted pulse beam and the reflected pulse beam.

[0037] like Figure 6 As shown, if in the first period T p1 T inside h1 During the time period, the pulsed beam is reflected to the first photoelectric conversion element and received after a flight time t. Then, in time T... h1 During a given time period, the first photoelectric conversion element receives the first sub-beam and the pulse beam of the first predetermined wavelength band, generating a charge Q. A1 Collected by the first tap, Q A1 Including the sub-charge Q generated by the first sub-beam in the broadband beam. a11 and the sub-charge Q generated by receiving the pulsed beam a12 ; in T h2 During a certain period, the first photoelectric conversion element simultaneously receives both the first sub-beam and the pulsed beam; during another period, it receives only the first sub-beam, thus generating a charge Q. B1 The charge Q is collected by the second tap. C1 Including the charge Q generated by receiving the first sub-beam. b11 and the amount of charge Q generated by receiving a pulsed beam b12 ; in T h3 During the time period, the first photoelectric conversion element only receives the first sub-beam and generates a charge Q. C1 Collected by the third tap; among them, Q a11 =Q b11 =Q C1 Q B1 -Q C1 This represents the amount of charge (Q) generated by the pulsed beam and collected by the second tap. b12 ), Q A1 +Q B1 -2QC1 Indicates a pulse width T h The amount of charge (Q) generated by the pulsed beam and collected internally. a12 +Q b12 If n=1 and m=0, then... The spectral response is the target's response to the first sub-beam; therefore, the spectral data S1 = Q. C1 .

[0038] If in T h1 During the time period, the pulsed beam was not reflected to the first photoelectric conversion element, while at T h2 When the time period begins to reflect to the first photoelectric conversion element, then at T h1 During the time period, the first photoelectric conversion element only receives the first sub-beam and generates a charge Q. A1 Collected by the first tap, T h2 During certain time periods, only the first sub-beam is received; during other time periods, both the first sub-beam and the pulse beam are received, generating a charge Q. B1 The charge Q is collected by the second tap. B1 Including the charge Q generated by receiving the first sub-beam b11 and the charge Q generated by the received pulse beam b12 T h3 During the time period, the system receives both the first sub-beam and the pulse beam; during other time periods, it receives only the first sub-beam, generating a charge Q. C1 The charge Q is collected by the third tap. C1 Including the charge Q generated by receiving the first sub-beam c11 and the charge Q generated by the received pulse beam c12 Among them, Q A1 =Q b11 =Q c11 Q C1 -Q A1 This represents the amount of charge (Q) generated by the pulsed beam and collected by the third tap. c12 ), Q C1 +Q B1 -2Q A1 Indicates a pulse width T h The amount of charge (Q) generated by the pulsed beam and collected internally. b12 +Q c12 ), and then T can be calculated. h2 Flight time within the time period, then add T h1 The total flight time t can be obtained from the time interval, that is, the above n=2, m=1, S1 = Q A1 .

[0039] If in T h1 and T h2During the period, the pulsed beam was not reflected back to the first photoelectric conversion element, while at T h3 The time period begins with reflection onto the first photoelectric conversion element and continues until the next cycle T. p2 T h1 Time period. That is, a single frame measurement includes two periods Tp, in T h2 During the time period, the first photoelectric conversion element only receives the charge Q generated by the first sub-beam. B1 Collected by the second tap, at T h3 During certain time periods, the first photoelectric conversion element receives only the first sub-beam; during other time periods, it receives both the first sub-beam and the pulsed beam, thereby generating a charge Q. C1 The charge Q is collected by the third tap. C1 Including the charge Q generated by receiving the first sub-beam c11 and the charge Q generated by the received pulse beam c12 In the next period T p2 T h1 During certain time periods, the first photoelectric conversion element receives both the first sub-beam and the pulsed beam; during other time periods, it receives only the first sub-beam, generating a charge Q. A1 The charge Q is collected by the first tap. A1 Including the charge Q generated by receiving the first sub-beam a11 and the charge Q generated by the received pulse beam a12 Among them, Q B1 =Q c11 =Q a11 Q A1 -Q B1 Q represents the amount of charge generated by the pulsed beam collected by the first tap. A1 +Q C1 -2Q B1 Indicates a pulse width T h The amount of charge generated by the pulsed beam is collected internally, and T can then be calculated. h3 The flight time t within the time period, then added to T h1 and T h2 The total flight time can then be obtained, i.e., when n=3 and m=2 above. S1 = Q B1 .

[0040] The second pixel 222, the third pixel 223, the fourth pixel 224, the fifth pixel 225, the sixth pixel 226, the seventh pixel 227, the eighth pixel 228, and the ninth pixel 229 are similar to the first pixel 221, except that there are certain differences in the received sub-beams. Referring to the calculation method of the first pixel 221, the depth d2 of the second pixel 222 and its response to a broadband light source S2, the depth d3 of the third pixel 223 and its response to a broadband light source S3, the depth d4 of the fourth pixel 224 and its response to a broadband light source S4, the depth d5 ​​of the fifth pixel 225 and its response to a broadband light source S5, the depth d6 of the sixth pixel 226 and its response to a broadband light source S6, the depth d7 of the seventh pixel 227 and its response to a broadband light source S7, the depth d8 of the eighth pixel 228 and its response to a broadband light source S8, and the depth d9 of the ninth pixel 229 and its response to a broadband light source S9 can be calculated.

[0041] The above method allows for the calculation of nine depth information values ​​for pixel unit 220. The spectral responses of these nine pixels constitute one spectral information value for pixel unit 220, which is a 1*9 vector [S1, S2, S3, S4, S5, S6, S7, S8, S9]. The number of pixels 221 to 229 in pixel unit 220 can be one or more, without limitation. After calculating the depth and spectral information of all pixel units 220, a depth image can be generated based on the depth values ​​of all pixels, and a multispectral image can be generated based on the spectral responses of all pixels. The multispectral image is a 9-channel image with spectral data in nine different bands, thus providing the target's depth and spectral information.

[0042] In one embodiment, if a pixel cannot be calculated for depth information, for example, if a pixel cannot receive a pulse beam or receives a small number of pulse beams, the control processor 30 can calculate the depth information of the pixel by interpolating the depth values ​​of neighboring pixels.

[0043] Please see Figures 1 to 2 In some embodiments, the acquisition module 20 further includes a lens 21 and a lens barrel 23. The image sensor 22 is mounted inside the lens barrel 23, and the lens 21 is mounted in the lens barrel 23 and located on the light-incident side of the image sensor 22, for converging the incident light beam to the image sensor 22. The pixel also includes a microlens 2203 disposed on the filter 2201 for converging the light beam.

[0044] The present invention also proposes a spectral and depth measurement method, which uses the above-described multispectral depth camera 100 to image the target. The specific structure of the multispectral depth camera 100 is as described in the above embodiments. Since this spectral and depth measurement method adopts all the technical solutions of all the above embodiments, it also has all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0045] like Figure 6 As shown, the spectral and depth measurement method includes the following steps:

[0046] 601. Control the broadband light source to continuously emit a broadband beam of light toward the target, and the pulse light source to emit a pulse beam of light toward the target;

[0047] 602. Control the image sensor to receive the reflected sub-beams and pulse beams. The image sensor includes multiple pixels, which are used to receive multiple sub-beams of different predetermined bands in the broadband beam emitted by the target, and all receive the reflected pulse beams.

[0048] 603. Receive the charge signals generated by each pixel, and calculate the spectral data and depth data of the target based on the charge signals.

[0049] The specific details of each step in the spectral and depth measurement method can be found in the relevant content of the above-mentioned multispectral depth camera embodiment, and will not be repeated here.

[0050] This invention also proposes a terminal, which includes a multispectral depth camera. The specific structure of the multispectral depth camera is as described in the above embodiments. Since this terminal adopts all the technical solutions of all the above embodiments, it also has all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here. The terminal can be a mobile phone, smart door lock, access control device, facial recognition payment device, etc., which will not be listed here.

[0051] It should be noted that, as Figure 4 As shown in the illustration, this application embodiment only uses the example of pixel unit 220 including nine different types of pixels, each pixel including three taps, for explanation. In other embodiments, pixel unit 220 may also include other numbers of pixels, such as 4, 5, 6, 7, 8 or more, and each pixel may also include 2, 4, 5 or more taps. The working principle and calculation method are similar, and will not be elaborated or limited here. In reality, when a multispectral depth camera measures the spectrum and depth of a target, there will also be the influence of ambient light. The above calculation of depth values ​​and spectral response does not consider ambient light.

[0052] The above are merely optional embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multispectral depth camera, characterized in that, include: The emission module includes a broadband light source and a pulsed light source. The broadband light source is used to continuously emit a broadband light beam toward the target, and the pulsed light source is used to emit a pulsed light beam of a predetermined wavelength toward the target. The wavelength range of the broadband light beam is 400 nm to 1000 nm, and the wavelength of the pulsed light beam is 850 nm or 940 nm. The acquisition module includes an image sensor, which includes multiple pixels. The multiple pixels are used to receive multiple sub-beams of different predetermined bands in the broadband beam reflected back by the target, and to receive the pulse beam reflected back by each of them. as well as A control processor is configured to receive charge signals generated by each pixel and calculate the depth data and spectral data of the target based on the charge signals. The pixel includes three taps, namely a first tap, a second tap, and a third tap. During the process of the broadband light source continuously emitting a broadband light beam toward the target, the broadband light source is continuously turned on. The first tap and the pulse light source are both turned on during the Th1 period, the second tap is turned on during the Th2 period, and the third tap is turned on during the Th3 period.

2. The multispectral depth camera as described in claim 1, characterized in that, The pixel includes a filter and a photosensitive chip. The filter allows sub-beams of a predetermined wavelength to pass through. The filter curves of the filters of different pixels are different, but all allow the pulse beam to pass through. The photosensitive chip is used to receive the beam after it has been filtered by the filter.

3. The multispectral depth camera as described in claim 2, characterized in that, The photosensitive chip includes a photoelectric conversion element, which is used to receive the reflected sub-beam and the charge generated by the pulse beam. The three taps are staggered in sequence to collect the charge to obtain a first charge, a second charge, and a third charge. The control processor is used to calculate the depth value and spectral response of the corresponding pixel based on the first charge, the second charge, and the third charge.

4. The multispectral depth camera as described in claim 3, characterized in that, The control processor calculates the depth value corresponding to the pixel according to the following formula: Wherein, Q1 is the charge collected by the first tap that collects the charge generated by the pulse beam, Q2 is the charge collected by the second tap that collects the charge generated by the pulse beam, Q0 is the charge collected by the tap that collects only the charge generated by the sub-beam, m=n-1, n is the sequence number of the tap corresponding to Q1, T h C represents the pulse width of the emission module and the three taps, and C represents the speed of light.

5. The multispectral depth camera as described in claim 4, characterized in that, The control processor is used to take Q0 as the spectral response corresponding to the pixel, and the spectral data of the target includes the spectral response of each pixel.

6. A multispectral and depth measurement method, using a multispectral depth camera as described in any one of claims 1-5, characterized in that, The multispectral and depth measurement method includes the following steps: Control a broadband light source to continuously emit a broadband light beam toward a target, and a pulsed light source to emit a pulsed light beam toward a target; the wavelength range of the broadband light beam is 400nm to 1000nm, and the wavelength of the pulsed light beam is 850nm or 940nm; The image sensor is controlled to receive the reflected sub-beams and the pulse beams. The image sensor includes multiple pixels, which are used to receive multiple sub-beams of different predetermined bands in the broadband beam emitted by the target, and to receive the reflected pulse beams. Receive the charge signals generated by each pixel, and calculate the spectral data and depth data of the target based on the charge signals; The pixel includes three taps, namely the first tap, the second tap, and the third tap. During the process of the broadband light source continuously emitting a broadband light beam toward the target, the broadband light source is continuously turned on. The first tap and the pulse light source are both turned on during the Th1 period, the second tap is turned on during the Th2 period, and the third tap is turned on during the Th3 period.

7. The multispectral and depth measurement method as described in claim 6, characterized in that, The pixel includes a filter and a photosensitive chip. The filter allows sub-beams of a predetermined wavelength to pass through. The filter curves of the filters of different pixels are different, but all allow the pulsed beam to pass through. The photosensitive chip is used to receive the beam filtered by the filter. The photosensitive chip includes a photoelectric conversion element. The photoelectric conversion element receives the reflected sub-beams and the pulsed beam to generate charge. Three taps are sequentially staggered to collect the charge to obtain a first charge, a second charge, and a third charge. The calculation of the target's spectral data and depth data based on the charge signals includes: The depth value and spectral response of the corresponding pixel are calculated based on the first charge, the second charge, and the third charge.

8. The multispectral and depth measurement method as described in claim 7, characterized in that, The depth value is calculated using the following formula: Wherein, Q1 is the charge collected by the first tap that collects the charge generated by the pulse beam, Q2 is the charge collected by the second tap that collects the charge generated by the pulse beam, Q0 is the charge collected by the tap that collects only the charge generated by the sub-beam, m=n-1, n is the sequence number of the tap corresponding to Q1, T h C represents the pulse width of the emission module and the three taps, and C represents the speed of light.

9. The multispectral and depth measurement method as described in claim 8, characterized in that, The spectral response of the pixel is Q0, and the spectral data of the target includes the spectral response of each pixel.

Images