Multispectral depth camera, spectral and depth measurement method
By designing a multispectral depth camera that emits pulse beams of different wavelengths and calculates spectral and depth data, the problem of single-function cameras is solved, and the integration of depth and spectral measurements is achieved, thus expanding the scope of applications.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN ORBBEC CO LTD
- Filing Date
- 2023-02-16
- Publication Date
- 2026-05-15
AI Technical Summary
Existing cameras have limited functionality and require the integration of depth cameras and multispectral cameras to measure the depth and spectral information of a target.
Design a multispectral depth camera, including a transmitting module and a acquiring module. The transmitting module emits pulse beams of different wavelengths, and the acquiring module receives and calculates the spectral and depth data of the target. Data processing is performed through a control and processor.
This technology enables a single camera to perform both depth and spectral measurements simultaneously, expanding the camera's application scenarios and improving the accuracy and precision of the measurements.
Smart Images
Figure CN116165679B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of optical measurement technology, and particularly relates to a multispectral depth camera and a method for measuring spectrum and depth. Background Technology
[0002] Depth information enables the analysis of a target's three-dimensional information, while spectral information reveals the target's response to various spectra, facilitating qualitative and even quantitative analysis. However, current cameras have limited functionality; depth cameras can only measure the target's depth, and multispectral cameras can only measure its spectral information. Therefore, a terminal must integrate both depth and multispectral cameras to measure both depth and spectral information of the target. Summary of the Invention
[0003] The purpose of this application is to provide a multispectral depth camera and a method for measuring spectrum and depth, aiming to solve the problem that the camera has a single function and the terminal must integrate a depth camera and a multispectral camera to measure the depth and spectral information of the target.
[0004] To achieve the above objectives, the technical solution adopted in this application is as follows:
[0005] In a first aspect, a multispectral depth camera is provided, comprising a transmission module, an acquisition module, and a control and processor; the transmission module is used to transmit a first pulse beam, a second pulse beam, and a third pulse beam of different wavelengths toward a target; the acquisition module includes an image sensor, the image sensor including a first pixel, a second pixel, and a third pixel for receiving the first pulse beam, the second pulse beam, and the third pulse beam reflected back from the target; the control and processor is used to receive charge signals generated by the first pixel, the second pixel, and the third pixel, and calculate the spectral data and depth data of the target based on the charge signals.
[0006] In some embodiments, the transmitting module includes a light source, which includes a first sub-light source emitting a first pulse beam, a second sub-light source emitting a second pulse beam, and a third sub-light source emitting a third pulse beam. In some embodiments, the wavelengths of the first pulse beam, the second pulse beam, and the third pulse beam are a first wavelength, a second wavelength, and a third wavelength, respectively. The first pixel includes a first filter and a photosensitive chip, the second pixel includes a second filter and a photosensitive chip, and the third pixel includes a third filter and a photosensitive chip. The center wavelengths of the first filter, the second filter, and the third filter are the first wavelength, the second wavelength, and the third wavelength, respectively.
[0007] In some embodiments, the photosensitive chip includes a photoelectric conversion element and three taps. The photoelectric conversion element receives ambient light and reflected pulsed light beams to generate charge. The three taps are staggered to acquire the charge generated by the photoelectric conversion element to obtain a first charge, a second charge, and a third charge. The control and processor calculate the depth data and spectral data of the corresponding pixel based on the first charge, the second charge, and the third charge. In some embodiments, the control and processor calculate the depth data according to the following formula: Where Q1 is the charge collected by the first tap that collects the charge generated by the pulsed beam, Q2 is the charge collected by the second tap that collects the charge generated by the pulsed beam, Q0 is the charge collected by the tap that only collects the charge generated by the ambient light, m = n-1, where n is the tap number corresponding to Q1, T h This refers to the pulse width of the transmitting module and its three taps. In some embodiments, the controller and processor calculate the spectral data according to the following formula: S = Q2 + Q1 - 2Q0, where the spectral data includes the spectral responses of the first pixel, the second pixel, and the third pixel.
[0008] Secondly, a method for measuring spectrum and depth is provided, comprising: controlling a transmitting module to simultaneously emit a first pulse beam, a second pulse beam, and a third pulse beam toward a target; controlling a receiving module to receive the first pulse beam, the second pulse beam, and the third pulse beam reflected back from the target; an image sensor including a first pixel, a second pixel, and a third pixel that respectively receive the first pulse beam, the second pulse beam, and the third pulse beam; receiving charge signals generated by the first pixel, the second pixel, and the third pixel; and calculating the spectral data and depth data of the target based on the charge signals.
[0009] In some embodiments, the first pixel, the second pixel, and the third pixel each include a photoelectric conversion element and three taps. The photoelectric conversion element receives ambient light and a reflected pulsed light beam to generate a charge. The three taps sequentially and staggeredly collect the charge to obtain a first charge, a second charge, and a third charge. The spectral data and depth data of the target are calculated based on the charge signals, including: calculating the depth value and spectral response of the corresponding pixel based on the first charge, the second charge, and the third charge. In some embodiments, the depth data of the pixel is calculated using the following formula: Where Q1 is the charge collected by the first tap that collects the charge generated by the pulsed beam, Q2 is the charge collected by the second tap that collects the charge generated by the pulsed beam, Q0 is the charge collected by the tap that only collects the charge generated by the ambient light, m = n-1, where n is the tap number corresponding to Q1, T hThis refers to the pulse width of the transmitting module and the three taps. In some embodiments, the spectral data S of the pixel is calculated according to the following formula: S = Q2 + Q1 - 2Q0, where the spectral data includes the spectral responses of the first pixel, the second pixel, and the third pixel.
[0010] The beneficial effects of this application are as follows: the transmitting module can project a first pulse beam, a second pulse beam, and a third pulse beam with different wavelengths toward the target; the first pixel, the second pixel, and the third pixel in the image sensor can receive the reflected first pulse beam, the second pulse beam, and the third pulse beam, respectively; the control and processor receive and calculate the target's depth data and spectral data based on the charge signals of the first pixel, the second pixel, and the third pixel, thereby realizing depth measurement and spectral measurement of the target. Thus, the multispectral depth camera of this application has two functions: spectral measurement and depth measurement, and its application scenarios are more extensive. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or exemplary technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a schematic diagram of the structure of the multispectral depth camera provided in the embodiments of this application;
[0013] Figure 2 This is a schematic diagram of the structure of the image sensor provided in the embodiments of this application;
[0014] Figure 3 This is a schematic diagram of the structure of a pixel unit provided in an embodiment of this application;
[0015] Figure 4 This is a timing diagram of the operation of the light source provided in the embodiments of this application;
[0016] Figure 5 This is a transmittance curve diagram of the first filter, second filter, and third filter provided in the embodiments of this application;
[0017] Figure 6 This is a schematic diagram illustrating the principle of the first pulse beam transmission and reception of the multispectral depth camera provided in the embodiments of this application;
[0018] Figure 7 This is a flowchart illustrating a spectral and depth measurement method provided in another embodiment of this application.
[0019] The following are the labeling elements in the figure:
[0020] 100. Multispectral depth camera; 10. Transmission module; 101. First pulse beam; 102. Second pulse beam; 103. Third pulse beam; 11. Diffuser; 12. Light source; 13. Housing; 20. Acquisition module; 21. Lens; 22. Image sensor; 23. Lens barrel; 30. Control and processor; 220. Pixel unit; 221. First pixel; 222. Second pixel; 223. Third pixel; 2211. First microlens; 2212. First filter; 2213. First photosensitive chip; 2221. Second microlens; 2222. Second filter; 2223. Second photosensitive chip; 2231. Third microlens; 2232. Third filter; 2233. Third photosensitive chip. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of this application.
[0022] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly or indirectly attached to that other component. When a component is referred to as "connected to" another component, it can be directly or indirectly connected to that other component. The terms "upper," "lower," "left," "right," etc., indicate orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, and are for ease of description only, not to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features. "A plurality" means two or more, unless otherwise explicitly defined.
[0023] Please see Figure 1 This application provides a multispectral depth camera, which has both spectral measurement and depth measurement functions. The camera has a higher degree of integration and a wider range of applications.
[0024] Please see Figures 1 to 3The multispectral depth camera includes a transmission module 10, a data acquisition module 20, and a control and processor 30. The transmission module 10 simultaneously emits a first pulse beam 101, a second pulse beam 102, and a third pulse beam 103 of different wavelengths toward the target. The data acquisition module 20 includes an image sensor 22, which includes a first pixel 221, a second pixel 222, and a third pixel 223. The first pixel 221, the second pixel 222, and the third pixel 223 are respectively used to receive the first pulse beam 101, the second pulse beam 102, and the third pulse beam 103 reflected back from the target. The control and processor 30 receives the charge signals generated by the first pixel 221, the second pixel 222, and the third pixel 223, and calculates the target's spectral and depth data based on the charge signals.
[0025] The multispectral depth camera provided in this application embodiment, through the design of the transmitting module 10 and the acquisition module 20, enables the transmitting module 10 to project a first pulse beam 101, a second pulse beam 102, and a third pulse beam 103 toward the target. The first pixel 221, the second pixel 222, and the third pixel 223 can respectively receive the reflected first pulse beam 101, the second pulse beam 102, and the third pulse beam 103. The control and processor 30 receives and analyzes the charge signals generated by the first pixel 221, the second pixel 222, and the third pixel 223, calculates the target's spectral data and depth data, and then generates the target's depth data and spectral data. Thus, a single camera integrates the functions of spectral measurement and depth measurement, making the camera's application scenarios more extensive.
[0026] Please see Figure 1The transmitting module 10 includes a light source 12, a diffuser 11, and a housing 13. The light source 12 is installed inside the housing 13, and the diffuser 11 is installed in the housing 13 and located in the emission path of the light source 12. The diffuser 11 is used to diffuse the pulsed light beam generated by the light source 11 to form a uniform surface light. The light source 12 includes a first sub-light source that emits a first pulsed light beam 101, a second sub-light source that emits a second pulsed light beam 102, and a third sub-light source that emits a third pulsed light beam 103. The first, second, and third sub-light sources can be staggered to ensure that the emitted first pulsed light beam 101, second pulsed light beam 102, and third pulsed light beam 103 are uniformly distributed. The wavelengths of the first pulsed light beam 101, second pulsed light beam 102, and third pulsed light beam 103 are the first wavelength, second wavelength, and third wavelength, respectively, and the first, second, and third wavelengths are all different. After the first pulsed light beam 101, second pulsed light beam 102, and third pulsed light beam 103 are projected onto the target, they are reflected by the target to the acquisition module 20 and received by the acquisition module 20. The first, second, and third sub-light sources can be laser sources such as vertical-cavity surface-emitting lasers (VCSELs). In one embodiment, the first, second, and third sub-light sources are all infrared laser sources.
[0027] like Figure 4 As shown, Figure 4 L1, L2, and L3 in the diagram represent the operating timing of the first, second, and third sub-light sources, respectively. The operating timing of the first, second, and third sub-light sources is the same, all occurring at T... h1 The time period begins, T h2 and T h3 When the time period is closed, the first, second, and third pulse beams are synchronized, improving the accuracy of depth and multispectral data.
[0028] Please see Figure 1 and Figure 2The acquisition module 20 also includes a lens 21 and a lens barrel 23. The lens 21 is mounted on the lens barrel 23 and located on the light-incident side of the image sensor 22, for converging the light beam to the image sensor 22. The image sensor 22 includes at least one pixel unit 220. Each pixel unit 220 includes a first pixel 221, a second pixel 222, and a third pixel 223, which are arranged linearly in sequence. In other embodiments, the three pixels can also be arranged in other ways, such as the second pixel 222, the third pixel 223, and the first pixel 221 being arranged linearly in sequence. The number of first pixels 221, second pixels 222, and third pixels 223 in a pixel unit 220 is not limited to one; it can be two, three, or more.
[0029] Please see Figures 1 to 3 Along the incident direction of the light beam, the first pixel 221 includes a first filter 2212 and a first photosensitive chip 2213, the second pixel 222 includes a second filter 2222 and a second photosensitive chip 2223, and the third pixel 223 includes a third filter 2232 and a third photosensitive chip 2233. The center wavelengths of the first filter 2212, the second filter 2222, and the third filter 2232 are the first wavelength, the second wavelength, and the third wavelength, respectively. In this way, the first pixel 221, the second pixel 222, and the third pixel 223 can receive the first pulse beam 101, the second pulse beam 102, and the third pulse beam 103, respectively, and it is less likely that the second pixel 222 will receive the first pulse beam 101.
[0030] In one embodiment, a first filter 2212 allows the first pulsed light beam 101 to pass through while blocking the second pulsed light beam 102 and the third pulsed light beam 103; a second filter 2222 allows the second pulsed light beam 102 to pass through while blocking the first pulsed light beam 101 and the third pulsed light beam 103; and a third filter 2232 allows the third pulsed light beam 103 to pass through while blocking the first pulsed light beam 101 and the second pulsed light beam 102. Thus, the charge signals generated by the first pixel 221, the second pixel 222, and the third pixel 223 have relatively high precision, resulting in more accurate multispectral and depth images subsequently generated.
[0031] like Figure 5 As shown, in one embodiment, the first wavelength is 760 nm, the second wavelength is 850 nm, and the third wavelength is 940 nm. Figure 5 (a) Figure 5 (b) and Figure 5(c) The filtering curves of the first filter 2212, the second filter 2222, and the third filter 2232 are shown respectively. All three filters have narrow-band responses, with center wavelengths of 760nm, 850nm, and 940nm respectively. Therefore, the data acquired by the first pixel 221, the second pixel 222, and the third pixel 223 are relatively accurate. In other embodiments, the first, second, and third wavelengths can be selected according to actual conditions, and no limitation is imposed here.
[0032] Please see Figures 1 to 3 In some embodiments, the first pixel 221 further includes a first microlens 2211 disposed on the first filter 2212, the second pixel 222 further includes a second microlens 2221 disposed on the second filter 2222, and the third pixel 223 further includes a third microlens 2231 disposed on the third filter 2232. The first microlens 2211, the second microlens 2221, and the third microlens 2231 are capable of converging the first pulse beam 101, the second pulse beam 102, or the third pulse beam 103, respectively.
[0033] Since ambient light also includes light beams with wavelengths of the first, second, and third wavelengths, the ambient light of the first wavelength and the reflected first pulse beam can both be incident on the first photosensitive chip 2213 and received. The ambient light of the second wavelength and the reflected second pulse beam can both be incident on the second photosensitive chip 2223 and received. The ambient light of the third wavelength and the reflected third pulse beam can both be incident on the third photosensitive chip 2233 and received. The first photosensitive chip 2213, the second photosensitive chip 2223, and the third photosensitive chip 2233 have the same structure and working principle, the difference being that they belong to different pixels. They all include photoelectric conversion elements and three taps.
[0034] The first photosensitive chip 2213 includes a first photoelectric conversion element, a first tap, a second tap, and a third tap. The first tap, the second tap, and the third tap sequentially collect the charge generated by the first photoelectric conversion element according to a set working sequence to obtain a first charge Q. A1 Second charge Q B1 and the third charge Q C1 The second photosensitive chip 2223 includes a second photoelectric conversion element, a first tap, a second tap, and a third tap. The first tap, the second tap, and the third tap sequentially collect the charge generated by the second photoelectric conversion element according to a set working sequence to obtain a first charge Q. A2 Second charge Q B2 and the third charge Q C2The third photosensitive chip 2233 includes a third photoelectric conversion element, a first tap, a second tap, and a third tap. The first tap, the second tap, and the third tap sequentially collect the charge generated by the third photoelectric conversion element according to a set working sequence to obtain a first charge Q. A3 Second charge Q B3 and the third charge Q C3 .
[0035] like Figure 6 As shown, Figure 6 L1 represents the operating timing of the first sub-light source; a high level indicates the emission of the first pulse beam. R1 represents the reflected first pulse beam. A1 represents the operating timing of the first tap. B1 represents the operating timing of the second tap. C1 represents the operating timing of the third tap; a high level indicates the tap is activated. The pulse period of light source 12 is T. p The pulse width of light source 12 is T. h T p =3T h During pulse period T p Inside, the first taps of the three photosensitive chips operate with the same timing sequence and are synchronized with the light source 12, all at T h1 When the time period begins, the second taps of the three image sensors operate in the same timing sequence, all within T... h2 When the time period begins, the third tap of each of the three image sensors operates in the same timing sequence, all within T... h3 The time slot has started.
[0036] The control and processor 30 is connected to the transmitting module 10 and the acquiring module 20, and can control the transmitting module 10 and the acquiring module 20 to turn on and off. The control and processor 30 receives the charge Q transmitted by the first pixel 221. A1 Q B1 and Q C1 According to Q A1 Q B1 Q C1 Calculate the depth value d1 and spectral response S1 of the first pixel 221; receive the charge Q transmitted by the second pixel 222. A2 Q B2 and Q C2 According to Q A2 Q B2 Q C2 Calculate the depth value d2 and spectral response S2 of the second pixel 222; receive the charge Q transmitted by the third pixel 223. A3 Q B3 and Q C3 According to Q A3 Q B3 and Q C3The depth value d3 and spectral response S3 of the third pixel 223 are calculated. For example, the controller and processor 30 calculates based on the charge Q. A1 Q B1 and Q C1 Calculate the phase deviation of the first pulse beam from transmission to reception, calculate the time of flight based on the phase deviation, and calculate the depth value based on the time of flight; based on the charge Q... A1 Q B1 and Q C1 Calculate the target's response to the first pulse beam.
[0037] The controller and processor 30 calculates the depth data d according to the following formula (1) and the spectral data S according to the following formula (2).
[0038]
[0039] S = Q2 + Q1 - 2Q0; (2)
[0040] In equations (1) and (2) above, Q1 is the charge collected by the first tap that collects the charge generated by the pulsed beam, Q2 is the charge collected by the second tap that collects the charge generated by the pulsed beam, Q0 is the charge collected by the tap that collects the charge generated only by receiving ambient light, m = n-1, n is the number of the tap corresponding to Q1, and T h The pulse width of the transmission module 10 and its three taps is given by C, which represents the speed of light. The numbers of the first tap, the second tap, and the third tap are 1, 2, and 3, respectively.
[0041] Taking the first pixel 221 as an example, the control and processor 30 according to Q A1 Q B1 Q C1 Calculate the specific details of the depth data d1 and spectral response data S1 of the first pixel 221. For example... Figure 6 As shown, both the first tap and the light source 12 are at T. h1 The time period begins, and the second tap occurs at T. h2 The time period begins, and the third tap occurs at T. h3 When the time period begins, the first photoelectric conversion element is in T h1 T h2 T h3 The receiving beam is activated and generates charge in all three time periods. There is a certain time difference between the first pulse beam being emitted and received, which is the flight time t.
[0042] like Figure 6 As shown, if in the first period T p1 T inside h1 During the time period, the first pulse beam is reflected to the first photoelectric conversion element and received after a flight time t. Then, in time T...h1 During the time period, the first photoelectric conversion element receives ambient light and the first pulse beam, generating a charge Q. A1 Collected by the first tap, Q A1 Including the sub-charge Q generated by receiving ambient light a11 and the sub-charge Q generated by receiving the first pulse beam a12 ; in T h2 During the time period, the first photoelectric conversion element simultaneously receives ambient light and the first pulse beam during some periods, and receives the ambient beam during other periods, generating a charge Q. B1 The charge Q is collected by the second tap. C1 Including the amount of charge Q generated by receiving ambient light. b11 and the charge Q generated by receiving the first pulse beam b12 ; in T h3 The first photoelectric conversion element in the time period only receives ambient light and generates a charge Q. C1 Collected by the third tap; among them, Q a11 =Q b11 =Q C1 Q B1 -Q C1 This represents the amount of charge (Q) generated by the first pulse beam and collected by the second tap. b12 ), Q A1 +Q B1 -2Q C1 Indicates a pulse width T h The amount of charge (Q) generated by the first pulse beam and collected internally. a12 +Q b12 If n=1 and m=0, then... The spectral data represents the target's response to the first pulse beam; therefore, the spectral data S1 = Q. A1 +Q B1 -2Q C1 .
[0043] If in T h1 During the time period, the first pulse beam was not reflected to the first photoelectric conversion element, while at T h2 When the time period begins to reflect to the first photoelectric conversion element, then at T h1 The first photoelectric conversion element in the time period only receives ambient light and generates a charge Q. A1 Collected by the first tap, T h2 During certain time periods, the system receives only ambient light; during other time periods, it receives both ambient light and the first pulse beam, generating a charge Q. B1 The charge Q is collected by the second tap. B1 Including the amount of charge Q generated by receiving ambient light b11 and the charge Q generated by receiving the first pulse beamb12 T h3 During the time period, ambient light and the first pulse beam are received in some time periods, while only ambient light is received in other time periods, generating a charge Q. C1 The charge Q is collected by the third tap. C1 Including the amount of charge Q generated by receiving ambient light c11 and the charge Q generated by receiving the first pulse beam c12 Among them, Q A1 =Q b11 =Q c11 Q C1 -Q A1 This represents the amount of charge (Q) generated by the first pulse beam and collected by the third tap. c12 ), Q C1 +Q B1 -2Q A1 Indicates a pulse width T h The amount of charge (Q) generated by the first pulse beam and collected internally. b12 +Q c12 ), and then T can be calculated. h2 Flight time within the time period, then add T h1 The total flight time t can be obtained from the time interval, that is, the above n=2, m=1, S1 = Q C1 +Q B1 -2Q A1 .
[0044] If in T h1 and T h2 During the time period, the first pulse beam was not reflected back to the first photoelectric conversion element, while at T h3 The time period begins with reflection onto the first photoelectric conversion element and continues until the next cycle T. p2 T h1 Time period. That is, a single frame measurement includes two periods T. p (T p1 +T p2 ), in T h2 The first photoelectric conversion element in the time period only receives ambient light to generate charge Q. B1 Collected by the second tap, at T h3 During some periods of the time, the first photoelectric conversion element only receives ambient light; during other periods, it receives both ambient light and the first pulse beam, thus generating a charge Q. C1 The charge Q is collected by the third tap. C1 Including the amount of charge Q generated by receiving ambient light c11 and the charge Q generated by receiving the first pulse beam c12 In the next period T p2 T h1During certain time periods, the first photoelectric conversion element receives both ambient light and the first pulse beam; during other time periods, it receives only ambient light, while generating a charge Q. A1 The charge Q is collected by the first tap. A1 Including the amount of charge Q generated by receiving ambient light a11 and the charge Q generated by receiving the first pulse beam a12 Among them, Q B1 =Q c11 =Q a11 Q A1 -Q B1 Q represents the amount of charge generated by the first pulse beam collected by the first tap. A1 +Q C1 -2Q B1 Indicates a pulse width T h The amount of charge generated by the first pulse beam, collected internally, can then be used to calculate T. h3 The flight time t within the time period, then added to T h1 and T h2 The total flight time can then be obtained, i.e., when n=3 and m=2 above.
[0045] S1 = Q A1 +Q C1 -2Q B1 .
[0046] The second photoelectric conversion element receives ambient light and the reflected second pulse beam 102. The charge quantities collected by the first tap, second tap, and third tap are Q, respectively. A2 Q B2 and Q C2 The calculation method for the depth value and spectral response of the second pixel 222 is similar to that of the first pixel 221. Referring to the relevant content of the first pixel 221, the tap that first collects the second pulse beam among the first, second, and third taps is determined, and then the corresponding calculation method is used to calculate the depth value d2 and the spectral response S2. The calculation methods for the depth value d2 and the spectral response S2 of the second pixel 222 are as follows:
[0047] When Q C2 When the value is at its minimum, that is, when the first tap is the first to collect the charge generated by the second pulse beam, then... S2 = Q A2 +Q B2 -2Q C2 ;
[0048] When Q A2 When the value is at its minimum, that is, when the second tap is the first to collect the charge generated by the second pulse beam, then... S2 = QC2 +Q B2 -2Q A2 ;
[0049] When Q B2 When the value is at its minimum, that is, when the third tap is the first to collect the charge generated by the second pulse beam, then... S2 = Q A2 +Q C2 -2Q B2 .
[0050] The second photoelectric conversion element receives ambient light and the reflected third pulse beam 103. The charge collected by the first tap, second tap, and third tap is Q. A3 Q B3 and Q C3 The calculation method for the depth value and spectral response of the third pixel 223 is similar to that of the first pixel 221. Referring to the relevant content of the first pixel 221, the tap that first collects the second pulse beam among the first, second, and third taps is determined, and then the corresponding calculation method is used to calculate the depth value d3 and the spectral response S3. The calculation methods for the depth value d3 and the spectral response S3 of the third pixel 223 are as follows:
[0051] When Q C3 When the value is at its minimum, that is, when the first tap is the first to collect the charge generated by the second pulse beam, then... S3 = Q A3 +Q B3 -2Q C3 ;
[0052] When Q A3 When the value is at its minimum, that is, when the second tap is the first to collect the charge generated by the second pulse beam, then... S3 = Q C3 +Q B3 -2Q A3 ;
[0053] When Q B3 When the value is at its minimum, that is, when the third tap is the first to collect the charge generated by the second pulse beam, then... S3 = Q A3 +Q C3 -2Q B3 .
[0054] The depth data and spectral response of the first pixel 221, the second pixel 222, and the third pixel 223 in a pixel unit 220 are calculated. The spectral responses of the first pixel 221, the second pixel 222, and the third pixel 223 constitute the spectral information spec = [S1, S2, S3] of pixel unit 220. In subsequent applications, qualitative or quantitative analysis such as liveness detection can be performed based on this set of spectral information. Based on the above method, the depth value and spectral response of each pixel in each pixel unit 220 in the entire multispectral image are calculated to obtain the depth information and spectral information of the entire image. A depth image can be generated based on the depth values of all pixels, and a multispectral image can be generated based on the spectral responses of all pixels. The multispectral image includes three channels. The depth information and spectral information of the target can be obtained based on the depth image and the multispectral image.
[0055] In summary, the embodiments of this application design the transmitting module 10 to emit three pulse beams of different wavelengths: a first pulse beam, a second pulse beam, and a third pulse beam. The image sensor 22 of the acquisition module 20 is designed to have a first pixel 221, a second pixel 222, and a third pixel 223 that respectively receive the first pulse beam, the second pulse beam, and the third pulse beam. Then, the controller and processor 30 calculate the corresponding depth data and spectral data from the charge signals generated by the first pixel 221, the second pixel 222, and the third pixel 223.
[0056] This enables the simultaneous output of depth and spectral information from a multispectral depth camera. The multispectral image contains data from three channels, which can be used for applications such as face recognition, liveness detection, and palm print and vein recognition. This facilitates target analysis from a spectral dimension, and the combination with depth images can further improve the accuracy of recognition.
[0057] like Figure 7 As shown, the present invention also proposes a spectral and depth measurement method, which uses the above-mentioned multispectral depth camera to acquire the depth and spectral information of the target. Since this spectral and depth measurement method adopts all the technical solutions of all the above embodiments, it also has all the beneficial effects brought about by the technical solutions of the above embodiments, and will not be repeated here.
[0058] Spectroscopic and depth measurement methods include the following steps:
[0059] 701: Control the firing module to simultaneously fire the first pulse beam, the second pulse beam, and the third pulse beam toward the target;
[0060] 702: Control the acquisition module to receive the first pulse beam, the second pulse beam, and the third pulse beam reflected back from the target;
[0061] 703: Receives the charge signals generated by the first pixel, the second pixel and the third pixel, and calculates the spectral data and depth data of the target based on the charge signals.
[0062] The specific details of each step in the spectral and depth measurement method can be found in the above-described embodiment of the multispectral depth camera, and will not be repeated here.
[0063] In this application, the emitting module 10 emits light beams of three different wavelengths, the image sensor 22 includes three types of pixels, and the photosensitive chip includes three taps as examples for illustrative purposes. The emitting module 10 is not limited to emitting light beams of three different wavelengths, and the image sensor 22 is not limited to including three types of pixels. In other embodiments, the emitting module 10 can emit light beams of more different wavelengths, such as a fourth pulse beam and a fifth pulse beam. The corresponding pixel unit 220 can also include a fourth pixel, a fifth pixel, or more types of pixels. The number of taps can also be two, four, or other numbers. There are no restrictions here, and they can be selected according to actual needs.
[0064] For example, when the transmitting module 10 emits beams of m wavelengths, the image sensor 22 includes m types of pixels, where m > 3 and is a positive integer. The number of taps can specifically be N, where N is a natural number and N ≥ 2, corresponding to T. p =N*T h .
[0065] This invention also proposes a terminal that includes the multispectral depth camera described in the above embodiments. Since this terminal employs all the technical solutions of all the above embodiments, it also possesses all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here. The terminal can be a mobile phone, smart lock, robot, access control system, facial recognition payment device, etc.
[0066] The above are merely optional embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A multispectral depth camera, characterized in that, include: The transmitting module is used to emit first pulse beams, second pulse beams, and third pulse beams of different wavelengths toward the target; The acquisition module includes an image sensor, which includes a first pixel, a second pixel, and a third pixel for receiving the first pulse beam, the second pulse beam, and the third pulse beam reflected back from the target, respectively. as well as A controller and processor are configured to receive charge signals generated by the first pixel, the second pixel, and the third pixel, and calculate the spectral data and depth data of the target based on the charge signals. The multispectral depth camera uses first, second, and third pulse beams of different wavelengths to obtain and analyze information in different spectral dimensions.
2. The multispectral depth camera as described in claim 1, characterized in that, The transmitting module includes a light source, which includes a first sub-light source that emits the first pulse beam, a second sub-light source that emits the second pulse beam, and a third sub-light source that emits the third pulse beam.
3. The multispectral depth camera as described in claim 1, characterized in that, The wavelengths of the first pulse beam, the second pulse beam, and the third pulse beam are respectively a first wavelength, a second wavelength, and a third wavelength; the first pixel includes a first filter and a photosensitive chip, the second pixel includes a second filter and a photosensitive chip, and the third pixel includes a third filter and a photosensitive chip; the center wavelengths of the first filter, the second filter, and the third filter are respectively the first wavelength, the second wavelength, and the third wavelength.
4. The multispectral depth camera as described in claim 3, characterized in that, The photosensitive chip includes a photoelectric conversion element and three taps. The photoelectric conversion element is used to receive ambient light and reflected pulse beams to generate charge. The three taps are staggered in sequence to collect the charge to obtain a first charge, a second charge, and a third charge. The control and processor are used to calculate the depth value and spectral response of the corresponding pixel based on the first charge, the second charge, and the third charge.
5. The multispectral depth camera as described in claim 4, characterized in that, The controller and processor calculate the depth value of the corresponding pixel according to the following formula: Wherein, Q1 is the charge collected by the first tap that collects the charge generated by the pulsed light, Q2 is the charge collected by the second tap that collects the charge generated by the pulsed light, Q0 is the charge collected by the tap that only collects the charge generated by ambient light, m=n-1, n is the sequence number of the tap corresponding to Q1, T h The pulse width is defined by the transmitting module and the three taps.
6. The multispectral depth camera as described in claim 5, characterized in that, The controller and processor calculate the spectral response of the corresponding pixel according to the following formula: S = Q2 + Q1 - 2Q0; The spectral data includes the spectral responses of the first pixel, the second pixel, and the third pixel.
7. A method for measuring spectrum and depth, using a multispectral depth camera as described in any one of claims 1-6, characterized in that, The spectral and depth measurement method includes the following steps: The control module emits first, second, and third pulse beams of different wavelengths toward the target; The control acquisition module receives the first pulse beam, the second pulse beam, and the third pulse beam reflected back from the target. The image sensor includes a first pixel that receives the first pulse beam, a second pixel that receives the second pulse beam, and a third pixel that receives the third pulse beam. The system receives charge signals generated by the first pixel, the second pixel, and the third pixel, and calculates the spectral data and depth data of the target based on the charge signals.
8. The spectral and depth measurement method according to claim 7, characterized in that, The first pixel, the second pixel, and the third pixel each include a photoelectric conversion element and three taps. The photoelectric conversion element is used to receive ambient light and reflected pulse beams to generate charge. The three taps are staggered in sequence to collect the charge, obtaining a first charge, a second charge, and a third charge. The calculation of the target's spectral data and depth data based on the charge signals includes: The depth value and spectral response of the corresponding pixel are calculated based on the first charge, the second charge, and the third charge.
9. The spectral and depth measurement method according to claim 8, characterized in that, The depth value is calculated using the following formula. Wherein, Q1 is the charge collected by the first tap that collects the charge generated by the pulsed light, Q2 is the charge collected by the second tap that collects the charge generated by the pulsed light, Q0 is the charge collected by the tap that only collects the charge generated by ambient light, m=n-1, n is the sequence number of the tap corresponding to Q1, T h The pulse width is defined by the transmitting module and the three taps.
10. The spectral and depth measurement method as described in claim 9, characterized in that, The spectral response is calculated using the following formula: S = Q2 + Q1 - 2Q0; The spectral data includes the spectral responses of the first pixel, the second pixel, and the third pixel.