Depth sensor

Abstract

To provide a depth sensor that can accurately detect the position of projected points (dots) when incident light (e.g., laser) reflected back from an object is projected onto an image sensor. [Solution] One embodiment of the present invention provides a depth sensor comprising: a regulator for adjusting a region of interest corresponding to the position of a projected dot, which varies with the depth to a subject, based on an image having at least one projected dot; and a detector for detecting the projected dot based on the region of interest and the image.

Description

Technical Field

[0001] The present invention relates to semiconductor design technology, and more particularly, to a depth sensor for measuring the depth from a subject.

Background Art

[0002] LiDAR is one of the depth sensors mainly used for measuring the distance (i.e., depth) from a subject. The LiDAR accumulates the hit count value of incident light (e.g., laser) reflected back from the subject in a plurality of time bins, and obtains the distance based on the time bin having the largest hit count value among the plurality of time bins.

[0003] The incident light reflected back from the subject is projected in dot form through an image sensor mounted on the LiDAR. The positions of the dots projected on the image sensor vary differently according to the distance and / or the pixel area of the image sensor.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Embodiments of the present invention provide a depth sensor that can accurately detect the positions of the dots when incident light (e.g., laser) reflected back from a subject is projected onto an image sensor.

Means for Solving the Problems

[0005] According to one aspect of the present invention, a depth sensor may include a regulator for adjusting a region of interest corresponding to the position of a projected dot, which varies with the depth to the subject, based on an image having at least one projected dot, and a detector for detecting the projected dot based on the region of interest and the image.

[0006] According to another aspect of the present invention, the depth sensor may include a mode selector for generating a mode selection signal corresponding to one of a plurality of depth output modes based on a mode control signal; a regulator for adjusting at least one region of interest corresponding to the positions of some or all of the plurality of projected dots, which vary with the depth to the subject, based on an image having a plurality of projected dots and the mode selection signal; and a detector for detecting some or all of the plurality of projected dots based on the region of interest and the image.

[0007] According to yet another aspect of the present invention, the depth sensor may comprise: a light emitter for emitting output light; an image sensor for sensing input light reflected from a subject by the output light and generating an image having projected dots corresponding to the input light; and an image processor for generating a depth map based on the image and, when generating the depth map, adjusting each region of interest for detecting some or all of the projected dots according to the depth relative to the subject. [Effects of the Invention]

[0008] Embodiments of the present invention have the effect of improving the operational reliability of a depth sensor by accurately detecting the position of projected points (dots) when incident light (e.g., laser) reflected back from an object is projected onto an image sensor. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram of a depth sensor according to an embodiment of the present invention. [Figure 2] Figure 1 is a block diagram showing an example of a pixel array included in the image sensor. [Figure 3] Figure 1 is a block diagram showing an example of an image processor. [Figure 4] This is a diagram illustrating the operation of a depth sensor according to an embodiment of the present invention. [Figure 5] This is a diagram illustrating the operation of a depth sensor according to an embodiment of the present invention. [Figure 6] This is a diagram illustrating the operation of a depth sensor according to an embodiment of the present invention. [Figure 7] This is a diagram illustrating the operation of a depth sensor according to an embodiment of the present invention. [Figure 8] This is a diagram illustrating the operation of a depth sensor according to an embodiment of the present invention. [Figure 9] This is a diagram illustrating the operation of a depth sensor according to an embodiment of the present invention. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings in order to explain in detail to the extent that a person with ordinary skill in the art to which the present invention pertains can easily implement the technical idea of ​​the present invention.

[0011] Furthermore, throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other elements in between. Also, when a part is described as "including" or "equipping" a certain component, this does not exclude other components unless specifically contradicted, but rather means that it may further include or equip with other components. Moreover, even if a certain component is described singularly throughout the specification, it will be clear that the present invention is not limited to that, and that the component may consist of multiple parts.

[0012] Figure 1 shows a block diagram of a depth sensor 10 according to an embodiment of the present invention.

[0013] As shown in Figure 1, the depth sensor 10 can measure the distance (i.e., depth) to at least one object (not shown) distributed within the field of view. For example, the depth sensor 10 may include a LiDAR. The depth sensor 10 may comprise a light emitter 100, an image sensor 200, and an image processor 300.

[0014] The light emitter 100 can emit output light TL toward the observation field. For example, the light emitter 100 may include a VCSEL (vertical cavity surface emitting laser) driver. When the output light TL strikes the object, incident light (e.g., laser) RL may be reflected from the object.

[0015] The image sensor 200 can sense incident light RL and generate an input image IMG. The incident light RL can be represented as projected dots (e.g., laser dots) projected onto the image sensor 200. These projected dots are connected to the image sensor 200 in a mesh form, and the image sensor 200 can generate an input image IMG having these projected dots.

[0016] The position of each of the projected points can be varied (i.e., shifted) by a mediating variable associated with the image sensor 200. The mediating variable can include a first mediating variable corresponding to the depth between the subject and the image sensor 200, and a second mediating variable corresponding to the position of the pixel region included in the image sensor 200.

[0017] The image processor 300 generates a depth map DMAP based on the image IMG and the mode control signal MD, and when generating the depth map DMAP, can flexibly adjust each region of interest for detecting some or all of the projected points. In an embodiment of the present invention, the image processor 300 is described by taking as an example receiving the mode control signal MD, but is not necessarily limited thereto, and by design, the image processor 300 may not receive the mode control signal MD (see FIG. 3).

[0018] FIG. 2 shows a block configuration diagram showing an example of the pixel array 210 included in the image sensor 200 shown in FIG. 1.

[0019] As shown in FIG. 2, the pixel array 210 can include a plurality of pixels PX for sensing the projected points. The plurality of pixels PX can be arranged at the intersections of a plurality of columns and a plurality of rows. For example, the pixel array 210 can include a SPAD (single-photon avalanche diode) array, and each of the plurality of pixels PX can include a SPAD. Hereinafter, the plurality of pixels PX are grouped into N×M pixels each (where “N” is a natural number of 1 or more, and “M” is a natural number of 2 or more), and the grouped pixel groups are each referred to as the pixel region for explanation.

[0020] FIG. 3 shows a block diagram illustrating an example of the image processor 300 shown in FIG. 1.

[0021] As shown in FIG. 3, the image processor 300 may include a mode selector 310, a register 320, an adjuster 330, a detector 340, and a depth map generator 250.

[0022] The mode selector 310 can generate a mode selection signal MS corresponding to one of the first distance output mode and the second distance output mode based on the mode control signal MD. The first distance output mode is a low speed mode and can be used when generating a depth map DMAP with high resolution. The second distance output mode is a high speed mode and can be used when generating a depth map DMAP with low resolution.

[0023] The register 320 can store threshold information TH for dividing the depth range. For example, the threshold information TH may include a first threshold TH1 and a second threshold TH2 for each pixel region, or may include a first threshold TH1 and a second threshold TH2 for each pixel region group. The pixel region group may include pixel regions grouped by the arrangement position of the pixel regions, or may include pixel regions grouped by the degree of movement of the projected points.

[0024] The adjuster 330 can flexibly adjust each region of interest corresponding to some or all of the positions of the projected points that vary according to the depth based on the image IMG and the mode selection signal MS. The adjuster 330 can determine the number of target points corresponding to some or all of the projected points for each pixel region, and can flexibly adjust each region of interest corresponding to the positions of the target points for each pixel region.

[0025] For example, the tuner 330 can determine all of the projected points by the first distance output mode as the target points and flexibly adjust each of the regions of interest corresponding to the target points. Alternatively, the tuner 330 can determine only a portion of the projected points by the second distance output mode as the target points and flexibly adjust each of the regions of interest corresponding to the target points.

[0026] The tuner 330 can adjust the size of the region of interest based on threshold information TH. For example, when the depth corresponds to a short distance, the tuner 330 can adjust the size of the region of interest to a first size ROI1; when the depth corresponds to a medium distance, it can adjust the size of the region of interest to a second size ROI2, which is smaller than the first size ROI1; and when the depth corresponds to a long distance, it can adjust the size of the region of interest to a third size ROI3, which is smaller than the second size ROI2. In other words, the tuner 330 can adjust the size of the region of interest to be larger the closer the depth is, and smaller the size of the region of interest the farther the depth is. The short distance, medium distance, and long distance can be distinguished by a first threshold TH1 and a second threshold TH2, and these can be the same or different for each pixel region (see Figure 6).

[0027] The adjuster 330 can fix either the vertical size of the region of interest or the horizontal size of the region of interest, and adjust the other. If the light emitter 100 and the image sensor 200 are physically arranged as shown in Figure 1 (i.e., the image sensor 200 is positioned to the right of the light emitter 100), the projected points may tend to shift to the right as the depth increases. In such a case, the adjuster 330 can fix the vertical size of the region of interest and adjust the horizontal size of the region of interest.

[0028] The detector 340 can detect part or all of the projected points based on the region of interest and the image IMG. For example, in the first distance output mode, the detector 340 can generate point information DS corresponding to all of the projected points. Although not shown in the drawings, the point information DS generated in the first distance output mode can be used when generating the high-resolution depth map DMAP. Alternatively, in the second distance output mode, the detector 340 can generate point information DS corresponding to part of the projected points. Although not shown in the drawings, the point information DS generated in the second distance output mode can be used when generating the low-resolution depth map DMAP.

[0029] The depth map generator 350 can generate a depth map DMAP based on point information DS. For example, in the first distance output mode, the depth map generator 350 can generate a depth map DMAP with high resolution based on point information DS. Alternatively, in the second distance output mode, the depth map generator 350 can generate a depth map DMAP with low resolution based on point information DS.

[0030] The embodiments of the present invention have been described using the example of a configuration including a mode selector 310, but are not necessarily limited to this configuration, and the mode selector 310 may not be included. If the mode selector 310 is not included, the regulator 330 can be operated by design in one of the first distance output modes and the second distance output modes, which are set in advance.

[0031] The operation of the depth sensor 10 according to the embodiment of the present invention having the above configuration will be described below with reference to Figures 4 to 9.

[0032] Figure 4 shows a flowchart illustrating the operation of the depth sensor 10 shown in Figure 1.

[0033] As shown in Figure 4, the depth sensor 10 can set threshold information TH by the parameter (S100). For example, the parameter may include a first parameter corresponding to the depth between the subject and the image sensor 200, and a second parameter corresponding to the position of the pixel region included in the image sensor 200. For example, the threshold information TH may include a first threshold TH1 and a second threshold TH2 for each pixel region, or a first threshold TH1 and a second threshold TH2 for each pixel region group. The pixel region group may include pixel regions grouped by a pixel region arrangement pattern, or may include pixel regions with similar degrees of movement of projected points.

[0034] The depth sensor 10 can select one of the first distance output mode and the second distance output mode based on the mode control signal MD (S102). The first distance output mode is the low speed mode, which can be used when generating a high-resolution depth map DMAP. The second distance output mode is the high speed mode, which can be used when generating a low-resolution depth map DMAP.

[0035] The depth sensor 10 can flexibly adjust the region of interest corresponding to some or all of the projected points based on threshold information TH (S104). For example, in the first distance output mode, the depth sensor 10 can determine all of the projected points as the target points and flexibly adjust the region of interest corresponding to the target points. In contrast, in the second distance output mode, the adjuster 330 can determine some of the projected points as the target points and flexibly adjust the region of interest corresponding to the target points.

[0036] The depth sensor 10 can accurately detect the point projected by the region of interest (S106). For example, the depth sensor 10 can adjust the size of the region of interest to be larger as the depth approaches, and adjust the size of the region of interest to be smaller as the depth approaches. The depth can be divided into short distance, medium distance, and long distance by a first threshold TH1 and a second threshold TH2.

[0037] Of the operations shown in Figure 4 (S100 to S106), the operation to set the threshold information TH (S100) can be performed in a mode prior to normal mode (e.g., test mode, simulation mode, etc.), and of the operations shown in Figure 4 (S100 to S106), the remaining operations (S102 to S106) can be performed in the normal mode.

[0038] Figures 5 and 6 show diagrams that provide supplementary explanations for the operation of setting the threshold information TH (i.e., S100) among the operations described in Figure 4.

[0039] Figure 5 shows a schematic diagram of the pixel array 210 including the pixel regions, and Figure 6 shows a table for explaining the threshold information TH of samples RP1 to RP10 among the pixel regions. For the sake of explanation, please note that Figures 5 and 6 only show the threshold information TH of some of the pixel regions RP1 to RP10 included in the pixel array 210. For reference, the table shown in Figure 6 may include points where the projected points on some of the pixel regions RP1 to RP10 have changed as a result of the depth sensor 10 capturing a flat object several times at 10 cm intervals, and threshold information TH based on those changed points.

[0040] As shown in both Figures 5 and 6, the points projected onto the aforementioned pixel regions RP1 to RP10 can shift in a certain direction as the depth increases. The threshold information TH can be set according to the degree of shift due to the depth. A representative example of the point projected onto the first pixel region RP1 among the aforementioned pixel regions RP1 to RP10 is as follows.

[0041] When the depth corresponds to the first change point (40 cm), the position of the point projected onto the first pixel region RP1 corresponds to "coordinate (3, 5)". When the depth corresponds to the second change point (50 cm), the position of the point projected onto the first pixel region RP1 corresponds to "coordinate (4, 5)". When the depth corresponds to the third change point (70 cm), the position of the point projected onto the first pixel region RP1 corresponds to "coordinate (5, 5)". When the depth corresponds to the fourth change point (190 cm), the position of the point projected onto the first pixel region RP1 corresponds to "coordinate (6, 5)". When the depth corresponds to the fifth change point (420 cm), the position of the point projected onto the first pixel region RP1 can correspond to "coordinate (7, 5)". As the depth increases, only the value of the "X" axis among the position values ​​included in the "coordinate (X, Y)" changes. In particular, as the depth increases, the value of the "X" axis becomes larger, so it can be seen that the position of the point projected onto the first pixel region RP1 shifts relatively to the right.

[0042] At this point, it can be seen that the interval between the third change point (70 cm) and the fourth change point (190 cm) is relatively large, and the interval between the fourth change point (190 cm) and the fifth change point (420 cm) is also relatively large. Therefore, as threshold information TH for the first pixel region RP1, a first threshold TH1 corresponding to the third change point (70 cm) and a second threshold TH2 corresponding to the fourth change point (190 cm) can be set.

[0043] As shown in Figure 6, the points projected onto some of the pixel regions RP1 to RP10 may differ depending on the depth and / or the pixel region. Therefore, the threshold information TH for some of the pixel regions RP1 to RP10 may differ depending on the pixel region.

[0044] Figures 7 and 8 show diagrams that provide supplementary explanations for the operation described in Figure 4, specifically the operation of selecting one of the first distance output mode and the second distance output mode (i.e., S102).

[0045] Figure 7 shows a schematic diagram of an arbitrary pixel region to illustrate the first distance output mode. For example, the arbitrary pixel region can correspond to each pixel region included in the pixel array 210.

[0046] As shown in Figure 7, four points (i.e., 2 × 2 laser dots) can be projected onto any given pixel region. The first distance output mode is the low-speed mode, which can be used when generating the high-resolution depth map (DMAP). In the first distance output mode, the depth sensor 10 can detect and utilize all four points simultaneously or sequentially.

[0047] As shown in Figure 8, four points (i.e., the 2x2 laser dots) can be projected onto the arbitrary pixel region. The second distance output mode is the high-speed mode, which can be used when generating the low-resolution depth map (DMAP). In the second distance output mode, the depth sensor 10 can detect and utilize some (e.g., one) of the four points.

[0048] Figure 9 shows diagrams to supplement the operation described in Figure 4, specifically the operation of flexibly adjusting the region of interest corresponding to some or all of the projected points (i.e., S104). For example, Figure 9 shows a schematic diagram illustrating the operation of adjusting the size of the region of interest.

[0049] As shown in Figure 9, the depth sensor 10 can fix either the vertical size of the region of interest or the horizontal size of the region of interest and adjust the other. For example, if the projected points gradually shift to the right as the depth increases, the depth sensor 10 can fix the vertical size of the region of interest and adjust the horizontal size of the region of interest. The method for fixing the vertical size of the region of interest and adjusting the horizontal size of the region of interest is as follows.

[0050] The depth sensor 10 can generate regions of interest having first to third sizes ROI1, ROI2, and ROI3 via a first basic region BR1 having a low-size RS and a first column size CS1, and a second basic region BR2 having a low-size RS and a second column size CS2. The first column size CS1 and the second column size CS2 can be the same or different. For example, the depth sensor 10 can generate the regions of interest having first to third sizes ROI1, ROI2, and ROI3 by overlapping a right portion of the first basic region BR1 and a left portion of the second basic region BR2. The first size ROI1 can correspond to an overall region including an overlapping region OR of the remaining portion of the first basic region BR1 and the remaining portion of the second basic region BR2. The overlapping region OR can be the overlapping region of the right portion of the first basic region BR1 and the left portion of the second basic region BR2. The second size ROI2 can correspond to the second basic region BR2. The third size ROI3 can correspond to the remaining region after excluding the region OR that overlapped in the second basic region BR2 (i.e., the remaining part of the second basic region BR2).

[0051] In other words, the depth sensor 10 can generate regions of interest having first to third sizes ROI1, ROI2, and ROI3 based on the low size RS and first column size CS1 of the first basic region CS1, the low size RS and second column size CS2 of the second basic region CS2, the low start point RSTR, the first column start point CSTR01, and the second column start point CSTR2.

[0052] When the depth corresponds to the short distance, the region of interest having a first size ROI1 can be used. As described above in Figure 6, the projected point exhibits greater variability at the short distance, so the region of interest having a relatively large first size ROI1 can be used. When the depth corresponds to the long distance, the region of interest having a third size ROI3 can be used. As described above in Figure 6, the projected point exhibits less variability at the long distance, so the region of interest having a relatively small third size ROI3 can be used.

[0053] According to this embodiment of the present invention, when the depth of field with respect to the subject is short, the variability of the projected point can be covered to the maximum extent by adjusting the size of the region of interest to be relatively large, and when the depth of field is long, the noise component reflected in the projected point can be minimized by adjusting the size of the region of interest to be relatively small.

[0054] While the technical concept of the present invention has been specifically described by the embodiments described above, it should be noted that the embodiments described above are for illustrative purposes only and not to limit them. Furthermore, a typical expert in the technical field of the present invention will understand that various embodiments are possible through various substitutions, modifications, and changes within the scope of the technical concept of the present invention. [Explanation of symbols]

[0055] 10 Depth Sensor 100 light emitters 200 Image Sensors 300 Image Processors

Claims

1. A regulator for adjusting a region of interest corresponding to the position of a projected dot, which varies with the depth of field relative to the subject, based on an image having at least one projected dot. A detector for detecting the projected point based on the region of interest and the image, A depth sensor equipped with [features / equipment].

2. The depth sensor according to claim 1, wherein the adjuster adjusts the size of the region of interest based on threshold information for dividing the range of distances.

3. The depth sensor according to claim 2, further comprising a register for storing the threshold information.

4. The depth sensor according to claim 1, wherein the adjuster adjusts the size of the region of interest to a first size when the distance corresponds to a short distance, adjusts the size of the region of interest to a second size smaller than the first size when the distance corresponds to a medium distance, and adjusts the size of the region of interest to a third size smaller than the second size when the distance corresponds to a long distance.

5. The first size corresponds to a third region formed by overlapping a part of the first region and a part of the second region. The second size corresponds to the second region, The depth sensor according to claim 4, wherein the third size corresponds to a fourth region obtained by removing a portion of the second region.

6. The depth sensor according to claim 1, wherein the adjuster fixes either the vertical size of the region of interest or the horizontal size of the region of interest, and adjusts the remaining one.

7. A mode selector for generating a mode selection signal corresponding to one of several depth output modes based on a mode control signal, A regulator for adjusting at least one region of interest corresponding to some or all of the positions of the multiple projected dots, which vary with the depth of the subject, based on an image having multiple projected dots and the mode selection signal, A detector for detecting some or all of the plurality of projected points based on the region of interest and the image, A depth sensor equipped with [features / equipment].

8. The depth sensor according to claim 7, wherein the adjuster determines the number of target points corresponding to some or all of the plurality of projected points for each pixel region, and adjusts the region of interest corresponding to the position of the target points for each pixel region.

9. The depth sensor according to claim 7, wherein the adjuster adjusts the size of the region of interest based on threshold information for dividing the range of distances.

10. The depth sensor according to claim 9, further comprising a register for storing the threshold information.

11. The depth sensor according to claim 7, wherein the adjuster adjusts the size of the region of interest to a first size when the distance corresponds to a short distance, adjusts the size of the region of interest to a second size smaller than the first size when the distance corresponds to a medium distance, and adjusts the size of the region of interest to a third size smaller than the second size when the distance corresponds to a long distance.

12. The first size corresponds to a third region formed by overlapping a part of the first region and a part of the second region. The second size corresponds to the second region, The depth sensor according to claim 11, wherein the third size corresponds to a fourth region obtained by removing a portion of the second region.

13. The depth sensor according to claim 7, wherein the adjuster fixes either the vertical size of the region of interest or the horizontal size of the region of interest, and adjusts the remaining one.

14. The detector, in the low-speed mode of the multiple distance output modes, generates point information corresponding to all of the multiple projected points, and in the low-speed mode, the point information is used to generate a depth map having high resolution. The depth sensor according to claim 7, wherein the detector generates point information corresponding to a portion of the plurality of projected points in the high-speed mode of the plurality of distance output modes, and the point information in the high-speed mode is used when generating a depth map having low resolution.

15. A light emitter for diverting output light, An image sensor for sensing input light reflected from a subject and generating an image having projected dots corresponding to the input light, An image processor for generating a depth map based on the aforementioned image, and for adjusting each region of interest for detecting part or all of the projected points according to the depth relative to the subject when generating the depth map, A depth sensor equipped with [features / equipment].

16. The depth sensor according to claim 15, wherein the image processor adjusts the size of the corresponding region of interest to a first size when the distance corresponds to a short distance, adjusts the size of the corresponding region of interest to a second size smaller than the first size when the distance corresponds to a medium distance, and adjusts the size of the corresponding region of interest to a third size smaller than the second size when the distance corresponds to a long distance.

17. The first size corresponds to a third region formed by overlapping a part of the first region and a part of the second region. The second size corresponds to the second region, The depth sensor according to claim 16, wherein the third size corresponds to a fourth region obtained by removing a portion of the second region.

18. The depth sensor according to claim 15, wherein the image processor fixes one of the vertical size and horizontal size of each region of interest and adjusts the other.

19. The aforementioned image processor is Based on the aforementioned image, a regulator for adjusting each of the regions of interest corresponding to the position of the projected points by the distance, A detector for detecting the projected points based on each of the aforementioned regions of interest and the image, The depth sensor according to claim 15, comprising:

20. The image processor further includes registers for storing threshold information for dividing the distance range, The depth sensor according to claim 19, wherein the adjuster adjusts the size of each region of interest based on the threshold information.

21. The aforementioned image processor is A mode selector for generating a mode selection signal corresponding to high-speed mode or low-speed mode based on a mode control signal, A tuner for adjusting, based on the image and the mode selection signal, to adjust each region of interest corresponding to some of the positions of the projected points on a pixel-by-pixel basis in high-speed mode, or to adjust each region of interest corresponding to all of the positions of the projected points on a pixel-by-pixel basis in low-speed mode, A detector for detecting the projected points based on each of the aforementioned regions of interest and the image, The depth sensor according to claim 15, comprising:

22. The image processor further includes registers for storing threshold information for dividing the distance range, The depth sensor according to claim 21, wherein the adjuster adjusts the size of each region of interest based on the threshold information.

23. In the low-speed mode, the detector generates point information corresponding to all of the projected points, and in the low-speed mode, the point information is used to generate a depth map having high resolution. The depth sensor according to claim 21, wherein the detector generates point information corresponding to a portion of the projected points in the high-speed mode, and the point information in the high-speed mode is used when generating a depth map having low resolution.

Images