Depth data measurement head, measurement device, and measurement method

Abstract

The measurement head (300) includes a projection device (310) for projecting line light onto the imaging region, and image sensors (320_1, 320_2) including N memory cell groups, where each memory cell group is exposed with an exposure switch period t that is separated from each other by 2π / N in phase. e The projection device (310) completes a scan in one pattern within the scan period, and the scan period includes a plurality of repeated sub-cycles. In each sub-cycle, the projection period t p includes N waveform projection regions with a width of 2π / N and the light intensity encoded based on the imaging pattern. When completing a scan in one pattern within the scan period, the N memory cell groups of the image sensors (320_1, 320_2) image one different stripe light pattern respectively, and there is a phase shift of 2π / N between the N stripe light patterns.

Description

[Technical Field]

[0001] This invention relates to the field of three-dimensional imaging, and more specifically to a depth data measurement head, a measurement device, and a measurement method. [Background technology]

[0002] A depth camera is a data acquisition device that collects depth information of a target object. Such cameras are widely applied in fields such as three-dimensional scanning and three-dimensional modeling. For example, an increasing number of smartphones now come equipped with depth imaging devices for facial recognition. Conventional technology can achieve high-precision imaging by using stripe optical coding. However, stripe optical coding requires capturing multiple stripe images and combining them into a single depth image. Consequently, the resulting depth image has a low frame rate and cannot meet the requirements for real-time, high-precision dynamic imaging.

[0003] Therefore, improved depth data measurement methods are needed. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The technical problem that this disclosure aims to solve is to provide an improved depth data measurement means that utilizes an image sensor in which each pixel includes multiple memory cell groups for phase-shift exposure, enabling imaging of line light having continuous phase shifts in different projected subcycles, and in a single scan of the line light, each of the N memory cell groups of the image sensor can acquire a different phase-shift stripe image, thereby enabling the acquisition of N stripe images in a single line light scan. This significantly improves the speed of depth map synthesis without compromising the resolution of the image sensor and is suitable for photographing moving targets. [Means for solving the problem]

[0005] TIFF0007829959000001.tif78169

[0006] Optionally, the projection period t of the line light p is synchronized with the exposure switch period t of the first memory cell group e and synchronized with it.

[0007] Optionally, in each sub-phase T i the projection waveforms within each projection period t of the line light p are the same, and are rectangular waves with a bright region of 2π / N phase and a dark region of 6π / N phase, and the N stripe light patterns are stripe light patterns in which the bright and dark regions are repeated.

[0008] TIFF0007829959000002.tif27169

[0009] Optionally, in each sub-phase T i the line light is projected m times with a projection period t p and the duration of each sub-phase T i is longer than the residence time t c and is longer than it.

[0010] TIFF0007829959000003.tif51169

[0011] TIFF0007829959000004.tif45169

[0012] TIFF0007829959000005.tif56169

[0013] Optionally, the projection device includes a light emitting device for generating line light, and a reflection device for reflecting the line light and projecting the line light that moves in a direction perpendicular to the stripe direction at a predetermined frequency onto the imaging region, wherein the longitudinal direction of the line light is the longitudinal direction of the projected stripe, and the reflection device includes one of a mechanical vibration mirror that reciprocates at the predetermined frequency, a micromirror device that reciprocates at a predetermined frequency, and a mechanical rotation mirror that rotates in one direction at a predetermined frequency.

[0014] Optionally, the image sensor includes a first image sensor and a second image sensor with a fixed relative position, the first image sensor and the second image sensor each include the N memory cell groups, and are exposed synchronously with each other.

[0015] TIFF0007829959000006.tif69169

[0016] TIFF0007829959000007.tif65169

[0017] TIFF0007829959000008.tif38169

[0018] According to a third aspect of the present disclosure, there is provided a depth data measurement device including a first depth imaging measurement head and a second depth imaging measurement head with a fixed relative position, and the first depth imaging measurement head and the second depth imaging measurement head are the depth data measurement heads described in the first aspect. Here, after the first depth imaging measurement head completes a scan in a first pattern, the second depth imaging measurement head performs a second pattern scan, and a first group of N-step phase shift patterns obtained by the scan in the first pattern and a second group of N-step phase shift patterns obtained by the scan in the second pattern are combined into depth information of an imaging object in the imaging region based on the relative position.

[0019] TIFF0007829959000009.tif95169

[0020] TIFF0007829959000010.tif139169

[0021] Optionally, the method further includes the steps of: the depth data measuring instrument capturing a plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns that move relative to the imaging target; and the depth data measuring instrument synthesizing depth information generated from the plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns with model information of the imaging target based on calibration points. [Effects of the Invention]

[0022] As a result, the depth imaging measurement head of the present invention can simultaneously acquire an N-step phase shift pattern in a single scan of line light while maintaining image resolution, thereby improving imaging speed. [Brief explanation of the drawing]

[0023] TIFF0007829959000011.tif247169 [Modes for carrying out the invention]

[0024] Preferred embodiments of the Disclosure will be described in more detail below with reference to the drawings. While preferred embodiments of the Disclosure are shown in the drawings, it should be understood that the Disclosure can be implemented in a variety of ways and should not be limited to the embodiments described herein. These embodiments are provided to make the Disclosure thorough and complete and to fully convey the scope of the Disclosure to those skilled in the art.

[0025] As can be seen from the measurement principle of structured light, the ability to accurately determine the scan angle α is a key point for the entire measurement system. Structured light consisting of points and lines can have its scan angle calculated and determined by mechanical devices such as rotating mirrors, and the significance of image encoding and decoding lies in determining the scan angle of the encoded structured light (i.e., surface structured light) system. Figure 1 shows the principle of depth imaging using stripe code structured light. To facilitate understanding, the encoding principle of stripe structured light is simply explained in the figure using a 3-bit binary time code with two grayscales. The projection device can sequentially project three patterns as shown in the figure onto the object to be measured in the imaging area, and each of the three patterns uses two grayscales, light and dark, to divide the projection space into eight regions. Each region corresponds to its respective projection angle, and it can be assumed that the bright region corresponds to code "1" and the dark region corresponds to code "0". The code values ​​of the three encoded patterns for a single point on the scene in the projection space are combined according to the projection order to obtain the region code value of that point, thereby identifying the region where the point is located, and the scan angle of that point is decoded and obtained.

[0026] To improve matching accuracy, the number of projection patterns in the timecode can be increased. Figure 2 shows another example of projecting structured light with a stripe code. Specifically, the figure shows a 5-bit binary timecode with two grayscale images. For example, in a binocular imaging application, this means that each pixel in each of the left and right image frames is either 0 or 1, thereby enabling left-right image matching with higher accuracy (e.g., at the pixel level). Assuming the projection speed of the projection device does not change, the example in Figure 2 represents achieving higher accuracy image matching with a higher time-domain cost compared to the three code patterns in Figure 1.

[0027] Figure 3 is a schematic diagram illustrating the principle of acquiring depth data by projecting a stripe image using line light. As shown in Figure 3, the depth data measurement head 300 includes a projection device 310 and two image sensors 320_1 and 320_2. In a monocular embodiment, the depth data measurement head 300 can also perform imaging using a single image sensor.

[0028] The projection device 310 is used to scan and project structured light having a stripe code onto the imaging area. For example, within the projection period of three consecutive image frames, the projection device 310 can project three patterns in succession, as shown in Figure 1, and the imaging results of the three patterns are used to generate depth data. The first and second image sensors 320_1 and 320_2, which can be called the first and second image sensors, have a predetermined relative positional relationship for imaging the imaging area and acquiring the first and second two-dimensional image frames, respectively, under the illumination of structured light. For example, when the projection device 310 projects three patterns as shown in Figure 1, the first image sensor 320_1 and the second image sensor 320_2 can each image the imaging area onto which these three patterns are projected (e.g., the imaging plane in Figure 3 and a certain range before and after it) within the imaging period of three synchronized image frames.

[0029] As shown in Figure 3, the projection device 310 can project a line of light extending in the x-direction in the z-direction (i.e., toward the imaging area). The projected line of light can be continuously moved in the y-direction to cover the entire imaging area. The lower part of Figure 3 is a more easily understandable illustration of the line of light scanning, corresponding to a perspective view of the imaging area.

[0030] In this disclosure, the direction in which the light beam exits the measuring head is defined as the z-direction, the vertical direction of the imaging plane as the x-direction, and the horizontal direction as the y-direction. Therefore, the striped structured light projected by the projection device may be the result of line light extending in the x-direction moving in the y-direction. In other embodiments, synchronization and imaging processing can also be performed on striped structured light obtained by line light extending in the horizontal y-direction moving in the x-direction, but in this disclosure, vertical striped light will be used preferentially in the explanation.

[0031] Figures 4A and 4B show enlarged examples of the projection device shown in Figure 3. Specifically, as shown in Figure 3, in the projection device 310, the laser emitted by the laser light-emitting element (e.g., the laser light-emitting element 411 shown in detail in Figures 4A and 4B) is scanned and projected onto the imaging area (the gray area in Figure 3) by the projection mechanism (e.g., the projection mechanism 412 shown in detail in Figures 4A and 4B), and is used to actively project structured light onto the object to be measured (e.g., the human in Figure 3) within the imaging area. A pair of image sensors 320_1 and 320_2 perform imaging on the imaging area, thereby obtaining image frames necessary for calculating depth data. As shown in Figure 3, the dashed line emanating from the projection device 310 is used to indicate its projection range, and the dashed lines emanating from the image sensors 320_1 and 320_2 are used to indicate their respective imaging ranges. The imaging area is generally located in the overlapping area of ​​the projection and imaging ranges of these three components.

[0032] In practical applications, the laser-emitting element is used to generate line and / or infrared lasers, and the laser-emitting element rapidly switches to scan and project alternating light and dark structured light corresponding to a stripe code. The rapid switching may include rapid switching of the laser-emitting element and rapid code switching.

[0033] In one embodiment, the laser light-emitting element continuously emits lasers of the same intensity, and the projected stripe light pattern may be realized by turning the laser light-emitting element on and off. In this case, the laser light-emitting element projects only light of one intensity with duty cycles of different periods, and each pixel of the image sensor integrates the projected light to determine the "presence or absence" of the irradiated light; therefore, the image sensor used may be a monochrome image sensor.

[0034] In another embodiment, the laser light-emitting element itself may emit a laser whose light intensity changes, for example, a laser whose emitted light intensity undergoes a sinusoidal transformation within one large period depending on the applied power. The sinusoidal transformation laser can be combined with stripe projection to scan and project a pattern that alternates between light and dark stripes, with different luminances between the brighter stripes. In this case, the image sensor must have the ability to distinguish between different light intensities and perform imaging, and therefore may be a multi-level grayscale image sensor. Clearly, grayscale projection and imaging can provide more accurate inter-pixel matching compared to black and white projection and imaging, thereby improving the accuracy of depth data measurement.

[0035] In one embodiment, the laser light-emitting element 411 may be a line laser light-emitting element that generates a line of light extending in the x-direction (in the direction perpendicular to the plane of the paper in Figures 4A-B). The line of light is then projected onto the imaging plane by a reflective mechanism 412 that can swing along the x-axis. The swing range of the reflective mechanism 412 is as shown in Figure 4B, and the projection mechanism 412 (e.g., a mirror) can scan within the range of angle α, thereby realizing a line of light scanning that reciprocates within the AB range of the imaging plane.

[0036] What needs to be understood is that in order to achieve the projection of a stripe light pattern, the line light itself needs to change in brightness (or, in a simple embodiment, blink) as the line light moves continuously in the y-direction. For example, if it is necessary to scan the first pattern in Figure 1, the laser light-emitting element 411 remains off when the projection mechanism 412 scans over an angle of 0 to α / 2, and turns on when it scans over an angle of α / 2 to α, thereby achieving a pattern where the left side is dark and the right side is bright. If it is necessary to scan the second pattern in Figure 1, the laser light-emitting element 411 remains off when the projection mechanism 412 scans over an angle of 0 to α / 4, turns on when it scans over an angle of α / 4 to α / 2, turns off again when it scans over an angle of α / 2 to 3α / 4, and turns on when it scans over an angle of 3α / 4 to α. This achieves a dark-bright-dark-bright pattern. Similarly, more frequent changes based on the rotation angle can result in the third pattern in Figure 1 and the pattern with finer stripe lights shown in Figure 2.

[0037] In one embodiment, the reflection mechanism 412 may be a micromirror device (also called a digital micromirror device or DMD), or it may be implemented as a MEMS (micro-electromechanical system). Figure 5 is a simplified perspective diagram of the projection device used in the present invention. As shown in Figure 5, the spot laser generated by the laser element can obtain line light through a lens (corresponding to the line laser light-emitting element 411 in Figure 4), and this line light is further reflected by a MEMS-type micromirror device, and the reflected line light is further projected into the external space through an optical window. Micromirror devices have very high performance; for example, commercially available DMDs can perform extremely stable reciprocating vibrations at a frequency of 2 kHz, thereby forming the basis for high-performance depth imaging.

[0038] To obtain a high-precision depth map, the depth data measurement head shown in Figure 3 needs to sequentially project multiple different stripe light patterns. In other words, conventional methods that synthesize depth maps using captured stripe light patterns sacrifice accuracy in the time domain. Furthermore, different stripe light patterns captured in N consecutive imaging cycles are used to synthesize a single depth map. Therefore, conventional depth data measurement methods are only applicable when the captured object remains stationary throughout the N imaging cycles, significantly limiting the application range of techniques that derive depth data using actively projected stripe images.

[0039] In view of this, the present invention provides a novel depth data measurement means that utilizes an image sensor equipped with different memory cell groups capable of phase-shift exposure, and by skillfully setting the brightness and darkness changes of the projected line light, different memory cell groups of the image sensor can each acquire different phase-shift stripe images in a single scan of the line light, thereby enabling the acquisition of multiple stripe images in a single line light scan. This significantly improves the speed of depth map synthesis and is suitable for photographing moving targets.

[0040] In one embodiment, the present invention can be realized as a depth imaging measurement head including a projection device and an image sensor. The projection device is used to project a line of light moving along a first direction (e.g., the y-direction in Figure 3) onto an imaging area, the longitudinal direction of the line of light being a second direction (e.g., the x-direction in Figure 3) perpendicular to the first direction. In one embodiment, the projection device may have the realization structure shown in Figure 5, which includes a line of light generator and a projection mechanism that reflects and projects the line of light and can change the projection direction within a certain angle.

[0041] Each pixel of the image sensor includes N memory cells, and the N memory cells of each pixel belong to one of N memory cell groups, and each memory cell group has an exposure switch period t separated from each other by 2π / N phase. e and is exposed at this time, where N is an integer greater than 1.

[0042] Here, for the sake of clarity, taking N = 4 as an example, the structure and exposure of the image sensor will be described. FIGS. 6A to 6B are examples of N memory cell groups included in the image sensor used in the present invention. The N memory cell groups included in the image sensor may be such that each pixel includes N memory cells, or the image sensor may include N pixel groups, and each pixel may correspond to one memory cell. FIG. 6A is a structure of one pixel column in which each pixel includes N memory cells. As shown in FIG. 6A, one pixel column 621 can include k pixels P1 to P k (in the example of a resolution of 600×800, k = 600). Each pixel has the same structure, that is, includes one photosensitive cell, N switches, and N memory cells, and each switch controls corresponding to the charge accumulation of one memory cell. Specifically, pixel P1622 can include a photodiode 624 as a photosensitive cell, N switches 626, and N memory cells 628. Pixel P k 623 can include a photodiode 625 as a photosensitive cell, N switches 6 , and N memory cells 629.

[0043] The memory cell is, for example, a cell that accumulates the charge generated by the charge accumulation photodiode based on the received light and outputs based on the charge accumulation amount. The N memory cells corresponding to the same pixel all acquire charge from the same photosensitive cell, but each has an exposure switch period t separated from each other by 2π / N phase. eExposure is performed using this method. In each pixel, one memory cell corresponds to one of N memory cell groups. Thus, in a 600x800 pixel image sensor, assuming N=4, there are 600x800x4 memory cells. These 600x800x4 memory cells belong to 4 groups, each group containing 600x800 memory cells, each corresponding to one different pixel. Each memory cell group can be exposed at the same period, and there is a phase difference of π / 2 between adjacent groups. Because N memory cells share one photosensitive cell, it is possible to acquire multiple images in a single scan without reducing the original resolution of the image sensor.

[0044] Figure 6B shows an example of an image sensor containing N pixel groups. For ease of explanation, Figure 6B shows an example with 16x24 pixels. It should be understood that image sensors used in practice may have more pixels, for example, 600x800 pixels. The image sensor shown in Figure 6B contains 4 (N=4) groups of pixels uniformly distributed across the entire imaging surface, represented by the numbers 1, 2, 3, and 4 in the illustrated blocks. Here, "uniformly distributed" across the entire imaging surface means that when line light is scanned and projected in the y direction, each type of pixel illuminates the same (or approximately the same) number of pixels within the current illumination area. In a preferred embodiment, these four pixel groups are spaced apart from each other, with one pixel as the unit, as shown in Figure 6B. That is, the image sensor shown in Figure 6B contains multiple "pixel units" (in the example of Figure 6B, as indicated by the thick black outlines in the figure, there may be 8x12 pixel units with the same structure), and the pixels contained in each pixel unit may be considered to belong to one of the four pixel groups. In other embodiments, each pixel group may be spaced apart from each other, with each group consisting of two pixels (for example, two pixels adjacent to each other in the x-direction).

[0045] Figure 7 shows an example of the relative exposure period relationships between different memory cell groups in the same image sensor. In the example in Figure 7, the four memory cell groups have the same exposure switch period t e The memory cells have the same exposure switch waveform, and are all switched with a 50% duty cycle, meaning that all memory cells of the image sensor have the same exposure switch waveform. The difference is that there is the same phase difference of π / 2 between the waveforms of consecutive different memory cell groups. In one embodiment, the exposure switch period t e For example, it takes a typical value of 20ns. This means that each memory cell in the image sensor is operated with an interval of 10ns on to receive exposure and 10ns off, but the on timing of the second group of memory cells is 5ns later than the first group, the on timing of the third group of memory cells is 5ns later than the second group, and the on timing of the fourth group of memory cells is 5ns later than the third group (it can also be considered 5ns earlier than the first group of memory cells).

[0046] If the image sensor used is capable of performing grouped phase-shift exposure as shown in Figures 6A-B and 7, multiple images can be acquired in a single scan by cleverly setting the line light projection of the projection device.

[0047] TIFF0007829959000012.tif34169

[0048] TIFF0007829959000013.tif47169

[0049] TIFF0007829959000014.tif32169

[0050] In one embodiment, one N-step phase shift pattern group is a sinusoidal 4-step phase shift pattern, and each projection period t is based on exposure corresponding to the waveform projection region of the N memory cell groups. p The light intensity values ​​for each waveform projection region are derived, where the light intensity values ​​are greater than or equal to zero.

[0051] TIFF0007829959000015.tif30169

[0052] P1 = Σ(Q1 + Q2) P2 = Σ(Q2 + Q3) P3 = Σ(Q3 + Q4) P4 = Σ(Q1 + Q4) (1)

[0053] This allows us to identify the respective values ​​of Q1 to Q4 according to the pattern type corresponding to the required 4-step phase-shift imaging.

[0054] TIFF0007829959000016.tif32169

[0055] P1 = Q / 2 * sin int + Q / 2 P2 = -Q / 2 * cost + Q / 2 P3 = -Q / 2 * sin int + Q / 2 P4 = Q / 2 * cost + Q / 2 (2)

[0056] TIFF0007829959000017.tif30169

[0057] Equation (2) allows us to inversely determine the luminance values ​​of Q1 to Q4 with respect to t. In equation (2), there are N=4 unknowns Q1 to Q4, and the rank of equation (2) is N-1=3, so Q1 to Q4 actually have infinitely many solutions, which give the solution values ​​as follows.

[0058] Q1 = A / 2 * sin int + A / 2 Q2=0 Q3 = -A / 2 * cost + A / 2 Q4 = A / 2(cost - sint) (3)

[0059] Here, O*A=Q, where O is, for example, the number of exposure cycles each pixel receives during a line light scan (for example, in the following example, each pixel row can complete 100 exposure cycles in the 2us it is scanned by the line light, in which case O=100).

[0060] However, since the light intensity must not be negative, it is necessary to ensure that the values ​​of Q1 to Q4 are not negative. Equation (3) holds in the range t=0 to π / 4, and when the line light scan is performed in the range t=π / 4 to π / 2, other solutions can be obtained based on equation (2).

[0061] Q1 = A / 2 * cost + A / 2 Q2 = A / 2 (sint - cost) Q3 = -A / 2 * sin int + A / 2 Q4=0 (4)

[0062] Figure 9 shows an example of the relative relationship between the values ​​of the projected light waveform and the exposure periods of memory cell groups 1-4 when imaging a sinusoidal 4-step phase shift pattern.

[0063] As shown in the figure, and as can be seen by referring to Figure 8 and equation (1), when t=0, the values ​​that P1 to P4 take may correspond to Q / 2, 0, Q / 2, and Q, respectively.

[0064] TIFF0007829959000018.tif44169

[0065] TIFF0007829959000019.tif35169

[0066] TIFF0007829959000020.tif80169

[0067] Returning to Fig. 9, in the process of the line light scanning from the position corresponding to t = 0 to the position corresponding to t = π / 6, for example, the values of Q1 to Q4 can be obtained respectively based on Equation (3). When t = π / 6, the line light can be regarded as scanning the 5th pixel column among the 60 pixel columns of the current cycle, and based on Equation (3), the corresponding Q1 = 3A / 4, Q2 = 0, Q3 = 0.183A, Q4 = 0.067A at t = π / 6 can be obtained. As the line light scans from the position corresponding to t = π / 6 to the position corresponding to t = π / 3, the values of Q1 to Q4 can continue to be obtained respectively. When t = π / 4, cost = sint, and since cost < sint in the subsequent process from π / 4 to 3π / 4, the formula (3) for obtaining Q4 < 0 is no longer applicable. At this time, based on Equation (4), the corresponding Q1 = 3A / 4, Q2 = 0.183A, Q3 = 0.067A, Q4 = 0 at t = π / 3 can be obtained. And when t = π / 3, the line light can be regarded as scanning the 10th pixel column among the 60 pixel columns of the current cycle.

[0068] As the line light scans from the position corresponding to t = π / 3 to the position corresponding to t = π / 2, the values of Q1 to Q4 can continue to be obtained respectively based on Equation (4). When t = π / 2, the line light can be regarded as scanning the 15th pixel column among the 60 pixel columns of the current cycle, and based on Equation (4), the corresponding Q1 = A / 2, Q2 = A / 2, Q3 = 0, Q4 = 0 at t = π / 2 can be obtained. Note that for pixels 1 to 4 to perform 4-step phase-shifting imaging, for each projection period t p there is always Q1 + Q2 + Q3 + Q4 = A, and pixels 1 to 4 are exposed with a phase difference of 2π / N from each other in the projection period t p [[ID=!0]]and the on-time accounts for 10%, so for each projection period t p in pixels 1 to 4, a total integrated luminance of 2A * t p / 4 can be obtained.

[0069] The above describes examples of Q1-Q4 values ​​for the first π / 2 in a 2π cycle of a sinusoidal N-step phase-shift pattern, and examples of imaging for each of pixels 1-4, with reference to Figure 9. Those skilled in the art can then determine examples of Q1-Q4 values ​​for the last 3π / 2 of the 2π cycle based on equation (2) and the example in Figure 9.

[0070] TIFF0007829959000021.tif43169

[0071] TIFF0007829959000022.tif107169

[0072] TIFF0007829959000023.tif85169

[0073] To make it easier to understand, we will explain using α=2 and N=4 as examples. That is, the exposure switch periods t are separated by a phase of 2π / N from each other. e By using an image sensor containing four pixel groups that are exposed, and performing two scans, an 8-step phase shift pattern can be acquired.

[0074] TIFF0007829959000024.tif77169

[0075] Furthermore, for the sake of explanation, an example with N=4 was shown, but in other embodiments, N may be a different value. Specifically, N=2 n The formula is such that n is an integer greater than or equal to 1. This makes it possible to achieve, for example, an 8-step phase shift using 8 memory cell image sensor groups with higher accuracy, or a 16-step phase shift using 16 memory cell image sensor groups.

[0076] TIFF0007829959000025.tif46169

[0077] TIFF0007829959000026.tif31169

[0078] When P1=Q and t=0 to π; when P1=0 and t=π to 2π When P2=0 and t=0 to π / 2; when P2=Q and t=π / 2 to 3π / 2; when P2=0 and t=3π / 2 to 2π When P3=0 and t=0~π; when P3=Q and t=π~2π When P4=Q and t=0 to π / 2; when P3=0 and t=π / 2 to 3π / 2; when P3=Q and t=3π / 2 to 2π (5)

[0079] Therefore, based on equations (1) and (5), the optimal solutions for Q1 to Q4 are as follows.

[0080] When Q1=A and t=0 to π / 2; when Q1=0 and t=π / 2 to 2π When Q2=0 and t=0 to π / 2; when Q2=A and t=π / 2 to π; when Q2=0 and t=π to 2π When Q3=0 and t=0~π; when Q3=A and t=π~3π / 2; when Q3=0 and t=3π / 2~2π When Q4=0 and t=0 to 3π / 2; when Q4=A and t=3π / 2 to 2π (6)

[0081] Here, O*A=Q, where O is, for example, the number of exposure periods each pixel receives during line light scanning.

[0082] TIFF0007829959000027.tif35169

[0083] TIFF0007829959000028.tif29169

[0084] As shown in Figures 13A-D, the projected laser has an exposure switch period t e Projection period t of the same length of time p It is projected by a laser, and each laser projection period t p The exposure switch period t of the first memory cell group is e Maintain constant synchronization with (laser projection period t) pThe laser is turned on at different phases in different subphases, and is switched with a duty cycle of 25% (i.e., the bright region is 2π / N), meaning the waveform is a square wave with a duty cycle of 25%. In the four subphases T1 to T4, each projection period t of the projection laser p The projection on times within each cell are synchronized with the exposure on times for the 1st to 4th memory cell groups.

[0085] Specifically, as shown in 13A, in subphase T1, the projection laser is used for each projection period t p It remains ON within the first π / 2 phase, and at this time, the first memory cell group and the fourth memory cell group are also ON, so as shown by the gray rectangle in the figure, each projection period t p The projected light reflected within the first π / 2 phase can be exposed, thereby allowing charge accumulation in the corresponding memory cell. After completing a predetermined m1 projection period, subphase T1 ends, and the projection laser is turned off for each projection period t p The system enters subphase T2, which maintains the ON state within the π / 2~π phase.

[0086] As shown in 13B, in subphase T2, the projection laser is used for each projection period t p It remains ON within the π / 2~π phase, and at this time, the first memory cell group and the second memory cell group are also ON, so as shown by the gray rectangle in the figure, each projection period t p The projected light reflected within the π / 2~π phase can be exposed, thereby enabling charge accumulation in the corresponding memory cell. After completing projection for a predetermined m2 periods, subphase T2 ends, and the projection laser is turned off for each projection period t p The system enters subphase T3, which maintains the ON state within a phase of π~3π / 2.

[0087] As shown in 13C, in subphase T3, the projection laser is used for each projection period t pIt remains ON within a phase of π~3π / 2, and at this time, the second and third memory cell groups are also ON, so as shown by the gray rectangle in the figure, each projection period t p The projected light reflected within the π~3π / 2 phase can be exposed, thereby allowing charge accumulation in the corresponding pixels. After completing projection for a predetermined m3 periods, subphase T3 ends, and the projection laser is turned off for each projection period t p It enters subphase T4, which maintains the ON state within a phase of 3π / 2 to 2π.

[0088] TIFF0007829959000029.tif42169

[0089] TIFF0007829959000030.tif103168

[0090] TIFF0007829959000031.tif72168

[0091] TIFF0007829959000032.tif89168

[0092] To achieve a 4-step phase shift, each subphase T i The duration of the line light must be longer than the time it takes for the line light to scan one pixel unit row. Using the 1920-row image sensor in the above example, if we acquire a 4-step phase-shift pattern with 32 stripes (16 light stripes and 16 dark stripes) as shown in Figure 10, each stripe covers 60 pixel rows (1920 / 32=60). As shown in Figure 9, each subphase T i Since it corresponds to half of the stripe, it covers 30 pixel rows, and the duration is 2us x 30 = 60us. Therefore, in this example, m1=m2=m3=m4=m=60us / 20ns=3000. Line light projection period t p The exposure switch period t for each memory cell group is eBecause the time is the same length, when the line of light scans and passes through half the distance of the stripe, each corresponding memory cell group is also switched 3000 times.

[0093] TIFF0007829959000033.tif72168

[0094] The above embodiment, which obtains the light and dark stripes shown in Figure 14 based on a rectangular wave with a duty cycle of 100 / N% and no change in brightness, uses a line light projection period t p Each 2π / N phase can be adjusted individually accordingly, and this allows the phase difference to be 2π / N and the exposure period to be similarly t p This can be considered an exception to the imaging method of the present invention, which generates a single N-step phase-shift pattern group in combination with an image sensor equivalent to [a certain value].

[0095] TIFF0007829959000034.tif55168

[0096] Furthermore, in order to realize scan projection, the projection apparatus of the present invention includes a light-emitting device for generating line light and a reflecting device for reflecting the line light and projecting line light moving perpendicular to the stripe direction at a predetermined frequency onto the imaging area, wherein the longitudinal direction of the line light is the longitudinal direction of the projected stripe, and the reflecting device includes one of a mechanical vibrating mirror that vibrates back and forth at a predetermined frequency, a micromirror device that reciprocates at a predetermined frequency, and a mechanical rotating mirror that rotates in one direction at a predetermined frequency. Here, the projected line light may be line light exhibiting a high-dimensional Gaussian distribution or a flat-top Gaussian distribution, thereby providing a highly uniform brightness distribution in the width direction of the line light.

[0097] TIFF0007829959000035.tif59168

[0098] Figure 15 is a schematic diagram of a depth data measuring device according to one embodiment of the present invention. As shown in the figure, the measuring device 1500 may include the measuring head and processor 1530 as described above. The measuring head includes a projection device 1510 and two image sensors 1520.

[0099] The processor 1530 is connected to the measuring head, for example, to the projection device 1510 and each of the two image sensors 1520, and is used to determine depth data of the object to be photographed in the imaging area based on predetermined relative positions of the first image sensor 1520_1 and the second image sensor 1520_2 and N first two-dimensional image frames and N second two-dimensional image frames obtained by them imaging the structured light.

[0100] Figures 16A and 16B are schematic flowcharts of a depth data measurement method according to one embodiment of the present invention. This method can be carried out using the depth data measurement head and measuring device of the present invention.

[0101] Figure 16A shows the execution method using an image sensor for each pixel, each containing N memory cells.

[0102] TIFF0007829959000036.tif52169

[0103] In step S1620, the imaging region is captured using an image sensor in which each pixel contains N memory cells, thereby obtaining N image frames under the line light scan projection, where each of the N memory cells in each pixel belongs to one of N memory cell groups, and each memory cell group is separated from each other by a phase of 2π / N with an exposure switch period t e Expose to light.

[0104] In step S1630, depth data of the object to be measured within the imaging region is derived based on the image frame.

[0105] Figure 16B shows an implementation method using an image sensor containing N memory cell groups corresponding to N pixel groups.

[0106] TIFF0007829959000037.tif34168

[0107] In step S1620', an imaging region is captured using an image sensor containing N pixel groups evenly distributed on the imaging surface, thereby acquiring N image frames, where each pixel group has an exposure switch period t that is 2π / N apart from each other. e Expose to light.

[0108] In step S1630', depth data of the object to be measured within the imaging area is derived based on the first and second two-dimensional image frames.

[0109] TIFF0007829959000038.tif35168

[0110] Furthermore, when performing imaging using actively projected structured light, problems of shadows and blind spots arise. Figure 17 shows an example where some depth information of the imaging target is lost when a single camera performs imaging using structured light. As shown in the figure, due to the unevenness of the structure of the imaging target itself, the structured light received by part of its structure is blocked, creating a shadow in the area of ​​the imaging target facing the camera, and since the structured light is not illuminating the shadowed area, depth information for the corresponding part is lost when deriving subsequent depth information (corresponding to "Missing 1" in the figure). Also, regardless of the structure of the imaging target itself, the part of the imaging target opposite the structured light generator cannot be illuminated by structured light, and depth information for the corresponding part is lost when deriving subsequent depth information (corresponding to "Missing 2" in the figure).

[0111] Figure 18 shows an example of dual cameras improving imaging by mitigating the loss of some depth information from the object being imaged. As shown in the figure, by placing cameras on both the left and right sides, more of the object being imaged can be illuminated with structured light, thereby enabling the measurement of more depth information from the object being imaged.

[0112] In view of this, the present invention can be realized in particular as a depth data measuring device with dual cameras (i.e., dual measuring heads). Figure 19 is a schematic diagram of a depth data measuring device according to one embodiment of the present invention. As shown in the figure, the depth data measuring device 1900 may include a first depth imaging measuring head 1910 and a second depth imaging measuring head 1920 whose relative positions are constant. Specifically, the first depth imaging measuring head 1910 and the second depth imaging measuring head 1920 can be mounted in the same outer case to ensure that their relative positions are fixed.

[0113] Furthermore, the measurement heads 1910 and 1920 can perform continuous structured light projection and imaging under the control of the same control device (not shown, but for example, a processor). Here, continuous means that the measurement head 1910 first performs one scan and imaging, then the measurement head 1920 performs one scan and imaging, and so on, performing line light projection and imaging at different time intervals. Since the exposure period of each measurement head is shorter than the depth detection framing time (for example, if the exposure period of one depth frame is 5 ms and the depth detection frame rate is 30 fps, the interval between each depth frame is approximately 28 ms), the exposure of other measurement heads can be performed during the exposure interval of the current measurement head, and exposure imaging and depth calculation can be performed alternately. In other words, the depth data measuring instrument of the present invention may include the two measurement heads described above, and may also include more measurement heads that alternately perform exposure imaging and depth calculation, and the imaging information of these measurement heads may be combined in the same xyz axis space. By using inter-frame interleaved detection, the depth detection frame rate can still be achieved at 30fps without affecting the system's frame rate or detection frame rate.

[0114] As shown in the figure, the measurement head 1910 and the measurement head 1920 may have the same configuration. For example, they may be two measurement heads with completely identical configurations, and they may be installed to continuously capture images of the same imaging area (the object to be imaged within the imaging area) from different angles. The measurement head 1910 and the measurement head 1920 include N memory cell groups as described above, and can be realized as measurement heads that complete N image imaging in a single scan.

[0115] Here, continuous imaging means that after the measurement head 1910 completes the scan in the first pattern (the projection device 1911 performs scan projection of line light, and the image sensor 1912 performs imaging on the projected line light), the measurement head 1920 performs the scan in the second pattern (the projection device 1921 performs scan projection of line light, and the image sensor 1922 performs imaging on the projected line light), and the first group of N-step phase shift patterns obtained from the scan in the first pattern and the second group of N-step phase shift patterns obtained from the scan in the second pattern are combined with the depth information of the imaging target within the imaging region based on the relative position.

[0116] Furthermore, the outer casing of the depth data measuring device 1900 may include a handle 1931 structure, enabling it to be implemented as a portable modeling device. In this case, the depth data measuring device captures a plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns that move relative to the imaging target, and synthesizes the depth information generated from the plurality of first groups of N-step phase shift patterns and the plurality of second groups of N-step phase shift patterns with the model information of the imaging target based on calibration points.

[0117] Figure 20 is a schematic diagram of a depth data measuring device according to one embodiment of the present invention. As shown in the figure, the measuring device 2000 may include two measuring heads 2010 and 2020 as described above, and a processor 2030. Measuring head 2010 includes a projection device 2011 and an image sensor 2012. Measuring head 2020 includes a projection device 2021 and an image sensor 2022.

[0118] The processor 2030 is connected to two measurement heads, for example, to projection device 2011 and image sensor 2012, and to projection device 2021 and image sensor 2022. Projection device 2011 can perform a single scan projection as described above under the control of processor 2030, and the N memory cell groups of image sensor 2012 perform corresponding imaging, thereby obtaining one N-step phase-shift image group after a single scan projection. Next, projection device 2021 can perform a single scan projection as described above under the control of processor 2030, and the N memory cell groups of image sensor 2022 perform corresponding imaging, thereby obtaining one N-step phase-shift image group after a single scan projection. These two N-step phase-shift image groups may each be used to synthesize a depth map of the imaging target, and these two depth maps may be combined based on the positional relationship of the two measurement heads, thereby obtaining more comprehensive data of the imaging target.

[0119] Figure 21 is a schematic flowchart of a depth data measurement method according to one embodiment of the present invention. This method can be carried out using the depth data measurement device of the present invention.

[0120] In step S2110, the first group of N-step phase shift patterns is obtained by performing a scan in the imaging region using the first depth imaging measurement head of the depth data measurement device in the first pattern.

[0121] In step S2120, a second group of N-step phase shift patterns is obtained by performing a scan in the imaging region using the second depth imaging measurement head of the depth data measurement device in a second pattern.

[0122] In step S2130, depth information of the imaging target within the imaging region is synthesized from the N-step phase shift pattern of the first group and the N-step phase shift pattern of the second group, based on the relative positions of the first depth imaging measurement head and the second depth imaging measurement head.

[0123] TIFF0007829959000039.tif98168

[0124] Steps S1910 to S1930 described above can be repeated, for example, to dynamically image an imaging target during motion. In this case, the depth data measuring device of the present invention can be held in place without changing its position.

[0125] In another embodiment, the depth data measuring device of the present invention may actively change its position. As shown in Figure 19, the depth data measuring device of the present invention can be realized as a portable device including a handle 1931. In this case, the depth data measuring device can capture a plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns that move relative to the imaging target, and can synthesize depth information generated from the plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns with model information of the imaging target based on calibration points.

[0126] The depth data measurement head, measurement device, and measurement method of the present invention have been described in detail above with reference to the drawings. The depth data measurement means of the present invention utilizes an image sensor equipped with different pixel groups capable of phase-shift exposure, performs imaging on projected phase-shift line light, and in a single scan of the line light, different pixel groups of the image sensor can acquire different phase-shift stripe images, thereby enabling the acquisition of multiple stripe images in a single line light scan. This significantly improves the speed of depth map synthesis and is suitable for photographing moving targets.

[0127] The flowcharts and block diagrams in the figures illustrate feasible architectures, functions, and operations of systems and methods of multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or part of code, and the module, segment, or part of code may contain executable commands for realizing a defined logical function. In some alternative embodiments, the functions described within a block may be performed in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially simultaneously, or they may sometimes be executed in reverse order depending on the functions they relate to. Furthermore, each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can also be realized by a dedicated system based on hardware for performing a defined function or operation, or by a combination of dedicated hardware and computer commands.

[0128] While embodiments of the present invention have been described, the above description is illustrative, not exhaustive, and not limited to the embodiments disclosed. Many modifications and changes made will be obvious to those skilled in the art, as long as they do not deviate from the scope and spirit of the embodiments described. The terms used herein have been selected to best describe the principles, practical applications, or improvements to the art in the market of each embodiment, or to enable those skilled in the art to understand each embodiment disclosed herein.

Claims

【Request Item 1】

2. The projection period t of the line light ｐ The exposure switch period t of the memory cell group is ｅ A depth data measuring head according to claim 1, which is synchronized with the

3. N=2 ｎ The depth data measuring head according to claim 1, wherein n is an integer of 1 or more.

4.

5. The aforementioned N-step phase shift pattern group is a sinusoidal 4-step phase shift pattern, and each projection period t is based on exposure corresponding to the waveform projection region of the N memory cell groups. ｐ The depth data measuring head according to claim 4, wherein the light intensity values ​​of each waveform projection region are derived, and the light intensity values ​​are zero or greater.

6.

7. The depth data measuring head according to claim 1, wherein each pixel in the image sensor contains N memory cells, and each of the N memory cells in each pixel belongs to one of the N memory cell groups.

8.

9.

10.

11. The projection device, A light-emitting device for generating line light, Includes a reflecting device for reflecting line light and projecting line light moving perpendicular to the stripe direction at a predetermined frequency onto the imaging area, The longitudinal direction of the line light is the longitudinal direction of the projected stripe, The aforementioned reflective device is A mechanical vibrating mirror that vibrates back and forth at a predetermined frequency, A micromirror device that reciprocates at a predetermined frequency, and A depth data measuring head according to claim 1, comprising one of a set of mechanically rotating mirrors that rotate in one direction at a predetermined frequency.

12. The depth data measuring head according to claim 1, wherein the image sensor includes a first image sensor and a second image sensor having constant relative positions, the first image sensor and the second image sensor each include the N memory cell groups and are exposed in synchronous manner with respect to each other.

13.

14.

15.

16. It includes a first depth imaging measurement head and a second depth imaging measurement head whose relative positions are constant. Furthermore, the first depth imaging measurement head and the second depth imaging measurement head are depth data measurement heads according to any one of claims 1 to 14. Herein, the depth data measuring device is configured such that after the first depth imaging measuring head completes a scan in the first pattern, the second depth imaging measuring head performs a scan in the second pattern, and the first group of N-step phase shift patterns obtained from the scan in the first pattern and the second group of N-step phase shift patterns obtained from the scan in the second pattern are combined with depth information of the imaging target within the imaging region based on the relative position.

17.

18.

19. The depth data measuring device captures a plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns that move relative to the imaging target, The depth data measurement method according to claim 18, further comprising the step of synthesizing depth information generated from a plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns with model information of the imaging target based on a calibration point.

Images