Depth data measurement head, measurement device, and measurement method

Abstract

The measurement head (300) includes a projection device (310) for projecting line light onto the imaging region, and image sensors (320_1, 320_2) including N memory cell groups, where each memory cell group is exposed with an exposure switch period t that is separated from each other by 2π / N in phase. e The projection device (310) completes a scan in one pattern within the scan period, and the scan period includes a plurality of repeated sub-cycles. In each sub-cycle, the projection period t p includes N waveform projection regions with a width of 2π / N and the light intensity encoded based on the imaging pattern. When completing a scan in one pattern within the scan period, the N memory cell groups of the image sensors (320_1, 320_2) image one different stripe light pattern respectively, and there is a phase shift of 2π / N between the N stripe light patterns.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional imaging, and more particularly to a depth data measurement head, a measuring device, and a measuring method.

Background Art

[0002] A depth camera is a device that collects depth information of a target object. Such cameras are widely used in fields such as three-dimensional scanning and three-dimensional modeling. For example, currently, more and more smartphones are equipped with depth imaging devices for face recognition. In the prior art, high-precision imaging can be achieved by using stripe light encoding. Stripe light encoding requires taking a plurality of stripe images and synthesizing them into a single depth image. Therefore, the obtained depth image has a low frame rate and cannot meet the requirements of real-time and high-precision dynamic imaging.

[0003] Therefore, there is a need for improved depth data measurement means.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technical problem to be solved by the present disclosure is to provide improved depth data measurement means. By using an image sensor in which each pixel includes a plurality of memory cell groups for phase shift exposure, it is possible to image line light having continuous phase shifts in different sub-cycles of the projected light. In a single scan of the line light, N memory cell groups of the image sensor can each obtain a different phase shift stripe image, thereby realizing the acquisition of N stripe images by a single line light scan. As a result, without sacrificing the resolution of the image sensor, the synthesis speed of the depth map can be significantly improved, and it is suitable for photographing a moving target object.

Means for Solving the Problems

[0005] TIFF2025518634000002.tif78169

[0006] Optionally, the projection period t of the line light p is synchronized with the exposure switch period t of the first memory cell group e

[0007] Optionally, in each sub-phase T i the projection waveforms within each projection period t of the line light p are the same, and are rectangular waves with a bright region of 2π / N phase and a dark region of 6π / N phase, and the N stripe light patterns are stripe light patterns in which the bright and dark regions are repeated

[0008] TIFF2025518634000003.tif27169

[0009] Optionally, in each sub-phase T i the line light is projected m times with a projection period t p and the duration of each sub-phase T i is longer than the residence time t c

[0010] TIFF2025518634000004.tif51169

[0011] TIFF2025518634000005.tif45169

[0012] TIFF2025518634000006.tif56169

[0013] ​​Optionally, the projection device includes a light emitting device for generating line light, and a reflecting device for reflecting the line light and projecting the line light moving in a direction perpendicular to the stripe direction at a predetermined frequency onto the imaging region, wherein the longitudinal direction of the line light is the longitudinal direction of the projected stripe, and the reflecting device includes one of a mechanical vibration mirror reciprocating at the predetermined frequency, a micromirror device reciprocating at a predetermined frequency, and a mechanical rotating mirror rotating in one direction at a predetermined frequency.

[0014] Optionally, the image sensor includes a first image sensor and a second image sensor with a fixed relative position, the first image sensor and the second image sensor each include the N memory cell groups, and are exposed synchronously with each other.

[0015] TIFF2025518634000007.tif69169

[0016] TIFF2025518634000008.tif65169

[0017] TIFF2025518634000009.tif38169

[0018] According to a third aspect of the present disclosure, there is provided a depth data measuring device including a first depth imaging measurement head and a second depth imaging measurement head with a fixed relative position, and the first depth imaging measurement head and the second depth imaging measurement head are the depth data measurement heads described in the first aspect. Here, after the first depth imaging measurement head completes scanning in a first pattern, the second depth imaging measurement head performs a second pattern scan, and a first group of N-step phase shift patterns obtained by scanning in the first pattern and a second group of N-step phase shift patterns obtained by scanning in the second pattern are combined into depth information of an imaging object in the imaging region based on the relative position.

[0019] TIFF2025518634000010.tif95169

[0020] TIFF2025518634000011.tif139169

[0021] Optionally, the method further includes: the depth data measuring device capturing a plurality of first group of N-step phase shift patterns and a plurality of second group of N-step phase shift patterns in which the depth data measuring device moves relative to the imaging object; and the depth data measuring device synthesizing the depth information generated from the plurality of first group of N-step phase shift patterns and the plurality of second group of N-step phase shift patterns into the model information of the imaging object based on calibration points.

Advantages of the Invention

[0022] Thereby, the depth imaging measurement head of the present invention can achieve simultaneous acquisition of N-step phase shift patterns by a single scan of line light while maintaining the image resolution, thereby improving the imaging speed.

Brief Description of the Drawings

[0023] TIFF2025518634000012.tif247169

Embodiments for Carrying Out the Invention

[0024] Preferred embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited to the embodiments described herein. These embodiments are provided to make the present disclosure thorough and complete and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] As can be understood from the measurement principle of structured light, whether the scan angle α can be accurately identified is a key point of the entire measurement system. The structured light formed by points and lines can calculate and identify the scan angle by mechanical devices such as rotating mirrors. The significance of image encoding and decoding is to identify the scan angle of the encoded structured light (i.e., surface structured light) system. Figure 1 shows the principle of depth imaging using stripe code structured light. For the sake of easy understanding, the encoding principle of stripe structured light is briefly explained in the figure by a 3-bit binary time code with two gray scales. The projection device can project three patterns as shown in the figure onto the object to be measured in the imaging area in sequence. In the three patterns, two gray scales of light and dark are used to divide the projection space into eight regions. It can be assumed that each region corresponds to a respective projection angle, the bright region corresponds to code "1", and the dark region corresponds to code "0". Combine the code values in the three encoded patterns of a point on the scene in the projection space according to the projection order to obtain the region code value of the point, thereby identifying the region where the point is located and decoding and obtaining the scan angle of the point.

[0026] In order to improve the matching accuracy, the number of projection patterns in the time code can be increased. Figure 2 shows another example of projecting stripe code structured light. Specifically, a 5-bit binary time code with two gray scales is shown in the figure. For example, in the application scenario of binocular imaging, this means that each pixel in each of the left and right image frames contains five region code values that are either 0 or 1, so that the left and right image matching can be realized with higher accuracy (for example, at the pixel level). When the projection speed of the projection device does not change, compared with the three code patterns in Figure 1, the example in Figure 2 corresponds to realizing higher-precision image matching at a higher time-domain cost.

[0027] FIG. 3 is a schematic diagram of the principle of obtaining depth data by projecting a stripe image using line light. As shown in FIG. 3, the depth data measurement head 300 includes a projection device 310 and two image sensors 320_1 and 320_2. In a monocular embodiment, the depth data measurement head 300 can also perform imaging using one image sensor.

[0028] The projection device 310 is used to scan and project structured light having a stripe code onto the imaging area. For example, within the projection period of three consecutive image frames, the projection device 310 can continuously project three patterns as shown in FIG. 1, and the imaging results of the three patterns are used to generate depth data. 320_1 and 320_2, which can be called the first and second image sensors respectively, have a predetermined relative positional relationship for imaging the imaging area and obtaining the first and second two-dimensional image frames respectively under the irradiation of structured light. For example, as shown in FIG. 1, when the projection device 310 projects three patterns, the first image sensor 320_1 and the second image sensor 320_2 can image the imaging area (for example, the imaging plane in FIG. 3 and the areas within a certain range before and after it) where these three patterns are projected within the imaging periods of the three synchronized image frames.

[0029] As shown in FIG. 3, the projection device 310 can project line light extending in the x direction in the z direction (i.e., toward the imaging area). The projected line light can be continuously moved in the y direction so as to cover the entire imaging area. The lower part of FIG. 3 is a more understandable illustration of the scan of the line light corresponding to the perspective view of the imaging area.

[0030] In the present disclosure, the direction in which the light beam exits from the measurement head is defined as the z-direction, the vertical direction of the imaging plane is defined as the x-direction, and the horizontal direction is defined as the y-direction. Therefore, the stripe structured light projected by the projection device may be the result of the line light extending in the x-direction moving in the y-direction. In other embodiments, synchronization and imaging processing can also be performed on the stripe structured light obtained by moving the line light extending in the horizontal y-direction in the x-direction. However, in the present disclosure, the description will be preferentially made using vertical stripe light.

[0031] Figures 4A - B are enlarged operation examples of the projection device shown in Figure 3. Specifically, as shown in Figure 3, in the projection device 310, the laser emitted by the laser emitting element (for example, the laser emitting element 411 shown in detail in Figures 4A - B) is scanned and projected onto the imaging area (the gray area in Figure 3) by the projection mechanism (for example, the projection mechanism 412 shown in detail in Figures 4A - B), and is used to actively project structured light onto the measurement object (for example, the human in Figure 3) within the imaging area. The pair of image sensors 320_1 and 320_2 perform imaging on the imaging area, thereby obtaining the image frames necessary for calculating the depth data. As shown in Figure 3, the dashed lines extending from the projection device 310 are used to indicate its projection range, and the dashed lines extending from the image sensors 320_1 and 320_2 are used to indicate their respective imaging ranges. The imaging area is generally located in the overlapping area of the projection and imaging ranges of these three components respectively.

[0032] In actual applications, the laser emitting element is used to generate line and / or infrared lasers, and the laser emitting element performs high-speed switching so as to scan and project structured light with alternating light and dark corresponding to the stripe code. The high-speed switching may include high-speed switching of the laser emitting element and high-speed code switching.

[0033] In one embodiment, the laser emitting element may continuously emit lasers with the same intensity, and the projected stripe light pattern may be realized by turning the laser emitting element on and off. In this case, the laser emitting element only projects light of one intensity with different cycle duty cycles, and each pixel of the image sensor integrates the projected light to determine the "presence or absence" of the irradiated light. Therefore, the arranged image sensor may be a black-and-white image sensor.

[0034] In another embodiment, the laser emitting element itself may emit a laser with a changing light intensity, for example, a laser whose emitted light intensity exhibits a sine transformation within one large cycle according to the applied power. The sine-transformed laser can be combined with stripe projection, thereby scanning and projecting a pattern that alternates between light and dark and has different luminances between bright stripes. In this case, the image sensor needs to have the ability to distinguish different light intensities for imaging. Therefore, it may be a multi-tone gray-scale image sensor. Obviously, gray-scale projection and imaging can provide more accurate pixel-to-pixel matching compared to black-and-white projection and imaging, thereby improving the accuracy of depth data measurement.

[0035] In one embodiment, the laser emitting element 411 may be a line laser emitting element that generates line light extending in the x direction (the direction perpendicular to the paper surface in FIGS. 4A - B). The line light is then projected onto the imaging plane by a reflection mechanism 412 that can swing along the axis in the x direction. The swinging range of the reflection mechanism 412 is as shown in FIG. 4B, and the projection mechanism 412 (for example, a reflecting mirror) can scan within the range of the angle α, thereby realizing a line light scan that reciprocates within the AB range of the imaging plane.

[0036] It should be understood that in order to realize the projection of the stripe light pattern, it is necessary for the line light to change in brightness (or in a simple embodiment, to change in blinking) continuously in the y direction during the process of moving continuously. For example, when it is necessary to scan the first pattern in FIG. 1, when the projection mechanism 412 scans over an angle of 0 to α / 2, the laser emitting element 411 remains off, and when scanning over an angle of α / 2 to α, the laser emitting element 411 turns on, thereby realizing a pattern with a dark left side and a bright right side. When it is necessary to scan the second pattern in FIG. 1, when the projection mechanism 412 scans over an angle of 0 to α / 4, the laser emitting element 411 remains off, when scanning over an angle of α / 4 to α / 2, the laser emitting element 411 turns on, when scanning over an angle of α / 2 to 3α / 4, the laser emitting element 411 turns off again, and when scanning over an angle of 3α / 4 to α, the laser emitting element 411 turns on. Thereby, a dark-bright-dark-bright pattern is realized. Similarly, more frequent changes based on the rotation angle can realize the third pattern in FIG. 1 and the pattern with finer stripe light shown in FIG. 2.

[0037] In one embodiment, the reflection mechanism 412 may be a micromirror device (also referred to as a digital micromirror device, DMD), and may also be realized as a MEMS (microelectromechanical system). FIG. 5 is a simplified perspective principle diagram of the projection device used in the present invention. As shown in FIG. 5, the spot laser generated by the laser element can obtain line light through a lens (corresponding to the line laser emitting element 411 in FIG. 4), the line light is further reflected by a micromirror device in the form of MEMS, and the reflected line light is further projected into the external space through an optical window. The micromirror device has very high performance. For example, a commercially available DMD can perform extremely stable reciprocating vibrations at a frequency of 2k, thereby laying the foundation for high-performance depth imaging.

[0038] In order to obtain a high-precision depth map, the depth data measurement head shown in FIG. 3 needs to sequentially project a plurality of different stripe light patterns. That is, in the conventional method of synthesizing a depth map using the captured stripe light pattern, accuracy is obtained by sacrificing the time domain. Further, different stripe light patterns captured in N consecutive imaging cycles are used for the synthesis of one depth map. Therefore, the conventional depth data measurement method is applicable only when the object being captured remains stationary during the process of N imaging cycles, greatly limiting the application range of the technology for deriving depth data using actively projected stripe images.

[0039] In view of this, the present invention provides a novel depth data measurement means, which utilizes an image sensor provided with different memory cell groups capable of phase shift exposure, and by skillfully setting the brightness change of the projected line light, in a single scan of the line light, different memory cell groups of the image sensor can respectively obtain different phase shift stripe images, thereby realizing the acquisition of a plurality of stripe images by a single line light scan. Thereby, the synthesis speed of the depth map can be greatly improved, and it is suitable for photographing a moving target object.

[0040] In one embodiment, the present invention can be realized as a depth imaging measurement head including a projection device and an image sensor. The projection device is used to project a line light moving along a first direction (for example, the y direction in FIG. 3) onto an imaging region, and the longitudinal direction of the line light is a second direction (for example, the x direction in FIG. 3) perpendicular to the first direction. In one embodiment, the projection device may have the implementation structure shown in FIG. 5, and the implementation structure includes a line light generating device and a projection mechanism that reflects and projects the line light and can change the projection direction within a certain angle.

[0041] Each pixel of the image sensor includes N memory cells, and the N memory cells of each pixel belong to one of N memory cell groups respectively. The N memory cell groups are exposed with an exposure switch period t that is separated from each other by 2π / N phase. e Here, N is an integer greater than 1.

[0042] Here, for clarity, taking N = 4 as an example, the structure and exposure of the image sensor will be described. FIGS. 6A to 6B are examples of N memory cell groups included in the image sensor used in the present invention. The N memory cell groups included in the image sensor may be such that each pixel includes N memory cells, or the image sensor may include N pixel groups and each pixel may correspond to one memory cell. FIG. 6A is a structure of one pixel column in which each pixel includes N memory cells. As shown in FIG. 6A, one pixel column 621 can include k pixels P1 to P k (in the example of a resolution of 600×800, k = 600). Each pixel has the same structure, that is, includes one photosensitive cell, N switches, and N memory cells. Each switch controls corresponding to the charge accumulation of one memory cell. Specifically, pixel P1622 can include a photodiode 624 as a photosensitive cell, N switches 626, and N memory cells 628. Pixel P k 623 can include a photodiode 625 as a photosensitive cell, N switches 627, and N memory cells 629.

[0043] A memory cell is, for example, a cell that accumulates the charge generated by a charge accumulation photodiode based on the received light and outputs based on the charge accumulation amount. The N memory cells corresponding to the same pixel all acquire charge from the same photosensitive cell, but each has an exposure switch period t that is separated from each other by 2π / N phase. eExposure is performed. In each pixel, one memory cell corresponds to one of N memory cell groups. Thus, in an image sensor of 600x800 pixels, assuming N = 4, 600x800x4 memory cells are included. These 600x800x4 memory cells belong to four groups, each group including 600x800 memory cells and corresponding to one different pixel respectively. Each memory cell group can be exposed in the same period, and there is a phase difference of π / 2 between adjacent groups. Since N memory cells share one photosensitive cell, it is possible to acquire multiple images by one scan without reducing the original resolution of the image sensor.

[0044] Figure 6B is an example where the image sensor includes N pixel groups. For convenience of explanation, Figure 6B shows an example of 16x24 pixels. It should be understood that the actually used image sensor may have more pixels, for example, 600x800 pixels. The image sensor shown in Figure 6B includes 4 (N = 4) groups of pixels uniformly distributed over the entire imaging surface, and are respectively denoted by the numbers 1, 2, 3, and 4 within the illustrated blocks. Here, the fact that the four types of pixels are "uniformly distributed" over the entire imaging surface means that when the line light is scanned and projected in the y direction, within the current irradiation area, each type of pixel irradiates the same (or approximately the same) number of pixels. In a preferred embodiment, these four pixel groups are arranged at intervals of one pixel as shown in Figure 6B. That is, the image sensor shown in Figure 6B includes a plurality of "pixel units" (in the example of Figure 6B, it may include 8x12 pixel units with the same structure as shown by the thick black frames in the figure), and the pixels included in each pixel unit may be regarded as belonging to one of the four pixel groups. In other embodiments, each pixel group may be arranged at intervals of two pixels (for example, two pixels arranged adjacent to each other in the x direction).

[0045] FIG. 7 is an example of the relative relationship of exposure periods between different memory cell groups of the same image sensor. In the example of FIG. 7, four memory cell groups have the same exposure switch period t e and are all switched at a 50% duty cycle, that is, all the memory cells of the image sensor all have the same exposure switch waveform. The difference is that there is the same phase difference of π / 2 between the waveforms of consecutive different memory cell groups. In one embodiment, the exposure switch period t e takes, for example, a typical value of 20 ns. This means that each memory cell in the image sensor is operated at intervals of being on for 10 ns to receive exposure and off for 10 ns, but the on-timing of the memory cells in the second group is 5 ns later than that of the first group, the on-timing of the memory cells in the third group is 5 ns later than that of the second group, and the on-timing of the memory cells in the fourth group is 5 ns later than that of the third group (it can also be regarded as 5 ns earlier than that of the memory cells in the first group).

[0046] When the image sensor used can perform phase-shifted exposure grouped as shown in FIGS. 6A - B and FIG. 7, by skillfully setting the line light projection of the projection device, it is possible to realize the acquisition of a plurality of images by one scan.

[0047] TIFF2025518634000013.tif34169

[0048] TIFF2025518634000014.tif47169

[0049] TIFF2025518634000015.tif32169

[0050] In one example, one N-step phase-shifted pattern group is a sine wave 4-step phase-shifted pattern, and based on the exposure corresponding to the waveform projection regions of N memory cell groups, the value of the light intensity of each waveform projection region in each projection period t p is derived, where the value of the light intensity is zero or more.

[0051] TIFF2025518634000016.tif30169

[0052] P1 = Σ(Q1 + Q2) P2 = Σ(Q2 + Q3) P3 = Σ(Q3 + Q4) P4 = Σ(Q1 + Q4) (1)

[0053] Accordingly, the respective values of Q1 to Q4 can be specified according to the pattern type corresponding to the required four-step phase-shifting imaging.

[0054] TIFF2025518634000017.tif32169

[0055] P1 = Q / 2 * sint + Q / 2 P2 = -Q / 2 * cost + Q / 2 P3 = -Q / 2 * sint + Q / 2 P4 = Q / 2 * cost + Q / 2 (2)

[0056] TIFF2025518634000018.tif30169

[0057] From equation (2), the luminance values of Q1 to Q4 with respect to t can be inversely obtained. In equation (2), the unknowns Q1 to Q4 have N = 4, and since the rank of equation (2) is N - 1 = 3, Q1 to Q4 actually have an infinite number of solutions, and the values of the solutions are given as follows.

[0058] Q1 = A / 2 * sint + A / 2 Q2 = 0 Q3 = -A / 2 * cost + A / 2 Q4 = A / 2(cost - sint) (3)

[0059] Here, O*A = Q, where O is, for example, the number of exposure cycles each pixel receives during line light scanning (for example, in the following example, each pixel column can complete 100 exposure ons during the 2 us of line light scanning, and in this case, O can be regarded as 100).

[0060] However, since the light intensity must not be negative, each value of Q1 to Q4 needs to be kept non - negative. Equation (3) holds within the range of t = 0 to π / 4. When line light scanning is performed up to the range of t = π / 4 to π / 2, other solutions can be obtained based on Equation (2).

[0061] Q1 = A / 2*cost + A / 2 Q2 = A / 2(sint - cost) Q3 = -A / 2*sint + A / 2 Q4 = 0 (4)

[0062] Figure 9 is an example of the relative relationship between the values of the projection light waveform when imaging a sine - wave 4 - step phase - shift pattern and the exposure cycles of memory cell groups 1 to 4.

[0063] As shown in the figure and as can be seen by referring to Figure 8 and Equation (1), when t = 0, the values taken by P1 to P4 may respectively correspond to Q / 2, 0, Q / 2, and Q.

[0064] TIFF2025518634000019.tif44169

[0065] TIFF2025518634000020.tif35169

[0066] TIFF2025518634000021.tif80169

[0067] Returning to Fig. 9, in the process of the line light scanning from the position corresponding to t = 0 to the position corresponding to t = π / 6, for example, the values of Q1 to Q4 can be obtained respectively based on Equation (3). When t = π / 6, the line light can be regarded as scanning the 5th pixel column among the 60 pixel columns of the current cycle, and based on Equation (3), the corresponding Q1 = 3A / 4, Q2 = 0, Q3 = 0.183A, Q4 = 0.067A at t = π / 6 are obtained. As the line light scans from the position corresponding to t = π / 6 to the position corresponding to t = π / 3, the values of Q1 to Q4 can continue to be obtained respectively. When t = π / 4, cost = sint, and since cost < sint in the subsequent process from π / 4 to 3π / 4, the formula (3) for obtaining Q4 < 0 is no longer applicable. At this time, based on Equation (4), the corresponding Q1 = 3A / 4, Q2 = 0.183A, Q3 = 0.067A, Q4 = 0 at t = π / 3 can be obtained. And when t = π / 3, the line light can be regarded as scanning the 10th pixel column among the 60 pixel columns of the current cycle.

[0068] As the line light scans from the position corresponding to t = π / 3 to the position corresponding to t = π / 2, the values of Q1 to Q4 can continue to be obtained respectively based on Equation (4). When t = π / 2, the line light can be regarded as scanning the 15th pixel column among the 60 pixel columns of the current cycle. Based on Equation (4), the corresponding Q1 = A / 2, Q2 = A / 2, Q3 = 0, Q4 = 0 at t = π / 2 are obtained. Note that for pixels 1 to 4 to perform 4-step phase-shifting imaging, for each projection period t p there is always Q1 + Q2 + Q3 + Q4 = A, and pixels 1 to 4 are exposed with a phase difference of 2π / N from each other in the projection period t p and the on-time accounts for 10%. Therefore, in each projection period t p for pixels 1 to 4, the integrated luminance of 2A * t p / 4 can be obtained.

[0069] The above described the examples of the values of Q1 to Q4 in the first π / 2 of the 2π cycle of the sine wave N-step phase shift pattern and the examples of imaging for each of pixels 1 to 4 with reference to FIG. 9. Those skilled in the art can continue to obtain the examples of the values of Q1 to Q4 in the last 3π / 2 of the 2π cycle based on Equation (2) and the example in FIG. 9.

[0070] TIFF2025518634000022.tif43169

[0071] TIFF2025518634000023.tif107169

[0072] TIFF2025518634000024.tif85169

[0073] For clarity, an example with α = 2 and N = 4 will be described. That is, an image sensor including four pixel groups exposed at exposure switch periods t separated from each other by a phase of 2π / N is used, and by scanning twice, acquisition of an 8-step phase shift pattern is realized. e In addition, for convenience of explanation, an example with N = 4 is shown, but in other embodiments, N may be other values. Specifically, N = 2

[0074] TIFF2025518634000025.tif77169

[0075] where n is an integer of 1 or more. Thereby, for example, 8-step phase shift using eight memory cell image sensor groups with higher precision, 16-step phase shift using 16 memory cell image sensor groups, etc. can be realized. n

[0076] TIFF2025518634000026.tif46169

[0077] TIFF2025518634000027.tif31169

[0078] ​When P1 = Q for t = 0 to π; P1 = 0 for t = π to 2π When P2 = 0 for t = 0 to π / 2; P2 = Q for t = π / 2 to 3π / 2; P2 = 0 for t = 3π / 2 to 2π When P3 = 0 for t = 0 to π; P3 = Q for t = π to 2π When P4 = Q for t = 0 to π / 2; P3 = 0 for t = π / 2 to 3π / 2; P3 = Q for t = 3π / 2 to 2π (5)

[0079] Therefore, when obtaining Q1 to Q4 based on equations (1) and (5), the optimal solution is as follows.

[0080] When Q1 = A for t = 0 to π / 2; Q1 = 0 for t = π / 2 to 2π When Q2 = 0 for t = 0 to π / 2; Q2 = A for t = π / 2 to π; Q2 = 0 for t = π to 2π When Q3 = 0 for t = 0 to π; Q3 = A for t = π to 3π / 2; Q3 = 0 for t = 3π / 2 to 2π When Q4 = 0 for t = 0 to 3π / 2; Q4 = A for t = 3π / 2 to 2π (6)

[0081] Here, O * A = Q, and O is, for example, the number of exposure cycles that each pixel receives during line light scanning.

[0082] TIFF2025518634000028.tif35169

[0083] TIFF2025518634000029.tif29169

[0084] As shown in FIGS. 13A to D, the projected laser has a projection period t e of the same length as the exposure switch period t p and is laser-projected, and each laser projection period t p always maintains synchronization with the exposure switch period t of the first memory cell group e (laser projection period t pis turned on with different phases in different sub - phases), and is switched with a duty cycle of 25% (i.e., the bright area is 2π / N), that is, the waveform is a rectangular wave with a duty cycle of 25%. In the four sub - phases T1 to T4, the projection - on time within each projection period t of the projection laser p is combined with the exposure - on times of the first to fourth memory cell groups respectively.

[0085] Specifically, first, as shown in Fig. 13A, in sub - phase T1, the projection laser is kept on within the first π / 2 phase of each projection period t p At this time, since the first memory cell group and the fourth memory cell group are also on, as shown by the gray rectangles in the figure, the projection light reflected within the first π / 2 phase of each projection period t p can be exposed, and thereby, charge accumulation can be performed in the corresponding memory cells. After completing the projection for a predetermined number of m1 cycles, sub - phase T1 ends, and the projection laser enters sub - phase T2, which keeps the laser on within the π / 2 - π phase of each projection period t p

[0086] As shown in Fig. 13B, in sub - phase T2, the projection laser is kept on within the π / 2 - π phase of each projection period t p At this time, since the first memory cell group and the second memory cell group are also on, as shown by the gray rectangles in the figure, the projection light reflected within the π / 2 - π phase of each projection period t p can be exposed, and thereby, charge accumulation can be performed in the corresponding memory cells. After completing the projection for a predetermined number of m2 cycles, sub - phase T2 ends, and the projection laser enters sub - phase T3, which keeps the laser on within the π - 3π / 2 phase of each projection period t p

[0087] As shown in Fig. 13C, in sub - phase T3, the projection laser is kept on within each projection period t pis held on within the phase of π to 3π / 2. At this time, since the second memory cell group and the third memory cell group are also on, as shown by the gray rectangles in the figure, each projection period t p the projection light reflected within the phase of π to 3π / 2 can be exposed, whereby charge accumulation can be performed within the corresponding pixel. After the projection of a predetermined m3 number of cycles is completed, the sub-phase T3 ends, and the projection laser enters the sub-phase T4 which is held on within the phase of 3π / 2 to 2π of each projection period t p .

[0088] TIFF2025518634000030.tif42169

[0089] TIFF2025518634000031.tif103168

[0090] TIFF2025518634000032.tif72168

[0091] TIFF2025518634000033.tif89168

[0092] To achieve a 4-step phase shift, the duration length of each sub-phase T i needs to be longer than the time for the line light to scan one pixel unit column. When using the 1920-column image sensor in the above example to obtain a 4-step phase shift pattern with 32 stripes (16 bright stripes and 16 dark stripes) shown in FIG. 10, each stripe covers 60 pixel columns (1920 / 32 = 60). As shown in FIG. 9, since each sub-phase T i corresponds to half of the stripe, it covers 30 pixel columns and the duration is 2 us x 30 = 60 us. Therefore, in this example, m1 = m2 = m3 = m4 = m = 60 us / 20 ns = 3000. The line light projection period t p is the exposure switch period t eSince it is the same length of time, when the line light scans and passes through half the distance of the stripe, each corresponding memory cell group is also switched 3000 times.

[0093] TIFF2025518634000034.tif72168

[0094] The above embodiment of obtaining the light and dark stripes shown in FIG. 14 based on a rectangular wave with a duty cycle of 100 / N % and a constant brightness, the projection period t of the line light p According to, each 2π / N phase can be individually adjusted, and thereby the phase difference is 2π / N and the exposure period is also t p Equal to, in combination with an image sensor, it can be regarded as a special case of the imaging method of the present invention that generates one N-step phase shift pattern group.

[0095] TIFF2025518634000035.tif55168

[0096] Furthermore, in order to realize the scan projection, the projection device of the present invention includes a light emitting device for generating line light, and a reflection device for reflecting the line light and projecting the line light moving in a direction perpendicular to the stripe direction at a predetermined frequency onto the imaging area. The longitudinal direction of the line light is the longitudinal direction of the projected stripe, and the reflection device includes one of a mechanical vibration mirror that reciprocates at the predetermined frequency, a micromirror device that reciprocates at a predetermined frequency, and a mechanical rotation mirror that rotates in one direction at a predetermined frequency. Here, the projected line light may be a line light presenting a high-dimensional Gaussian distribution or a flat-top Gaussian distribution, thereby providing a highly uniform luminance distribution in the width direction of the line light.

[0097] TIFF2025518634000036.tif59168

[0098] FIG. 15 is a schematic diagram of a depth data measurement device according to an embodiment of the present invention. As shown in the figure, the measurement device 1500 can include the measurement head and the processor 1530 as described above. The measurement head includes a projection device 1510 and two image sensors 1520.

[0099] The processor 1530 is connected to the measurement head, for example, connected to each of the projection device 1510 and the two image sensors 1520, and based on the predetermined relative positions of the first image sensor 1520_1 and the second image sensor 1520_2 and the N first two-dimensional image frames and the N second two-dimensional image frames obtained by imaging the structured light, it is used to identify the depth data of the object to be photographed in the photographing area.

[0100] FIGS. 16A - B are schematic flowcharts of a method for measuring depth data according to an embodiment of the present invention. This method can be implemented by the depth data measurement head and the measurement device of the present invention.

[0101] FIG. 16A shows an execution method by an image sensor for each pixel, each including N memory cells.

[0102] TIFF2025518634000037.tif52169

[0103] In step S1620, each pixel uses an image sensor including N memory cells to photograph the imaging area, thereby obtaining N image frames under the line light scan projection. Here, the N memory cells of each pixel all belong to one of the N memory cell groups, and each memory cell group is exposed with an exposure switch period t separated by 2π / N in phase from each other. e for exposure.

[0104] In step S1630, based on the image frames, the depth data of the object to be measured in the imaging area is derived.

[0105] FIG. 16B shows a method of execution by an image sensor including N memory cell groups corresponding to N pixel groups.

[0106] TIFF2025518634000038.tif34168

[0107] In step S1620’, by using an image sensor including N pixel groups evenly arranged on the imaging surface to capture an imaging region, N image frames are acquired, where each pixel group is exposed with an exposure switch period t that is separated from each other by 2π / N in phase. e for exposure.

[0108] In step S1630’, based on the first and second two-dimensional image frames, depth data of the object to be measured in the captured region is derived.

[0109] TIFF2025518634000039.tif35168

[0110] Furthermore, when imaging is performed using actively projected structured light, problems of shadows and dead angles are encountered. FIG. 17 shows an example in which part of the depth information of the imaging object is missing when a single camera performs imaging using structured light. As shown in the figure, due to the unevenness of the structure of the imaging object itself, the structured light received by a part of its own structure is blocked, a shadow is generated in the range towards the camera of the imaging object, and since the structured light is not irradiated on the shadow part, when deriving the subsequent depth information, the depth information of the corresponding part will be missing (corresponding to “missing 1” in the figure). Also, no matter what structure the imaging object itself has, the part on the opposite side of the structured light generator of the imaging object cannot receive the irradiation of the structured light, and when deriving the subsequent depth information, the depth information of the corresponding part will be missing (corresponding to “missing 2” in the figure).

[0111] FIG. 18 shows an example in which a dual camera improves and images a part of the missing depth information of an imaging target. As shown in the figure, by installing cameras on both the left and right sides, more parts of the imaging target can receive structured light irradiation, thereby realizing measurement of more depth information for the imaging target.

[0112] In view of this, the present invention can be realized particularly as a depth data measurement device for a dual camera (i.e., a dual measurement head). FIG. 19 is a schematic diagram of a depth data measurement device according to an embodiment of the present invention. As shown in the figure, the depth data measurement device 1900 can include a first depth imaging measurement head 1910 and a second depth imaging measurement head 1920 with a fixed relative position. Specifically, the first depth imaging measurement head 1910 and the second depth imaging measurement head 1920 can be mounted in the same outer case, thereby ensuring fixation of the relative position.

[0113] Furthermore, the measurement head 1910 and the measurement head 1920 can perform continuous structured light projection and imaging under the control of the same control device (not shown, for example, a processor). Here, "continuous" means that the measurement head 1910 first performs one scan and imaging, and then the measurement head 1920 performs one scan and imaging, and line light projection and imaging are performed in different time zones respectively. Since the exposure period of each measurement head is shorter than the framing time of depth detection (for example, when the exposure period of one depth frame is 5 ms and the depth detection frame rate is 30 fps, the interval between each depth frame is about 28 ms), the exposure of other measurement heads can be performed during the exposure interval period of the current measurement head, and exposure imaging and depth calculation can be performed alternately. That is, the depth data measuring device of the present invention may include two measurement heads as described above, and may further include more measurement heads that alternately perform exposure imaging and depth calculation, and the imaging information of these measurement heads may be combined in the same xyz-axis space. Since frame interleaving detection is used, the depth detection frame rate can still reach 30 fps without affecting the frame rate and detection frame rate of the system.

[0114] As shown in the figure, the measurement head 1910 and the measurement head 1920 may have the same configuration. For example, both may be two measurement heads with exactly the same configuration, and may be installed to continuously capture images of the same imaging region (the imaging object within the imaging region) from different angles. The measurement head 1910 and the measurement head 1920 can be realized as a measurement head that includes N memory cell groups as described above and completes N image imagings in one scan.

[0115] Here, continuous shooting means that after the measurement head 1910 completes the scan in the first pattern (the projection device 1911 performs line light scan projection and the image sensor 1912 images the projected line light), the measurement head 1920 performs a scan in the second pattern (the projection device 1921 performs line light scan projection and the image sensor 1922 images the projected line light), and the first group of N-step phase shift patterns obtained by the scan in the first pattern and the second group of N-step phase shift patterns obtained by the scan in the second pattern are synthesized into the depth information of the imaging target within the imaging region based on the relative position.

[0116] Furthermore, the outer case of the depth data measuring device 1900 may include a handle 1931 structure and can be realized as a portable modeling device. In this case, the depth data measuring device captures a plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns that move relative to the imaging target, and synthesizes the depth information generated from the plurality of first groups of N-step phase shift patterns and the plurality of second groups of N-step phase shift patterns into the model information of the imaging target based on the calibration points.

[0117] FIG. 20 is a schematic diagram of a depth data measuring device according to an embodiment of the present invention. As shown in the figure, the measuring device 2000 can include two measurement heads 2010 and 2020 as described above, and a processor 2030. The measurement head 2010 includes a projection device 2011 and an image sensor 2012. The measurement head 2020 includes a projection device 2021 and an image sensor 2022.

[0118] Processor 2030 is connected to two measurement heads, for example, connected to projection device 2011 and image sensor 2012, and also connected to projection device 2021 and image sensor 2022. Projection device 2011 can perform one scan projection as described above under the control of processor 2030, and the N memory cell groups of image sensor 2012 perform corresponding imaging, thereby obtaining one group of N-step phase shift images after one scan projection. Next, projection device 2021 can perform one scan projection as described above under the control of processor 2030, and the N memory cell groups of image sensor 2022 perform corresponding imaging, thereby obtaining one group of N-step phase shift images after one scan projection. These two groups of N-step phase shift images may each be used to synthesize one depth map of the imaging object, and these two depth maps may be combined based on the positional relationship between the two measurement heads, thereby obtaining more comprehensive data of the imaging object.

[0119] Figure 21 is a schematic flowchart of a method for measuring depth data according to an embodiment of the present invention. This method can be implemented by the depth data measurement device of the present invention.

[0120] In step S2110, a first group of N-step phase shift patterns is obtained by scanning the imaging area with a first pattern using the first depth imaging measurement head of the depth data measurement device.

[0121] In step S2120, a second group of N-step phase shift patterns is obtained by scanning the imaging area with a second pattern using the second depth imaging measurement head of the depth data measurement device.

[0122] In step S2130, based on the relative positions of the first depth imaging measurement head and the second depth imaging measurement head, depth information of an imaging target within the imaging region is synthesized from the N-step phase shift patterns of the first group and the N-step phase shift patterns of the second group.

[0123] TIFF2025518634000040.tif98168

[0124] The above steps S1910 to S1930 can be repeated, for example, to dynamically image an imaging target during movement. At this time, the depth data measurement device of the present invention can be held without changing its position.

[0125] In another embodiment, the depth data measurement device of the present invention may actively change its position. As shown in FIG. 19, the depth data measurement device of the present invention can be realized as a portable device particularly including a handle 1931. In this case, the depth data measurement device captures a plurality of first-group N-step phase shift patterns and a plurality of second-group N-step phase shift patterns that move relative to the imaging target, and the depth information generated from the plurality of first-group N-step phase shift patterns and the plurality of second-group N-step phase shift patterns can be synthesized into the model information of the imaging target based on calibration points.

[0126] As described above, with reference to the drawings, the depth data measurement head, measurement device, and measurement method of the present invention have been described in detail. The depth data measurement means of the present invention utilizes an image sensor provided with different pixel groups capable of phase shift exposure, images the projected phase shift line light, and in a single scan of the line light, different pixel groups of the image sensor can each obtain different phase shift stripe images, thereby realizing the acquisition of a plurality of stripe images by a single line light scan. Thereby, the synthesis speed of the depth map can be significantly improved, and it is suitable for photographing a target object during movement.

[0127] The flowcharts and block diagrams in the figures illustrate the architecture, functions, and operations of the systems and methods of multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, segment, or part of the code, and the module, segment, or part of the code includes executable commands for implementing the defined logical function. It should be noted that in some alternative embodiments, the functions described within the blocks may be performed in a different order than that shown in the figures. For example, two consecutive blocks may actually be executed substantially simultaneously, or sometimes in the reverse order depending on the functions they are related to. Also, each block in the block diagram and / or flowchart diagram, and combinations of blocks in the block diagram and / or flowchart diagram, can be implemented by a dedicated system based on hardware for performing the defined function or operation, or a combination of dedicated hardware and computer commands.

[0128] Although each embodiment of the present invention has been described, the above description is exemplary, not exhaustive, and not limited to each disclosed embodiment. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of each embodiment, the practical applications, or improvements to the technology in the market, or to enable those skilled in the art to understand each embodiment disclosed in this specification.

Claims

1.

2. The projection period t of the line light ｐ is synchronized with the exposure switch period t of the first memory cell group, and the depth data measurement head according to Claim 1. ｅ

3. N = 2 ｎ and n is an integer of 1 or more, and the depth data measurement head according to Claim 1.

4.

5. The one N-step phase shift pattern group is a sine wave 4-step phase shift pattern, and based on the exposure corresponding to the waveform projection region of N memory cell groups, the value of the light intensity of each waveform projection region in each projection period t ｐ is derived, where the value of the light intensity is zero or more, and the depth data measurement head according to Claim 4.

6.

7. In each sub-phase T ｉ the line light is projected m times with a projection period t ｐ and the duration of each sub-phase T ｉ is longer than the residence time t ｃ and the depth data measurement head according to Claim 5.

8. Each pixel in the image sensor includes N memory cells, and the N memory cells of each pixel respectively belong to one of the N memory cell groups, and the depth data measurement head according to Claim 1.

9.

10.

11.

12. The projection device includes a light emitting device for generating line light and a reflection device for reflecting the line light and projecting the line light moving in a direction perpendicular to the stripe direction at a predetermined frequency onto the imaging region, and the longitudinal direction of the line light is the longitudinal direction of the projected stripe, The reflection device is a mechanical vibration mirror that reciprocates at the predetermined frequency, a micromirror device that reciprocates at a predetermined frequency, and one of a mechanical rotation mirror that rotates in one direction at a predetermined frequency, the depth data measurement head according to claim 1.

13. The image sensor includes a first image sensor and a second image sensor with a constant relative position, the first image sensor and the second image sensor each include the N memory cell groups, and are exposed synchronously with each other, the depth data measurement head according to claim 1.

14.

15.

16.

17. includes a first depth imaging measurement head and a second depth imaging measurement head with a constant relative position, and the first depth imaging measurement head and the second depth imaging measurement head are the depth data measurement heads according to any one of claims 1 to 15, where, after the first depth imaging measurement head completes scanning in the first pattern, the second depth imaging measurement head performs scanning in the second pattern, and the first group of N-step phase shift patterns obtained by scanning in the first pattern and the second group of N-step phase shift patterns obtained by scanning in the second pattern are synthesized into depth information of the imaging object in the imaging region based on the relative position, a depth data measurement device.

18.

19.

20. the step of the depth data measurement device capturing a plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns that move relative to the imaging object, The method according to claim 19, further comprising the step of synthesizing depth information generated from a plurality of N-step phase shift patterns of a first group and a plurality of N-step phase shift patterns of a second group by the depth data measuring device into model information of the imaging target based on calibration points.

Images