Depth data measurement head, measurement device and measurement method

Abstract

A depth data measurement head, measurement device, and measurement method are disclosed. The measurement head includes: a projection device for projecting linear light onto an imaging area; and an image sensor where each pixel includes N storage units, with each of the N storage units belonging to one of N groups of storage units, and each group of storage units operating on an exposure switching period t with a phase interval of 2π / N. e Exposure is performed. The projection device completes one pattern scan within a scanning cycle, which includes multiple cyclic sub-cycles. In each cyclic sub-cycle, the projection period t... p It includes N waveform projection areas, each 2π / N wide, with light intensity encoded based on the imaging pattern. Thus, during a single pattern scan within a scanning cycle, each of the N storage units of the image sensor images a different fringe pattern, and there is a 2π / N phase shift between the N fringe patterns. This invention enables the acquisition of multiple images in a single linear light scan, significantly improving the speed of depth map synthesis, and is suitable for capturing moving targets.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional imaging, and more specifically, to a depth data measurement head, a measurement device, and a measurement method. Background Technology

[0002] A depth camera is a device that acquires depth information of a target object. These cameras are widely used in fields such as 3D scanning and 3D modeling. For example, an increasing number of smartphones are now equipped with depth cameras for facial recognition. In existing technologies, stripe light coding can be used to achieve high-precision imaging. However, stripe light coding requires capturing multiple images of different stripes and then synthesizing a single depth image. Therefore, the resulting depth image has a low frame rate and cannot meet the requirements of real-time, high-precision dynamic imaging.

[0003] Therefore, an improved depth data measurement scheme is needed. Summary of the Invention

[0004] One technical problem this disclosure aims to solve is to provide an improved depth data measurement scheme that utilizes an image sensor with multiple sets of storage units, each pixel including phase-shift exposure, to image linear light projected with successive phase shifts in different sub-cycles. This allows each of the N sets of storage units in the image sensor to acquire different phase-shift fringe images during a single scan of the linear light, thereby achieving the acquisition of N fringe images from a single scan of the linear light. This significantly improves the speed of depth map synthesis without sacrificing image sensor resolution and is suitable for capturing moving targets.

[0005] According to a first aspect of this disclosure, a depth imaging measurement head is provided, comprising: a projection device for projecting linear light moving along a first direction onto an imaging region, wherein the longitudinal direction of the linear light is a second direction perpendicular to the first direction; and an image sensor, wherein each pixel in the image sensor package includes N storage units, the N storage units of each pixel belonging to one of N groups of storage units, and each group of storage units being spaced apart by an exposure switching period t with a phase interval of 2π / N. e Exposure is performed, where N is an integer greater than 1, and the projection device is in the scanning cycle. One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In this process, the linear light has a projection period t p The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. pIt includes N waveform projection regions with a width of 2π / N, and the projected light intensity of each waveform projection region is encoded so that during the scanning period... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

[0006] Optionally, the projection period t of the linear light p The exposure switching period t of the first group of storage cells e synchronous.

[0007] Optionally, in each sub-cycle T i In the process, each projection period t of the linear light... p The internally projected waveforms are identical and are rectangular waves with a 2π / N phase bright area and a 6π / N phase dark area, wherein the N stripe patterns are stripe patterns that repeat in the bright and dark areas.

[0008] Optionally, the linear optical scan corresponds to the dwell time t on each column of pixels. c Not less than the scan period Divided by the column number C, the dwell time t c The projection period t is p More than 10 times.

[0009] Optionally, in each sub-cycle T i In this process, the linear light has a projection period t p Projected m times, and each sub-period T i The duration is greater than the dwell time t c .

[0010] Optionally, each pixel includes N storage units, which are N charge storage units. The exposure positions of the N charge storage units corresponding to the same pixel are the same, but their exposure periods are spaced apart by 2π / N phase, and during the scan period... When a pattern scan is completed, a set of N-step phase-shift patterns is obtained from N sets of charge storage units. The set of N-step phase-shift patterns is used to generate a depth map of the imaging region.

[0011] Optionally, the projection device in the first scanning cycle The image sensor completes one pattern scan within a certain period, so that each of the N storage units of the image sensor images a different stripe pattern, and the N stripe patterns form a Gray code pattern; the projection device completes one pattern scan in the second scanning cycle. A pattern scan is completed within the time frame so that each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute the set of N-step phase shift patterns. A depth map of the imaging region is generated from the set of N-step phase shift patterns based on the set of Gray code patterns.

[0012] Optionally, the projection device includes: a light-emitting device for generating linear light; and a reflective device for reflecting the linear light to project linear light moving in a direction perpendicular to the stripe direction onto the shooting area at a predetermined frequency, wherein the length direction of the linear light is the length direction of the projected stripe, and the reflective device includes one of the following: a mechanical galvanometer that reciprocates at the predetermined frequency; a micromirror device that reciprocates at the predetermined frequency; and a mechanical rotating mirror that rotates unidirectionally at the predetermined frequency.

[0013] Optionally, the image sensor includes a first image sensor and a second image sensor with fixed relative positions, wherein the first image sensor and the second image sensor each include the N sets of storage units and are exposed synchronously with each other.

[0014] Optionally, the projection device performs α scanning cycles. The pattern is scanned α times within each scan cycle. It includes multiple cyclic sub-cycles. In each cycle sub-cycle In, there are N sub-cycles T1-T N In each sub-cycle T i In this process, the linear light has a projection period t p The projection period t is used to perform brightness and darkness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p Including the bright area, among which, in the sub-cycle T1-T N During the projection period t, the bright area is... p The position of the [something] changes with a phase interval of 2π / αN, such that in each scan cycle... During one pattern scan, the N storage units of the image sensor each image a different stripe pattern, and within α scan cycles... When α pattern scans are completed, αN stripe patterns form a set of αN-step phase shift patterns with a phase shift of 2π / αN between them, where α is an integer greater than or equal to 2.

[0015] Optionally, the projection device during the scanning cycle One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In, there are N sub-cycles T1-T N In each sub-cycle T i In this process, the linear light has a projection period t p The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p Including the bright area, among which, in the sub-cycle T1-T N During the projection period t, the bright area is... p The position of the [something] changes with a phase interval of 2π / N, such that during the scan cycle... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different fringe pattern, and the N fringe patterns constitute a set of N-step phase-shift patterns with a phase shift of 2π / N between them. Optionally, N=2 n n is an integer greater than or equal to 1.

[0016] According to a second aspect of this disclosure, a depth data measurement apparatus is provided, comprising: a depth data measurement head according to the first aspect, and a processor connected to the depth data measurement head for use in the scanning cycle. When a pattern scan is completed, a depth map of the imaging region is obtained from the acquired N stripe patterns.

[0017] According to a third aspect of this disclosure, a depth data measurement device is provided, including a first depth imaging measurement head and a second depth imaging measurement head with fixed relative positions, wherein the first depth imaging measurement head and the second depth imaging measurement head are depth data measurement heads as described in the first aspect, wherein after the first depth imaging measurement head completes a first pattern scan, the second depth imaging measurement head performs a second pattern scan, and a first set of N-step phase shift patterns obtained from the first pattern scan and a second set of N-step phase shift patterns obtained from the second pattern scan are synthesized based on the relative positions to form depth information of an imaging object within the imaging area.

[0018] According to a fourth aspect of this disclosure, a depth data measurement method is provided, comprising: projecting a linear light moving along a first direction onto an imaging region, wherein the length direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light is in a scanning cycle. One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In this process, the linear light has a projection period tp The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p The system includes N waveform projection areas, each 2π / N wide, and encodes the projected light intensity for each area, where N is an integer greater than 1. An image sensor, with each pixel comprising N storage units, is used to capture images of the imaging area to obtain N image frames under the linear light scan projection. Each pixel's N storage units belong to one of N groups of storage units, and each group of storage units is separated by an exposure switching period t with a phase interval of 2π / N. e Exposure is performed; and depth data of the object under test within the imaging area is obtained based on the image frame, wherein the projected light intensity of each waveform projection area is encoded such that during the scanning cycle... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

[0019] According to a fifth aspect of this disclosure, a depth data measurement method is provided, comprising: using a first depth imaging measurement head of a depth data measurement device to perform a first pattern scan on an imaging region to obtain a first set of N-step phase shift patterns; using a second depth imaging measurement head of the depth data measurement device to perform a second pattern scan on the imaging region to obtain a second set of N-step phase shift patterns; and synthesizing depth information of an imaging object within the imaging region from the first set of N-step phase shift patterns and the second set of N-step phase shift patterns based on the relative positions of the first depth imaging measurement head and the second depth imaging measurement head, wherein the scanning imaging of the first depth imaging measurement head and the second depth imaging measurement head each comprises: projecting a linear light moving along a first direction onto the imaging region, wherein the length direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light during the scanning cycle... One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In this process, the linear light has a projection period t p The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. pThe system includes N waveform projection areas, each 2π / N wide, and encodes the projected light intensity for each area, where N is an integer greater than 1. An image sensor, with each pixel comprising N storage units, is used to capture images of the imaging area to obtain N image frames under the linear light scan projection. Each pixel's N storage units belong to one of N groups of storage units, and each group of storage units is separated by an exposure switching period t with a phase interval of 2π / N. e Exposure is performed; and depth data of the object under test within the imaging area is obtained based on the image frame, wherein the projected light intensity of each waveform projection area is encoded such that during the scanning cycle... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

[0020] Optionally, the method further includes: the depth data measurement device capturing multiple sets of first N-step phase shift patterns and multiple sets of second N-step phase shift patterns that are in relative motion with respect to the imaging object; and synthesizing the depth information generated from the multiple sets of first N-step phase shift patterns and multiple sets of second N-step phase shift patterns into model information of the imaging object based on calibration points.

[0021] Therefore, the depth imaging measurement head of the present invention can acquire N-step phase shift maps in a single linear light scan while maintaining image resolution, thereby improving imaging speed. Attached Figure Description

[0022] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0023] Figure 1 This demonstrates the principle of depth imaging using stripe-coded structured light.

[0024] Figure 2 This shows another example of projected stripe-coded structured light.

[0025] Figure 3 A schematic diagram of the composition of a depth data measurement head according to an embodiment of the present invention is shown.

[0026] Figure 4A -B shows Figure 3 Example of magnification operation of the projection device shown.

[0027] Figure 5 A simplified perspective diagram of the projection device used in this invention is shown.

[0028] Figure 6 An example of the pixel structure of the image sensor used in this invention is shown.

[0029] Figure 7 An example is shown showing the relative exposure cycles between different groups of storage units of the same image sensor.

[0030] Figure 8 A cyclic sub-cycle is shown. A schematic diagram of the imaging of different groups of storage units.

[0031] Figure 9 An example is shown of the relative relationship between the projected light waveform and the exposure cycles of memory cell group 1-4 during sinusoidal four-step phase-shift pattern imaging.

[0032] Figure 10 This shows the completion of one scan cycle. An example of patterns 1-4 obtained from storage cell groups 1-4 respectively.

[0033] Figure 11 An example of depth map imaging using Gray code combined with four-step phase shifting is shown.

[0034] Figure 12 A cyclic sub-cycle is shown. A schematic diagram of the imaging of different groups of storage units.

[0035] Figure 13A -D shows the relative relationship between the projected light waveforms of sub-cycles T1-T4 and the exposure cycles of memory cell groups 1-4.

[0036] Figure 14 This shows the completion of one scan cycle. An example of patterns 1-4 obtained from storage cell groups 1-4 respectively.

[0037] Figure 15 A schematic diagram of a depth data measurement device according to an embodiment of the present invention is shown.

[0038] Figure 16 A schematic flowchart of a depth data measurement method according to an embodiment of the present invention is shown.

[0039] Figure 17 This illustrates an example where partial depth information of the imaged object is missing when using structured light imaging with a single camera.

[0040] Figure 18 An example is shown where dual-camera imaging improves the imaging of objects that lack partial depth information.

[0041] Figure 19A schematic diagram of a depth data measurement device according to an embodiment of the present invention is shown.

[0042] Figure 20 A schematic diagram of the composition of a depth data measurement device according to an embodiment of the present invention is shown.

[0043] Figure 21 A schematic flowchart of a depth data measurement method according to an embodiment of the present invention is shown. Detailed Implementation

[0044] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0045] According to the principle of structured light measurement, the key to the entire measurement system is whether the scanning angle α can be accurately determined. The scanning angle of point and line structured light can be calculated and determined by mechanical devices such as rotating mirrors. The significance of image encoding and decoding lies in determining the scanning angle of the encoded structured light, i.e., the surface structured light system. Figure 1 This diagram illustrates the principle of depth imaging using striped coded structured light. For ease of understanding, the diagram briefly explains the coding principle of striped structured light using a two-grayscale, three-bit binary time-coding method. The projection device sequentially projects three patterns, as shown in the diagram, onto the object being measured in the imaging area. Each pattern divides the projection space into eight regions using two grayscale levels (bright and dark). Each region corresponds to its own projection angle; it can be assumed that bright regions correspond to code "1" and dark regions correspond to code "0". By combining the coding values ​​of a point on the object in the three coded patterns according to the projection order, the region coding value of that point is obtained. This determines the region where the point is located and, consequently, the scanning angle of that point is decoded.

[0046] To improve matching accuracy, the number of projected patterns in time coding can be increased. Figure 2 Another example of projected stripe-coded structured light is shown. Specifically, the figure illustrates a five-bit binary time-coded light with two gray levels. In applications such as binocular imaging, this means that each pixel in each frame of the left and right images contains five region-coded values ​​of either 0 or 1, thereby enabling left-right image matching with higher precision (e.g., pixel-level). With the projection rate of the projection device remaining constant, compared to... Figure 1 The three coded patterns Figure 2 The example is equivalent to achieving higher-precision image matching at a higher temporal cost.

[0047] Figure 3A schematic diagram illustrates the principle of using linear light to project fringe images to obtain depth data. (Example) Figure 3 As shown, the depth data measurement head 300 includes a projection device 310 and two image sensors 320_1 and 320_2. In a monocular implementation, the depth data measurement head 300 can also use a single image sensor for image capture.

[0048] The projection device 310 is used to scan and project structured light with stripe coding onto the imaging area. For example, during three consecutive image frame projection cycles, the projection device 310 can project such... Figure 1 The three patterns shown can be used to generate depth data. Images 320_1 and 320_2, which can be referred to as first and second image sensors respectively, have a predetermined relative positional relationship and are used to capture images of the imaging area to obtain first and second two-dimensional image frames under structured light illumination, respectively. For example, when projected onto the projection device 310... Figure 1 In the case of the three patterns shown, the first and second image sensors 320_1 and 320_2 can respectively image the areas projected with the three patterns (e.g., ...) within three synchronized image frame imaging cycles. Figure 3 Imaging is performed on the imaging plane and the area within a certain range before and after it.

[0049] like Figure 3 As shown, the projection device 310 can project a linear light extending in the x direction in the z direction (i.e., toward the imaging area). The projected linear light can move continuously in the y direction to cover the entire imaging area. Figure 3 The lower part provides a more easily understood illustration of the scanning of linear light using a perspective view of the shooting area.

[0050] In this disclosure, the direction of the light emanating from the measuring head is defined as the z-direction, the vertical direction of the imaging plane is defined as the x-direction, and the horizontal direction is defined as the y-direction. Therefore, the fringe structured light projected by the projection device can be the result of a linear light extending in the x-direction moving in the y-direction. Although in other embodiments, fringe structured light obtained by moving a linear light extending in the horizontal y-direction in the x-direction can also be synchronized and imaged, this disclosure preferably uses vertical fringe light for illustration.

[0051] Figure 4A -B shows Figure 3 An example of magnification operation of the projection device shown. Specifically, as... Figure 3 As shown, in the projection device 310, the laser generator (such as...) Figure 4A The laser emitted by the laser generator 411 (shown in detail in section B) is projected through a projection mechanism (such as...) Figure 4A The projection mechanism 412, as detailed in section -B, scans and projects onto the shooting area ( Figure 3The gray area in the image is used to measure the object to be tested in the shooting area (e.g., Figure 3 Active structured light projection is performed on the area being photographed (in the image). A pair of image sensors 320_1 and 320_2 image the area being photographed, thereby acquiring the image frames required for depth data calculation. Figure 3 As shown, the dashed lines emitted by the projection device 310 represent its projection range, while the dashed lines emitted by the image sensors 320_1 and 320_2 represent their respective imaging ranges. The imaging area is typically located in the overlapping region of the projection and imaging ranges of these three devices.

[0052] In practical applications, the laser generator is used to generate linear and / or infrared laser light, and the laser generator performs high-speed switching to scan the projected structured light, which corresponds to alternating bright and dark stripes, with the stripe coding. High-speed switching may include high-speed switching of the laser generator and high-speed coding switching.

[0053] In one embodiment, the laser generator can emit laser light of uniform intensity, and the projected stripe pattern is achieved by turning the laser generator on and off. In this case, since the laser generator projects light of only one intensity with different periodic duty cycles, each pixel of the image sensor integrates the projected light to determine the "presence" or "absence" of the illumination light; therefore, the equipped image sensor can be a monochrome image sensor.

[0054] In another embodiment, the laser generator itself can emit laser light with varying intensity, for example, a laser whose emitted light intensity varies sinusoidally over a large period depending on the applied power. This sinusoidally varying laser light can be combined with stripe projection, thereby scanning and projecting a pattern of alternating bright and dark areas with varying brightness between the bright stripes. In this case, the image sensor needs to be capable of distinguishing and imaging different light intensities, and therefore can be a multi-level grayscale image sensor. Clearly, grayscale projection and imaging can provide more accurate pixel-to-pixel matching than black-and-white projection and imaging, thereby improving the accuracy of depth data measurement.

[0055] In one embodiment, the laser generator 411 may be a linear laser generator, generating linear light extending in the x-direction. Figure 4A -B (perpendicular to the plane of the paper). This linear light is then projected onto the imaging plane by a reflective mechanism 412 that can oscillate along an axis in the x-direction. The oscillation of the reflective mechanism 412 is shown in the attached figure. Figure 4B As shown, the projection mechanism 412 (e.g., a reflector) is capable of scanning within the range of angle α, thereby achieving reciprocating linear light scanning within the range AB of the imaging plane.

[0056] It should be understood that in order to achieve the projection of a striped pattern, the linear light itself needs to undergo changes in brightness (or, in a simpler implementation, changes in brightness and darkness) as it continuously moves in the y-direction. For example, in situations requiring scanning... Figure 1 When scanning the first pattern, the laser generator 411 remains off as the projection mechanism 412 scans through the front α / 2 angle. When scanning to the rear α / 2 angle, the laser generator 411 turns on, thus creating a pattern with a dark left side and a bright right side. And when scanning... Figure 1 When scanning the second pattern, the laser generator 411 remains off as the projection mechanism 412 scans through an angle of 0 to α / 4. When scanning to an angle of α / 2 to α / 2, the laser generator 411 turns on. When scanning through an angle of α / 2 to 3α / 4, the laser generator 411 turns off again. When scanning to an angle of 3α / 4 to α, the laser generator 411 turns on again. This achieves a dark-light-dark-light pattern. Similarly, this can be achieved with more frequent changes based on the rotation angle. Figure 1 The third pattern and Figure 2 The pattern shown has finer stripes.

[0057] In one embodiment, the aforementioned reflection mechanism 412 may be a micromirror device (also known as a digital micromirror device, DMD) and may be implemented as a MEMS (microelectromechanical system). Figure 5 A simplified perspective diagram of the projection device used in this invention is shown. Figure 5 As shown, the point laser generated by the laser can be transformed into linear light (corresponding to the linear laser generator 411 in Figure 4) through a lens. This linear light is then reflected by a micromirror device in the form of a MEMS, and the reflected linear light is then projected into the external space through an optical window. Micromirror devices have extremely high performance; for example, commercially available DMDs can reciprocate at a highly stable frequency of 2 kHz, thus laying the foundation for high-performance depth imaging.

[0058] In order to obtain a high-precision depth map, Figure 3 The depth data measurement head shown needs to project multiple different fringe patterns sequentially. In other words, existing methods for synthesizing depth maps using captured fringe patterns sacrifice accuracy for time-domain performance. Furthermore, since different fringe patterns captured in N consecutive imaging cycles are used to synthesize a single depth map, existing depth data measurement methods are only applicable when the subject remains stationary during these N imaging cycles. This significantly limits the application scope of techniques for obtaining depth data using actively projected fringe images.

[0059] In view of this, the present invention proposes a novel depth data measurement scheme. This scheme utilizes an image sensor equipped with different sets of storage units capable of phase-shift exposure. By cleverly configuring the changes in brightness and darkness of the projected linear light, the different sets of storage units of the image sensor can each acquire different phase-shifted fringe images during a single scan of the linear light, thereby achieving the acquisition of multiple fringe images from a single linear light scan. This significantly improves the speed of depth map synthesis and is suitable for photographing moving targets.

[0060] In one embodiment, the present invention can be implemented as a depth imaging measurement head, comprising: a projection device and an image sensor. The projection device can be used to project an image onto the imaging area along a first direction (e.g., ...). Figure 3 A linear light moving in the y-direction, wherein the length direction of the linear light is a second direction perpendicular to the first direction (e.g., ...). Figure 3 (x-direction in the image). In one embodiment, the projection device may have... Figure 5 The implementation structure shown includes a linear light generating device and a projection mechanism for reflecting and projecting linear light, and capable of changing the projection direction within a certain angle.

[0061] The image sensor, wherein each pixel comprises N storage units, and the N storage units of each pixel belong to one of N groups of storage units. Each group of storage units is separated from the others by an exposure switching period t with a phase interval of 2π / N. e Exposure is performed, where N is an integer greater than 1.

[0062] For ease of understanding, we will use N=4 as an example to illustrate the structure and exposure of the image sensor. Figure 6 An example of the pixel structure of the image sensor used in this invention is shown. An example is an image sensor with a resolution of 600x800 pixels. Figure 6 The structure of one of the pixel columns is shown. For example... Figure 6 As shown, a pixel column 621 can include k pixels P1-P k (In the 600x800 resolution example, k = 600). Each pixel includes the same structure: one photosensitive unit, N switches, and N storage units, where each switch controls the charge storage of one storage unit. Specifically, pixel P1622 may include a photodiode 624 serving as the photosensitive unit, N switches 626, and N storage units 628. Pixel P k 623 may include a photodiode 625 used as a photosensitive unit, N switches 627 and N memory units 629.

[0063] A storage cell is, for example, a unit used to accumulate and store charge. A photodiode generates charge based on received light and outputs the charge based on the amount stored. N storage cells corresponding to the same pixel all acquire charge from the same photosensitive unit, but each is spaced apart by an exposure switching period t with a phase interval of 2π / N. e Exposure is performed. Each pixel has one storage unit corresponding to one of N groups of storage units. Therefore, in a 600x800 pixel image sensor, assuming N=4, there are 600x800x4 storage units. These 600x800x4 storage units belong to 4 groups, each group containing 600x800 storage units, each corresponding to a different pixel. Each group of storage units can be exposed at the same period, with a phase difference of π / 2 between adjacent groups. Since N storage units share one photosensitive unit, multiple images can be acquired in a single scan without reducing the original resolution of the image sensor.

[0064] Figure 7 An example illustrating the relative exposure cycles between different groups of storage cells within the same image sensor is shown. Figure 7 In the example, the four sets of memory cells have the same exposure switching period t. e All cells switch with a 50% duty cycle; in other words, all memory cells of the image sensor have the same exposure switching waveform. The difference lies in the fact that the waveforms of successive groups of memory cells have the same phase difference of π / 2. In one implementation, the exposure switching period t... e For example, a typical value of 20ns is taken. This means that each storage unit in the image sensor operates at an interval of 10ns to receive exposure and 10ns to turn off, but the second group of storage units turns on 5ns later than the first group, the third group of storage units turns on 5ns later than the second group, and the fourth group of storage units turns on 5ns later than the third group (which can also be regarded as 5ns earlier than the first group of storage units).

[0065] The image sensor used can be as Figure 6 and Figure 7 As shown, when performing grouped phase-shift exposure, the linear light projection of the projection device can be cleverly set to achieve the acquisition of multiple images in a single scan.

[0066] Specifically, the projection device can be in one scanning cycle (e.g., denoted as the scanning cycle). A pattern scan is completed within ) days. Within the scanning cycle Inside, it is assumed that linear light sweeps across the imaging region at a uniform speed, and in multiple cyclic sub-periods. This process is repeated. The following will describe each cycle sub-cycle. A light projection embodiment is described in detail below.

[0067] In each cycle sub-cycle In this process, the linear light has a projection period t p The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The duration is the same. Projection period t p It includes N waveform projection regions with a width of 2π / N, and the projected light intensity of each waveform projection region is encoded so that during the scanning period... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

[0068] In one embodiment, each waveform projection region corresponds to a projected rectangular wave of width 2π / N or 0 (0 can also be considered as a rectangular wave with zero intensity), and a set of N-step phase shift patterns corresponds to one cyclic sub-period. The light intensity distribution is used to determine the light intensity of each waveform projection area.

[0069] In one embodiment, a set of N-step phase-shift patterns is a sinusoidal four-step phase-shift pattern, and each projection period t is obtained based on the exposure of the waveform projection area corresponding to the N sets of storage cells. p The light intensity value of each waveform projection area is not less than zero.

[0070] For ease of explanation, we assume N = 4, and that the line laser scanning projection period is t. p The laser intensities in each 1 / 4 cycle within that time period are Q1 / Q2 / Q3 / Q4, respectively. Therefore, in one cyclic sub-cycle... The integral brightness of the four pixels is as follows:

[0071] P1 = Σ(Q1 + Q2)

[0072] P2 = Σ(Q2 + Q3)

[0073] P3 = Σ(Q3 + Q4)

[0074] P4=Σ(Q1+Q4) (1)

[0075] Therefore, the values ​​of Q1 to Q4 can be determined according to the pattern type corresponding to the required 4-step phase-shift imaging.

[0076] Figure 8 A cyclic sub-cycle is shown. A schematic diagram of imaging different groups of memory cells. Figure 8In the example shown, the desired phase shift pattern is a sinusoidal light waveform with a phase difference of π / 2. In this case, the values ​​of the four memory cells P1 to P4 can be:

[0077] P1 = Q / 2 * sint + Q / 2

[0078] P2 = -Q / 2 * cost + Q / 2

[0079] P3 = -Q / 2 * sint + Q / 2

[0080] P4 = Q / 2 * cost + Q / 2 (2)

[0081] Here, Q can be seen as Figure 8 The integrated brightness at the brightest point of the phase-shifted fringe shown can be represented by the value of t, which corresponds to a single cycle. At different angular positions, the value of t when linear light scans at a constant speed can correspond to pixels at different positions in the image sensor.

[0082] The brightness values ​​of Q1 to Q4 with respect to t can be obtained by reversing equation (2). Since there are N = 4 unknowns Q1 to Q4 in equation (2), and the rank of equation (2) is N-1 = 3, Q1 to Q4 can actually have infinitely many solutions. One solution value is given below:

[0083] Q1 = A / 2 * sint + A / 2

[0084] Q2 = 0

[0085] Q3 = -A / 2 * cost + A / 2

[0086] Q4 = A / 2(cost - sint) (3)

[0087] Where O*A=Q, O is, for example, the number of exposure cycles received by each pixel when the linear light sweeps (for example, in the following example, each pixel column can complete 100 exposures in 2us of linear light sweep, which can be regarded as O=100).

[0088] However, since light intensity cannot be negative, the values ​​of Q1 to Q4 must remain non-negative. Equation (3) holds true in the range of t = 0 to π / 4, but when the linear light scans to the range of t = π / 4 to π / 2, another solution value can be given based on equation (2):

[0089] Q1 = A / 2 * cost + A / 2

[0090] Q2 = A / 2(sint - cost)

[0091] Q3 = -A / 2 * sint + A / 2

[0092] Q4 = 0 (4)

[0093] Figure 9 An example is shown showing the relative relationship between the waveform value of the projected light and the exposure cycles of storage cell group 1-4 when performing sinusoidal four-step phase-shift pattern imaging.

[0094] As shown in the figure and combined with Figure 8 According to equation (1), at t = 0, the values ​​of P1 to P4 can correspond to Q / 2, 0, Q / 2 and Q respectively.

[0095] Assuming scan cycle The latency is 3.84 ms, and the image sensor includes 1920 columns with a linear light projection period t. p The value is 20ns, which is the time it takes for each pixel to be scanned by the linear light, i.e., the dwell time t. c It can be equal to the scan period. Divide by the column number C, i.e., 3.84ms / 1920 = 2us, or, considering that the linear light has a certain width, the dwell time t c The exposure time is no less than 2µs, so each pixel column can complete 100 exposures within 2µs of linear light sweep. At this time, O = 100.

[0096] In one embodiment, linear light can be made to... Initially, maintain the first 100 projection cycles t p It has waveforms corresponding to Q1 = A / 2, Q2 = Q3 = 0, and Q4 = A / 2 at t = 0 (where A = Q / 0 = Q / 100). In a preferred embodiment, the waveform can be adjusted based on a small change in t during each projection period t. p The values ​​of Q1 to Q4 are fine-tuned, for example, by solving the value of equation (2).

[0097] As mentioned earlier, repeating the sub-cycle Reaching a predetermined number of scans, for example, M scans, and thus completing one scan cycle. Figure 10 This shows the completion of one scan cycle. Patterns 1-4 are obtained from storage cell groups 1-4 respectively. For example, in Figure 10 In this example, M can be equal to 16, that is, the sub-cycle is repeated 16 times. This yields a four-step phase shift diagram with a phase difference of π / 2. Clearly, Figure 10 An example of a 4-step phase-shift pattern for a sine wave is shown. This was acquired using an image sensor with 1920 columns, as in the example above. Figure 10When a four-step phase-shift map with 32 repeating sinusoidal fringes is shown, each sinusoidal fringe covers 60 pixel columns (1920 / 32 = 60). In other words, a linear optical scan covers one cyclic sub-cycle. 60 * 100 = 6000 projections are required. In each projection cycle t... p In the embodiment where the values ​​of Q1 to Q4 are fine-tuned for example, as shown in equation (3), each projection period t p Compared to the previous projection period t p For each t, ​​there exists a small increment of Δt = 2π / 6000 = π / 3000, which leads to different values.

[0098] Back Figure 9 As the linear light scans from the position corresponding to t=0 to the position corresponding to t=π / 6, the values ​​of Q1 to Q4 can be solved based on, for example, equation (3). At t=π / 6, it can be assumed that the linear light has scanned to the 5th pixel column in the current cycle of 60 pixel columns, and based on equation (3), the values ​​corresponding to t=π / 6 are Q1=3A / 4, Q2=0, Q3=0.183A, and Q4=0.067A.

[0099] As the linear light scans from the position corresponding to t = π / 6 to the position corresponding to t = π / 3, the values ​​of Q1 to Q4 can be continuously solved. Since cost = sint at t = π / 4, and cost < sint in the subsequent π / 4 to 3π / 4 process, equation (3) showing Q4 < 0 is no longer applicable. At this time, based on equation (4), we can solve for Q1 = 3A / 4, Q2 = 0.183A, Q3 = 0.067A, and Q4 = 0 at t = π / 3. Furthermore, at t = π / 3, it can be considered that the linear light has scanned to the 10th pixel column in the current cycle of 60 pixel columns.

[0100] As the linear light scans from the position corresponding to t = π / 3 to the position corresponding to t = π / 2, the values ​​of Q1 to Q4 can be solved based on equation (4). At t = π / 2, it can be assumed that the linear light has scanned to the 15th pixel column in the current cycle of 60 pixel columns, and based on equation (4), the values ​​of Q1 = A / 2, Q2 = A / 2, Q3 = 0, and Q4 = 0 corresponding to t = π / 2 can be solved. It should be noted that since pixels 1-4 undergo 4-step phase-shift imaging, the values ​​of Q1 to Q4 in each projection cycle t = π / 2 can be solved. p All of them have Q1+Q2+Q3+Q4=A, and since pixels 1-4 are spaced 2π / N apart, their phases also have a projection period t. p Exposure is performed and the on-time accounts for 10%, therefore, in each projection cycle t p A total of 2A*t can be obtained from pixels 1-4. p / 4 of the integral brightness.

[0101] As described above Figure 9 Examples of Q1 to Q4 values ​​for the first π / 2 of a 2π cycle in a sinusoidal N-step phase shift image are described, with imaging examples for pixels 1-4 respectively. Those skilled in the art can apply equation (2) and... Figure 9 For example, continue to find the values ​​of Q1 to Q4 in the last 3π / 2 of a 2π cycle.

[0102] In addition, it should be understood that although Figure 8 and Figure 9 To facilitate the explanation of the changes in the stripe image and waveform, sub-periods T1-T4 are shown (each sub-period T1-T4 corresponds to a 2π cyclic sub-period). The 0 to π / 2 part, π / 2 to π part, π to 3π / 2 part and 3π / 2 to 2π part in the above, but when realizing the above 4-step phase shift pattern of sine wave, it is only necessary to obtain one cyclic sub-period according to the above formula (2). The change in the brightness value of the internally projected laser does not require additional sub-period T1-T4 division.

[0103] In one embodiment, a more accurate depth image can be obtained by combining other fringe lights with N-step phase-shift imaging. Figure 11 This illustrates an example of depth map imaging using Gray code combined with four-step phase shifting. In practical applications, this can be achieved by including more cyclic sub-periods within each four-step phase shifting pattern. This improves the imaging accuracy of the depth map. However, due to the periodic repetition of the 4-step phase-shift pattern, there is a problem of not being able to identify depth jumps across cycles. In this case, Gray code can be used to perform preliminary imaging of the object within the imaging space, followed by 4-step phase-shift imaging. Specifically, the projection device can... The projection device completes one pattern scan within a certain period (e.g., by solving the corresponding waveform of each linear light projection cycle using the illustrated brightness and darkness diagram, so that each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a Gray code pattern). The projection device completes one pattern scan within the second scan cycle. A pattern scan is completed within the time frame so that each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute the set of N-step phase shift patterns. A depth map of the imaging region is generated from the set of N-step phase shift patterns based on the set of Gray code patterns.

[0104] In addition, multiple scan cycles were introduced. At this time, not only can other stripe patterns plus N-step phase shift patterns be implemented, but also αN-step phase shift patterns can be implemented. Here, α is an integer greater than or equal to 2. Specifically, the projection device performs α scan cycles. The pattern is scanned α times within each scan cycle. It includes multiple cyclic sub-cycles. In each cycle sub-cycle In, there are N sub-cycles T1-T N In each sub-cycle T i In this process, the linear light has a projection period t p The projection period t is used to perform brightness and darkness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p Including the bright area, among which, in the sub-cycle T1-T N During the projection period t, the bright area is... p The position of the [something] changes with a phase interval of 2π / αN, such that in each scan cycle... During one pattern scan, the N storage units of the image sensor each image a different stripe pattern, and within α scan cycles... When α pattern scans are completed, αN stripe patterns form a set of αN-step phase shift patterns with a phase shift of 2π / αN between them, where α is an integer greater than or equal to 2.

[0105] For ease of understanding, let's take α = 2 and N = 4 as an example. That is, using exposure switching periods t, which include 4 sets of phases spaced 2π / N apart,... e An image sensor that exposes pixels and performs two scans is used to acquire an eight-step phase-shift pattern.

[0106] To acquire an eight-step phase-shift pattern using four sets of image sensors with storage units, the projection device needs to perform two scans. In the first scan, i.e., during the first scan cycle… The middle includes multiple cyclic sub-cycles. In each cycle sub-cycle The projection includes four sub-periods T1-T4, and the bright area is in the projection period t. p The position of the [something] changes at π / 4 phase intervals, such that in the first scan cycle... During a single pattern scan, the first four images of the eight-step phase-shift pattern are acquired. In the second scan, i.e., during the second scan cycle... The middle also includes multiple cyclic sub-cycles. In each cycle sub-cycle Similarly, it includes N sub-periods T1-T4, and the bright area is in the projection period t. p The position of the bright area changes at π / 4 phase intervals (but the phase of the bright area is different from that in the first scan), so that in the second scan cycle... During a single pattern scan, the last four images of the eight-step phase-shift pattern are acquired. This allows for the acquisition of the eight-step phase-shift pattern in just two scans, thus achieving higher depth imaging accuracy.

[0107] Furthermore, although the example of N=4 was given above for ease of explanation, N can take other values ​​in other embodiments. Specifically, N=2 n n is an integer greater than or equal to 1. Therefore, it is possible to achieve, for example, higher precision 8-step phase shifting using an image sensor with 8 memory units, or 16-step phase shifting using an image sensor with 16 memory units.

[0108] Each pixel comprises N storage units, which are N charge storage units. The N charge storage units corresponding to the same pixel have the same exposure position, but their exposure periods are spaced 2π / N phase apart, and during the scan period... When a pattern scan is completed, the set of N-step phase-shift patterns is obtained from the N sets of charge storage units. The set of N-step phase-shift patterns is used to generate a depth map of the imaging region.

[0109] Figure 12 A cyclic sub-cycle is shown. A schematic diagram of imaging different groups of memory cells. Figure 12 In the example shown, the desired phase shift pattern is a bright and dark stripe waveform with a phase difference of π / 2. In this case, the values ​​of the four memory cells P1 to P4 can be:

[0110] P1=Q, when t=0~π; =0, when t=π~2π

[0111] P2=0, when t=0~π / 2; =Q, when t=π / 2~3π / 2; =0, when t=3π / 2~2π

[0112] P3=0, when t=0~π; =Q, when t=π~2π

[0113] P4=Q, when t=0~π / 2; =0, when t=π / 2~3π / 2; =Q, when t=3π / 2~2π (5)

[0114] Therefore, based on equations (1) and (5), we solve for Q1 to Q4 and obtain the following optimized solution:

[0115] Q1=A, when t=0~π / 2; =0, when t=π / 2~2π

[0116] Q2=0, when t=0~π / 2; =A, when t=π / 2~π; =0, when t=π~2π

[0117] Q3=0, when t=0~π; =A, when t=π~3π / 2; =0, when t=3π / 2~2π

[0118] Q4=0, when t=0~3π / 2; =A, when t=3π / 2~2π (6)

[0119] Where O*A=Q, O is, for example, the number of exposure cycles received by each pixel when the linear light sweeps across.

[0120] In the optimization solution based on equation (6), each cyclic sub-cycle can be... The timeline is divided into four sub-cycles, T1-T4, each corresponding to a 2π cyclic sub-cycle. The bright area is defined in the 0 to π / 2 portion, the π / 2 to π portion, the π to 3π / 2 portion, and the 3π / 2 to 2π portion, and the bright area is defined in the projection period t. p The position of the element changes at 2π / 4 phase intervals.

[0121] In the example of N=4, each cycle sub-cycle It includes four sub-cycles, T1-T4. Figure 13A -D shows the relative relationship between the projected light waveforms of sub-cycles T1-T4 and the exposure cycles of memory cell groups 1-4.

[0122] like Figure 13A As shown in -D, the projected laser is synchronized with the exposure switching period t. e The same projection period t p Laser projection is performed, and each laser projection cycle t can be considered as... p They always maintain the same exposure switching period t as the first set of memory cells. e Synchronization (although the laser projection period t) p The laser is switched on at different phases in different sub-cycles and on and off with a duty cycle of 25% (i.e., the bright area is 2π / N), meaning the waveform is a rectangular wave with a 25% duty cycle. In the four sub-cycles T1-T4, the projected laser is... p The projection activation time within each memory cell is aligned with the exposure activation time of the first to fourth memory cells.

[0123] Specifically, firstly, as Figure 13A As shown, in sub-period T1, the projected laser is in each projection period t p It remains on during the first π / 2 phase. During this time, since the first and fourth memory groups are also on, it can remain on during each projection cycle t.p The reflected projected light is exposed within the first π / 2 phase, allowing charge accumulation in the corresponding storage cells, as shown by the gray rectangles in the figure. After completing a predetermined m1 projection cycles, the sub-cycle T1 ends, and the projected laser enters each projection cycle t. p The sub-period T2 remains active within the π / 2 to π phase.

[0124] like Figure 13B As shown, in sub-period T2, the projected laser is in each projection period t p It remains on during the π / 2 to π phase. During this time, since both the first and second sets of memory cells are also on, it can remain on during each projection cycle t. p The reflected projected light is exposed within the π / 2 to π phase, allowing charge accumulation in the corresponding storage cells, as shown by the gray rectangles in the figure. After completing a predetermined m² projection cycles, the sub-cycle T2 ends, and the projected laser enters each projection cycle t. p The sub-period T3 remains active within the π to 3π / 2 phase.

[0125] like Figure 13C As shown, in sub-period T3, the projected laser is in each projection period t p It remains on during the π to 3π / 2 phase. During this time, since the second and third sets of memory cells are also on, it can remain on during each projection cycle t. p The reflected projected light is exposed within a phase of π to 3π / 2, allowing charge accumulation to occur within the corresponding pixels, as shown by the gray rectangles in the figure. After completing a predetermined m³ projection cycles, the sub-cycle T3 ends, and the projected laser enters each projection cycle t. p The sub-period T4 remains active within the 3π / 2 to 2π phase.

[0126] like Figure 13D As shown, in sub-period T4, the projected laser is in each projection period t p It remains on during the 3π / 2 to 2π phase. During this time, since the third and fourth memory groups are also on, it can remain on in each projection cycle t. p The reflected projected light is exposed within the 3π / 2 to 2π phase range, allowing charge accumulation to occur within the corresponding pixels, as shown by the gray rectangles in the figure. After completing the predetermined m4 projection cycles, sub-cycle T4 ends. At this point, one cycle sub-cycle is completed. The projection, and then proceed to the next cycle sub-cycle. The sub-cycle T1.

[0127] In a simple implementation, the number of projection cycles for each sub-cycle can be made the same, i.e., m1 = m2 = m3 = m4, meaning the duration of T1-T4 is the same. In this case, if the linear light sweeps across the imaging plane at a uniform speed, then the following can be obtained: Figure 12 The phase shift diagram is shown. Combined with, as shown... Figure 13A As shown in -D, in sub-cycle T1, the illumination range of the projected laser falls within the exposure range of the first and fourth sets of storage cells, therefore the first and fourth sets of storage cells correspond to bright fringes; in sub-cycle T2, the illumination range of the projected laser falls within the exposure range of the first and second sets of storage cells, therefore the first and second sets of storage cells correspond to bright fringes; in sub-cycle T3, the illumination range of the projected laser falls within the exposure range of the second and third sets of storage cells, therefore the second and third sets of storage cells correspond to bright fringes; in sub-cycle T4, the illumination range of the projected laser falls within the exposure range of the third and fourth sets of storage cells, therefore the third and fourth sets of storage cells correspond to bright fringes. The above sub-cycles are repeated. Reaching a predetermined number of scans, for example, M scans, and thus completing one scan cycle. Figure 14 This shows the completion of one scan cycle. Patterns 1-4 are obtained from storage cell groups 1-4 respectively. For example, in Figure 14 In this example, M can be equal to 16, that is, the sub-cycle is repeated 16 times. This yields a four-step phase shift diagram of bright and dark fringes with a phase difference of π / 2.

[0128] Therefore, when obtaining the N-step phase shift map of bright and dark fringes, in each cycle sub-cycle... In, there are N sub-cycles T1-T N In each sub-cycle T i In this process, the linear light has a projection period t p The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p Including the bright area, among which, in the sub-cycle T1-T N During the projection period t, the bright area is... p The position of the element changes with a phase interval of 2π / N, thereby allowing the scanning period to... When a pattern scan is completed, each of the N storage units of the image sensor images a different fringe pattern, and there is a phase shift of 2π / N between the N fringe patterns. Thus, a single scan of linear light, for example from left to right (corresponding to one scan cycle), is transmitted. It can directly obtain a set of N-step phase shift patterns from the N sets of storage units of the image sensor.

[0129] Linear optical scanning corresponds to the dwell time t on each column of pixels. c Not less than the scan period Divide by the column number C. To ensure sufficient exposure, the dwell time t c The projection period t is p More than 10 times, preferably, the dwell time t c The projection period t is p More than 50 times. For example, assuming a scan cycle The frame rate is 3.84ms (i.e., the frame rate is 1000 / 3.84≈260 frames / s), and the image sensor includes 1920 columns (in a configuration with...). Figure 6 In the case of the pixel unit shown (with 960 columns of pixels), the time it takes for each pixel to be swept by the linear light, i.e., the dwell time t, is... c It can be equal to the scan period. Divide by the column number C, i.e., 3.84ms / 1920 = 2us, or, considering that the linear light has a certain width, the dwell time t c Not less than 2µs. Online light projection period t p With a wavelength of 20ns and a duty cycle of 25%, each pixel column can complete 100 exposures within a 2µs linear light scan, and this is consistent with the current projection period t. p The two sets of storage units corresponding to the bright areas can achieve an exposure time of 5ns x 100 = 0.5us. In addition, since the time it takes for the linear light to scan through a column of pixels is long enough compared to the exposure period of the pixels (e.g., 100 times in the example above), for the currently illuminated column of pixels, the linear light itself can be approximated as a light source whose position does not move during these 2us.

[0130] To achieve a four-step phase shift, each sub-cycle T requires i The duration of the light sweep is not less than the time it takes for the linear light to sweep across one column of pixel units. This is achieved using an image sensor with 1920 columns as shown in the example above. Figure 10 The four-step phase-shift map shown has 32 stripes (16 bright stripes and 16 dark stripes), with each stripe covering 60 pixel columns (1920 / 32 = 60). For example... Figure 9 As shown, due to each sub-cycle T i Corresponding to half a stripe, this covers 30 pixel columns, with a duration of 2µs x 30 = 60µs. Therefore, in this example, m1 = m2 = m3 = m4 = m = 60µs / 20ns = 3000. This is because the linear light projection period t... p Exposure switching period t of each memory cell groupe Since the duration is the same, when the linear optical scan passes through half the distance of the stripe, the corresponding memory units have also been switched on and off 3000 times.

[0131] As described above Figures 12-14 The linear light being projected is given by the projection period t. p An example of a rectangular wave with a bright region of 2π / N phase. In this example, the projected linear light is always the same as the projection period t. p The duty cycle is 100 / N% (in the example above where N=4, the duty cycle is 25%, corresponding to the bright area of ​​the 2π / N phase) and each different projection period t p The projected light intensity in the inner bright region preferably has the same rectangular wave. In each sub-cycle T... i In the process, each projection period t of the linear light... p The internally projected waveforms are identical and are rectangular waves with a 2π / N phase bright area and a 6π / N phase dark area (e.g., each projection period t). p In the middle, the laser is in t p It is activated within a time interval of / N, and (N-1)*t p (Turn off within / N time intervals). Between successive sub-cycles, the projection period of the linear light and the ratio of bright to dark areas remain unchanged; only the position of the bright area changes. And this constitutes one scan cycle. Multiple cyclic sub-cycles Internal repetition. Therefore, the resulting set of N-step phase-shift patterns is: Figure 14 The striped pattern shown has a clear distinction between bright and dark areas and repeats multiple times.

[0132] The above is based on a rectangular wave with a 100 / N% duty cycle and constant brightness. Figure 14 The illustrated example of bright and dark stripes can be considered as an embodiment of the present invention based on the linear light projection period t. p Within this, each 2π / N phase is individually adjustable, thus combining a phase difference of 2π / N and an exposure period equal to t. p An image sensor is a special case of an imaging scheme that generates a set of N-step phase-shift patterns.

[0133] The projection and imaging scheme of the present invention can be used in a monocular scheme (i.e., a scheme equipped with one image sensor) or a binocular scheme (i.e., a scheme equipped with two image sensors with fixed relative positions for synchronous imaging). When the image sensors include a first image sensor and a second image sensor with fixed relative positions, the first image sensor and the second image sensor can each include the N sets of storage units and are exposed synchronously with each other. In other words, in the binocular scheme, during the scanning cycle... When completing one pattern scan, the first image sensor can acquire N stripe images, the second image sensor can acquire N stripe images, and these 2N stripe images can be used to obtain depth data.

[0134] Furthermore, to achieve scanning projection, the projection device of the present invention includes: a light-emitting device for generating linear light; and a reflective device for reflecting the linear light, which projects linear light moving in a direction perpendicular to the stripe direction onto the imaging area at a predetermined frequency, wherein the length direction of the linear light is the length direction of the projected stripes. The reflective device includes one of the following: a mechanical galvanometer that reciprocates at the predetermined frequency; a micromirror device that reciprocates at the predetermined frequency; and a mechanical rotating mirror that rotates unidirectionally at the predetermined frequency. Here, the projected linear light can be linear light with a high-order Gaussian or flat-top Gaussian distribution, thereby providing a highly uniform brightness distribution in the width direction of the linear light.

[0135] The present invention also discloses a measuring device using the aforementioned measuring head. Specifically, a depth data measuring device may include the depth data measuring head described above, and a processor connected to the depth data measuring head, for use in the scanning cycle. When a pattern scan is completed, a depth map of the imaging region is obtained from the acquired N stripe patterns. In a binocular scheme, the processor can determine the depth data of the object in the imaging region based on the predetermined relative positions of the first and second image sensors and the N first two-dimensional image frames and N second two-dimensional image frames obtained from the structured light imaging. In different embodiments, the measuring head can have a relatively independent package or can be packaged together with the processor in the measuring device.

[0136] Figure 15 A schematic diagram of a depth data measurement apparatus according to an embodiment of the present invention is shown. As shown, the measurement apparatus 1500 may include a measurement head and a processor 1530 as described above. The measurement head includes a projection device 1510 and two image sensors 1520.

[0137] The processor 1530 is connected to the measuring head, for example, to the projection device 1510 and each of the two image sensors 1520, for determining the depth data of the object in the shooting area based on the predetermined relative positions of the first and second image sensors 1520_1 and 1520_2 and the N first two-dimensional image frames and N second two-dimensional image frames obtained by the structured light imaging.

[0138] Figure 16 A schematic flowchart of a depth data measurement method according to an embodiment of the present invention is shown. This method can be implemented using the depth data measurement head and measuring device of the present invention.

[0139] In step S1610, a linear light beam moving along a first direction is projected onto the imaging area, wherein the length direction of the linear light beam is a second direction perpendicular to the first direction, and the projected linear light beam is within the scanning cycle. One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In this process, the linear light has a projection period t p The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p It includes N waveform projection areas with a width of 2π / N, and the projected light intensity of each waveform projection area is encoded, where N is an integer greater than 1.

[0140] In step S1620, an image sensor comprising N storage units for each pixel is used to capture images of the imaging area to obtain N image frames under the linear light scan projection. Each pixel's N storage units belong to one of N groups of storage units, and each group of storage units is spaced apart by an exposure switching period t with a phase interval of 2π / N. e To expose it.

[0141] In step S1630, the depth data of the object under test within the imaging area is obtained based on the image frame.

[0142] The intensity of the projected light in each waveform projection region is encoded so that during the scanning cycle... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

[0143] Furthermore, when using actively projected structured light for imaging, issues such as shadows and blind spots arise. Figure 17 This illustrates an example of missing depth information in a single-camera object using structured light imaging. As shown in the figure, the unevenness of the object's structure can obstruct the reception of structured light, creating shadows in the area facing the camera. Because these shadowed areas lack structured light illumination, depth information is missing in the corresponding region during subsequent depth information calculation (corresponding to "Missing 1" in the figure). Furthermore, regardless of the object's structure, the portion of the object facing away from the structured light generator cannot receive structured light, resulting in missing depth information in that region during subsequent depth information calculation (corresponding to "Missing 2" in the figure).

[0144] Figure 18 This illustrates an example of how dual-camera imaging improves the reduction of partial depth information in an imaged object. As shown in the figure, by using dual cameras on the left and right sides, more parts of the imaged object can receive structured light illumination, thereby enabling the measurement of more depth information about the object.

[0145] In view of this, the present invention can be particularly implemented as a depth data measurement device with dual cameras (i.e., dual measuring heads). Figure 19 A schematic diagram of a depth data measurement device according to an embodiment of the present invention is shown. As shown, the depth data measurement device 1900 may include a first depth imaging measurement head 1910 and a second depth imaging measurement head 1920 with their relative positions fixed. Specifically, the first depth imaging measurement head 1910 and the second depth imaging measurement head 1920 may be mounted in the same housing, thereby ensuring the fixation of their relative positions.

[0146] Furthermore, measuring heads 1910 and 1920 can perform sequential structured light projection and imaging under the control of the same control device (e.g., a processor, not shown in the figure). Here, "sequential" means that measuring head 1910 first performs a scan and imaging, and measuring head 1920 then performs another scan and imaging, performing linear light projection and imaging at different time intervals. Since the exposure cycle of each measuring head is shorter than the framing time for depth detection (e.g., the exposure cycle for one depth frame is 5ms, the depth detection frame rate is 30fps, and the interval between each depth frame is approximately 28ms), exposures of other measuring heads can be performed within the current measuring head's exposure interval, interleaving exposure imaging and depth calculation. In other words, the depth data measurement device of the present invention can include the two measuring heads described above, or it can include more measuring heads that interleave exposure imaging and depth calculation, and the imaging information of these measuring heads can be stitched together in the same xyz axis space. Due to the use of inter-frame interleaving detection, the system frame rate detection frame rate is not affected, and the depth detection frame rate can still reach 30fps.

[0147] As shown in the figure, measuring head 1910 and measuring head 1920 can have the same configuration. For example, they can be two measuring heads with completely identical configurations, and are set to successively capture images of the same imaging area (the imaging object in the imaging area) from different angles. Measuring head 1910 and measuring head 1920 can be implemented as measuring heads that include N sets of storage units and are capable of completing N image imaging in a single scan, as described above.

[0148] Here, sequential imaging means that after the measuring head 1910 completes the first pattern scan (the projection device 1911 scans and projects linear light, and the image sensor 1912 images the projected linear light), the measuring head 1920 performs the second pattern scan (the projection device 1921 scans and projects linear light, and the image sensor 1922 images the projected linear light). The first set of N-step phase-shift patterns obtained from the first pattern scan and the second set of N-step phase-shift patterns obtained from the second pattern scan are synthesized based on the relative positions to form the depth information of the imaging object within the imaging area.

[0149] Furthermore, the housing of the depth data measurement device 1900 may also include a handle structure 1931, and can be implemented as a handheld modeling device. In this implementation, the depth data measurement device captures multiple sets of first N-step phase shift patterns and multiple sets of second N-step phase shift patterns that are in relative motion with respect to the imaging object, and can synthesize the depth information generated from the multiple sets of first N-step phase shift patterns and multiple sets of second N-step phase shift patterns into model information of the imaging object based on calibration points.

[0150] Figure 20 A schematic diagram of a depth data measurement apparatus according to an embodiment of the present invention is shown. As shown, the measurement apparatus 2000 may include two measuring heads 2010 and 2020 as described above, and a processor 2030. The measuring head 2010 includes a projection device 2011 and an image sensor 2012. The measuring head 2020 includes a projection device 2021 and an image sensor 2022.

[0151] The processor 2030 is connected to two measuring heads, for example, to a projection device 2011 and an image sensor 2012, and to a projection device 2021 and an image sensor 2022. Under the control of the processor 2030, the projection device 2011 performs a single-scan projection as described above, and the N sets of storage units of the image sensor 2012 perform corresponding imaging, thereby obtaining a set of N-step phase-shifted images after the single-scan projection. Subsequently, under the control of the processor 2030, the projection device 2021 performs a single-scan projection as described above, and the N sets of storage units of the image sensor 2022 perform corresponding imaging, thereby obtaining a set of N-step phase-shifted images after the single-scan projection. Each of these two sets of N-step phase-shifted images can be used to synthesize a depth map of the imaging object, and these two depth maps can be stitched together based on the positional relationship between the two measuring heads, thereby obtaining more comprehensive data about the imaging object.

[0152] Figure 21 A schematic flowchart of a depth data measurement method according to an embodiment of the present invention is shown. This method can be implemented by the depth data measurement device of the present invention.

[0153] In step S2110, the first depth imaging measurement head of the depth data measurement device performs a first pattern scan on the imaging area to obtain a first set of N-step phase shift patterns.

[0154] In step S2120, the second depth imaging measurement head of the depth data measurement device is used to perform a second pattern scan on the imaging area to obtain a second set of N-step phase shift patterns.

[0155] In step S2130, depth information of the imaging object within the imaging area is synthesized from the first set of N-step phase shift patterns and the second set of N-step phase shift patterns based on the relative positions of the first depth imaging measurement head and the second depth imaging measurement head.

[0156] The scanning imaging of the first depth imaging measurement head and the second depth imaging measurement head may each include: projecting a linear light moving along a first direction onto the imaging area, wherein the length direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light is in the scanning cycle. One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In this process, the linear light has a projection period t p The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p The system includes N waveform projection areas with a width of 2π / N, and the projected light intensity of each waveform projection area is encoded, where N is an integer greater than 1. An image sensor comprising N sets of storage units uniformly distributed on the imaging surface is used to capture images of the imaging area to obtain N image frames under the linear light scan projection, wherein each set of storage units has an exposure switching period t with a phase interval of 2π / N between them. e Exposure is performed; depth data of the object under test within the imaging area is obtained based on the image frame. The projected light intensity of each waveform projection area is encoded to ensure that during the scanning cycle... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

[0157] Steps S1910 to S1930 can be repeated to perform dynamic imaging of, for example, a moving imaging object. During this process, the depth data measurement device of the present invention can maintain its position.

[0158] In another embodiment, the depth data measurement device of the present invention can actively change its position. As previously stated... Figure 19 As shown, the depth data measurement device of the present invention can be implemented in particular as a handheld device including a handle 1931. In this implementation, the depth data measurement device captures multiple sets of first N-step phase shift patterns and multiple sets of second N-step phase shift patterns that are in relative motion with respect to the imaging object, and can synthesize the depth information generated from the multiple sets of first N-step phase shift patterns and multiple sets of second N-step phase shift patterns into model information of the imaging object based on calibration points.

[0159] The depth data measurement head, measuring device, and measuring method according to the present invention have been described in detail above with reference to the accompanying drawings. The depth data measurement scheme of the present invention utilizes an image sensor equipped with different groups of pixels capable of phase-shift exposure to image projected phase-shifted linear light. This allows different groups of pixels in the image sensor to acquire different phase-shifted fringe images during a single scan of the linear light, thereby achieving the acquisition of multiple fringe images in a single linear light scan. This significantly improves the speed of depth map synthesis and is suitable for photographing moving targets.

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0161] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A depth data measurement head, comprising: A projection device for projecting linear light moving along a first direction onto an imaging area, wherein the longitudinal direction of the linear light is a second direction perpendicular to the first direction; The image sensor, wherein each pixel comprises N storage units, and the N storage units of each pixel belong to one of N groups of storage units. Each group of storage units is separated from the others by an exposure switching period t with a phase interval of 2π / N. e Exposure is performed, where N is an integer greater than 1. The projection device during the scanning cycle One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In this process, the linear light has a projection period t p The projection period t performs brightness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p It includes N waveform projection regions with a width of 2π / N, and the projected light intensity of each waveform projection region is encoded so that during the scanning period... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

2. The depth data measurement head as described in claim 1, wherein, The projection period t of the linear light p The exposure switching period t of the first group of memory cells in the N groups of memory cells e synchronous.

3. The depth data measurement head as described in claim 1, wherein, N=2 n , n It is an integer greater than or equal to 1.

4. The depth data measurement head as described in claim 1, wherein, Each waveform projection region corresponds to a rectangular wave of width 2π / N or 0, and a set of N-step phase shift patterns corresponds to one cyclic sub-cycle. The light intensity distribution is used to determine the light intensity of each waveform projection area.

5. The depth data measurement head as described in claim 4, wherein, The set of N-step phase shift patterns is a sinusoidal four-step phase shift pattern, and each projection period t is obtained based on the exposure of the waveform projection area corresponding to the N sets of storage units. p The light intensity value of each waveform projection area is not less than zero.

6. The depth data measurement head as described in claim 1, wherein, Linear optical scanning corresponds to the dwell time t on each column of pixels. c Not less than the scan period Divided by the column number C, the dwell time t c The projection period t is p More than 10 times.

7. The depth data measurement head as described in claim 6, wherein, In each sub-cycle T i In this process, the linear light has a projection period t p Projected m times, and each sub-period T i The duration is greater than the dwell time t c .

8. The depth data measurement head as described in claim 1, wherein, Each pixel comprises N storage units, which are N charge storage units. The N charge storage units corresponding to the same pixel have the same exposure position, but their exposure periods are spaced 2π / N phase apart. In the scan cycle When a pattern scan is completed, the set of N-step phase-shift patterns is obtained from the N sets of charge storage units. The set of N-step phase-shift patterns is used to generate a depth map of the imaging region.

9. The depth data measurement head as described in claim 1, wherein, The projection device in the first scanning cycle The image sensor completes one pattern scan so that each of the N storage units of the image sensor images a different stripe pattern, and the N stripe patterns form a Gray code pattern. The projection device in the second scanning cycle The image sensor completes one pattern scan within a time frame, so that each of the N sets of storage units images a different stripe pattern, and the N stripe patterns constitute the set of N-step phase-shift patterns. Specifically, a depth map of the imaging region is generated from the set of Gray code patterns based on the set of N-step phase shift patterns.

10. The depth data measurement head as claimed in claim 1, wherein, The projection device includes: Light-emitting device for generating linear light; and A reflecting device for reflecting linear light, which is projected at a predetermined frequency onto a shooting area and moves in a direction perpendicular to the direction of the stripes, wherein the longitudinal direction of the linear light is the longitudinal direction of the projected stripes, and the reflecting device includes one of the following: A mechanical galvanometer that reciprocates at the predetermined frequency; Micromirror devices that reciprocate at a predetermined frequency; and A mechanical rotating mirror that rotates unidirectionally at a predetermined frequency.

11. The depth data measurement head as claimed in claim 1, wherein, The image sensor includes a first image sensor and a second image sensor with fixed relative positions, wherein the first image sensor and the second image sensor each include the N sets of storage units and are exposed synchronously with each other.

12. The depth data measurement head as claimed in claim 1, wherein, The projection device performs α scanning cycles The pattern is scanned α times within each scan cycle. It includes multiple cyclic sub-cycles. In each cycle sub-cycle In, there are N sub-cycles T1-T N In each sub-cycle T i In this process, the linear light has a projection period t p The projection period t is used to perform brightness and darkness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p Including the bright area, among which, in the sub-cycle T1-T N During the projection period t, the bright area is... p The position of the [something] changes with a phase interval of 2π / αN, such that in each scan cycle... During one pattern scan, the N storage units of the image sensor each image a different stripe pattern, and within α scan cycles... When α pattern scans are completed, αN stripe patterns form a set of αN-step phase shift patterns with a phase shift of 2π / αN between them, where α is an integer greater than or equal to 2.

13. The depth data measurement head as described in claim 1, wherein, In each cycle sub-cycle In, there are N sub-cycles T1-T N In each sub-cycle T i In this process, the linear light has a projection period t p The projection period t is used to perform brightness and darkness changes. p The duration is related to the exposure switching period t e The durations are the same, and the projection period t is the same. p Including the bright area, among which, in the sub-cycle T1-T N During the projection period t, the bright area is... p The position of the [something] changes with a phase interval of 2π / N, such that during the scan cycle... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute an N-step phase shift pattern of bright and dark stripes with a phase shift of 2π / N between them.

14. A depth data measurement device, comprising: The depth data measurement head as described in any one of claims 1-13, and A processor connected to the depth data measurement head is used in the scanning cycle. When a pattern scan is completed, a depth map of the imaging region is obtained from the acquired N stripe patterns.

15. A depth data measurement device, comprising a first depth imaging measurement head and a second depth imaging measurement head fixed in relative positions, wherein the first depth imaging measurement head and the second depth imaging measurement head are depth data measurement heads as described in any one of claims 1-13. in, After the first depth imaging measurement head completes the first pattern scan, the second depth imaging measurement head performs the second pattern scan. The first set of N-step phase shift patterns obtained from the first pattern scan and the second set of N-step phase shift patterns obtained from the second pattern scan are synthesized into depth information of the imaging object within the imaging area based on the relative position.

16. A depth data measurement method, comprising: A linear light beam moving along a first direction is projected onto the imaging area, wherein the length direction of the linear light beam is a second direction perpendicular to the first direction, and the projected linear light beam is within the scanning cycle. One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In this process, the linear light has a projection period t p The projection period t performs brightness changes. p Duration and exposure switching cycle t e The durations are the same, and the projection period t is the same. p It includes N waveform projection areas with a width of 2π / N, and the projected light intensity of each waveform projection area is encoded, where N is an integer greater than 1; The imaging area is captured using an image sensor comprising N storage units for each pixel to obtain N image frames under the linear light scan projection. Each pixel's N storage units belong to one of N groups of storage units, and each group of storage units is spaced apart by an exposure switching period t with a phase interval of 2π / N. e Expose; and The depth data of the object under test within the imaging area is obtained based on the image frame. The intensity of the projected light in each waveform projection region is encoded such that during the scanning cycle... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

17. A depth data measurement method, comprising: A first depth imaging measurement head using a depth data measurement device performs a first pattern scan on the imaging area to obtain a first set of N-step phase shift patterns. The second depth imaging measurement head of the depth data measurement device is used to perform a second pattern scan on the imaging area to obtain a second set of N-step phase shift patterns. Based on the relative positions of the first depth imaging measurement head and the second depth imaging measurement head, depth information of the imaging object within the imaging area is synthesized from the first set of N-step phase shift patterns and the second set of N-step phase shift patterns. The scanning imaging of the first depth imaging measurement head and the second depth imaging measurement head each includes: A linear light beam moving along a first direction is projected onto the imaging area, wherein the length direction of the linear light beam is a second direction perpendicular to the first direction, and the projected linear light beam is within the scanning cycle. One pattern scan is completed within a certain period of time, and the scanning cycle is... It includes multiple cyclic sub-cycles. In each cycle sub-cycle In this process, the linear light has a projection period t p The projection period t performs brightness changes. p Duration and exposure switching cycle t e The durations are the same, and the projection period t is the same. p It includes N waveform projection areas with a width of 2π / N, and the projected light intensity of each waveform projection area is encoded, where N is an integer greater than 1; The imaging area is captured using an image sensor comprising N storage units for each pixel to obtain N image frames under the linear light scan projection. Each pixel's N storage units belong to one of N groups of storage units, and each group of storage units is spaced apart by an exposure switching period t with a phase interval of 2π / N. e Expose; and The depth data of the object under test within the imaging area is obtained based on the image frame. The intensity of the projected light in each waveform projection region is encoded such that during the scanning cycle... When a pattern scan is completed, each of the N sets of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a set of N-step phase shift patterns with a 2π / N phase shift between each other.

18. The method of claim 17, further comprising: The depth data measurement device captures multiple sets of first N-step phase shift patterns and multiple sets of second N-step phase shift patterns that are in relative motion with respect to the imaging object; as well as Based on the calibration points, the depth information generated from multiple sets of first group N-step phase shift patterns and multiple sets of second group N-step phase shift patterns will be synthesized into model information of the imaging object.

Images