Depth data measuring head, measuring apparatus, and measuring method

Abstract

A depth data measuring head (300), a measuring apparatus (1500), and a measuring method. The measuring head (300) comprises: a projection apparatus (310) for projecting linear light to an imaging area; and image sensors (320_1, 320_2) comprising N groups of memory cells, each group of memory cells being exposed at an exposure switching period te spaced apart by a 2 π / N phase. The projection apparatus (310) implements one-time pattern scanning in a scanning period , wherein the scanning period comprises a plurality of cycle sub-periods ; in each cycle sub-period , a projection period tp comprises N waveform projection areas of 2π / N width and of a light intensity encoded based on an imaging pattern. When one-time pattern scanning is implemented within the scanning period , the N groups of memory cells of the image sensors (320_1, 320_2) each form a different stripe pattern, and there is phase shift of 2π / N between the N stripe patterns. Thus, multiple images of single linear optical scanning can be obtained, greatly increasing the speed of synthesising a depth map, and being suitable for photographing a target object in motion.

Description

Depth data measuring head, measuring device and measuring method Technical Field

[0001] The present invention relates to the field of three-dimensional imaging, and in particular to a depth data measuring head, a measuring device and a measuring method. Background Art

[0002] Depth cameras are devices that capture depth information about target objects. These cameras are widely used in fields such as 3D scanning and 3D modeling. For example, an increasing number of smartphones are now equipped with depth cameras for facial recognition. Existing technology can use fringe light coding to achieve high-precision imaging. However, fringe light coding requires capturing multiple fringe images to synthesize a single depth image. As a result, the resulting depth image has a low frame rate and cannot meet the requirements for real-time, high-precision dynamic imaging.

[0003] Therefore, an improved depth data measurement scheme is needed.

[0004] Summary of the Invention

[0005] The present disclosure addresses a technical problem by providing an improved depth data measurement solution. This solution utilizes an image sensor with multiple groups of memory cells for phase-shifted exposure per pixel to image projected line light with successive phase shifts in different sub-periods. This allows the image sensor's N groups of memory cells to each capture a different phase-shifted fringe image during a single scan of the line light, thereby enabling the acquisition of N fringe images from a single line light scan. This significantly increases the speed of depth map synthesis without sacrificing image sensor resolution, making it suitable for capturing moving objects.

[0006] According to a first aspect of the present disclosure, a depth imaging measurement head is provided, comprising: a projection device for projecting a linear light moving along a first direction toward an imaging area, wherein the length direction of the linear light is a second direction perpendicular to the first direction; an image sensor, wherein the image sensor comprises N groups of storage units, each group of storage units having an exposure switching period t ... e Exposure is performed, wherein N is an integer greater than 1, wherein the projection device is in a scanning period A pattern scan is completed within one scan period. There are multiple cyclic sub-periods in In each sub-cycle In the example, the linear light is projected with a period t p Perform light and dark changes, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period tp It includes N waveform projection areas with a width of 2π / N, and the projection light intensity of each waveform projection area is encoded so that When one pattern scan is completed, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

[0007] Optionally, the projection period t of the linear light p The exposure switching period t of the first group of memory cells e synchronous.

[0008] Optionally, in each sub-period T i In the projection period t of the linear light p The internally projected waveform is the same and is a rectangular wave with a 2π / N phase bright area and a 6π / N phase dark area, wherein the N stripe patterns are stripe patterns with repeated bright and dark areas.

[0009] Optionally, the line light scan corresponds to a dwell time t on each column of pixels. c Not less than the scanning period Divided by the number of columns C, the residence time t c is the projection period t p More than 10 times.

[0010] Optionally, in each sub-period T i In the example, the linear light is projected with a period t p Project m times, and each sub-period T i The duration is greater than the residence time t c .

[0011] Optionally, each pixel in the image sensor includes N storage units, and the N storage units of each pixel belong to one of the N groups of storage units. Optionally, the N storage units included in each pixel are N charge storage units, and the exposure positions of the N charge storage units corresponding to the same pixel are the same, but the exposure periods are spaced 2π / N apart, and during the scanning period When one pattern scan is completed, a set of N-step phase-shift patterns is obtained from the N sets of charge storage units, and the set of N-step phase-shift patterns is used to generate a depth map of the imaging area.

[0012] Optionally, the image sensor includes N groups of pixels evenly distributed on the imaging surface, each pixel includes a corresponding storage unit, and during the scanning period When one pattern scan is completed, the set of N-step phase-shift patterns is obtained from the N groups of storage units corresponding to the N groups of pixels, and the set of N-step phase-shift patterns is used to generate a depth map of the imaging area.

[0013] Optionally, the projection device in the first scanning period A pattern scan is completed within a second scanning period, so that each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of Gray code patterns; the projection device A pattern scan is completed within 10 seconds, so that each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute the set of N-step phase-shift patterns, wherein a depth map of the imaging area is generated from the set of N-step phase-shift patterns based on the set of Gray code patterns.

[0014] Optionally, the projection device includes: a light-emitting device for generating linear light; and a reflecting device for reflecting the linear light, so as to project the linear light moving in the direction perpendicular to the stripe direction to the shooting area at a predetermined frequency, the length direction of the linear light is the length direction of the projected stripe, and the reflecting device includes one of the following: a mechanical galvanometer that vibrates back and forth at the predetermined frequency; a micromirror device that reciprocates at a predetermined frequency; and a mechanical rotating mirror that rotates unidirectionally at a predetermined frequency.

[0015] Optionally, the image sensor includes a first image sensor and a second image sensor fixed in relative position, wherein the first image sensor and the second image sensor each include the N groups of storage units and are exposed synchronously with each other.

[0016] Optionally, the projection device is configured to perform a scanning cycle. Complete α pattern scans within a period, each of the scanning cycles There are multiple cyclic sub-periods in In each sub-cycle Including N sub-cycles T1-T N , in each sub-cycle T i In the example, the linear light is projected with a period t p Perform light and dark variation projection, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period t p Includes bright areas, where in subperiods T1-T N In the projection period t, the bright area p The position of the phase is changed at intervals of 2π / αN, so that in each scanning cycle When a pattern scan is completed within α, each of the N groups of storage units of the image sensor images a different stripe pattern, and within α scanning cycles When α times of pattern scanning is completed within αN, αN stripe patterns constitute a group of αN-step phase-shift patterns with a 2π / αN phase shift between each other, where α is an integer greater than or equal to 2.

[0017] Optionally, the projection device is configured to: A pattern scan is completed within one scan period. There are multiple cyclic sub-periods in In each sub-cycle Including N sub-cycles T1-T N , in each sub-cycle T i In the example, the linear light is projected with a period t p Perform light and dark changes, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period t p Includes bright areas, where in subperiods T1-T N In the projection period t, the bright area p The position in the scan cycle is changed at intervals of 2π / N phases. When a pattern scan is completed within 1000, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other. Optionally, N=2 n , n is an integer greater than or equal to 1.

[0018] According to a second aspect of the present disclosure, a depth data measuring device is provided, comprising: the depth data measuring head according to the first aspect, and a processor connected to the depth data measuring head, configured to: When one pattern scan is completed within a period of time, a depth map of the imaging area is obtained from the N acquired fringe patterns.

[0019] According to a third aspect of the present disclosure, a depth data measurement device is provided, comprising a first depth imaging measurement head and a second depth imaging measurement head fixed in relative position, and the first depth imaging measurement head and the second depth imaging measurement head are the depth data measurement heads as described in the first aspect, wherein, after the first depth imaging measurement head completes a first pattern scan, the second depth imaging measurement head performs a second pattern scan, and a first group of N-step phase-shifted patterns obtained by the first pattern scan and a second group of N-step phase-shifted patterns obtained by the second pattern scan are synthesized into depth information of an imaging object in an imaging area based on the relative position.

[0020] According to a fourth aspect of the present disclosure, a depth data measurement method is provided, comprising: projecting a linear light moving along a first direction toward an imaging area, wherein the length direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light is A pattern scan is completed within one scan period. There are multiple cyclic sub-periods in In each sub-cycle In the example, the linear light is projected with a period t p Perform light and dark changes, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period t p The invention comprises N waveform projection areas with a width of 2π / N, and the projection light intensity of each waveform projection area is encoded, where N is an integer greater than 1; an image sensor comprising N groups of storage units is used to photograph the imaging area to obtain N image frames under the linear light scanning projection, wherein each group of storage units has an exposure switching period t that is spaced 2π / N apart from each other. e exposure; and obtaining the depth data of the object under test in the imaging area based on the image frame, wherein the projection light intensity of each waveform projection area is encoded so that in the scanning period When one pattern scan is completed, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

[0021] According to a fifth aspect of the present disclosure, a depth data measurement method is provided, comprising: using a first depth imaging measurement head of a depth data measurement device to perform a first pattern scan on an imaging area to obtain a first group of N-step phase-shift patterns; using a second depth imaging measurement head of the depth data measurement device to perform a second pattern scan on the imaging area to obtain a second group of N-step phase-shift patterns; synthesizing depth information of an imaging object in the imaging area from the first group of N-step phase-shift patterns and the second group of N-step phase-shift patterns based on the relative positions of the first depth imaging measurement head and the second depth imaging measurement head, wherein the scanning imaging of the first depth imaging measurement head and the second depth imaging measurement head each comprises: projecting a linear light moving along a first direction onto the imaging area, wherein the length direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light is projected in a scanning period A pattern scan is completed within one scan period. There are multiple cyclic sub-periods in In each sub-cycle In the example, the linear light is projected with a period t p Perform light and dark changes, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period t p The invention comprises N waveform projection areas with a width of 2π / N, and the projection light intensity of each waveform projection area is encoded, where N is an integer greater than 1; an image sensor comprising N groups of storage units is used to photograph the imaging area to obtain N image frames under the linear light scanning projection, wherein each group of storage units has an exposure switching period t that is spaced 2π / N apart from each other. e exposure; and obtaining the depth data of the object under test in the imaging area based on the image frame, wherein the projection light intensity of each waveform projection area is encoded so that in the scanning period When one pattern scan is completed, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

[0022] Optionally, the method also includes: the depth data measuring device photographing multiple groups of first-group N-step phase-shift patterns and multiple groups of second-group N-step phase-shift patterns in relative motion with respect to the imaging object; and synthesizing depth information generated from the multiple groups of first-group N-step phase-shift patterns and multiple groups of second-group N-step phase-shift patterns into model information of the imaging object based on calibration points.

[0023] Therefore, the depth imaging measurement head of the present invention can simultaneously acquire N-step phase shift images under a single scan of linear light while maintaining image resolution, thereby improving imaging speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0025] Figure 1 shows the principle of depth imaging using fringe-coded structured light.

[0026] FIG2 shows another example of projecting stripe-coded structured light.

[0027] FIG3 shows a schematic diagram of the composition of a depth data measurement head according to an embodiment of the present invention.

[0028] 4A-B illustrate an enlarged operation example of the projection device shown in FIG. 3 .

[0029] FIG. 5 shows a simplified perspective schematic diagram of a projection device used in the present invention.

[0030] 6A-B show examples of N groups of memory cells included in the image sensor used in the present invention.

[0031] FIG. 7 shows an example of the relative relationship between exposure periods of different groups of storage units in the same image sensor.

[0032] Figure 8 shows a cyclic sub-period Schematic diagram of imaging of different groups of storage units.

[0033] FIG9 shows an example of the relative relationship between the projection light waveform and the exposure period of the storage unit groups 1-4 when performing sinusoidal wave four-step phase shift pattern imaging.

[0034] Figure 10 shows the completion of a scan cycle An example of patterns 1-4 obtained from memory cell groups 1-4 respectively.

[0035] FIG11 shows an example of depth map imaging using Gray code combined with four-step phase shifting.

[0036] Figure 12 shows a cyclic sub-period Schematic diagram of imaging of different groups of storage units.

[0037] 13A-D show the relative relationship between the projection light waveforms of sub-periods T1-T4 and the exposure periods of memory cell groups 1-4.

[0038] Figure 14 shows the completion of a scan cycle An example of patterns 1-4 obtained from memory cell groups 1-4 respectively.

[0039] FIG. 15 is a schematic diagram showing a depth data measuring device according to an embodiment of the present invention.

[0040] 16A-B are schematic flow charts showing a depth data measurement method according to an embodiment of the present invention.

[0041] FIG17 shows an example in which part of the depth information of the imaging object is missing when a single camera uses structured light imaging.

[0042] FIG18 shows an example of using dual-camera imaging to improve the partial depth information loss of the imaging object.

[0043] FIG. 19 shows a schematic diagram of a depth data measuring device according to an embodiment of the present invention.

[0044] FIG20 is a schematic diagram showing the composition of a depth data measuring device according to an embodiment of the present invention.

[0045] FIG21 shows a schematic flow chart of a depth data measurement method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0047] According to the principle of structured light measurement, accurately determining the scanning angle α is crucial to the entire measurement system. Point and line structured light can be calculated and determined using mechanical devices such as rotating mirrors. The significance of image encoding and decoding lies in determining the scanning angle of coded structured light, or surface structured light systems. Figure 1 illustrates the principle of depth imaging using fringe-coded structured light. For ease of understanding, the encoding principle of fringe structured light is briefly explained using a two-grayscale, three-bit binary time code. A projection device sequentially projects three patterns onto the object in the image capture area. Each pattern uses two grayscales, light and dark, to divide the projection space into eight regions. Each region corresponds to a specific projection angle, where it can be assumed that the bright region corresponds to the code "1" and the dark region corresponds to the code "0." The code values ​​of a point on the object in the projection space in the three coded patterns are combined in the order of projection to obtain the region code value for that point. This determines the region in which the point is located and is then decoded to obtain the scanning angle for that point.

[0048] In order to improve the matching accuracy, the number of projected patterns in the time coding can be increased. Figure 2 shows another example of projecting stripe coded structured light. Specifically, the figure shows a five-bit binary time coding of two gray levels. In an application scenario such as binocular imaging, this means that, for example, each pixel in each left and right image frame contains 5 area coding values ​​of either 0 or 1, thereby enabling left and right image matching to be achieved with higher accuracy (for example, pixel level). When the projection rate of the projection device remains unchanged, compared with the three coding patterns in Figure 1, the example of Figure 2 is equivalent to achieving higher-precision image matching at a higher time domain cost.

[0049] Figure 3 shows a schematic diagram of how to use linear light to project a stripe image to acquire depth data. As shown in Figure 3, a depth data measurement head 300 includes a projection device 310 and two image sensors 320_1 and 320_2. In a monocular implementation, the depth data measurement head 300 can also use a single image sensor for capture.

[0050] The projection device 310 is used to scan and project structured light with stripe coding onto the capture area. For example, within three consecutive image frame projection cycles, the projection device 310 can successively project the three patterns shown in Figure 1. The imaging results of these three patterns can be used to generate depth data. 320_1 and 320_2, which can be respectively referred to as the first and second image sensors, have a predetermined relative positional relationship and are used to capture the capture area to obtain first and second two-dimensional image frames under the illumination of the structured light. For example, when the projection device 310 projects the three patterns shown in Figure 1, the first and second image sensors 320_1 and 320_2 can respectively image the capture area (e.g., the imaging plane in Figure 3 and the area within a certain range before and after it) where the three patterns are projected within three synchronized image frame imaging cycles.

[0051] As shown in Figure 3 , projection device 310 can project a linear light extending in the x-direction in the z-direction (i.e., toward the imaging area). The projected linear light can continuously move in the y-direction to cover the entire imaging area. The lower portion of Figure 3 provides a more understandable illustration of the scanning of the linear light, with a perspective view of the imaging area.

[0052] In this disclosure, the direction of light exiting the measurement head is defined as the z-direction, the vertical direction of the imaging plane is defined as the x-direction, and the horizontal direction is defined as the y-direction. Therefore, the stripe structured light projected by the projection device can be the result of linear light extending in the x-direction moving in the y-direction. While other embodiments can also synchronize and image the stripe structured light generated by linear light extending in the horizontal y-direction moving in the x-direction, this disclosure still prefers to use vertical stripe light for illustration.

[0053] Figures 4A-B illustrate an enlarged example of the operation of the projection device shown in Figure 3. Specifically, as shown in Figure 3, in the projection device 310, a laser is emitted by a laser generator (such as the laser generator 411 shown in detail in Figures 4A-B) and is scanned and projected by a projection mechanism (such as the projection mechanism 412 shown in detail in Figures 4A-B) onto a shooting area (the gray area in Figure 3) to perform active structured light projection on the object to be measured in the shooting area (for example, the person in Figure 3). A pair of image sensors 320_1 and 320_2 images the shooting area, thereby obtaining the image frames required for depth data calculation. As shown in Figure 3, the dotted line emitted by the projection device 310 represents its projection range, while the dotted lines emitted by the image sensors 320_1 and 320_2 represent their respective imaging ranges. The shooting area is generally located in the overlapping area of ​​the projection and imaging ranges of the three.

[0054] In practical applications, a laser generator is used to generate linear and / or infrared laser light, and the laser generator is switched at high speed to scan and project light that alternates between light and dark, corresponding to the stripe code. The high-speed switching can include high-speed on / off of the laser generator and high-speed code switching.

[0055] In one embodiment, the laser generator can emit laser light of uniform intensity, and the projected fringe pattern is achieved by turning the laser generator on and off. In this case, because the laser generator projects light of only one intensity at varying duty cycles, each pixel of the image sensor integrates the projected light to determine the presence or absence of illumination. Therefore, the image sensor can be a black-and-white image sensor.

[0056] In another embodiment, the laser generator itself can emit laser light with varying light intensity. For example, the laser light intensity can be sinusoidally alternating over a large period depending on the applied power. This sinusoidally alternating laser light can be combined with fringe projection to create a scanning pattern that alternates between light and dark, with the brightness of the bright fringes varying between them. In this case, the image sensor needs to be able to discriminate between different light intensities and can therefore be a multi-level grayscale image sensor. Clearly, grayscale projection and imaging can provide more precise pixel-to-pixel matching than black-and-white projection and imaging, thereby improving the accuracy of depth data measurement.

[0057] In one embodiment, laser generator 411 can be a linear laser generator that generates a linear beam extending in the x-direction (a direction perpendicular to the paper in Figures 4A-B). This linear beam is then projected onto the imaging plane by a reflective mechanism 412 that can swing along an axis in the x-direction. The swinging diagram of reflective mechanism 412 is shown in Figure 4B. Projective mechanism 412 (e.g., a reflector) is capable of scanning within an angle range of α, thereby achieving reciprocating linear beam scanning within a range AB of the imaging plane.

[0058] It should be understood that in order to achieve the projection of a stripe pattern, the linear light itself needs to change brightness (or, in a simple implementation, light on and off) as it continues to move in the y-direction. For example, when scanning the first pattern in FIG. 1 , while the projection mechanism 412 scans through the first α / 2 angle, the laser generator 411 remains off. When scanning through the last α / 2 angle, the laser generator 411 turns on, thereby achieving a pattern with a dark left side and a bright right side. When scanning the second pattern in FIG. 1 , while the projection mechanism 412 scans through the 0 to α / 4 angles, the laser generator 411 remains off. When scanning through the α / 2 to α / 2 angles, the laser generator 411 turns on. When scanning through the α / 2 to 3α / 4 angles, the laser generator 411 turns off again. When scanning through the 3α / 4 to α angles, the laser generator 411 turns on. This achieves a dark-bright-dark-bright pattern. Similarly, the third pattern of FIG. 1 and the pattern with finer stripes shown in FIG. 2 can be realized with more frequent changes based on the rotation angle.

[0059] In one embodiment, the above-mentioned reflection mechanism 412 can be a micromirror device (also known as a digital micromirror device, DMD), and can be implemented as a MEMS (micro-electromechanical system). Figure 5 shows a simplified perspective schematic diagram of the projection device used in the present invention. As shown in Figure 5, the point laser generated by the laser can obtain linear light through a lens (corresponding to the linear laser generator 411 in Figure 4), and the above-mentioned linear light is then reflected by a micromirror device in the form of MEMS, and the reflected linear light is then projected into the external space through a light window. Micromirror devices have extremely high performance. For example, commercially available DMDs can perform highly stable reciprocating vibrations at a frequency of 2k, thus laying the foundation for high-performance depth imaging.

[0060] To obtain a high-precision depth map, the depth data measurement head shown in Figure 3 must sequentially project multiple different fringe patterns. In other words, existing approaches that use captured fringe patterns to synthesize depth maps trade accuracy for time. Furthermore, because different fringe patterns captured in N consecutive imaging cycles are used to synthesize a depth map, existing depth data measurement methods are only applicable when the subject remains motionless during these N imaging cycles. This significantly limits the application scope of techniques for extracting depth data using actively projected fringe images.

[0061] In light of this, the present invention proposes a novel depth data measurement scheme. This scheme utilizes an image sensor equipped with different groups of storage units capable of phase-shifted exposure. By cleverly configuring the light and dark variations of the projected line pattern, different groups of storage units in the image sensor can each capture different phase-shifted fringe images during a single scan of the line pattern, thereby enabling the acquisition of multiple fringe images from a single line pattern scan. This significantly increases the speed of depth map synthesis and is suitable for capturing moving objects.

[0062] In one embodiment, the present invention can be implemented as a depth imaging measurement head, comprising: a projection device and an image sensor. The projection device can be used to project a linear light moving along a first direction (e.g., the y-direction in FIG. 3 ) toward an imaging area, wherein the longitudinal direction of the linear light is a second direction perpendicular to the first direction (e.g., the x-direction in FIG. 3 ). In one embodiment, the projection device can have the implementation structure shown in FIG. 5 , comprising a linear light generating device and a projection mechanism for reflecting and projecting the linear light, capable of varying the projection direction within a certain angle.

[0063] An image sensor, wherein each pixel in the image sensor comprises N storage units, each of the N storage units of each pixel belongs to one of N groups of storage units, and each group of storage units has an exposure switching period t that is spaced 2π / N apart from each other. e Exposure is performed, where N is an integer greater than 1.

[0064] Here, for ease of understanding, N=4 is taken as an example to illustrate the structure and exposure of the image sensor. Figures 6A-B show an example of N groups of storage units included in the image sensor used in the present invention. The N groups of storage units included in the image sensor can be N storage units included in each pixel; or the image sensor can be composed of N groups of pixels, each pixel corresponding to a storage unit. Figure 6A shows the structure of a pixel column in which each pixel includes N storage units. As shown in Figure 6A, a pixel column 621 can include k pixels P1-P k (In the example of 600x800 resolution, k=600.) Each pixel includes the same structure, namely, a photosensitive unit, N switches, and N storage units, where each switch controls the charge storage of one storage unit. Specifically, pixel P1622 may include a photodiode 624 serving as a photosensitive unit, N switches 626, and N storage units 628. Pixel P k 623 may include a photodiode 625 serving as a light-sensing unit, N switches 627 , and N storage units 629 .

[0065] The storage unit is, for example, a unit for accumulating and storing the charge generated by the charge photodiode based on the received light and outputting the charge based on the charge storage amount. The N storage units corresponding to the same pixel all obtain charge from the same photosensitive unit, but each has an exposure switching period t that is 2π / N phases apart from each other. e For exposure. Each pixel has a storage cell corresponding to one of N groups of storage cells. Therefore, in a 600x800 pixel image sensor, assuming N = 4, there are 600x800x4 storage cells. These 600x800x4 storage cells belong to four groups, each of which contains 600x800 storage cells, each corresponding to a different pixel. Each group of storage cells can be exposed with the same period, with a phase difference of π / 2 between adjacent groups. Because N storage cells share a single photosensitive unit, multiple images can be acquired in a single scan without reducing the original resolution of the image sensor.

[0066] Figure 6B illustrates an example of an image sensor comprising N groups of pixels. For ease of illustration, Figure 6B shows an example of 16x24 pixels. It should be understood that an image sensor in actual use may have more pixels, for example, 600x800 pixels. The image sensor shown in Figure 6B includes four (N=4) groups of pixels evenly distributed across the entire imaging surface, represented by the symbols 1, 2, 3, and 4 in the blocks shown. Here, "evenly distributed" across the entire imaging surface means that when linear light is projected and scanned in the y-direction, the same (or approximately the same) number of pixels of each type are illuminated within the currently illuminated area. In a preferred embodiment, the four groups of pixels, as shown in Figure 6B, are spaced apart from each other in units of one pixel. That is, the image sensor shown in Figure 6B can be considered to include multiple "pixel units" (as indicated by the bold black boxes in the figure; in the example of Figure 6B, this may include 8x12 identically constructed pixel units), each pixel unit including one pixel from each of the four groups. In other embodiments, each group of pixels may also be spaced apart in units of two pixels (e.g., two pixels adjacent to each other in the x-direction).

[0067] FIG7 shows an example of the relative relationship between the exposure cycles of different groups of storage units of the same image sensor. In the example of FIG7 , the four groups of storage units have the same exposure switching cycle t e , and all are switched at a duty cycle of 50%. In other words, all memory cells of the image sensor have the same exposure switching waveform. However, the difference is that the waveforms of different groups of memory cells have the same phase difference of π / 2. In one implementation, the exposure switching period t eFor example, a typical value of 20ns means that each memory cell in the image sensor operates with a 10ns open time for exposure and a 10ns close time. However, the second group of memory cells opens 5ns later than the first group, the third group of memory cells opens 5ns later than the second group, and the fourth group of memory cells opens 5ns later than the third group (which can also be considered 5ns earlier than the first group of memory cells).

[0068] When the image sensor used is capable of performing group phase-shift exposure as shown in FIG6A-B and FIG7 , the linear light projection of the projection device can be cleverly set to achieve multi-image acquisition in a single scan.

[0069] Specifically, the projection device can perform a scan cycle (for example, ) completes a pattern scan. It is assumed that the linear light scans the imaging area at a uniform speed and in multiple cycles. Repeat. The following will be for each cycle sub-period A light projection embodiment is described in detail.

[0070] In each sub-cycle In the example, the linear light is projected with a period t p Perform light and dark changes, the projection period t p The duration of the exposure switching cycle t e The projection period t p It includes N waveform projection areas with a width of 2π / N, and the projection light intensity of each waveform projection area is encoded so that When one pattern scan is completed, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

[0071] In one embodiment, each waveform projection area projects a rectangular wave with a width of 2π / N or 0 (0 can also be regarded as a rectangular wave with a light intensity of 0), and corresponds to a cyclic sub-period based on a set of N-step phase shift patterns. The light intensity distribution is used to determine the light intensity of each waveform projection area.

[0072] In one embodiment, a set of N-step phase shift patterns is a sine wave four-step phase shift pattern, and each projection period t is obtained based on the exposure of N groups of storage cells corresponding to the waveform projection area. p The light intensity value of each waveform projection area is taken, wherein the light intensity value is not less than zero.

[0073] For the convenience of explanation, it is assumed that N=4 and the line laser scanning projection period t p The laser intensity of each 1 / 4 cycle is Q1 / Q2 / Q3 / Q4. The integrated brightness of the four pixels are as follows: P1 = Σ(Q1 + Q2) P2 = Σ(Q2 + Q3) P3 = Σ(Q3 + Q4) P4 = Σ(Q1 + Q4) (1)

[0074] Therefore, the values ​​of Q1 to Q4 can be determined according to the pattern type corresponding to the required 4-step phase-shift imaging.

[0075] Figure 8 shows a cyclic sub-period Schematic diagram of imaging of different groups of storage units. In the example shown in Figure 8, the desired phase shift pattern is a sinusoidal light waveform with a phase difference of π / 2. At this time, the values ​​of the four groups of storage units P1 to P4 can be: P1 = Q / 2*sint+Q / 2 P2 = -Q / 2*cost+Q / 2 P3 = -Q / 2*sint+Q / 2 P4 = Q / 2*cost+Q / 2 (2)

[0076] Here, Q can be regarded as the integrated brightness of the brightest part of the phase-shifted fringe shown in Figure 8, and the value of t can correspond to a cyclic sub-period. At different angular positions in the image sensor, the value of t when the linear light is scanned at a uniform speed can correspond to pixels at different positions in the image sensor.

[0077] The brightness values ​​of Q1 to Q4 with respect to t can be inversely calculated using equation (2). Since there are N = 4 unknowns Q1 to Q4 in equation (2), and the rank of equation (2) is N-1 = 3, Q1 to Q4 can actually have countless solutions. One solution is given below: Q1 = A / 2*sint+A / 2 Q2 = 0 Q3 = -A / 2*cost+A / 2 Q4 = A / 2(cost-sint) (3)

[0078] Wherein, O*A=Q, and O is the number of exposure cycles received by each pixel when the linear light is scanned (for example, in the following example, each pixel column can complete 100 exposure cycles in 2 μs when the linear light is scanned, and O can be regarded as 100 in this case).

[0079] However, since light intensity cannot be negative, the values ​​of Q1 to Q4 must remain non-negative. Equation (3) holds true in the range of t = 0 to π / 4. When the linear light is scanned to the range of t = π / 4 to π / 2, another solution can be given based on Equation (2): Q1 = A / 2*cost+A / 2 Q2 = A / 2(sint-cost) Q3 = -A / 2*sint+A / 2 Q4 = 0 (4)

[0080] FIG9 shows an example of the relative relationship between the projection light waveform values ​​and the exposure periods of the storage unit groups 1-4 when performing sinusoidal wave four-step phase shift pattern imaging.

[0081] As shown in the figure and in combination with FIG8 and formula (1), it can be seen that at t=0, the values ​​of P1 to P4 can correspond to Q / 2, 0, Q / 2 and Q respectively.

[0082] Assuming scanning cycle 3.84ms, when the image sensor includes 1920 columns and the linear light projection period t p is 20ns, because the time it takes for each pixel to be scanned by the linear light, that is, the dwell time t c Can be equal to the scan period Divide by the number of columns C, that is, 3.84ms / 1920=2us, or considering that the linear light has a certain width, the dwell time t c The time is not less than 2 μs, so each pixel column can complete 100 exposures in the 2 μs of the linear light scan.

[0083] In one embodiment, the linear light can be made to At the beginning, keep the first 100 projection cycles t p The waveform corresponding to Q1=A / 2, Q2=Q3=0, Q4=A / 2 at t=0 (here A=Q / O=Q / 100). In a preferred embodiment, the waveform can be changed in each projection period t based on a small change in t. p In the above, the values ​​of Q1 to Q4 are each fine-tuned, for example, by the solution of equation (2).

[0084] As mentioned before, the repeating sub-cycle A predetermined number of times, for example, M times, thereby completing a scanning cycle Figure 10 shows the completion of a scan cycle Patterns 1-4 are obtained from memory cell groups 1-4 respectively. For example, in the example of FIG10 , M may be equal to 16, i.e., the sub-cycles are repeated 16 times. Thus, a four-step phase shift pattern with a phase difference of π / 2 is obtained. Obviously, Figure 10 shows an example of a sine wave four-step phase shift pattern. When using the image sensor with 1920 columns in the above example to obtain the four-step phase shift pattern with 32 sinusoidal wave stripes shown in Figure 10, each sinusoidal wave stripe covers 60 pixel columns (1920 / 32=60). In other words, the linear light scans one cycle. It is necessary to complete 60*100=6000 projections. In each projection cycle t p In the embodiment where the values ​​of Q1 to Q4 are fine-tuned as in the solution of formula (3), each projection cycle t p Compared with the previous projection period t p , t has a small increment of Δt=2π / 6000=π / 3000, which leads to different values.

[0085] Returning to Figure 9, as the linear light scans from the position corresponding to t = 0 to the position corresponding to t = π / 6, the values ​​of Q1 to Q4 can be solved based on, for example, equation (3). At t = π / 6, it can be assumed that the linear light has scanned to the fifth pixel column of the 60 pixel columns in the current cycle, and based on equation (3), it can be solved that t = π / 6 corresponds to Q1 = 3A / 4, Q2 = 0, Q3 = 0.183A, and Q4 = 0.067A.

[0086] As the linear light scans from the position corresponding to t = π / 6 to the position corresponding to t = π / 3, the values ​​of Q1 to Q4 can be further solved. Since cost = sint at t = π / 4, and cost < sint from π / 4 to 3π / 4, equation (3) that indicates Q4 < 0 no longer applies. Based on equation (4), we can now solve for Q1 = 3A / 4, Q2 = 0.183A, Q3 = 0.067A, and Q4 = 0 at t = π / 3. Furthermore, at t = π / 3, it can be assumed that the linear light has scanned to the 10th pixel column of the 60 pixel columns in the current cycle.

[0087] As the linear light scans from the position corresponding to t = π / 3 to the position corresponding to t = π / 2, the values ​​of Q1 to Q4 can be solved based on equation (4). At t = π / 2, it can be considered that the linear light scans to the 15th pixel column among the 60 pixel columns of the current cycle, and based on equation (4), it is solved that when t = π / 2, Q1 = A / 2, Q2 = A / 2, Q3 = 0, and Q4 = 0. It should be noted that since pixels 1-4 perform 4-step phase shift imaging, in each projection cycle t p , there exists Q1+Q2+Q3+Q4=A, and since pixels 1-4 are spaced 2π / N apart from each other, the phases are also projected at a period t pExposure is performed and the open time is 10%, so in each projection cycle t p Pixels 1-4 can obtain a total of 2A*t p / 4 integrated brightness.

[0088] As described above, in conjunction with FIG9 , examples of the values ​​of Q1 to Q4 for the first π / 2 of a 2π cycle of a sinusoidal wave N-step phase shift diagram are described, using the imaging examples of pixels 1-4. Based on equation (2) and the example in FIG9 , those skilled in the art can further determine examples of the values ​​of Q1 to Q4 for the last 3π / 2 of a 2π cycle.

[0089] In addition, it should be understood that although FIG8 and FIG9 show the sub-periods T1-T4 for the convenience of explaining the changes in the fringe image and the waveform (the sub-periods T1-T4 each correspond to a 2π cycle sub-period 0~π / 2 part, π / 2~π part, π~3π / 2 part and 3π / 2~2π part in the above sine wave phase shift pattern), but when realizing the above sine wave 4-step phase shift pattern, only one cycle sub-period needs to be obtained according to the above formula (2) The change of the internal projected laser brightness value does not require additional division into sub-periods T1-T4.

[0090] In one embodiment, other stripe lights can be combined with N-step phase shift imaging to obtain a more accurate depth image. Figure 11 shows an example of Gray code combined with four-step phase shift to perform depth image imaging. In practical applications, more cyclic sub-periods can be included in each 4-step phase shift pattern. To improve the imaging accuracy of the depth map. However, due to the periodic repetition of the 4-step phase shift pattern, there is a problem that the depth jump across the period cannot be identified. In this case, the Gray code can be used to perform a preliminary imaging of the object in the shooting space, and then the 4-step phase shift imaging can be performed. Specifically, the projection device can perform a first scanning cycle. A pattern scan is completed within a period of 10 seconds (for example, the corresponding waveform of each linear light projection period is solved by the light-dark diagram shown in the figure, so that each of the N groups of storage cells of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of Gray code patterns. The projection device performs a pattern scan in the second scanning period. A pattern scan is completed within 10 seconds, so that each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute the set of N-step phase-shift patterns, wherein a depth map of the imaging area is generated from the set of N-step phase-shift patterns based on the set of Gray code patterns.

[0091] In addition, when introducing multiple scan cycles When the projection device is α, it can not only realize other stripe patterns plus N-step phase shift patterns, but also realize αN-step phase shift patterns. Here, α is an integer greater than or equal to 2. Specifically, the projection device is α scanning cycles. Complete α pattern scans within a period, each of the scanning cycles There are multiple cyclic sub-periods in In each sub-cycle Including N sub-cycles T1-T N , in each sub-cycle T i In the example, the linear light is projected with a period t p Perform light and dark variation projection, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period t p Includes bright areas, where in subperiods T1-T N In the projection period t, the bright area p The position of the phase is changed at intervals of 2π / αN, so that in each scanning cycle When a pattern scan is completed within α, each of the N groups of storage units of the image sensor images a different stripe pattern, and within α scanning cycles When α times of pattern scanning is completed within αN, αN stripe patterns constitute a group of αN-step phase-shift patterns with a 2π / αN phase shift between each other, where α is an integer greater than or equal to 2.

[0092] For ease of understanding, the following is explained using α=2, N=4 as an example. That is, the exposure switching period t is composed of 4 groups of exposure switching periods t with a phase interval of 2π / N. e The image sensor of the exposed pixels and two scans are performed to achieve the acquisition of the eight-step phase shift pattern.

[0093] In order to use 4 groups of storage unit image sensors to obtain the eight-step phase shift pattern, the projection device needs to perform two scans. In the first scan, that is, in the first scan cycle Including multiple cyclic sub-cycles In each sub-cycle In the projection period t, there are 4 sub-periods T1-T4. The bright area is in the projection period t p The position in the phase is changed at intervals of π / 4, so that in the first scanning cycle When one pattern scan is completed within 1 second, the first four patterns in the eight-step phase shift pattern are obtained. In the same way, it also includes multiple cyclic sub-periods In each sub-cycle In the projection period t, there are also N sub-periods T1-T4. p The position of the image is changed at intervals of π / 4 phase (but the phase of the bright area is different from that of the first scan), so that in the second scanning cycle When one pattern scan is completed within 1 second, the last four patterns in the eight-step phase-shift pattern are acquired. Thus, the eight-step phase-shift pattern can be acquired in two scans, thereby achieving higher depth imaging accuracy.

[0094] In addition, although N=4 is given as an example for the convenience of explanation, in other embodiments, N may also take other values. Specifically, N=2 n , n is an integer greater than or equal to 1. Thus, for example, a higher-precision 8-step phase shift using 8 groups of storage unit image sensors or a 16-step phase shift using 16 groups of storage unit image sensors can be achieved.

[0095] The image sensor may include N groups of pixels, corresponding to N groups of storage units. Alternatively, each pixel of the image sensor may include N storage units, each of which is a charge storage unit. The exposure positions of the N charge storage units corresponding to the same pixel are the same, but the exposure periods are spaced 2π / N apart, and the exposure period is 2π / N apart. When one pattern scan is completed, the set of N-step phase-shift patterns is obtained from the N sets of charge storage units, and the set of N-step phase-shift patterns is used to generate a depth map of the imaging area.

[0096] Figure 12 shows a cyclic sub-period Schematic diagram of imaging of different groups of storage units. In the example shown in Figure 12, the desired phase shift pattern is a light and dark stripe waveform with a phase difference of π / 2. At this time, the values ​​of the four groups of storage units P1 to P4 can be:

[0097] P1=Q, when t=0~π; =0, when t=π~2π

[0098] P2=0, when t=0~π / 2; =Q, when t=π / 2~3π / 2; =0, when t=3π / 2~2π

[0099] P3=0, when t=0~π; =Q, when t=π~2π

[0100] P4=Q, when t=0~π / 2; =0, when t=π / 2~3π / 2; =Q, when t=3π / 2~2π(5)

[0101] Therefore, based on equations (1) and (5), we solve Q1 to Q4 and obtain an optimal solution as follows:

[0102] Q1=A, when t=0~π / 2; =0, when t=π / 2~2π

[0103] Q2=0, when t=0~π / 2; =A, when t=π / 2~π; =0, when t=π~2π

[0104] Q3=0, when t=0~π; =A, when t=π~3π / 2; =0, when t=3π / 2~2π

[0105] Q4=0, when t=0~3π / 2; =A, when t=3π / 2~2π(6)

[0106] Wherein, O*A=Q, and O is the number of exposure cycles received by each pixel when the linear light scans past.

[0107] In the optimization solution based on formula (6), each cycle sub-period can be It is divided into 4 sub-periods T1-T4, each of which corresponds to a 2π cycle sub-period The 0 to π / 2 part, the π / 2 to π part, the π to 3π / 2 part and the 3π / 2 to 2π part, and the bright area is within the projection period t p The position in the phase changes at intervals of 2π / 4.

[0108] In the example of N=4, each cycle sub-period 13A-D show the relative relationship between the projection light waveform of the sub-periods T1-T4 and the exposure period of the memory cell groups 1-4.

[0109] As shown in FIG13A-D, the projected laser is at a time corresponding to the exposure switching period t e The projection period t of the same length p Laser projection is performed, and it can be considered that each laser projection cycle t p All of them keep the exposure switching period t of the first group of storage cells. e Synchronous (although the laser projection period t p In different sub-cycles, it will be turned on at different phases) and switched at a duty cycle of 25% (i.e., the bright area is 2π / N), that is, the waveform is a rectangular wave with a duty cycle of 25%. In the 4 sub-cycles T1-T4, the projected laser is turned on at each projection cycle t p The projection start time in the image is aligned with the exposure start time of the 1st to 4th groups of storage cells respectively.

[0110] Specifically, first, as shown in FIG13A , in the sub-period T1, the projected laser is projected at each projection period t pThe first π / 2 phase of the projected image is kept open. At this time, since the first and fourth groups of storage units are also open, it is possible to p The reflected projection light is exposed in the first π / 2 phase of the projection, so that the charge accumulation can be performed in the corresponding storage unit, as shown in the gray rectangle in the figure. After completing the predetermined m1 cycles of projection, the sub-cycle T1 ends and the projection laser enters the projection cycle t p The sub-period T2 remains open within the π / 2~π phase.

[0111] As shown in FIG13B , in the sub-period T2, the projected laser p The first group of storage units and the second group of storage units are also turned on, so it is possible to p The reflected projection light is exposed within the phase of π / 2 to π, so that the charge accumulation can exist in the corresponding storage unit, as shown by the gray rectangle in the figure. After completing the predetermined m2 cycles of projection, the sub-cycle T2 ends and the projection laser enters each projection cycle t p The sub-period T3 remains open within the phase of π to 3π / 2.

[0112] As shown in FIG13C , in the sub-period T3, the projected laser p The second and third groups of storage units are also turned on, so it is possible to p The reflected projection light is exposed within the phase of π to 3π / 2, so that charge accumulation can exist in the corresponding pixel, as shown by the gray rectangle in the figure. After completing the predetermined m3 cycles of projection, the sub-cycle T3 ends and the projection laser enters each projection cycle t p The sub-period T4 remains open within the 3π / 2~2π phase.

[0113] As shown in FIG13D , in the sub-period T4, the projected laser p The third and fourth groups of storage units are also turned on, so it is possible to p The reflected projection light is exposed within the 3π / 2 to 2π phase, so that charge accumulation can exist in the corresponding pixel, as shown by the gray rectangle in the figure. After completing the predetermined m4 cycles of projection, the sub-cycle T4 ends. At this point, a cycle sub-cycle is completed. Projection and enter the next cycle sub-period Subcycle T1.

[0114] In a simple implementation, the number of projection cycles in each sub-cycle can be made the same, that is, m1 = m2 = m3 = m4, that is, the durations T1-T4 are the same. In this case, if the linear light sweeps across the imaging plane at a uniform speed, a phase shift diagram as shown in FIG12 can be obtained. As shown in FIG13A-D , in sub-cycle T1, the lighting interval of the projected laser falls within the exposure interval of the first and fourth groups of storage cells, and therefore the first and fourth groups of storage cells correspond to bright fringes; in sub-cycle T2, the lighting interval of the projected laser falls within the exposure interval of the first and second groups of storage cells, and therefore the first and second groups of storage cells correspond to bright fringes; in sub-cycle T3, the lighting interval of the projected laser falls within the exposure interval of the second and third groups of storage cells, and therefore the second and third groups of storage cells correspond to bright fringes; in sub-cycle T4, the lighting interval of the projected laser falls within the exposure interval of the third and fourth groups of storage cells, and therefore the third and fourth groups of storage cells correspond to bright fringes. Repeat the above cycle A predetermined number of times, for example, M times, thereby completing a scanning cycle Figure 14 shows the completion of a scan cycle Patterns 1-4 are obtained from memory cell groups 1-4 respectively. For example, in the example of FIG14 , M may be equal to 16, i.e., the sub-cycles are repeated 16 times. Thus, a four-step phase shift diagram of light and dark fringes with a phase difference of π / 2 is obtained.

[0115] Therefore, when obtaining the N-step phase shift diagram of the light and dark fringes, in each cycle Including N sub-cycles T1-T N , in each sub-cycle T i In the example, the linear light is projected with a period t p Perform light and dark changes, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period t p Includes bright areas, where in subperiods T1-T N In the projection period t, the bright area p The position in the scan cycle is changed at intervals of 2π / N phases, thereby making it possible to When a pattern scan is completed within a period of 1 second, each of the N groups of storage cells of the image sensor images a different stripe pattern, and there is a phase shift of 2π / N between the N stripe patterns. ), a set of N-step phase-shift patterns can be directly obtained from N groups of storage units of the image sensor.

[0116] Linear light scanning corresponds to the dwell time t on each column of pixels c Not less than the scanning period Divide by the number of columns C. To ensure sufficient exposure, the dwell time t c is the projection period t p More than 10 times, preferably, the residence time t c is the projection period t p For example, assuming the scan cycle =3.84ms (i.e., the frame rate is 1000 / 3.84≈260 frames / s). When the image sensor includes 1920 columns (960 columns of pixel units in the case of the pixel unit shown in FIG6 ), the time for each pixel to be scanned by the linear light, i.e., the dwell time t c Can be equal to the scan period Divide by the number of columns C, that is, 3.84ms / 1920=2us, or considering that the linear light has a certain width, the dwell time t c Not less than 2us. Linear light projection period t p When the duty cycle is 20ns and 25%, each pixel column can complete 100 exposures in the 2us of the linear light scan, and the exposure time is the same as the current projection period t p The two groups of storage cells corresponding to the bright areas can achieve an exposure duration of 5ns x 100 = 0.5us. Furthermore, because the time it takes for a line light to scan a column of pixels is significantly longer than the pixel exposure period (e.g., 100 times as long as in the above example), the line light itself can be considered to be a fixed light source for the currently illuminated pixel column during these 2us.

[0117] In order to achieve four-step phase shift, each sub-cycle T i The duration of the linear light is not less than the time it takes to scan a pixel column. When using the image sensor with 1920 columns in the above example to obtain the four-step phase shift image with 32 stripes (16 bright stripes and 16 dark stripes) shown in Figure 10, each stripe covers 60 pixel columns (1920 / 32=60). As shown in Figure 9, since each sub-period T i Corresponding to half a stripe, thus covering 30 pixel columns, the duration is 2usx30=60us. Therefore, in this example, m1=m2=m3=m4=m=60us / 20ns=3000. Since the linear light projection period t p The exposure switching period t of each memory cell group e The duration is the same, so when the linear light scans through the distance of half a stripe, the corresponding groups of storage units have also been switched 3000 times.

[0118] As shown above in conjunction with FIG. 12 to FIG. 14, the linear light projected is projected with a period t p And an example of a rectangular wave with a 2π / N phase bright area. In this example, the projected linear light is always projected with a period t p , the duty cycle is 100 / N% (in the above example with N=4, the duty cycle is 25%, corresponding to the bright area of ​​2π / N phase) and each different projection period t p The projection light intensity of the inner bright area has a preferably uniform rectangular wave. i In the projection period t of the linear light p The internal projection waveform is the same and is a rectangular wave with a 2π / N phase bright area and a 6π / N phase dark area (for example, each projection cycle t p In the laser, the p / N time to open, (N-1)*t p / N time off). Between successive sub-cycles, the linear light projection cycle and the ratio of bright area to dark area remain unchanged, only the position of the bright area changes. And in the formation of a scanning cycle Multiple cyclic sub-periods of Thus, a set of N-step phase shift patterns is obtained, which is a stripe pattern with distinct bright and dark areas and repeated multiple times as shown in FIG14 .

[0119] The above embodiment of obtaining the light and dark stripes shown in FIG14 based on a rectangular wave with a duty cycle of 100 / N% and constant brightness can be regarded as an embodiment of the present invention based on a linear light projection period t p Within 2π / N, each phase is individually adjustable and thus the phase difference is 2π / N and the exposure period is also equal to t p A special case of an imaging scheme in which the image sensor generates a set of N-step phase-shift patterns.

[0120] The projection and imaging scheme of the present invention can be used in a monocular scheme (i.e., a scheme equipped with one image sensor) or a binocular scheme (i.e., a scheme equipped with two image sensors fixed in relative positions for synchronous imaging). When the image sensor includes a first image sensor and a second image sensor fixed in relative positions, the first image sensor and the second image sensor can each include the N groups of storage units and be exposed synchronously with each other. In other words, in the binocular scheme, during the scanning period When a pattern scan is completed, the first image sensor can obtain N fringe images, the second image sensor can obtain N fringe images, and these 2N fringe images can be used to obtain depth data.

[0121] Furthermore, to achieve scanning projection, the projection device of the present invention includes: a light-emitting device for generating a linear light; and a reflective device for reflecting the linear light, projecting the linear light at a predetermined frequency toward the capture area, the linear light moving in a direction perpendicular to the direction of the stripes, with the length of the projected stripes being the same. The reflective device includes one of the following: a mechanical galvanometer that reciprocates at the predetermined frequency; a micromirror device that reciprocates at a predetermined frequency; and a mechanical rotating mirror that rotates unidirectionally at a predetermined frequency. The projected linear light can have a high-order Gaussian or flat-top Gaussian distribution, thereby providing a highly uniform brightness distribution across the width of the linear light.

[0122] The present invention also discloses a measuring device using the above-mentioned measuring head. Specifically, a depth data measuring device may include the above-mentioned depth data measuring head and a processor connected to the depth data measuring head, for measuring the depth of the depth data during the scanning period. When a pattern scan is completed within a certain timeframe, a depth map of the imaging area is obtained from the N acquired fringe patterns. In a binocular solution, the processor can determine depth data of the object in the imaging area based on the predetermined relative positions of the first and second image sensors and the N first and N second two-dimensional image frames obtained from the structured light imaging. In various embodiments, the measurement head can be packaged independently or integrated with the processor into the measurement device.

[0123] FIG15 shows a schematic diagram of a depth data measurement device according to an embodiment of the present invention. As shown in the figure, the measurement device 1500 may include the measurement head described above and a processor 1530. The measurement head includes a projection device 1510 and two image sensors 1520.

[0124] The processor 1530 is connected to the measuring head, for example, to the projection device 1510 and each of the two image sensors 1520, and is used to determine the depth data of the object in the shooting area based on the predetermined relative positions of the first and second image sensors 1520_1 and 1520_2 and the N first two-dimensional image frames and N second two-dimensional image frames obtained by imaging the structured light.

[0125] 16A-B are schematic flow charts of a depth data measurement method according to an embodiment of the present invention, which can be implemented by the depth data measurement head and measurement device of the present invention.

[0126] FIG. 16A illustrates a method performed for an image sensor in which each pixel includes N memory cells.

[0127] In step S1610, a linear light moving along a first direction is projected onto the imaging area, wherein the longitudinal direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light is projected in a scanning period. A pattern scan is completed within one scan period. There are multiple cyclic sub-periods in In each sub-cycle In the example, the linear light is projected with a period t p Perform light and dark changes, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period t p It includes N waveform projection areas with a width of 2π / N, and encodes the projection light intensity of each waveform projection area, where N is an integer greater than 1.

[0128] In step S1620, the imaging area is photographed using an image sensor including N storage units per pixel to obtain N image frames under the linear light scanning projection, wherein the N storage units of each pixel belong to one of N groups of storage units, and each group of storage units has an exposure switching period t that is spaced 2π / N apart from each other. e Make an exposure.

[0129] In step S1630, depth data of the object to be measured in the imaging area is obtained based on the image frame.

[0130] FIG. 16B illustrates a method performed by an image sensor including N groups of memory cells corresponding to N groups of pixels.

[0131] In step S1610', a linear light is projected onto the imaging area, wherein the linear light is projected onto the imaging area during the scanning period. A pattern scan is completed within one scan period. There are multiple cyclic sub-periods in In each sub-cycle In the projection period t p It includes N waveform projection areas with a width of 2π / N and light intensity encoded based on the imaging pattern.

[0132] In step S1620', an image sensor including N groups of pixels evenly distributed on an imaging surface is used to capture an imaging area to obtain N image frames, wherein each group of pixels has an exposure switching period t1 separated by a phase of 2π / N. e Make an exposure.

[0133] In step S1630 ′, depth data of the object within the shooting area is obtained based on the first and second two-dimensional image frames.

[0134] Regardless of the storage scheme, the projected light intensity of each waveform projection area is encoded so that When one pattern scan is completed, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

[0135] Furthermore, when using actively projected structured light for imaging, problems with shadows and blind spots will be encountered. Figure 17 shows an example of a single camera missing partial depth information of an imaging object when using structured light imaging. As shown in the figure, due to the uneven structure of the imaging object itself, part of its own structure will be blocked from receiving structured light, thereby causing a shadow in the range facing the camera, and since the shadow part has no structured light illumination, the depth information of the corresponding part is missing when the depth information is subsequently obtained (corresponding to "missing 1" in the figure). Regardless of the structure of the imaging object itself, the part of the imaging object facing away from the structured light generator cannot receive the illumination of the structured light, resulting in the lack of depth information of the corresponding part when the depth information is subsequently obtained (corresponding to "missing 2" in the figure)

[0136] Figure 18 shows an example of how dual-camera imaging can improve the lack of depth information on a portion of the imaged object. As shown in the figure, by setting up dual cameras on both sides, more of the imaged object can be illuminated by structured light, thereby enabling the measurement of more depth information for the imaged object.

[0137] In view of this, the present invention can be implemented as a dual-camera (i.e., dual-head) depth data measurement device. Figure 19 shows a schematic diagram of a depth data measurement device according to one embodiment of the present invention. As shown, depth data measurement device 1900 may include a first depth imaging measurement head 1910 and a second depth imaging measurement head 1920, which are fixed in relative position. Specifically, first depth imaging measurement head 1910 and second depth imaging measurement head 1920 may be mounted in the same housing, thereby ensuring that their relative positions are fixed.

[0138] Furthermore, the measuring head 1910 and the measuring head 1920 can perform sequential structured light projection and imaging under the control of the same control device (e.g., a processor, not shown in the figure). Here, sequentially means that the measuring head 1910 first performs a scan and imaging, and the measuring head 1920 performs a scan and imaging again, and each performs linear light projection and imaging in different time periods. Since the exposure period of each measuring head is shorter than the framing time of depth detection (for example, the exposure period of a frame of depth is 5ms, and the depth detection frame rate is 30fps, then the interval between each frame of depth is approximately 28ms), the exposure of other measuring heads can be performed during the exposure interval of the current measuring head, and the exposure imaging and depth calculation are performed in an interlaced manner. In other words, the depth data measurement device of the present invention may include the two measuring heads described above, or may include more measuring heads that perform exposure imaging and depth calculation in an interlaced manner, and the imaging information of these measuring heads can be spliced ​​in the same xyz axis space. Due to the use of inter-frame interleaving detection, the system frame rate detection frame rate is not affected, and the depth detection frame rate can still reach 30fps.

[0139] As shown, measurement head 1910 and measurement head 1920 can have the same configuration. For example, they can be two measurement heads with exactly the same configuration and configured to sequentially capture the same imaging area (the imaging object within the imaging area) from different angles. Measurement head 1910 and measurement head 1920 can be implemented as measurement heads including N groups of storage units as described above and capable of completing N images in a single scan.

[0140] Here, successive shooting means that after the measuring head 1910 completes the first pattern scan (the projection device 1911 scans and projects the linear light, and the image sensor 1912 images the projected linear light), the measuring head 1920 performs the second pattern scan (the projection device 1921 scans and projects the linear light, and the image sensor 1922 images the projected linear light), and the first group of N-step phase-shifted patterns obtained by the first pattern scan and the second group of N-step phase-shifted patterns obtained by the second pattern scan are synthesized into depth information of the imaging object in the imaging area based on the relative position.

[0141] Furthermore, the housing of depth data measurement device 1900 may also include a handle 1931, and may be implemented as a handheld modeling device. In this implementation, the depth data measurement device captures multiple sets of first N-step phase-shift patterns and multiple sets of second N-step phase-shift patterns in relative motion with respect to the imaging object, and may synthesize depth information generated from the multiple sets of first N-step phase-shift patterns and multiple sets of second N-step phase-shift patterns into model information of the imaging object based on calibration points.

[0142] FIG20 shows a schematic diagram of a depth data measurement device according to an embodiment of the present invention. As shown, the measurement device 2000 may include two measurement heads 2010 and 2020 as described above, and a processor 2030. The measurement head 2010 includes a projection device 2011 and an image sensor 2012. The measurement head 2020 includes a projection device 2021 and an image sensor 2022.

[0143] Processor 2030 is connected to two measurement heads, for example, projection device 2011 and image sensor 2012, and to projection device 2021 and image sensor 2022. Under the control of processor 2030, projection device 2011 can perform a single scan projection as described above, and the N groups of storage cells of image sensor 2012 can perform corresponding imaging, thereby obtaining a set of N-step phase-shifted images after the single scan projection. Subsequently, projection device 2021 can perform a single scan projection as described above, under the control of processor 2030, and the N groups of storage cells of image sensor 2022 can perform corresponding imaging, thereby obtaining a set of N-step phase-shifted images after the single scan projection. These two sets of N-step phase-shifted images can each be used to synthesize a depth map of the imaged object, and these two depth maps can be spliced ​​based on the positional relationship between the two measurement heads, thereby obtaining more comprehensive data about the imaged object.

[0144] 21 shows a schematic flow chart of a depth data measurement method according to an embodiment of the present invention. The method can be implemented by the depth data measurement device of the present invention.

[0145] In step S2110 , a first depth imaging measurement head of a depth data measurement device is used to perform a first pattern scan on an imaging area to obtain a first set of N-step phase shift patterns.

[0146] In step S2120 , a second depth imaging measurement head of the depth data measurement device is used to perform a second pattern scan on the imaging area to obtain a second set of N-step phase shift patterns.

[0147] In step S2130 , depth information of the imaging object in the imaging area is synthesized from the first set of N-step phase shift patterns and the second set of N-step phase shift patterns based on the relative positions of the first depth imaging measurement head and the second depth imaging measurement head.

[0148] The scanning imaging of the first depth imaging measurement head and the second depth imaging measurement head may each include: projecting a linear light moving along a first direction toward the imaging area, wherein the length direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light is within a scanning period. A pattern scan is completed within one scan period. There are multiple cyclic sub-periods in In each sub-cycle In the example, the linear light is projected with a period t p Perform light and dark changes, the projection period t p The duration of the exposure switching cycle t e The duration is the same, and the projection period t p The invention comprises N waveform projection areas with a width of 2π / N, and the projection light intensity of each waveform projection area is encoded, where N is an integer greater than 1; an image sensor comprising N groups of storage units evenly distributed on an imaging surface is used to photograph the imaging area to obtain N image frames under the linear light scanning projection, wherein each group of storage units has an exposure switching period t that is spaced 2π / N apart from each other. e Expose; obtain the depth data of the object under test in the imaging area based on the image frame. The projection light intensity of each waveform projection area is encoded so that in the scanning cycle When one pattern scan is completed, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

[0149] The above steps S1910 to S1930 may be repeated to perform dynamic imaging of, for example, a moving imaging object. In this case, the depth data measurement device of the present invention may maintain a constant position.

[0150] In another embodiment, the depth data measurement device of the present invention can actively change its position. As shown in FIG. 19 , the depth data measurement device of the present invention can be implemented as a handheld device including a handle 1931. In this implementation, the depth data measurement device captures multiple sets of first N-step phase-shift patterns and multiple sets of second N-step phase-shift patterns in relative motion with respect to the imaging object, and can synthesize depth information generated from the multiple sets of first N-step phase-shift patterns and multiple sets of second N-step phase-shift patterns into model information of the imaging object based on calibration points.

[0151] The depth data measurement head, measurement device, and measurement method according to the present invention have been described in detail above with reference to the accompanying drawings. The depth data measurement scheme of the present invention utilizes an image sensor equipped with different groups of pixels capable of phase-shifted exposure to image projected phase-shifted line light. This allows different groups of pixels on the image sensor to capture different phase-shifted fringe images during a single scan of the line light, thereby enabling the acquisition of multiple fringe images from a single line light scan. This significantly improves the speed of depth map synthesis and is suitable for capturing images of moving objects.

[0152] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A depth imaging measurement head, comprising: A projection device, used for projecting a linear light moving along a first direction to an imaging area, wherein a length direction of the linear light is a second direction perpendicular to the first direction; The image sensor comprises N groups of storage units, each group of storage units has an exposure switching cycle t that is spaced 2π / N apart from each other. e Exposure is performed, wherein N is an integer greater than 1, Wherein, the projection device has a scanning period A pattern scan is completed within one cycle. There are multiple cyclic sub-periods in In each sub-cycle In the embodiment, the linear light is projected with a period t p The projection period t p The duration of the exposure switch cycle t e The duration is the same, and the projection period t p It includes N waveform projection areas with a width of 2π / N, and the projection light intensity of each waveform projection area is encoded so that in the scanning cycle When a pattern scan is completed within one second, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

2. The depth imaging measurement head according to claim 1, wherein: The projection period t of the linear light p The exposure switching period t of the first group of storage cells e synchronous.

3. The depth imaging measurement head according to claim 1, wherein: N=2 n , n is an integer greater than or equal to 1.

4. The depth imaging measurement head according to claim 1, wherein: Each waveform projection area projects a rectangular wave with a width of 2π / N or 0, and corresponds to a cyclic sub-period based on a set of N-step phase shift patterns. The light intensity distribution is determined to determine the light intensity of each waveform projection area.

5. The depth imaging measurement head according to claim 4, wherein: The set of N-step phase shift patterns is a sine wave four-step phase shift pattern, and each projection period t is obtained based on the exposure of the N sets of storage units corresponding to the waveform projection area. p The light intensity value of each waveform projection area is not less than zero.

6. The depth imaging measurement head according to claim 1, wherein: Linear light scanning corresponds to a dwell time t on each column of pixels c Not less than the scanning period Divided by the number of columns C, the residence time t c is the projection period t p More than 10 times.

7. The depth imaging measurement head according to claim 5, wherein: In each subcycle T i In the embodiment, the linear light is projected with a period t p Project m times, and each subperiod T i The duration is greater than the residence time t c .

8. The depth imaging measurement head according to claim 1, wherein: Each pixel in the image sensor includes N storage units, and the N storage units of each pixel belong to one of the N groups of storage units.

9. The depth imaging measurement head according to claim 8, wherein: Each pixel includes N storage units, which are N charge storage units. The exposure positions of the N charge storage units corresponding to the same pixel are the same, but the exposure cycles are spaced 2π / N apart from each other. In the scanning cycle When one pattern scan is completed within a period of time, the set of N-step phase-shift patterns is obtained from the N sets of charge storage units, and the set of N-step phase-shift patterns is used to generate a depth map of the imaging area.

10. The depth imaging measurement head according to claim 1, wherein: The image sensor includes N groups of pixels evenly distributed on the imaging surface, each pixel includes a corresponding storage unit, and during the scanning cycle When one pattern scan is completed, the set of N-step phase-shift patterns is obtained from the N groups of storage units corresponding to the N groups of pixels, and the set of N-step phase-shift patterns is used to generate a depth map of the imaging area.

11. The measuring head according to claim 1, wherein: The projection device is configured to perform a first scanning cycle. A pattern scan is completed within a period of 1 second, so that each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of Gray code patterns; The projection device is configured to perform a plurality of operations on the second scanning period. A pattern scan is completed within 10 seconds, so that each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute the group of N-step phase shift patterns, Wherein, based on the set of Gray code patterns, a depth map of the imaging area is generated from the set of N-step phase shift patterns.

12. The measuring head according to claim 1, wherein: The projection device comprises: A light emitting device for generating linear light; and A reflecting device, used for reflecting linear light, projects linear light moving in a direction perpendicular to the stripe direction to the shooting area at a predetermined frequency, wherein the length direction of the linear light is the length direction of the projected stripe, and the reflecting device comprises one of the following: a mechanical galvanometer that reciprocates at the predetermined frequency; a micromirror device that reciprocates at a predetermined frequency; and A mechanical mirror that rotates in one direction at a predetermined frequency.

13. The measuring head according to claim 1, wherein: The image sensor includes a first image sensor and a second image sensor which are fixed in relative position, wherein the first image sensor and the second image sensor each include the N groups of storage units and are exposed synchronously with each other.

14. The measuring head according to claim 1, wherein: The projection device has a scanning cycle Complete α pattern scans within a period, each of the scanning cycles There are multiple cyclic sub-periods in In each cycle In the process, there are N sub-cycles T1-T N , in each subcycle T i In the embodiment, the linear light is projected with a period t p Perform light and dark variation projection, the projection period t p The duration of the exposure switch cycle t e The duration is the same, and the projection period t p Includes bright areas, where in subperiods T1-T N In the projection period t, the bright area p The position in the scanning cycle is changed at an interval of 2π / αN phase. When a pattern scan is completed within α scanning cycles, each of the N groups of storage units of the image sensor images a different stripe pattern, and When α times of pattern scanning is completed within αN, αN stripe patterns constitute a group of αN-step phase shift patterns with a 2π / αN phase shift between them, where α is an integer greater than or equal to 2.

15. The measuring head according to claim 1, wherein: In each sub-cycle In the process, there are N sub-cycles T1-T N , in each subcycle T i In the embodiment, the linear light is projected with a period t p Perform light and dark variation projection, the projection period t p The duration of the exposure switch cycle t e The duration is the same, and the projection period t p Includes bright areas, where in subperiods T1-T N In the projection period t, the bright area p The position in the scanning cycle is changed at intervals of 2π / N phases. When a pattern scan is completed within one second, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute an N-step phase shift pattern of a group of light and dark stripes with a 2π / N phase shift between each other.

16. A depth data measuring device, comprising: The depth data measuring head according to any one of claims 1 to 15, and A processor connected to the depth data measuring head is used to When one pattern scan is completed within a period of time, a depth map of the imaging area is obtained from the N fringe patterns obtained.

17. A depth data measurement device, comprising a first depth imaging measurement head and a second depth imaging measurement head fixed in relative position, wherein the first depth imaging measurement head and the second depth imaging measurement head are the depth data measurement heads according to any one of claims 1 to 15, in, After the first depth imaging measurement head completes the first pattern scan, the second depth imaging measurement head performs a second pattern scan, and a first group of N-step phase-shift patterns obtained by the first pattern scan and a second group of N-step phase-shift patterns obtained by the second pattern scan are synthesized into depth information of an imaging object in an imaging area based on the relative positions.

18. A depth data measurement method, comprising: A linear light moving along a first direction is projected toward the imaging area, wherein the length direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light is A pattern scan is completed within one cycle. There are multiple cyclic sub-periods in In each sub-cycle In the embodiment, the linear light is projected with a period t p The projection period t p The duration of the exposure switch cycle t e The duration is the same, and the projection period t p It includes N waveform projection areas with a width of 2π / N, and the projection light intensity of each waveform projection area is encoded, where N is an integer greater than 1; The imaging area is photographed using an image sensor comprising N groups of storage units to obtain N image frames under the linear light scanning projection, wherein each group of storage units has an exposure switching cycle t that is spaced 2π / N apart from each other. e Make an exposure; and Obtaining depth data of the object to be measured in the imaging area based on the image frame, The projection light intensity of each waveform projection area is encoded so that in the scanning cycle When a pattern scan is completed within one second, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

19. A depth data measurement method, comprising: Use a first depth imaging measurement head of a depth data measurement device to perform a first pattern scan on an imaging area to obtain a first set of N-step phase shift patterns; Use a second depth imaging measurement head of the depth data measurement device to perform a second pattern scan on the imaging area to obtain a second group of N-step phase shift patterns; synthesizing depth information of an imaging object in the imaging area from the first group of N-step phase shift patterns and the second group of N-step phase shift patterns based on the relative positions of the first depth imaging measurement head and the second depth imaging measurement head, Wherein, the scanning imaging of the first depth imaging measurement head and the second depth imaging measurement head each include: A linear light moving along a first direction is projected toward the imaging area, wherein the length direction of the linear light is a second direction perpendicular to the first direction, and the projected linear light is A pattern scan is completed within one cycle. There are multiple cyclic sub-periods in In each sub-cycle In the embodiment, the linear light is projected with a period t p The projection period t p The duration of the exposure switch cycle t e The duration is the same, and the projection period t p It includes N waveform projection areas with a width of 2π / N, and the projection light intensity of each waveform projection area is encoded, where N is an integer greater than 1; The image sensor including N groups of storage units photographs the imaging area to obtain N image frames under the linear light scanning projection, wherein each group of storage units has an exposure switching cycle t that is spaced 2π / N apart from each other. e Make an exposure; and Obtaining depth data of the object to be measured in the imaging area based on the image frame, The projection light intensity of each waveform projection area is encoded so that in the scanning cycle When a pattern scan is completed within one second, each of the N groups of storage units of the image sensor images a different stripe pattern, and the N stripe patterns constitute a group of N-step phase shift patterns with a 2π / N phase shift between each other.

20. The method of claim 19, further comprising: The depth data measuring device photographs a plurality of first groups of N-step phase shift patterns and a plurality of second groups of N-step phase shift patterns in relative motion with respect to the imaging object; as well as Depth information generated from a plurality of first N-step phase shift patterns and a plurality of second N-step phase shift patterns are synthesized into model information of the imaging object based on the calibration points.

Images