Depth data measuring head, computing device and corresponding method

Abstract

A depth data measuring head, a calculation device and a corresponding method are provided. The measuring head (200) includes a projection device (210) and an image sensor (220), the projection device (210) is used to scan and project a set of structured light with different patterns onto an imaging area, the image sensor (220) is used to capture an image of the imaging area and obtain a set of image frames used for one-time depth data calculation of the imaging area under irradiation of the set of structured light, and the image sensor (220) includes at least two sub-image sensors (223, 224) that share at least a part of an optical path, and the at least two sub-image sensors (223, 224) are respectively used for imaging the different patterns of structured light projected by the projection device (210) successively. By using a coaxial monocular or multiple sets of binocular structures for successive imaging, problems such as the time required to capture multiple frames is too long to perform dynamic imaging and the frame rate of depth data being too low can be reduced in a scene where depth data is obtained by integrating multiple frames.

Description

[Technical field]

[0001] The present disclosure relates to the field of three-dimensional imaging, and in particular to a depth data measuring head, a computing device and corresponding methods. [Background technology]

[0002] A depth camera is a collection device that collects depth information of a target object, and such cameras are widely applied in the fields of 3D scanning, 3D modeling, etc. For example, more and more smartphones are now equipped with depth imaging devices for face recognition. Although 3D imaging has long been a research topic in related fields, current depth cameras still have many challenges, such as the inability to balance measurement accuracy and imaging speed, imaging performance and cost optimization.

[0003] Therefore, there is a need for improved means of measuring depth data. Summary of the Invention [Problem to be solved by the invention]

[0004] The technical problem to be solved by the present disclosure is to provide a depth data measurement means, which can reduce the problems such as the time required to acquire multiple frames is too long to perform dynamic imaging and the frame rate of depth data is too low in a scene where multiple coaxial sub-image sensors are used to perform continuous imaging and depth data is obtained by integrating multiple frames, etc. In addition, since an imaging device with a general frame rate is introduced, the accuracy and frame rate of depth imaging can be improved while avoiding the use of a high-cost high-frame rate imaging device. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a depth data measurement head including a projection device and an image sensor, wherein the projection device is used for scanning and projecting structured light sets of different patterns onto a shooting area, the structured light sets including at least two different patterns of structured light, the image sensor is used for photographing the shooting area and acquiring a set of image frames used for a single depth data calculation of the shooting area under irradiation with the structured light set, the image sensor includes at least two sub-image sensors sharing at least a portion of an optical path, and the at least two sub-image sensors are each used for imaging against the different patterns of structured light projected in succession by the projection device.

[0006] The depth data measuring head may be realized in a monocular form having one image sensor. In a preferred binocular structure form, the image sensor may be a first image sensor and a second image sensor having a predetermined relative positional relationship, which are used to capture the image of the shooting area and obtain an image frame set pair used for one depth data calculation of the shooting area under the irradiation of the structured light set, and the first image sensor and the second image sensor each include at least two sub-image sensors that share at least a part of the optical path, and the at least two sub-image sensors are each used to image different patterns of structured light projected successively by the projection device.

[0007] According to a second aspect of the present disclosure, there is provided a depth data calculation device including a depth data measurement head as described in the first aspect and a processor connected to the depth data measurement head, wherein the processor is used to determine depth data of an object to be photographed in the shooting area based on the image frame set obtained by imaging the structured light.

[0008] According to a third aspect of the present disclosure, a method for measuring depth data is provided, the method including the steps of: scan-projecting structured light onto a shooting area; using a first sub-image sensor to shoot the shooting area and obtain a first image frame under irradiation with structured light; scan-projecting a second structured light of a different pattern onto the shooting area; and using a second sub-image sensor to shoot the shooting area and obtain a second image frame under irradiation with second structured light, wherein the first and second sub-image sensors share at least a part of the optical path and constitute an image sensor, and the first and second image frames are used to calculate one depth data of the shooting area.

[0009] In a two-lens form, the method includes the steps of scanning and projecting structured light onto a shooting area, using a first sub-image sensor pair having a predetermined relative positional relationship to shoot the shooting area and obtain a first image frame pair under irradiation with structured light, scanning and projecting a second structured light of a different pattern onto the shooting area, and using a second sub-image sensor pair to shoot the shooting area and obtain a second image frame pair under irradiation with second structured light, wherein one sub-image sensor of each of the first and second sub-image sensor pairs shares at least a portion of the optical path and constitutes a first image sensor, and the other sub-image sensor of each of the first and second sub-image sensor pairs shares at least a portion of the optical path and constitutes a second image sensor, and the first and second image frame pairs are used to calculate one depth data of the shooting area.

[0010] According to a fourth aspect of the present disclosure, there is provided a method for calculating depth data, comprising: acquiring first and second image frames, or first, second and third image frames, based on the depth data measurement method described in the third aspect; and determining depth data of a subject to be photographed in the photographing area based on the predetermined relative position and the first and second image frames, or the first, second and third image frames. Effect of the Invention

[0011] Therefore, the present invention discloses a method for sequentially imaging different patterns with multiple coaxial sub-image sensors and synthesizing depth data based on the images, which can reduce the imaging time and improve the frame rate when integrating multi-frames to calculate depth data by introducing a coaxial monocular or binocular light-sensing unit. [Brief description of the drawings]

[0012] The above and other objects, features, and advantages of the present disclosure will become more apparent by describing in more detail exemplary embodiments of the present disclosure with reference to the drawings, in which like reference numerals generally refer to like elements. [Figure 1A] This is the principle of depth imaging using structured light of a stripe code. [Figure 1B] This is the principle of depth imaging using structured light of a stripe code. [Diagram 2] 1 is a schematic diagram showing the configuration of a depth data measurement head according to an embodiment of the present invention. [Diagram 3] 1 is a schematic diagram showing the configuration of a depth data measurement head according to an embodiment of the present invention. [Figure 4] FIG. 13 is a comparative timing diagram of two sets of coaxial binocular imaging and one set of binocular imaging. [Diagram 5] FIG. 13 is a timing diagram of three sets of coaxial twin-lens imaging. [Figure 6] 2 is a schematic diagram of a depth data measuring head performing column synchronous imaging according to one embodiment of the present invention; FIG. [Figure 7A] 7 is an example of an enlargement operation of the projection device shown in FIG. 6. [Figure 7B] 7 is an example of an enlargement operation of the projection device shown in FIG. 6. [Figure 8] FIG. 1 is a simplified perspective view of a projection device used in the present invention; [Figure 9] FIG. 2 is a schematic diagram of alternate pixel columns in an image sensor being turned on; [Figure 10]2 is a schematic flowchart of a depth measurement method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The preferred embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be realized in various forms and should not be limited to the embodiments described herein. These embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0014] In order to improve the imaging accuracy and versatility of depth data, active projection of multiple patterns (e.g., stripe-coded structured light) and preferably a binocular imaging structure can be combined to realize highly flexible pixel-level depth data imaging based on the superimposable properties of stripe-coded patterns and their independence from a specific imaging plane in binocular imaging.

[0015] As can be seen from the measurement principle of structured light, whether the scan angle α can be accurately determined is the key point of the whole measurement system, and the structured light by points and lines can calculate and identify the scan angle by mechanical devices such as rotating mirrors, and the significance of encoding and decoding the image is to identify the scan angle of the encoded structured light (i.e., surface structured light) system. Figures 1A-B show the principle of depth imaging using structured light with stripe code. For ease of understanding, Figure 1A briefly describes the encoding principle of stripe structured light by black and white 3-bit binary time code. The projection device can sequentially project three patterns as shown in the figure onto the object to be measured in the shooting area, and the projection space is divided into eight areas using two gray scales, light and dark, in each of the three patterns. Each area corresponds to a respective projection angle, and it can be assumed that the bright area corresponds to the code "1" and the dark area corresponds to the code "0". The code values ​​in the three code patterns of a point on the scene in the projection space are combined according to the projection order to obtain the area code value of the point, thereby determining the area in which the point is located, and the scan angle of the point is decoded and obtained.

[0016] In a monocular imaging system, the calculation of depth data may be performed by comparing the captured image with the reference image. In a binocular imaging system, the above decoding process may be simplified by directly matching the code values ​​of each point in the first and second image sensors. In order to improve the matching accuracy, the number of projection patterns in the time code may be increased. FIG. 1B is another example of projecting stripe code structured light. Specifically, the figure shows a black and white 5-bit binary time code. This means that in the application scenario of binocular imaging, for example, each pixel in each of the left and right image frames contains five region code values ​​that are all 0 or 1, so that the left and right image matching can be realized with higher accuracy (for example, pixel level). It should be noted that in other embodiments, for example, a 5-bit quaternary time code or an octal time code can also be realized by using different brightness levels.

[0017] When the projection speed of the projection device does not change, the example of FIG. 1B corresponds to realizing a higher accuracy of image matching at a higher time domain cost compared to the three code patterns of FIG. 1A. Although the inherent projection speed of the projection device is extremely fast (for example, the vibrating mirror device preferably used in the present invention), the frame rate of the depth data by multi-frame synthesis is low due to the limited frame rate of the image sensor, and the imaging time required for each frame is too long, making it difficult to realize dynamic video shooting. For example, in a situation where a conventional image sensor with a frame rate of 60 frames / second is used, when two-dimensional images captured every 5 frames (in a binocular embodiment, actually 5 sets of 10 frames of images) are synthesized into one frame of depth image, the frame rate per second drops to 12 frames / second, and the imaging time per frame is long, about 83 ms. Such a low frame rate and long imaging time make it difficult to meet the needs of dynamic imaging. However, while a relatively high frame rate of depth data frames can be achieved by directly increasing the frame rate of the image sensor, for example, a depth data frame rate of 30 frames / sec can be achieved by using a high-speed image sensor with 150 frames / sec, the introduction of high-performance devices significantly increases the cost of the equipment.

[0018] In order to simultaneously meet the requirements of high accuracy, high frame rate and low cost in the three-dimensional imaging field, the inventors of the present invention, considering the current technical situation that the current projection device can project at high speed, and the normal image sensor has a high imaging speed but the frame rate is reduced due to data transmission and processing, etc., put forward a means for imaging different patterns using multiple sets of coaxial sensor structures, thereby improving the overall frame rate of obtaining depth data based on multi-frames, and solving the problems of the prior art that the time required to obtain multi-frames is too long to perform dynamic imaging, and the frame rate of depth data is too low, etc. In addition, since the imaging device introduced is a general frame rate imaging device, the accuracy and frame rate of depth imaging can be improved while avoiding the use of high-cost high frame rate imaging devices.

[0019] Specifically, the present invention provides a depth data measurement head including a projection device and an image sensor, the projection device is used to scan and project a structured light set of different patterns onto an imaging area, the structured light set includes at least two different patterns of structured light, and the image sensor is used to image the imaging area and obtain a set of image frames used for one depth data calculation of the imaging area under the irradiation of the structured light set. In other words, the image sensor needs to image each pattern in the structured light set, and the images of the different patterns are used together to synthesize one piece of depth data. The image sensor of the present disclosure includes at least two sub-image sensors that share at least a part of an optical path, and the at least two sub-image sensors are used to image the structured light of different patterns projected successively by the projection device.

[0020] The sub-image sensors included in the image sensor, which share at least a part of the optical path, can be used to alternately image the successively projected patterns in the structured light set projected by the projection device until imaging of all patterns in the structured light set is completed. For example, the projection device projects one set of three structured lights as shown in FIG. 1, and the image sensor includes two sub-image sensors. At this time, the first sub-image sensor images the first pattern, the second sub-image sensor images the second pattern, and the first sub-image sensor further images the third pattern. In other words, when the number of patterns included in the structured light set is equal to or less than the number of sub-image sensors included in the image sensor, different patterns are respectively imaged by different image sensors included in the image sensor. When the number of patterns included in the structured light set is greater than the number of sub-image sensors included in the image sensor, imaging needs to be performed twice due to the number of sub-image sensors included in the image sensor, that is, the same sub-image sensor images different patterns at different times.

[0021] In some embodiments, the projected pattern may be a discrete spot pattern, for example the same diffraction pattern may be projected at different angles and imaged at different angles resulting in different patterns.

[0022] Alternatively, in a preferred embodiment of the present invention, the projected pattern may be a stripe light pattern as shown in Figures 1A-B. In contrast to a spatially modulated discrete spot pattern, imaging using a stripe light pattern can be considered as a temporal modulation means, i.e., synthesizing a single depth data map using different stripes projected at different times, and has a sufficiently high QE (quantum efficiency) to fill the charge of exposed pixels of an image sensor on a microsecond scale. This allows a depth measurement device with high interference resistance to be realized, especially in combination with a rolling shutter type image sensor.

[0023] In some embodiments, the image sensor may be implemented as one image sensor, i.e., a monocular system. In other embodiments, the image sensor may be implemented as two image sensors, i.e., a binocular system. In the following, monocular and binocular embodiments according to the principles of the present invention are described with reference to Figures 2 and 3, respectively.

[0024] FIG. 2 is a schematic diagram of a depth data measuring head according to an embodiment of the present invention. As shown in the figure, a depth data measuring head 200 based on a monocular principle includes a projection device 210 and an image sensor 220. Although not shown for convenience, the measuring head 200 may further include a housing surrounding the above devices, and the coupling structure 240 shown in FIG. 2 may be regarded as a mechanism for fixing and coupling the above devices to the housing. In some embodiments, the coupling structure 240 may be a circuit board including a control circuit. It should be noted that in other embodiments, the above devices 210 and 220 may be coupled to the housing in other ways, and perform corresponding data transmission and command receiving operations.

[0025] Here, the projection device 210 is used to scan and project a structured light set of different patterns onto an imaging area, and the structured light set includes at least two different patterns of structured light. The image sensor 220 is used to image the imaging area and obtain a set of image frames, which are used for one-time depth data calculation of the imaging area under the illumination of the structured light set.

[0026] For example, the projection device 210 sequentially projects three patterns as shown in Fig. 1. The three patterns are taken as a set, and an image sensor images each of the three patterns, thereby obtaining an image frame set including three frames. Based on the monocular imaging principle, the three frame images can be compared with the corresponding reference image frames, respectively, and used together to calculate one depth data of the shooting area, that is, to calculate one frame depth image.

[0027] Unlike an image sensor in a conventional measurement head that includes only one photosensitive unit and each photosensitive unit performs imaging three times to obtain a set of three image frames, in the present invention, the image sensor 220 includes at least two sub-image sensors that share at least a portion of the optical path, and the at least two sub-image sensors are each used to perform imaging on different patterns of structured light successively projected by the projection device.

[0028] FIG. 2 is an example in which the image sensor 220 includes two sub-image sensors (photosensitive units). As shown in the figure, the image sensor 220 includes sub-image sensors 223 and 224. The sub-image sensors 223 and 224 share an optical path to the beam splitter surface of the beam splitter 222, and are at equal or at least approximately equal distances from the area of ​​the beam splitter. In other words, the present invention introduces a sensor structure that is coaxial with each other. Here, the sub-image sensor 223 can be used to perform imaging for the structured light of, for example, the first pattern among the three patterns in FIG. 1. Then, the sub-image sensor 224 can be used to perform imaging for the structured light of, for example, the second pattern among the three patterns in FIG. 1. In other words, the sub-image sensor 224, which has the same optical path length (or a completely equivalent optical path) at this time, can be considered to perform imaging for the next pattern structured light instead of the image sensor 223 in situ. This means that the imaging interval between two adjacent frames does not depend on the frame interval of each image sensor, and imaging can be performed at a smaller interval.

[0029] Similarly, Figure 3 is a schematic diagram of a depth data measurement head according to an embodiment of the invention, in which an example image sensor configuration is shown in more detail, as compared to the schematic projection device.

[0030] As shown in FIG. 3, the depth data measuring head 300 based on the binocular principle includes a projection device 310, and a first image sensor 320 and a second image sensor 330 having a predetermined relative positional relationship. Although not shown for convenience, the measuring head 300 may further include a housing surrounding the above devices, and the coupling structure 340 shown in FIG. 3 may be regarded as a mechanism for fixing and coupling the above devices to the housing. In some embodiments, the coupling structure 340 may be a circuit board including a control circuit. It is to be noted that in other embodiments, the above devices 310-330 may be coupled to the housing in other ways, and perform corresponding data transmission and command receiving operations.

[0031] Here, the projection device 310 is used to scan and project different patterns of structured light sets onto an imaging area, and the structured light sets include at least two different patterns of structured light. The first image sensor 320 and the second image sensor 330 having a predetermined relative positional relationship are used to image the imaging area and obtain an image frame set pair, which is used for one-time depth data calculation of the imaging area under the irradiation of the structured light set.

[0032] For example, the projection device 310 sequentially projects three patterns as shown in Fig. 1. These three patterns are taken as one set, and an image sensor performs imaging on each of the three patterns to obtain an image frame set pair including three pairs (six frames). These six frames of images are used together to calculate one depth data of the shooting area, that is, one frame of depth image can be calculated.

[0033] Unlike the conventional twin-lens measurement head in which the first image sensor 320 and the second image sensor 330 each include only one photosensitive unit, and each photosensitive unit performs imaging three times to obtain three pairs (six frames) of image frame set pairs, in the present invention, each of the first and second image sensors includes at least two sub-image sensors that share at least a portion of the optical path, and the at least two sub-image sensors are each used to perform imaging on different patterns of structured light successively projected by the projection device.

[0034] FIG. 3 is an example in which the first and second image sensors each include two sub-image sensors (photosensitive units). As shown in the figure, the first image sensor 320 includes sub-image sensors 323 and 324, and the second image sensor 330 includes sub-image sensors 333 and 334. The sub-image sensors 323 and 324 share an optical path to the beam splitter surface of the beam splitter 322 and are at equal distances from the area of ​​the beam splitter. Similarly, the sub-image sensors 333 and 334 share an optical path to the beam splitter surface of the beam splitter 332 and are at equal distances from the area of ​​the beam splitter. In other words, the present invention introduces a binocular structure with multiple sets of coaxial lenses. Here, the sub-image sensors 323 and 333 belonging to different image sensors can be regarded as a first image sensor set (first binocular set) for imaging the structured light of the first pattern of the three patterns in FIG. 1, for example. Then, the sub-image sensors 324 and 334, which can be considered as a second image sensor set (second binocular set), are used to perform imaging for the structured light of, for example, the second pattern of the three patterns in Fig. 1. In other words, the sub-image sensors 324 and 334, which are coaxial with 323 and 333 at this time, respectively, can be considered to perform imaging for the next pattern structured light on the spot instead of the sub-image sensors 323 and 333. This makes it possible to perform imaging at a smaller interval, without the imaging interval between two adjacent frames depending on the frame interval of each image sensor.

[0035] For this purpose, the measurement head 200 or 300 may further include a synchronization device, which is used to make at least two sub-image sensors included in the image sensor 220 or the first image sensor 320 and the second image sensor 330, respectively, sequentially and synchronously image the at least two different patterns of structured light at the first interval, simultaneously with the projection device projecting the at least two different patterns of structured light at the first interval, which is shorter than the frame imaging interval of the sub-image sensors. Correspondingly, each sub-image sensor images its own next frame at a second interval equal to or greater than the frame imaging interval of the sub-image sensors (e.g., images at its own frame interval), and the imaging operation can be synchronized with the projection by the projection device under synchronization by the synchronization device.

[0036] Fig. 4 is a timing diagram comparing two sets of coaxial imaging and one set of imaging. For convenience of explanation, the frame rate of each photosensitive unit (sub-image sensor) is set to 100 frames / s, the frame interval is set to 10 ms, and the exposure time required for each photosensitive unit is set to 1 ms.

[0037] If the image sensor 220 or the first image sensor 320 and the second image sensor 330 are conventional image sensors including only one photosensitive unit, and the depth data calculation is performed using the three patterns shown in Fig. 1, three imaging operations are required at 0, 10, and 20 ms, as shown in the lower part of Fig. 4. Therefore, to synthesize each depth data image, the subject needs to remain still for 21 ms (so it is more difficult to photograph a moving subject), and the frame rate is also reduced from 100 frames / s to 33.3 frames / s.

[0038] In contrast, in the image sensor of the present invention, in which the image sensor 220 or the first image sensor 320 and the second image sensor 330 each include two photosensitive units (for example, the image sensor 220 includes the sub-image sensors 223 and 224, and the first image sensor 320 and the second image sensor 330 each include the sub-image sensors 323 and 324, and the sub-image sensors 333 and 334, respectively), calculates depth data using the three patterns shown in FIG. 1, as shown in the upper part of FIG. 4, the first photosensitive unit set performs imaging for pattern 1 at 0 ms, followed by the second photosensitive unit set performing imaging for pattern 2 at 1 ms, and then 10 ms later, the first photosensitive unit set performs imaging for pattern 3 at 10 ms, thereby completing three imagings required for one depth data image. After that, the second photosensitive unit set can start the next imaging for pattern 1 at 11 ms. At 20 ms, the first photosensitive unit set performs imaging for pattern 2. At 21 ms, the second photosensitive unit set further performs imaging for pattern 3. In this way, the interval between imaging by different photosensitive unit sets only needs to be the time required for imaging (e.g., 1 ms), but the interval until the same photosensitive unit set next images is still based on the frame interval time (e.g., 10 ms) corresponding to the frame rate. At this time, by introducing two sets of coaxial twin lenses, the target only needs to remain still for 11 ms to synthesize each depth data image (so it is easier to photograph a moving target), and the frame rate can be maintained at about 66.6 frames / s.

[0039] Although an example having two sets of coaxial (same optical axis) photosensitive units has been described with reference to Figs. 2 to 4, in other embodiments, each of the first and second image sensors may include more photosensitive units. Fig. 5 is a timing diagram of three sets of coaxial twin-lens imaging. In this case, each of the first and second image sensors may include three coaxial photosensitive units (sub-image sensors). As a result, as shown in Fig. 5, the first set of photosensitive units performs imaging for pattern 1 at 0 ms, immediately thereafter the second set of photosensitive units performs imaging for pattern 2 at 1 ms, and immediately thereafter the third set of photosensitive units performs imaging for pattern 3 at 2 ms. After that, imaging of the next three sets is started at 10 ms, imaging of the next three sets is started at 20 ms, and so on. At this time, by introducing three sets of coaxial twin lenses, the three pairs (six frames) of images required to synthesize one depth data image can be acquired in just 3 ms, and the subject only needs to remain still for 3 ms, greatly improving the shooting level for moving subjects, and the frame rate can be maintained at approximately 100 frames / s (in this example, it takes 1003 ms, or 1.003 seconds, to shoot 100 frames).

[0040] Therefore, it can be understood that by simply introducing one additional set of coaxial twin-lens structures (or monocular structures), the frame rate of depth data based on multi-frame synthesis can be doubled, and the imaging time of each frame can be reduced. In theory, it is possible to arrange the same number of coaxial twin-lens structures as the number of images projected by the projection device, so that the framing time of each depth frame and the frame interval of the sensor are related only to a multiple of the exposure time (if the frame interval is greater than the exposure time x the number of sets of coaxial structures). For example, when synthesizing depth frames based on four patterns, if two sets of coaxial twin-lens are used as shown in FIG. 3, the imaging time to acquire four frames is slightly extended to 12 ms, but the frame rate drops to about 50 frames / s. However, if four sets of coaxial twin-lens are used, the imaging time to acquire four frames is only 4 ms, and the frame rate is still kept at about 100 frames / s. However, the introduction of an excessive number of coaxial structures increases the difficulty of the structure of the image sensor, so it is necessary to consider a compromise in cost, feasibility, and imaging speed.

[0041] In order to realize a coaxial arrangement of different light-sensitive units in the same image sensor, it is necessary to design the optical path. In the example of Fig. 3, a coaxial arrangement realized based on beam splitting is shown (the example of Fig. 2 has a similar structure). In this case, taking the first image sensor 320 as an example, it can include a lens unit 321 for receiving the incident return structured light, a beam splitter 322 for splitting the incident return structured light into at least a first beam and a second beam, a first sub-image sensor 323 for imaging the first beam, and a second sub-image sensor 324 for imaging the second beam of return structured light corresponding to a different pattern.

[0042] In one embodiment, the beam splitter 322 is an optical prism, such as a cube prism or a triangular prism, such that the reflected infrared light of the incident light reaches the second sub-image sensor 324, and the unreflected visible light of the incident light travels straight to the first sub-image sensor 323.

[0043] As shown, the prism-form beam splitter 322 can split the incoming light into two beams with mutually orthogonal propagation directions, and correspondingly, the first sub-image sensor 323 and the second sub-image sensor 324 can also be arranged perpendicularly, so that the incoming visible and infrared light beams are received at perpendicular angles, respectively.

[0044] In order to eliminate parallax and realize pixel-level alignment, it is necessary that the components of the incident light have the same optical path length. Therefore, when a cube prism is used as the beam splitter 322, the first sub-image sensor 323 and the second sub-image sensor 324 can be disposed at equal distances from the beam splitting region of the beam splitter 322. When a triangular prism is used as the beam splitter 322, the distances between the two photosensitive units and the beam splitter 322, especially the beam splitting region, can be flexibly adjusted based on the ratio of the refractive index between air and the prism material.

[0045] The pixel-level alignment (or near-pixel-level alignment) between the first sub-image sensor 323 and the second sub-image sensor 324 is theoretically realized by making the incident light share most of the optical path and have the same optical path length. However, during the actual manufacturing process of the image sensor, the actual arrangement of the first sub-image sensor 323 and the second sub-image sensor 324 is not ideally perpendicular and equidistant, so that deviations occur between the two imaging. In this case, a forced software correction can be performed on the manufactured image sensor. For example, a calibration target is introduced and both the imaging of the first sub-image sensor 323 and the second sub-image sensor 324 are aligned with the calibration target to achieve reliable pixel-level correction.

[0046] As shown in the figure, the image sensor 320 of the present invention may be realized by a single module. Therefore, the image sensor 320 may further include a housing used to fix the relative positions of the lens unit, the beam splitter, and the two light-sensitive units. Preferably, the housing can be combined with the lens unit 321 to form a sealed body, thereby preventing the contained device from being contaminated by the external environment. In other embodiments, the image sensor 320 of the present invention may be a part of a larger module (e.g., a depth data measuring head), and the fixing between each element is realized by the housing of the larger module.

[0047] Preferably, the image sensor 320 may further include a cable connected to the first sub-image sensor 323 and the second sub-image sensor 324, respectively. The housing has an opening for inserting the cable. In one embodiment, the cable may be a flexible cable, such as a FPC (flexible printed circuit) cable.

[0048] In one embodiment, before the beam is incident on the first sub-image sensor 323 and the second sub-image sensor 324, it may be passed through a filter to further filter out the influence of light of other wavelengths. In one embodiment, the projection device may project infrared laser light, and therefore the filter disposed in the image sensor may be a corresponding infrared light transmission unit for transmitting infrared light of a specific frequency range, for example, infrared light with a wavelength of 780 to 1100 nm in the present invention. In another embodiment, the projection device may project visible light, for example, red laser light or blue laser light, such as 635 nm red light or 450 nm blue light. For example, compared with 830 nm infrared light, which has a QE of only 20%, 635 nm red light has a high QE of 90% to 95%. Although the ambient light may also contain red light or blue light, imaging with a high signal-to-noise ratio can also be performed with the help of a filter that transmits the corresponding red light or blue light, due to the short exposure time and strong instantaneous light intensity. If the projection device projects visible light, for example red light, the first and second sub-image sensors may be realized as corresponding visible light sensors.

[0049] Preferably, when the beam splitter is a cube prism, one side of the filter is in direct physical contact with the cube prism and the other side is in physical contact with the photosensitive unit, allowing the photosensitive unit and the cube prism to be locked within the housing, thereby ensuring a high degree of consistency of the relative positions of each device.

[0050] In some embodiments, particularly when the first and second sub-image sensors are infrared light sensors for receiving the projected infrared light pattern, an additional visible light sensing unit (not shown) may be disposed in the image sensor to capture image information of the measurement target object, so that the image captured by the image sensor includes not only image information of the measurement target object but also depth information. The visible light sensing unit may be a grayscale sensor or a color sensor. The grayscale sensor captures only luminance information, and the color sensor is used to capture color information of the measurement target object, in which case the visible light sensing unit may be composed of a three-primary color sensing unit, and the three primary colors may be red, green, blue (RGB) or cyan, magenta, yellow (CMY).

[0051] 3, the structure of the first image sensor 320 has been specifically described, but the second image sensor 330 may have the same structure. Also, 323 and 333 may be regarded as a first twin-lens set, and 324 and 334 may be regarded as a second twin-lens set, but 323 and 334 may also be regarded as a first twin-lens set, and 324 and 333 may also be regarded as a second twin-lens set, and in any case, it is sufficient that the sensor is turned on and imaged after the corresponding pattern is incident.

[0052] When beam splitting is used to share the optical path as shown in FIG. 3, the amount of light acquired by each photosensitive unit is reduced, so the imaging sensitivity or effective distance range can be ensured by increasing the projection brightness or enlarging the entrance aperture.

[0053] Therefore, alternatively, light path sharing can be realized based on light path conversion. In this case, the image sensor 220 or the first image sensor 320 and the second image sensor 330 can each include a lens unit for receiving the incident return structured light, a light path conversion device for sending the incident return structured light to at least a first sub-path and a second sub-path, a first sub-image sensor for imaging the return structured light on the first sub-path, and a second sub-image sensor for imaging the return structured light corresponding to a different pattern on the second sub-path. In one embodiment, the light path conversion device can be a rotating mirror, which can, for example, reflect the incident light to the light sensing unit 323 at 0 ms and reflect the incident light to the light sensing unit 324 at 1 ms. In other embodiments, the light path conversion device can be a device for performing light path conversion based on other mechanical, chemical, or electrical principles.

[0054] As mentioned above, the different patterns of structured light sets projected by the projection device 310 may be structured light sets with different coding stripes, such as the patterns of Figure 1 or Figure 2. In some cases, the projection device 310 may be a projection device that projects one complete pattern each time, and in other cases, the projection device 310 may be a device that completes a pattern by a linear light-dark scan or a two-dimensional scan of a spot.

[0055] For this purpose, the projection device 310 may include a laser light-emitting element used to generate linear and / or spot laser light, and the laser light-emitting element performs high-speed switching to scan-project the structured light of alternating light and dark corresponding to the stripe code. More specifically, the projection device includes a light-emitting device and a reflecting device, the light-emitting device is used to generate linear light, the laser light-emitting element may be included in the light-emitting device, and the reflecting device is used to reflect the linear light and project the linear light moving in a direction perpendicular to the stripe direction onto the shooting area. In the module 310 of FIG. 3, the light-emitting device is shown as a rectangle, and the reflecting device is shown as a diagonal line. The reflecting device may include one of a mechanical vibration mirror that reciprocates at a predetermined frequency and is used to scan-project the linear light onto the shooting area at the predetermined frequency, the length direction of the linear light being the length direction of the projection stripe, and a micromirror element that reciprocates at a predetermined frequency and is used to scan-project the linear light onto the shooting area at the predetermined frequency, the length direction of the linear light being the length direction of the projection stripe.

[0056] When the projection device performs scanning projection, the image sensor may be a global shutter image sensor (i.e., all pixels perform imaging simultaneously), but is preferably realized by a rolling shutter image sensor. For this reason, the measurement head 300 may further include a column synchronizer, which is used to synchronize and perform imaging between the sub-image sensor used for the current imaging and the corresponding pixel column in the stripe direction at the current scanning position based on the scanning position of the projection device.

[0057] In a preferred embodiment, a rolling shutter image sensor can generally only perform column exposure in one direction, for example, exposure from the 1st row to the 1000th row, and not vice versa. For example, if two sub-image sensors need to perform imaging consecutively, and at this time, assuming that the projection device performs linear light projection from left to right within a time of 1 ms, for example, when the fourth pattern of FIG. 1B is projected, the first sub-image sensor also performs column exposure from left to right correspondingly. If the mounting manner of the two sub-image sensors is the same, the projection device needs to return to the projection start position on the left side after completing the projection of the fourth pattern, and perform the projection of the fifth image of FIG. 1B correspondingly, the second sub-image sensor also performs column exposure from left to right correspondingly. Therefore, the two sub-image sensors mounted in one image sensor of the present invention may be rolling shutter image sensors mounted upside down, thereby realizing column exposure in opposite directions, and the vibrating mirror does not need to return to a specific fixed position to start projecting a new pattern, and both sides of the reciprocating movement can start projecting a new pattern.

[0058] For example, one stripe pattern can be scanned and projected by moving the vibrating mirror from left to right, and the first sub-image sensor can perform row exposure from left to right accordingly. After the projection of one stripe pattern is completed, the vibrating mirror does not need to return to the left side, and can directly scan and project the second stripe pattern from right to left, and at this time, since the second sub-image sensor is arranged upside down with respect to the first sub-image sensor, the second sub-image sensor can perform row exposure from right to left. It should be understood that the projection device of the measurement head of the present invention may have a structure other than that shown in FIG. 3, such as a diffraction grating on which a fixed pattern is printed.

[0059] Also, when imaging with a rolling shutter image sensor, it is possible to introduce a multi-mode imaging mechanism, for example, imaging a wider luminance range with lower luminance accuracy for high exposure rows, and imaging a narrower luminance range with higher luminance accuracy for low exposure rows.

[0060] 6 is a schematic diagram of a depth data measuring head performing column synchronous imaging according to an embodiment of the present invention. As shown in FIG. 6, the depth data measuring head 600 includes a projection device 610 and two image sensors 620 and 630.

[0061] The projection device 610 is used to scan-project structured light having a stripe code onto the photographing area. For example, within a projection period of three consecutive image frames, the projection device 610 can continuously project three patterns as shown in FIG. 1, and the imaging results of the three patterns are used to generate depth data. The first and second image sensors 620 and 630, which can be referred to as the first and second image sensors, respectively, have a predetermined relative positional relationship and are used to photograph the photographing area to obtain first and second two-dimensional image frames under the irradiation of the structured light, respectively. For example, as shown in FIG. 1, when the projection device 610 projects three patterns, the first image sensor 620 and the second image sensor 630 can respectively image the photographing area onto which the three patterns are projected (for example, the imaging plane in FIG. 3 and the area within a certain range before and after it) within three synchronized image frame imaging periods. A different set of photosensitive units is used for each of the three imaging times.

[0062] As shown in FIG. 6, the projection device 610 can project a linear light extending in the x direction in the z direction (i.e., toward the imaging area). In different embodiments, the projection of the linear light can be a shaped one (i.e., the emitted light itself is already a linear light) or a spot moving in the x direction (i.e., a linear light obtained by scanning). The projected linear light can be moved continuously in the y direction to cover the entire imaging area. The lower part of FIG. 6 is a more easily understood illustration of the scanning of the linear light, corresponding to a perspective view of the imaging area.

[0063] In the embodiment of the present invention, the direction in which the light beam is emitted from the measurement head is set as the z direction, the vertical direction of the imaging plane is set as the x direction, and the horizontal direction is set as the y direction. Therefore, the stripe-structured light projected by the projection device may be the result of linear light extending in the x direction moving in the y direction. In other embodiments, synchronization and imaging processing can be performed on the stripe-structured light obtained by moving linear light extending in the horizontal y direction in the x direction, but in the embodiment of the present invention, a vertical stripe light is preferably used for the description.

[0064] Further, the measurement head 600 includes a column synchronizer 650. The column synchronizer 650 is connected to the projection device 610, the first image sensor 620, and the second image sensor 630, respectively, to realize accurate synchronization among the three. Specifically, the column synchronizer 650 can synchronously turn on pixel columns in the stripe direction corresponding to the current scan position among the corresponding sets of photosensitive units of the first image sensor 620 and the second image sensor 630 according to the scan position of the projection device 610 to perform imaging. As shown in FIG. 3, the current stripe scans up to the central region of the imaging region. Therefore, in the corresponding sets of photosensitive units of the image sensors 620 and 630, pixel columns (e.g., three adjacent pixel columns) located in the central region are turned on to perform imaging. As the stripe moves in the y direction (indicated by the arrow in the lower perspective view of FIG. 3), the pixel columns turned on for imaging in the corresponding set of light-sensitive units of the image sensors 620 and 630 also move synchronously accordingly (indicated by the arrow above the matrix in the upper left block diagram of FIG. 3). This allows the one-dimensional characteristic of the stripe image to be used to control the range of pixel columns to be imaged at each time, reducing the adverse effect of ambient light on the measurement results. To further reduce the effect of ambient light, the projection device is particularly suitable for projecting light that is difficult to confuse with ambient light, such as infrared light. In addition, since the correspondence between the pixel columns and the scanning light is affected by multiple factors such as the width, power, and speed of the projection light, and the light-sensitive efficiency of the image sensor, the range (and the corresponding number) of pixel columns to be synchronously turned on each time can be determined, for example, based on a calibration operation.

[0065] 7A-B are examples of the enlargement operation of the projection device shown in FIG. 6. Specifically, as shown in FIG. 6, in the projection device 610, a laser light emitted by a laser light emitting element (for example, a laser light emitting element 711 shown in detail in FIG. 7A-B) is scan-projected by a projection mechanism (for example, a projection mechanism 712 shown in detail in FIG. 7A-B) into a shooting area (a gray area in FIG. 6), and is used to actively project structured light onto a measurement target object (for example, a person in FIG. 6) in the shooting area. A pair of image sensors 620 and 630 perform imaging on the shooting area, thereby acquiring image frames required for calculating depth data. As shown in FIG. 3, a dashed line coming from the projection device 610 is used to indicate its projection range, and dashed lines coming from the image sensors 620 and 630 are used to indicate their respective imaging ranges. The shooting area is generally located in an overlapping area of ​​the projection and imaging ranges of each of these three.

[0066] In practical application, a laser light emitting element is used to generate linear and / or infrared laser light, and the laser light emitting element performs high-speed switching to scan and project a structured light of alternating bright and dark corresponding to a stripe code. The high-speed switching may include high-speed switching of the laser light emitting element and high-speed coding switching.

[0067] In one embodiment, the laser emitter continuously emits laser light of the same intensity, and the projected stripe pattern may be realized by turning the laser emitter on and off. In this case, since the laser emitter only projects light of one intensity, each pixel of the image sensor only needs to record the "presence or absence" of light, and the deployed image sensor may be a black and white image sensor.

[0068] In another embodiment, the laser light emitting element itself may emit a laser light with a variable light intensity, for example, a laser light whose emitted light intensity exhibits a sine transformation according to the applied power. The sine transformed laser light may be combined with stripe projection, thus scanning and projecting a pattern with alternating light and dark and different brightness between the light stripes. In this case, the image sensor must be capable of distinguishing and imaging different light intensities, and therefore may be a multi-stage grayscale image sensor. Obviously, grayscale projection and imaging can provide more accurate pixel-to-pixel matching compared to black and white projection and imaging, improving the accuracy of depth data measurement.

[0069] In one embodiment, the laser light emitting element 711 may be a linear laser light emitting element that generates a linear light extending in the x direction (the direction perpendicular to the paper surface in Figs. 7A-B). The linear light is then projected onto the imaging plane by a reflecting mechanism 712 that can be swung along the axis in the x direction. The swing diagram of the reflecting mechanism 712 is as shown in Fig. 7B. This allows a linear light scan to be performed back and forth within the range AB of the imaging plane.

[0070] In one embodiment, the reflection mechanism 712 may be a micromirror device (also called a digital micromirror device, DMD), and may be realized as a MEMS (microelectromechanical system). FIG. 8 is a simplified perspective principle diagram of a projection device used in the present invention. As shown in FIG. 8, the spot laser light generated by the laser element can obtain linear light through a lens (corresponding to the linear laser light emitting element 711 in FIG. 7), and the linear light is further reflected through a MEMS-type micromirror device, and the reflected linear light is further projected into the external space through an optical window. The micromirror device has very high performance, for example, a commercially available MEMS can perform extremely stable reciprocating vibration at a frequency of 2k, thereby laying the foundation for high-performance depth imaging.

[0071] In another embodiment, the laser light to be scanned may be a spot laser light, so the projection mechanism needs to convert the projection direction into two-dimensional directions (two directions of x and y in the figure) accordingly. For example, the projection mechanism first scans the stripe light in the x direction, then displaces it in the y direction, and then scans it in the x direction at different y positions.

[0072] Whether a stripe light moving in the y direction is directly projected, or a spot light that moves in the x direction to form a stripe and needs to be displaced in the y direction is projected, what appears in the shooting area is a stripe that moves in the y direction over time. As the spot moves in the y direction, a certain pixel column among all the pixels for recording the image frame on the image sensor is turned on, so that the light reflected back at the corresponding position can be collected. FIG. 9 is a schematic diagram of pixel columns in the image sensor being turned on alternately. As shown in FIG. 9, when the stripe projected by the projector moves from the center of the imaging area to one side, the pixel columns turned on for imaging in the pixel array of the image sensor also move from the center to one side accordingly. This allows the pixel columns to perform imaging recording only during the time when the corresponding shooting area is scanned, and not to perform recording at other times. Since the intensity of the projected laser light needs to be higher than the intensity of the ambient light, in a situation where the ambient light is not accumulated under the synchronous on method of the present invention, very accurate imaging can be performed on the structured light itself. Conventional image sensors generally perform row exposure, so the image sensor used in the present invention that exposes each column (or multiple columns simultaneously) can be obtained by tilting the existing image sensor by 90 degrees. After tilting, it is necessary to add control for simultaneous exposure of all columns.

[0073] It should be noted that the pixel arrays shown in Fig. 6 and Fig. 9 are merely examples for illustrating the synchronization principle of the present invention. In practical applications, the pixel array of the image sensor often has a higher grade (e.g., 1000x1000), and the pixel rows that are simultaneously turned on each time may also have different ranges based on calibration (e.g., 3 rows are turned on each time, or different numbers of rows are turned on for different positions in the shooting area, etc.). Furthermore, the turning on of the pixel rows in the image sensor may only be related to the scanning position of the projection structure in the projection device, and may be unrelated to whether the stripe light is actually projected. In other words, the turning off and on of the laser light emitting element that is performed based on the distribution of the bright and dark stripes of the projected structured light does not affect the scanning projection operation of the projection structure, nor does it affect the turning on operation of the pixel rows of the image sensor that is synchronized with the scanning projection operation.

[0074] The projection device as described above may include a vibrating mirror, such as a MEMS mirror or a mechanical vibrating mirror, which is used to scan the shooting area at a predetermined frequency and project a linear laser light. The vibrating mirror can achieve a very high vibration frequency, for example, 2k per second, which is equivalent to scanning one complete projected structured light in 250 μs, so it is necessary to perform very accurate synchronization with the reflected light position. Due to the above accuracy, it is impossible to directly use the activation signal of the MEMS mirror to perform synchronization (because the delay is not reliable), so considering the characteristics of the phase vibration of the micromirror element, the synchronization device may include a measuring device for measuring the vibration phase of the vibrating mirror in real time, and the pixel row imaging is synchronized on based on the measurement result of the measuring device. Therefore, the synchronization of scanning and imaging at a very high frequency is guaranteed. In the embodiment of the multiple-set coaxial twin-lens of the present invention, the vibrating mirror can be required to scan one complete projected structured light within 1 millisecond.

[0075] In one embodiment, the measurement may be based on the emitted light itself. Thus, the measurement device may be one or more photosensors (e.g., two photodiodes PD), and the two photosensors are arranged in one of the following ways: on different emission paths of the projection device, on different reflection paths within the projection device, or on emission paths and reflection paths inside and outside the projection device, respectively. The arrangement manner of the photosensors can be reasonably selected to accurately measure the phase while not affecting the normal projection of the structured light. As shown in FIG. 5, the PD may be mounted inside the projection device to determine the instantaneous vibration phase by measuring the reflection angle when the laser light is emitted from the optical window. Since the vibration phase of the MEMS mirror is a sinusoidal distribution, one PD can determine the sinusoidal distribution information, but more PDs are helpful to measure the phase more accurately. In another embodiment, the PD may be mounted outside the projection device, for example, on the optical window, for example near the edge of the optical window, thereby preventing the influence on the projection in the shooting area. In another embodiment, other ways may be used to measure the phase, for example, by performing a capacitance measurement.

[0076] In one embodiment, each image sensor completes imaging of one image frame every time the projection device performs one scan projection. For example, the vibrating mirror completes imaging of one image frame (e.g., one pattern in FIG. 1 or FIG. 2) by completing a half-cycle of vibration to scan the stripe light in the x-direction from one side of the shooting area to the other side. When the projection output of the projection device is limited or the measurement target is far away from the measurement head, imaging is generally not possible with the charge amount acquired by the image sensor after one scan, and multiple scan imaging is required. Therefore, each image sensor completes imaging of one image frame every time the projection device performs a set number of scan projections. For example, the DMD scans the same structured light pattern within five consecutive vibration cycles, so that the image sensor can acquire a sufficient charge amount for imaging, and scans the same next structured light pattern within the next five vibration cycles, and so on.

[0077] FIG. 8 shows a vibrating mirror as a reflecting device. However, in another embodiment, the reflecting device may be a mechanical rotating mirror that rotates in one direction, and is used to scan and project the linear light generated by the laser light emitting element at the predetermined frequency onto the imaging area. Correspondingly, the measuring device included in the synchronizing device may be an angle meter for measuring the rotation angle of the motor of the reflecting device in real time. Therefore, the synchronizing device can synchronize on the pixel row imaging based on the measurement result of the angle meter.

[0078] In the above embodiment, the synchronization between the scan line and the column pixel exposure is realized by controlling the exposure of the image sensor. This can be used in the situation where the light source scanning is controllable (for example, the angle and rotation speed of the mechanical vibrating mirror can be controlled by voltage and current), and especially in the situation where the phase and speed of the light source scanning are not controllable (for example, in MEMS mirror or mechanical rotating mirror). Therefore, the MEMS mirror can detect the angle using PD or capacitor, and the mechanical rotating mirror can also realize the position detection by voltage detection or photoelectric encoding.

[0079] Whether it is a vibrating mirror whose rotational speed is controllable and which moves back and forth, or a rotating mirror whose rotation is not controllable and which moves at a constant speed in one direction, the angular velocity can be matched to the row sequential exposure speed of the rolling shutter image sensor, so that either can realize a row sequential exposure means based on line laser scanning.

[0080] According to another embodiment of the present invention, a depth data calculation device can be realized, comprising the depth data measurement head and a processor connected to the depth data measurement head, wherein the processor is used in a monocular manner to determine depth data of an object to be photographed in the shooting area based on a reference image of the image sensor and the image frame set obtained by imaging the structured light thereby, or in a binocular manner to determine depth data of an object to be photographed in the shooting area based on a predetermined relative position of the first and second image sensors and the image frame set pair obtained by imaging the structured light thereby.

[0081] According to another embodiment of the present invention, a method for measuring depth data can be realized. Figure 10 is a schematic flow chart of a depth measurement method according to an embodiment of the present invention. The method can be realized by using the depth data measurement head of the present invention.

[0082] In step S1010, a structured light (first pattern) is scan-projected onto the photographing area. In step S1020, a first sub-image sensor (e.g., sub-image sensor 223, or sub-image sensors 323 and 333) is used to photograph the photographing area and obtain a first image frame under the irradiation of the structured light. In step S1030, a second structured light (second pattern) having a different pattern is scan-projected onto the photographing area. In step S1040, a second sub-image sensor (e.g., sub-image sensor 224, or sub-image sensors 324 and 334) is used to photograph the photographing area and obtain a second image frame under the irradiation of the second structured light, the first and second sub-image sensors share at least a part of the optical path and constitute an image sensor, and the first and second image frames are used to calculate one-time depth data of the photographing area.

[0083] In a binocular embodiment, step S1020 may include using a first sub-image sensor pair (e.g., 323 and 333) having a predetermined relative positional relationship to capture the shooting area and obtain a first image frame pair under the illumination of structured light, and step S1040 may include using a second sub-image sensor pair (e.g., 324 and 334) to capture the shooting area and obtain a second image frame pair under the illumination of second structured light, where the first and second sub-image sensor pairs are coaxial. That is, one sub-image sensor of each of the first and second sub-image sensor pairs shares at least a part of the optical path and constitutes a first image sensor, and the other sub-image sensor of each of the first and second sub-image sensor pairs shares at least a part of the optical path and constitutes a second image sensor, and the first and second image frame pairs are used to calculate one-time depth data of the shooting area.

[0084] The step of scanning and projecting a second structured light of a different pattern onto the imaging area includes a step of projecting the second structured light at a first interval from the projection of the structured light that is shorter than the frame imaging interval of the sub-image sensor, and a time interval between capturing the first image frame (pair) by the first sub-image sensor (pair) and capturing the second image frame (pair) by the second sub-image sensor (pair) that is shorter than the frame imaging interval of the sub-image sensor.

[0085] In one embodiment, the method further includes a step of scanning and projecting a third structured light onto an imaging area, wherein a pattern of the third structured light is different from the structured light and the second structured light, and a step of using the first or second sub-image sensor (pair) to image the imaging area and obtain a third image frame (pair) under the illumination of the third structured light, wherein the third image frame (pair) is used for one depth data calculation of the imaging area, or a step of using a third sub-image sensor (pair) to image the imaging area and obtain a third image frame (pair) under the illumination of the third structured light.

[0086] The third sub-image sensor belongs to the image sensor and shares at least a part of the optical path with the other sub-image sensors therein. Specifically, in a twin-lens embodiment, one sub-image sensor of the third sub-image sensor pair belongs to the first image sensor and shares at least a part of the optical path with the other sub-image sensors therein, the other sub-image sensor of the third sub-image sensor pair belongs to the second image sensor and shares at least a part of the optical path with the other sub-image sensors therein, and the third image frame pair is used to calculate one-time depth data of the shooting area.

[0087] The step of scanning and projecting a third structured light onto the shooting area includes a step of projecting the third structured light at a second interval from the projection of the structured light that is equal to or greater than the frame imaging interval of the sub-image sensor, and using the first sub-image sensor (pair) to photograph the shooting area and obtain the third image frame (pair) under irradiation with the third structured light.

[0088] To achieve coaxiality, the first sub-image sensor (pair) and the second sub-image sensor (pair) respectively acquire split beams of the structured light and the second structured light, and the first sub-image sensor (pair) or the second sub-image sensor (pair) can be selectively turned on to capture images, or the optical path of the incident light can be controlled so that only the first sub-image sensor pair acquires the structured light and captures images, and only the second sub-image sensor pair acquires the second structured light and captures images.

[0089] The present invention further realises a depth data calculation method, which includes the steps of acquiring a first and second image frames (pair), or a first, second and third image frames (pair), based on the above-mentioned depth data measurement method, and determining depth data of the subject to be photographed in the photographing area based on the predetermined relative position and the first and second image frames (pair), or the first, second and third image frames (pair).

[0090] Above, the embodiments of the multi-set coaxial monocular and binocular of the present invention have been described in detail with reference to the drawings, which introduces the multi-set coaxial photosensitive units to reduce the imaging time and improve the frame rate when integrating multi-frames to calculate depth data.

[0091] The depth data measuring head and the corresponding depth data measuring means of the present invention are particularly suitable for the combination of linear light stripe pattern projection and rolling shutter exposure, thus realizing depth data measurement with high interference resistance. Specifically, the depth data measuring means of the present invention is particularly suitable for measuring target objects outdoors, for example, under sunlight, and can be realized as, for example, a damage determination device for automobiles, etc.

[0092] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described with reference to this disclosure may be implemented as electronic hardware, computer software, or a combination of both.

[0093] The flowcharts and block diagrams in the figures show possible architectures, functions and operations of the systems and methods of several embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, a program segment or a portion of code, which includes one or more executable commands for implementing a specified logical function. It is to be noted that in some alternative embodiments, the functions described in the blocks may be performed in a different order than that described in the figures. For example, two consecutive blocks may actually be executed substantially simultaneously, or may sometimes be executed in a reverse order depending on the functions involved. Also, each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be realized by a dedicated system based on hardware for implementing the specified functions or operations, or a combination of dedicated hardware and computer commands.

[0094] Although each embodiment of the present invention has been described, the above description is illustrative, not exhaustive, and not limited to each embodiment disclosed. Many modifications and changes will be obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of each embodiment, practical applications, or improvements to technology in the market, or to enable those skilled in the art to understand each embodiment disclosed in this specification.

Claims

1. a projection device and an image sensor; The projection device is used to scan and project structured light sets of different patterns onto an imaging region, the structured light sets including at least two different patterns of structured light; the image sensor is used to image the imaging region and obtain a set of image frames used for a single depth data calculation of the imaging region under illumination with the set of structured light; the image sensor includes at least two sub-image sensors that share at least a portion of an optical path, the at least two sub-image sensors being used to perform imaging on different patterns of structured light projected successively by the projection device, and a synchronization device used for sequentially and synchronously imaging the at least two different patterns of structured light on the at least two sub-image sensors at a first interval, the first interval being shorter than a frame imaging interval of the sub-image sensors, at the same time as the projection device projects the at least two different patterns of structured light on the at least two sub-image sensors at the first interval. Depth data measurement head.

2. the image sensor includes first and second image sensors having a predetermined relative positional relationship, and is used to capture the image of the imaging region and obtain a pair of image frame sets used for a single depth data calculation of the imaging region under illumination with the structured light set; The first and second image sensors each include at least two sub-image sensors that share at least a portion of an optical path, and the at least two sub-image sensors are each used to image different patterns of structured light projected successively by the projection device. The depth data measurement head of claim 1 .

3. the synchronization device is used for each sub-image sensor to image its respective next frame at a second interval equal to or greater than the frame imaging interval of the sub-image sensor and to synchronize with projection by the projection device. The depth data measurement head of claim 1 .

4. The image sensor includes: a lens unit for receiving the incident return structured light; a beam splitter for splitting the incident return structured light into at least a first beam and a second beam; a first sub-image sensor for imaging the first beam; a second sub-image sensor for imaging a second beam of returned structured light corresponding to a different pattern; Including, 3. A depth data measuring head according to claim 1 or 2.

5. The image sensor includes: a lens unit for receiving the incident return structured light; an optical path changing device for directing the incident return structured light to at least a first sub-path and a second sub-path; a first sub-image sensor configured to image the returning structured light on a first sub-path; a second sub-image sensor configured to image the returned structured light corresponding to a different pattern on a second sub-path; Including, 3. A depth data measuring head according to claim 1 or 2.

6. At least two sub-image sensors that share at least a portion of an optical path have optical paths of the same length.

3. A depth data measuring head according to claim 1 or 2.

7. At least two sub-image sensors sharing at least a portion of the optical path are aligned at the pixel level; 7. The depth data measurement head of claim 6.

8. At least two of the sub-image sensors that share at least a portion of the optical path are infrared light sensors; and / or The image sensor is used to perform imaging on the incident structured light and includes a visible light image sensor that shares at least a portion of an optical path with the first and / or second sub-image sensor.

3. A depth data measuring head according to claim 1 or 2.

9. The structured light sets projected by the projection device have different patterns, and the structured light sets have different coding stripes. The depth data measurement head of claim 1 .

10. the projection device includes a laser light emitting element for generating a linear and / or spot laser light, and the laser light emitting element performs high-speed switching to scan-project a structured light of alternating light and dark corresponding to a stripe code; The depth data measurement head of claim 1 .

11. The projection device includes a light emitting device and a reflecting device, The light emitting device is used to generate linear light, The reflecting device is used for reflecting the linear light and projecting the linear light moving in a direction perpendicular to the stripe direction of the stripe code onto the photographing area. The depth data measurement head of claim 10.

12. At least two of the sub-image sensors sharing at least a portion of the optical path are global shutter image sensors; or the at least two sub-image sensors are rolling shutter image sensors; Further, the depth data measurement head includes a column synchronization device, and the column synchronization device is used to synchronize and turn on a sub-image sensor used for current imaging and a pixel column in a stripe direction of the stripe code corresponding to the current scanning position based on a scanning position of the projection device, thereby performing imaging. The depth data measurement head of claim 10.

13. The at least two sub-image sensors are mounted upside down relative to each other.

13. The depth data measurement head of claim 12.

14. A depth data measuring head according to any one of claims 1 to 13, and a processor connected to the depth data measuring head, The processor is used to determine depth data of an object in the imaging region based on the set of image frames obtained by imaging the structured light. Depth data calculator.

15. A step of scanning and projecting structured light onto an imaging area; using a first sub-image sensor to capture an image of the capture area under illumination with structured light to obtain a first image frame; Scanning and projecting a second structured light having a different pattern onto the photographing area; using a second sub-image sensor to capture an image of the capture area under illumination with a second structured light to obtain a second image frame; Including, The first and second sub-image sensors share at least a part of an optical path and constitute an image sensor, and the first and second image frames are used to calculate depth data of the imaging area at one time; the step of scanning and projecting a second structured light of a different pattern onto an imaging area includes a step of projecting the second structured light at a first interval from the projection of the structured light, the first interval being shorter than a frame imaging interval of the first and second sub-image sensors, and a time interval between capturing the first image frame by the first sub-image sensor and capturing the second image frame by the second sub-image sensor being shorter than the frame imaging interval of the first and second sub-image sensors; How depth data is measured.

16. The step of photographing the shooting area using a first sub-image sensor and acquiring a first image frame under illumination with structured light includes a step of photographing the shooting area using a first sub-image sensor pair having a predetermined relative positional relationship and acquiring a first image frame pair under illumination with structured light, the step of photographing the shooting area using a second sub-image sensor to obtain a second image frame under illumination with the second structured light includes a step of photographing the shooting area using a second sub-image sensor pair to obtain a second image frame pair under illumination with the second structured light; One of the first and second sub-image sensor pairs shares at least a portion of an optical path and constitutes a first image sensor, and the other of the first and second sub-image sensor pairs shares at least a portion of an optical path and constitutes a second image sensor, and the first and second image frame pairs are used to calculate one-time depth data of the imaging area. The method of claim 15.

17. Scanning and projecting a third structured light onto an imaging region, the third structured light having a pattern different from the structured light and the second structured light; Using the first or second sub-image sensor, photograph the photographed area to obtain a third image frame under illumination with a third structured light, and the third image frame is used for one depth data calculation of the photographed area; or Using a third sub-image sensor to photograph the photographing area and obtain a third image frame under the illumination of a third structured light, the third sub-image sensor belongs to the image sensor and shares at least a part of an optical path with other sub-image sensors therein, and the third image frame is used to calculate one-time depth data of the photographing area; Further comprising: The method of claim 15.

18. the step of scanning and projecting a third structured light onto an imaging area includes a step of projecting the third structured light at a second interval from the projection of the structured light, the second interval being equal to or greater than a frame imaging interval of the sub-image sensor, and using the first sub-image sensor to capture an image of the imaging area and obtain the third image frame under irradiation with the third structured light; 20. The method of claim 17.

19. The first sub-image sensor and the second sub-image sensor respectively acquire the split beams of the structured light and the second structured light, and selectively turn on the first sub-image sensor or the second sub-image sensor to capture an image; or By controlling an optical path of the incident light, only the first sub-image sensor acquires the structured light and performs imaging, and only the second sub-image sensor acquires the second structured light and performs imaging. The method of claim 15.

20. The method includes: acquiring first and second image frames, or first, second and third image frames, based on the depth data measurement method according to any one of claims 15 to 19; and determining depth data of a subject in the shooting area based on the first and second image frames, or the first, second and third image frames. How depth data is calculated.

Images