Depth data measurement head, computing device and corresponding method

Abstract

A depth data measurement head, a computing device and a corresponding method thereof are disclosed. The measurement head includes: a projection device for scanning and projecting a set of structured light with different patterns onto a shooting area; an image sensor for shooting the shooting area to obtain a set of image frames under the irradiation of the set of structured light for single-time depth data calculation of the shooting area, wherein the image sensor includes at least two sub-image sensors sharing at least a partial optical path, and the at least two sub-image sensors are respectively used for imaging the structured light with different patterns successively projected by the projection device. By using a coaxial single-eye or multiple groups of binocular structures for continuous imaging, the present invention can reduce problems such as inability to perform dynamic imaging and too low frame rate of depth data caused by too long multi-frame acquisition time in the scenario of obtaining depth data by multi-frame merging.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional imaging, and more particularly, to a depth data measurement head, a computing device, and a corresponding method thereof. Background Art

[0002] A depth camera is a data acquisition device that captures depth information of a target object. Such cameras are widely used in fields such as three-dimensional scanning and three-dimensional modeling. For example, more and more smartphones are now equipped with depth camera devices for face recognition. Although three-dimensional imaging has been a research hotspot in the field for many years, existing depth cameras still have many problems, such as the inability to achieve both measurement accuracy and imaging speed, and the inability to achieve both imaging performance and cost optimization.

[0003] Therefore, an improved depth data measurement scheme is needed. Summary of the Invention

[0004] One technical problem to be solved by the present disclosure is to provide a depth data measurement scheme that uses multiple coaxial binoculars for continuous imaging, which can reduce problems such as the inability to perform dynamic imaging and the too low frame rate of depth data in the scenario of obtaining depth data by merging multiple frames due to the too long acquisition time of multiple frames. In addition, since a conventional frame rate imaging device is introduced, it is possible to improve the depth imaging accuracy and frame rate while avoiding the use of high-cost high-frame rate imaging devices.

[0005] According to a first aspect of the present disclosure, there is provided a depth data measurement head, including: a projection device for scanning and projecting a set of structured light with different patterns onto a shooting area, the set of structured light including at least two structured lights with different patterns; an image sensor for shooting the shooting area to obtain a set of image frames under the irradiation of the set of structured light for single-time depth data calculation of the shooting area, wherein the image sensor includes at least two sub-image sensors sharing at least a part of the optical path, and the at least two sub-image sensors are respectively used for imaging the structured light with different patterns successively projected by the projection device.

[0006] In a binocular structure, the image sensor may be a first and a second image sensor having a predetermined relative position relationship for shooting the shooting area to obtain a set of image frame pairs under the irradiation of the set of structured light for single-time depth data calculation of the shooting area, wherein each of the first and second image sensors includes at least two sub-image sensors sharing at least a part of the optical path, and the at least two sub-image sensors are respectively used for imaging the structured light with different patterns successively projected by the projection device.

[0007] According to a second aspect of the present disclosure, a depth data calculation device is provided, including: a depth data measurement head as described in the first aspect; and a processor connected to the depth data measurement head, configured to determine depth data of an object to be photographed in the photographed area according to a set of image frames obtained by photographing the structured light.

[0008] According to a third aspect of the present disclosure, a depth data measurement method is provided, including: scanning and projecting structured light onto a photographed area; using a first sub-image sensor to photograph the photographed area to obtain a first image frame under the illumination of the structured light; scanning and projecting a second structured light with a different pattern onto the photographed area; using a second sub-image sensor to photograph the photographed area to obtain a second image frame under the illumination of the second structured light, wherein the first and second sub-image sensors share at least part of an optical path and form an image sensor, and the first and second image frames are used for single-time depth data calculation of the photographed area.

[0009] In a binocular implementation, the method may include: scanning and projecting structured light onto a photographed area; using a pair of first sub-image sensors with a predetermined relative position relationship to photograph the photographed area to obtain a first pair of image frames under the illumination of the structured light; scanning and projecting a second structured light with a different pattern onto the photographed area; using a pair of second sub-image sensors to photograph the photographed area to obtain a second pair of image frames under the illumination of the second structured light, wherein one sub-image sensor in each of the first and second pairs of sub-image sensors shares at least part of an optical path and forms a first image sensor, and the other sub-image sensor in each of the first and second pairs of sub-image sensors shares at least part of an optical path and forms a second image sensor, and the first and second pairs of image frames are used for single-time depth data calculation of the photographed area.

[0010] According to a fourth aspect of the present disclosure, a depth data calculation method is provided, including: obtaining a first and a second image frame, or a first, a second, and a third image frame according to the depth data measurement method described in the third aspect; and determining depth data of an object to be photographed in the photographed area according to the predetermined relative position and the first and second image frames, or the first, second, and third image frames.

[0011] Thus, the present invention discloses multiple sets of coaxial binocular solutions. By introducing coaxial monocular or binocular photosensitive units, the solution can reduce the imaging time required for multi-frame merged calculation of depth data and improve the frame rate. Description of the Drawings

[0012] The above and other objects, features, and advantages of the present disclosure will become more apparent by describing the exemplary embodiments of the present disclosure in more detail with reference to the accompanying drawings, in which, in the exemplary embodiments of the present disclosure, the same reference numerals generally represent the same components.

[0013] Figure 1A -B shows the principle of depth imaging using structured light encoded with stripes.

[0014] Figure 2 Shows a schematic diagram of the composition of a depth data measurement head according to an embodiment of the present invention.

[0015] Figure 3 Shows a schematic diagram of the composition of a depth data measurement head according to an embodiment of the present invention.

[0016] Figure 4 Shows a comparison timing diagram of coaxial two sets of binocular imaging and single set of binocular imaging.

[0017] Figure 5 Shows a timing diagram of coaxial three sets of binocular imaging.

[0018] Figure 6 Shows a schematic diagram of column-synchronized imaging by a depth data measurement head according to an embodiment of the present invention.

[0019] Figure 7A -B shows Figure 6 An enlarged operation example of the projection device shown.

[0020] Figure 8 Shows a simplified perspective schematic diagram of the projection device used in the present invention.

[0021] Figure 9 Shows a schematic diagram of pixel rows in the image sensor being sequentially turned on.

[0022] Figure 10 Shows a schematic flowchart of a depth measurement method according to an embodiment of the present invention. Detailed implementation manners

[0023] Hereinafter, the preferred embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure will be more thorough and complete, and can fully convey the scope of the present disclosure to those skilled in the art.

[0024] To improve the imaging accuracy and flexibility of depth data, it is possible to achieve highly flexible pixel-level depth data imaging through the combination of actively projecting multiple patterns (e.g., stripe-encoded structured light) and binocular imaging, based on the superposable nature of the stripe-encoded patterns and the characteristic of binocular imaging that does not depend on a specific imaging plane.

[0025] According to the principle of structured light measurement, the key to the entire measurement system lies in accurately determining the scanning angle α. For point and line structured light, the scanning angle can be calculated and determined through mechanical devices such as a rotating mirror, while the significance of image encoding and decoding lies in determining the scanning angle of the encoded structured light, i.e., the surface structured light system. Figure 1A -B shows the principle of depth imaging using stripe-encoded structured light. For ease of understanding, Figure 1A The encoding principle of stripe structured light is briefly described using a two-gray-level three-bit binary time encoding. The projection device can sequentially project the three patterns shown in the figure onto the object to be measured in the shooting area. The three patterns divide the projection space into 8 regions using two gray levels of bright and dark. Each region corresponds to its own projection angle. It can be assumed that the bright region corresponds to the encoding "1" and the dark region corresponds to the encoding "0". The encoding values of a point on the scene in the projection space in the three encoded patterns are combined in the projection order to obtain the region encoding value of this point, thereby determining the region where this point is located and then decoding to obtain the scanning angle of this point.

[0026] In a monocular imaging system, depth data calculation can be performed by comparing the captured image with a reference image. In a binocular imaging system, the above decoding process can be simplified by directly matching the encoding values of each point in the first and second image sensors. To improve the matching accuracy, the number of projected patterns in the time encoding can be increased. Figure 1B Another example of projecting stripe-encoded structured light is shown. Specifically, a two-gray-level five-bit binary time encoding is shown in the figure. In the application scenario of binocular imaging, this means that for example, each pixel in each of the left and right image frames contains 5 region encoding values of 0 or 1, thereby enabling left-right image matching with higher accuracy (e.g., at the pixel level). It should be understood that in other embodiments, time encodings such as five-base four or even eight-base can also be achieved by using different brightness levels.

[0027] When the projection rate of the projection device remains unchanged, compared with Figure 1A the three encoded patterns, Figure 1BThe example is equivalent to achieving higher-precision image matching at a higher time-domain cost. Although the original projection rate of the projection device is extremely high (for example, the galvanometer device preferably used in the present invention), due to the limited frame rate of the image sensor, the frame rate of the multi-frame synthesized depth data is low, and the imaging time required for each frame is too long, making it difficult to capture dynamic videos. For example, in the case of using a conventional image sensor with a frame rate of 60 frames per second, if it is necessary to synthesize one frame of depth image from two-dimensional images captured every five frames (in binocular implementation, actually 5 groups of 10-frame images), the frame rate per second drops to 12 frames per second, and the imaging time for each frame is as long as 1 / 15. Such a low frame rate and such a long imaging time are difficult to meet the requirements of dynamic imaging. However, directly increasing the frame rate of the image sensor to achieve a relatively high frame rate of depth data, for example, using a high-speed image sensor with a frame rate of 150 frames per second to achieve a depth data frame rate of 30 frames per second, will significantly increase the device cost due to the introduction of high-performance components.

[0028] In order to meet the requirements of high precision, high frame rate, and low cost in the field of three-dimensional imaging, considering the current technical status that the current projection device can achieve high-speed projection, the imaging rate of the conventional image sensor is relatively high, but the frame rate is reduced due to data transmission and processing, etc., the inventors of the present invention proposed a method of using a multi-group sensor structure on the same axis to image different patterns respectively, so as to increase the overall frame rate of obtaining depth data based on multiple frames, thereby solving the problems in the prior art such as the inability to perform dynamic imaging and the too low frame rate of depth data due to the too long multi-frame acquisition time. In addition, since the introduced imaging device has a conventional frame rate, it is possible to avoid the use of high-cost high-frame-rate imaging devices while improving the depth imaging accuracy and frame rate.

[0029] Specifically, the present invention discloses a depth data measurement head, including: a projection device for scanning and projecting a set of structured light with different patterns onto a shooting area, where the set of structured light includes at least two structured lights with different patterns; an image sensor for shooting the shooting area to obtain a set of image frames under the irradiation of the set of structured light for single-time depth data calculation of the shooting area, wherein the image sensor includes at least two sub-image sensors sharing at least part of the optical path, and the at least two sub-image sensors are respectively used for imaging the structured light with different patterns successively projected by the projection device.

[0030] In some embodiments, the image sensor may be a single image sensor, that is, implemented as a monocular system. In other embodiments, the image sensor may be two image sensors, that is, implemented as a binocular system. The following will be combined with Figure 2 and Figure 3 to illustrate the monocular and binocular implementations based on the principles of the present invention respectively.

[0031] Figure 2 The schematic composition diagram of the depth data measurement head according to an embodiment of the present invention is shown. As shown in the figure, the depth data measurement head 200 based on the monocular principle includes a projection device 210 and an image sensor 220. Although not shown in the figure for convenience of description, the measurement head 200 may further include a housing for enclosing the above devices, and Figure 2 the shown connection structure 240 can be regarded as a mechanism for fixing the above devices and connecting them to the housing. In some embodiments, the connection structure 240 may be a circuit board on which a control circuit is included. It should be understood that in other implementations, the above devices 210 and 220 may be connected to the housing in other ways and perform corresponding data transmission and instruction receiving operations.

[0032] Here, the projection device 210 is used to scan and project a set of structured light with different patterns onto the shooting area, and the set of structured light includes at least two structured lights with different patterns. The image sensor 220 is used to capture the shooting area to obtain a set of image frames under the irradiation of the set of structured light for single-time depth data calculation of the shooting area.

[0033] For example, the projection device 210 sequentially projects three patterns as shown in FIG. 1. These three patterns form a set, and the image sensor respectively images each of them, thereby obtaining a set of image frames including 3 frames. According to the monocular imaging principle, these 3 frames of images can be respectively compared with their corresponding reference image frames and jointly used for the calculation of the depth data of the shooting area once, that is, a frame of depth image can be calculated.

[0034] Different from the conventional measurement head, the image sensor 220 only includes one photosensitive unit, and each photosensitive unit respectively performs three imaging operations to obtain a set of 3-frame image frames. In the present invention, the image sensor 220 includes at least two sub-image sensors sharing at least a part of the optical path, and the at least two sub-image sensors are respectively used to image the structured light with different patterns successively projected by the projection device.

[0035] Figure 2An example is shown in which the image sensor 220 includes two sub-image sensors (photosensitive units). As shown in the figure, the image sensor 220 includes sub-image sensors 223 and 224. Among them, the sub-image sensors 223 and 224 share the optical path up to the beam splitting surface of the beam splitting device 222 and are at equal distances from the above-mentioned beam splitting region. In other words, the present invention introduces a sensor structure that is coaxial with each other. Here, the sub-image sensor 223 can be used to image the structured light of the first pattern among the three patterns in FIG. 1, for example. Subsequently, the sub-image sensor 224 can be used to image the structured light of the second pattern among the three patterns in FIG. 1, for example. In other words, at this time, it can be regarded that the sub-image sensor 224 with the same optical path (or completely equivalent optical path) is in place and replaces 223 to image the structured light of the subsequent pattern. Thus, the imaging interval between two adjacent frames can be made smaller without depending on the frame interval of each image sensor.

[0036] Similarly, Figure 3 A schematic diagram of the composition of a depth data measurement head according to an embodiment of the present invention is shown. Compared with the schematically shown projection device, a more detailed composition example of the image sensor is given in the figure.

[0037] As Figure 3 shown, the depth data measurement head 300 based on the binocular principle includes a projection device 310 and a first image sensor 320 and a second image sensor 330 having a predetermined relative positional relationship. Although not shown in the figure for convenience of explanation, the measurement head 300 may further include a housing for enclosing the above devices, and Figure 3 the shown connection structure 340 can be regarded as a mechanism for fixing the above devices and connecting them to the housing. In some embodiments, the connection structure 340 may be a circuit board on which a control circuit is included. It should be understood that in other implementations, the above devices 310-330 may be connected to the housing in other ways and perform corresponding data transmission and instruction receiving operations.

[0038] Here, the projection device 310 is used to scan and project a set of structured light with different patterns onto the shooting area, and the set of structured light includes at least two structured lights with different patterns. The first image sensor 320 and the second image sensor 330 having a predetermined relative positional relationship are used to shoot the shooting area to obtain a set of image frame pairs under the irradiation of the set of structured light for single-depth data calculation of the shooting area.

[0039] For example, the projection device 310 sequentially projects three patterns as shown in FIG. 1. These three patterns form a group, and each of these patterns is imaged by an image sensor respectively, thereby obtaining a group of image frame pairs including three pairs (6 frames). These 6 frames of images are jointly used for the calculation of the depth data of a single shot area, that is, a depth image of one frame can be calculated.

[0040] Different from a conventional binocular measurement head, where the first image sensor 320 and the second image sensor 330 each only include one photosensitive unit, and each photosensitive unit performs three imaging operations respectively to obtain a group of image frame pairs of three pairs (6 frames), in the present invention, the first and second image sensors each include at least two sub-image sensors sharing at least a part of the optical path, and the at least two sub-image sensors are respectively used for imaging the structured light of different patterns successively projected by the projection device.

[0041] Figure 3 An example is shown where the first and second image sensors each include two sub-image sensors (photosensitive units). As shown in the figure, the first image sensor 320 includes sub-image sensors 323 and 324, and the second image sensor 330 includes sub-image sensors 333 and 334. Among them, the sub-image sensors 323 and 324 share the optical path until the beam splitting surface of the beam splitter 322, and are at equal distances from the above-mentioned beam splitting area. Similarly, the sub-image sensors 333 and 334 share the optical path until the beam splitting surface of the beam splitter 332, and are at equal distances from the above-mentioned beam splitting area. In other words, the present invention introduces multiple groups of coaxial binocular structures. Here, the sub-image sensors 323 and 333 can be regarded as the first group of image sensors (the first group of binoculars) for imaging the structured light of, for example, the first pattern among the three patterns in FIG. 1. Subsequently, the sub-image sensors 324 and 334, which can be regarded as the second group of image sensors (the second group of binoculars), are used for imaging the structured light of, for example, the second pattern among the three patterns in FIG. 1. In other words, at this time, it can be considered that the sub-image sensors 324 and 334 coaxial with 323 and 333 respectively are in place and replace 323 and 333 to perform the imaging of the structured light of the subsequent pattern. Thus, the imaging interval between adjacent two frames can be made smaller without relying on the frame interval of each image sensor.

[0042] To this end, the measurement head 200 or 300 may further include: a synchronization device, which, while the projection device projects structured light of at least two different patterns at a first interval smaller than the frame imaging interval of the sub-image sensor, enables the image sensor 220 or the first and second image sensors 320 and 330 each including at least two sub-image sensors to synchronously image the structured light of the at least two different patterns successively at the first interval. Accordingly, each sub-image sensor still performs the next frame imaging of itself at a second interval not less than the frame imaging interval of the sub-image sensor (for example, imaging at its own frame interval), and the above imaging operations can be synchronized with the projection of the projection device under the synchronization of the synchronization device.

[0043] Figure 4 A comparative timing diagram of coaxial two-component imaging and single-component imaging is shown. Here, for the convenience of explanation, it can be assumed that the frame rate of each photosensitive unit (sub-image sensor) is 100 frames / s, then its frame interval is 10 ms, and it can be assumed that the exposure time required for each photosensitive unit is 1 ms.

[0044] If the image sensor 220 or the first and second image sensors 320 and 330 are conventional image sensors each including only a single photosensitive unit, when using the three patterns shown in FIG. 1 for depth data calculation, then as Figure 4 shown in the lower part, three imaging operations are required at the 0th, 10th, and 20th milliseconds. To this end, the object to be photographed needs to remain stationary for 21 ms for synthesizing each depth data image (therefore it is more difficult to photograph a moving object), and the frame rate also drops from 100 frames / s to 33.3 frames / s.

[0045] In contrast, if the image sensor 220 or the first and second image sensors 320 and 330 are the image sensors of the present invention each including two photosensitive units (for example, the image sensor 220 includes sub-image sensors 223 and 224, and the first and second image sensors 320 and 330 each include sub-image sensors 323 and 324, and sub-image sensors 333 and 334), when using the three patterns shown in FIG. 1 for depth data calculation, then as Figure 4As shown above, the first group of photosensitive units captures an image of Pattern 1 at the 0th millisecond. Immediately afterwards, the second group of photosensitive units captures an image of Pattern 2 at the 1st millisecond. Subsequently, after a 10-ms interval, the first group of photosensitive units captures an image of Pattern 3 at the 10th millisecond, thus completing the three captures required for a depth data image. Subsequently, at the 11th millisecond, the second group of photosensitive units can start the next round of capturing an image of Pattern 1. At the 20th millisecond, the first group of photosensitive units captures an image of Pattern 2. At the 21st millisecond, the second group of photosensitive units captures an image of Pattern 3 again. In this way, the interval between captures of different groups of photosensitive units only needs to be the time required for capturing an image (e.g., 1 ms), while the interval between repeated captures of the same group of photosensitive units still follows the frame interval time corresponding to the frame rate (e.g., 10 ms). At this time, by introducing two sets of coaxial binoculars, it only takes 11 ms for the object to remain stationary to synthesize each depth data image (so it is easier to capture moving objects), and the frame rate can be maintained at nearly 66.6 frames / s.

[0046] Although the example Figures 2 - 4 describes having two sets of coaxial (same optical axis) photosensitive units, in other embodiments, the first and second image sensors may each further include more photosensitive units. Figure 5 shows a timing diagram of coaxial triple binocular imaging. At this time, the first and second image sensors may each include three coaxial photosensitive units (sub-image sensors). For this purpose, as Figure 5 shown, the first group of photosensitive units captures an image of Pattern 1 at the 0th millisecond. Immediately afterwards, the second group of photosensitive units captures an image of Pattern 2 at the 1st millisecond. Immediately afterwards, the third group of photosensitive units captures an image of Pattern 3 at the 2nd millisecond. Subsequently, the next round of triple captures starts at the 10th millisecond, and the next round of triple captures starts at the 20th millisecond, and so on. At this time, by introducing three sets of coaxial binoculars, it only takes 3 ms to obtain the three sets (6 frames) of images required to synthesize a depth data image, that is, the object only needs to remain stationary for 3 ms, so the shooting level for moving objects is greatly improved, and the frame rate can be maintained at nearly 100 frames / s (in this example, it takes 1003 ms, i.e., 1.003 seconds, to capture 100 frames).

[0047] Therefore, it should be understood that by simply introducing an additional set of coaxial binocular structures (or monocular structures), the frame rate of depth data based on multi-frame synthesis can be doubled, and the imaging time for each frame can be shortened. In theory, the same number of sets of coaxial binocular structures as the number of images projected by the projection device can be arranged, so that the framing time for each depth frame and the frame interval of the sensor are only related to the multiple of the exposure time (in the case where the frame interval is greater than the exposure time x the number of coaxial structures). For example, in the case of synthesizing a depth frame based on four patterns, ifFigure 3 For the two sets of coaxial binoculars shown, the imaging time for obtaining four frames slightly increases to 12 ms, but the frame rate drops to nearly 50 frames / s. However, if four sets of coaxial binoculars are used, the imaging time for obtaining four frames is only 4 ms, and the frame rate still remains at nearly 100 frames / s. However, introducing too many coaxial structures will increase the construction difficulty of the image sensor. Therefore, a compromise needs to be made among cost, feasibility, and imaging speed.

[0048] To achieve the coaxial configuration of different photosensitive units within the same image sensor, the optical path needs to be designed. In Figure 3 the example of, a coaxial arrangement based on beam splitting is shown ([[]] Figure 2 the example of also has a similar structure). At this time, taking the first image sensor 320 as an example, it may include: a lens unit 321 for receiving the incident returned structured light; a beam splitting device 322 for splitting the incident returned structured light into at least a first light beam and a second light beam; a first sub-image sensor 323 for imaging the first light beam; and a second sub-image sensor 324 for imaging the second light beam corresponding to the returned structured light of different patterns.

[0049] In one embodiment, the beam splitting device 322 is an optical prism, such as a quadrangular prism or a triangular prism. Thus, the reflected infrared light in the incident light reaches the second sub-image sensor 324, and the visible light that is not reflected in the incident light can propagate straight to the first sub-image sensor 323.

[0050] As shown in the figure, the beam splitting device 322 in the form of a prism can split the incident light into two light beams with perpendicular propagation directions. Correspondingly, the first sub-image sensor 323 and the second sub-image sensor 324 can also be arranged vertically so as to receive the incident visible light and infrared light beams at vertical angles respectively.

[0051] To eliminate parallax and achieve pixel-level alignment, the components in the incident light need to have the same optical path. For this reason, when using a quarter prism as the beam splitting device 322, the first sub-image sensor 323 and the second sub-image sensor 324 can be arranged at equal distances from the beam splitting area of the beam splitting device 322. When using a triangular prism as the beam splitting device 322, the distances between the two photosensitive units and the beam splitting device 322, especially the beam splitting area, can be flexibly adjusted according to the refractive index ratio of air to the prism material.

[0052] The pixel-level alignment between the first sub-image sensor 323 and the second sub-image sensor 324 can be theoretically achieved by making the incident light share most of the optical path and have the same optical path length. However, in the actual manufacturing process of the image sensor, deviation between the imaging of the two may occur due to the fact that the actual arrangement of the first sub-image sensor 323 and the second sub-image sensor 324 cannot present an ideal vertical and equidistant condition. At this time, forced software correction can be performed on the manufactured image sensor. For example, by introducing a calibration target and aligning the imaging of both the first sub-image sensor 323 and the second sub-image sensor 324 with the calibration target, true pixel-level correction can be achieved.

[0053] As shown in the figure, the image sensor 320 of the present invention can be implemented as a separate module. For this purpose, the image sensor 320 may further include a housing for fixing the relative positions of the lens unit, the beam splitting device, and the two photosensitive units. Preferably, the housing can be combined with the lens unit 321 to form a sealed body to avoid contamination of the contained devices by the external environment. In other embodiments, the image sensor 320 of the present invention can be a part of a larger module (for example, a depth data measurement head), and the housing of the larger module can be used to fix the elements to each other.

[0054] Preferably, the image sensor 320 may further include cables respectively connected to the first sub-image sensor 323 and the second sub-image sensor 324. The housing has openings for cable access.

[0055] In one embodiment, the cable can be a flexible cable, such as an FPC (flexible printed circuit) cable.

[0056] In one embodiment, before the light beam enters the first sub-image sensor 323 and the second sub-image sensor 324, it can also pass through a filter to further filter out the influence of light of other wavelengths. In one embodiment, the projection device can project infrared laser light, so the filter arranged in the image sensor can be a corresponding infrared light transmission unit for transmitting infrared light within a specific frequency range. For example, in the present invention, infrared light with a wavelength of 780 - 1100 nm is used. In other embodiments, the projection device can also project visible light, such as projecting red laser light or blue laser light, such as red light at 635 nm or blue light at 450 nm. Although the ambient light may also include red light or blue light, due to the short exposure time and the high instantaneous light intensity of the laser, high signal-to-noise ratio imaging can also be performed with the help of the corresponding filter for projecting red light or blue light.

[0057] Preferably, when the beam splitting device is a cube prism, one side of the filter can be directly in physical contact with the cube prism, and the other side is in physical contact with the photosensitive unit, while the photosensitive unit and the cube prism are snap-fitted in the housing, thereby ensuring the high immutability of the relative positions of the devices.

[0058] In some embodiments, especially when the first and second sub-image sensors are infrared light sensors for receiving the projected infrared pattern, additional visible light sensing units (not shown in the figure) can also be arranged in the image sensor to capture the image information of the object to be measured, so that the image captured by the image sensor contains both the image information and the depth information of the object to be measured. The visible light sensing unit can be a grayscale sensor or a color sensor. The grayscale sensor only captures the luminance information, while the color sensor can be used to capture the color information of the object to be measured. In this case, the visible light sensing unit can be composed of three primary color sensing units, and the three primary colors can be the red, green, and blue primary colors (RGB) or the cyan, magenta, and yellow primary colors (CMY).

[0059] It should be understood that although based on Figure 3 the structure of the specifically described first image sensor 320, the second image sensor 330 can also have the same structure. Additionally, it should be understood that 323 and 333 can be regarded as the first set of binoculars, and 324 and 334 as the second set of binoculars, but 323 and 334 can also be regarded as the first set of binoculars, and 324 and 333 as the second set of binoculars, as long as imaging is enabled after the corresponding pattern is incident.

[0060] In the case of Figure 3 realizing optical path sharing by using beam splitting as shown, since the light received by each sensing unit will decrease, the sensitivity or effective distance range of imaging can be ensured by increasing the projection brightness or expanding the incident aperture.

[0061] For this purpose, as an alternative, optical path sharing can also be realized based on optical path conversion. At this time, the image sensor 220 or the first and second image sensors 320 and 330 can each include: a lens unit for receiving the incident returned structured light; an optical path conversion device for delivering the incident returned structured light to at least a first sub-path and a second sub-path; a first sub-image sensor for imaging the returned structured light on the first sub-path; and a second sub-image sensor for imaging the returned structured light corresponding to different patterns on the second sub-path. In one embodiment, the optical path conversion device can be a rotating mirror, which can reflect the incident light to the sensing unit 323 at, for example, the 0th millisecond and reflect the incident light to the sensing unit 324 at the 1st millisecond, etc. In other embodiments, the optical path conversion device can also be a device that performs optical path conversion based on other mechanical, chemical, or electrical principles.

[0062] As shown previously, a set of structured light with different patterns projected by the projection device 310 can be a set of structured light with different encoded stripes, such as in FIG. 1 or Figure 2The pattern. In some cases, the projection device 310 can be a projection device that projects a complete pattern each time. In other cases, the projection device 310 can be a device that completes the pattern through the bright and dark scanning of line light or the two-dimensional scanning of light points.

[0063] For this purpose, the projection device 310 can include: a laser generator for generating line and / or point lasers, and the laser generator performs high-speed switching to scan and project the structured light with alternating light and dark corresponding to the stripe coding. More specifically, the projection device includes: a light-emitting device for generating line light, and the laser generator can be included in the light-emitting device; and a reflection device for reflecting the line light to project the line light moving in the direction perpendicular to the stripe direction onto the imaging area. In Figure 3 In the module 310, the light-emitting device is shown as a rectangle, and the reflection device is shown as a slant line. The reflection device can include one of the following: a mechanical galvanometer that reciprocates at a predetermined frequency for scanning and projecting the line light onto the imaging area at the predetermined frequency, where the length direction of the line light is the length direction of the projected stripe; a micromirror device that vibrates reciprocally at a predetermined frequency for scanning and projecting the line light onto the imaging area at the predetermined frequency, where the length direction of the line light is the length direction of the projected stripe.

[0064] In the case where the projection device performs scanning projection, although the image sensor can be a global image sensor (i.e., all pixels perform imaging simultaneously), it is preferably implemented as a rolling shutter image sensor. For this purpose, the measurement head 300 can further include: a column synchronization device for synchronously turning on the sub-image sensor currently used for imaging and the pixel columns in the corresponding stripe direction at the current scanning position based on the scanning position of the projection device for imaging.

[0065] In a preferred embodiment, since a rolling shutter image sensor usually can only perform unidirectional column exposure, for example, exposure from the first row to the 1000th row, rather than vice versa, the two sub-image sensors installed in an image sensor of the present invention can be rolling shutter image sensors installed upside down with respect to each other, thereby achieving column exposure in opposite directions, so that the galvanometer does not need to start projecting from a certain fixed position, but starts projecting on both sides of the reciprocating motion.

[0066] For example, the movement of the galvanometer from left to right can scan and project a stripe pattern, and correspondingly, the first sub-image sensor can perform column exposure from left to right. After completing the projection of one stripe pattern, the galvanometer does not need to return to the left side, but can directly scan and project the second stripe pattern from right to left. At this time, since the second sub-image sensor is arranged upside down relative to the first sub-image sensor, the second sub-image sensor can perform column exposure from right to left. It should also be understood that the projection device of the measuring head of the present invention may have Figure 3 a structure other than that shown, such as a grating printed with a fixed pattern, etc.

[0067] Figure 6 shows a schematic diagram of column-synchronized imaging of a depth data measuring head according to an embodiment of the present invention. As Figure 6 shown, the depth data measuring head 600 includes a projection device 610 and two image sensors 620 and 630.

[0068] The projection device 610 is used to scan and project structured light with stripe coding onto the shooting area. For example, within three successive image frame projection periods, the projection device 610 can successively project three patterns as shown in FIG. 1, and the imaging results of these three patterns can be used for the generation of depth data. 620 and 630, which can be respectively referred to as the first and second image sensors, have a predetermined relative positional relationship and are used to shoot the shooting area to respectively obtain the first and second two-dimensional image frames under the illumination of the structured light. For example, in the case where the projection device 610 projects the three patterns as shown in FIG. 1, the first and second image sensors 620 and 630 can respectively image the shooting area (for example, Figure 3 the imaging plane in and a certain range before and after it) onto which these three patterns are projected within three synchronous image frame imaging periods. These three imaging operations respectively use different groups of photosensitive units.

[0069] As Figure 6 shown, the projection device 610 can project a linear light extending in the x direction in the z direction (i.e., towards the shooting area). In different embodiments, the projection of the above linear light can be already formed (i.e., the outgoing light itself is a linear light), or it can be a light spot moving in the x direction (i.e., the linear light obtained by scanning). The projected linear light can continuously move in the y direction to cover the entire imaging area. Figure 6 The lower perspective view of the shooting area gives a more understandable illustration of the scanning of the linear light.

[0070] In an embodiment of the present invention, the direction of the light-emitting measurement head is defined as the z direction, the vertical direction of the imaging plane is the x direction, and the horizontal direction is the y direction. Thus, the stripe structured light projected by the projection device can be the result of the linear light extending in the x direction moving in the y direction. Although in other embodiments, stripe structured light obtained by moving the linear light extending in the horizontal y direction in the x direction can also be used for synchronization and imaging processing, in the embodiments of the present invention, vertical stripe light is still preferably used for description.

[0071] Furthermore, the measurement head 600 further includes a column synchronization device 650. The column synchronization device 650 is respectively connected to the projection device 610 and the first and second image sensors 620 and 630 to achieve precise synchronization among the three. Specifically, the column synchronization device 650 can, based on the scanning position of the projection device 610, synchronously turn on the pixel columns in the corresponding stripe direction corresponding to the current scanning position in a corresponding group of photosensitive units of the first and second image sensors 620 and 630 for imaging. As Figure 3 shown, the current stripe is being scanned to the central region of the imaging area. For this reason, in a corresponding group of photosensitive units of the image sensors 620 and 630, the pixel columns located in the central region (for example, 3 adjacent pixel columns) are turned on for imaging. As the stripe moves in the y direction (as Figure 3 indicated by the arrow in the lower perspective view), the pixel columns turned on for imaging in a corresponding group of photosensitive units of the image sensors 620 and 630 also move synchronously accordingly (as Figure 3 indicated by the arrow above the matrix in the upper left block diagram). Thus, the one-dimensional characteristic of the stripe image can be utilized to control the range of the pixel columns for imaging at each moment, thereby reducing the adverse influence of ambient light on the measurement result. To further reduce the influence of ambient light, the projection device is particularly suitable for projecting light that is not easily confused with ambient light, such as infrared light. In addition, since the correspondence between the pixel columns and the scanning light is affected by many factors such as the width, power, speed of the projected light, and the photosensitive efficiency of the image sensor, the range (and corresponding quantity) of the pixel columns synchronously turned on each time can be determined, for example, based on a calibration operation.

[0072] Figure 7A -B shows Figure 6 an enlarged operation example of the projection device shown. Specifically, as Figure 6 shown, in the projection device 610, the laser emitted by the laser generator (such as Figure 7A the laser generator 711 shown in detail in -B) is scanned and projected onto the imaging area ( Figure 7A the gray area in) by the projection mechanism (such as Figure 6 the projection mechanism 712 shown in detail in -B) to irradiate the object to be measured in the imaging area (for example, Figure 6Active structured light projection is performed on the person) inside. A pair of image sensors 620 and 630 image the shooting area, thereby obtaining the image frames required for depth data calculation. As Figure 3 shown, the dotted lines emitted by the projection device 610 are used to represent its projection range, while the dotted lines emitted by the image sensors 620 and 630 are used to represent their respective imaging ranges. The shooting area is usually located in the overlapping area of the projection and imaging ranges of these three.

[0073] In practical applications, a laser generator is used to generate linear and / or infrared lasers, and the laser generator performs high-speed switching to scan and project the light and dark alternating structured light corresponding to the stripe coding. The high-speed switching can include the high-speed switch of the laser generator and the high-speed coding switching.

[0074] In one embodiment, the laser generator can continuously emit lasers with the same intensity, and the projected stripe pattern is achieved by turning on and off the laser generator. In this case, since the laser generator only projects light of one intensity, each pixel of the image sensor only needs to record the "presence or absence" of light, so the equipped image sensor can be a black-and-white image sensor.

[0075] In another embodiment, the laser generator itself can emit lasers with varying light intensities. For example, the emitted light intensity is sinusoidally transformed according to the applied power. The above sinusoidally transformed laser can be combined with stripe projection, thereby scanning and projecting a pattern with alternating light and dark and different brightnesses between the bright stripes. In this case, the image sensor needs to have the ability to distinguish and image different light intensities, so it can be a multi-order gray-scale image sensor. Obviously, gray-scale projection and imaging can provide more accurate pixel-to-pixel matching than black-and-white projection and imaging, thereby improving the accuracy of depth data measurement.

[0076] In one embodiment, the laser generator 711 can be a linear laser generator, generating linear light extending in the x direction ( Figure 7A -the direction perpendicular to the paper plane in B). This linear light is then projected onto the imaging plane by a reflection mechanism 712 that can swing along the axis in the x direction. The swinging of the reflection mechanism 712 is shown in the accompanying drawings as Figure 7B shown. Thus, reciprocating linear light scanning can be performed within the range AB of the imaging plane.

[0077] In one embodiment, the above reflection mechanism 712 can be a micro-mirror device (also known as a digital micro-mirror device, DMD), and can be implemented as a kind of MEMS (micro-electro-mechanical system). Figure 8 shows a simplified perspective schematic diagram of the projection device used in the present invention. As Figure 8As shown, the dot laser generated by the laser can be converted into a line laser through a lens (corresponding to the line laser generator 711 in FIG. 7). The above line laser is then reflected by a micromirror device in the form of MEMS, and the reflected line laser is projected into the external space through an optical window. The micromirror device has extremely high performance. For example, commercially available DMDs can perform highly stable reciprocating vibrations at a frequency of 2k, thus laying a foundation for high-performance depth imaging.

[0078] In other embodiments, the scanned and projected laser can also be a dot laser. Therefore, the projection mechanism needs to correspondingly change the projection direction in two dimensions (the x and y directions in the figure). For example, the projection mechanism first scans out stripe light in the x direction, then makes a displacement in the y direction, and continues to scan in the x direction at different y positions.

[0079] Whether it is directly projecting stripe light that moves in the y direction or projecting dot light that needs to move in the x direction to form stripes and make a displacement in the y direction, what appears in the shooting area is stripes that move in the y direction over time. As the light spot moves in the y direction, specific pixel columns among all the pixels on the image sensor used to record the image frame are turned on so that they can collect the light reflected from the corresponding positions. Figure 9 The schematic diagram showing the pixel columns in the image sensor being turned on in turn is shown. As Figure 9 shown, when the stripes projected by the projection device move from the middle of the imaging area to one side, in the pixel array of the image sensor, the pixel columns turned on for imaging also move from the middle to one side accordingly. Thus, the pixel columns only perform imaging recording during the period when the corresponding shooting area is scanned, and do not perform recording at other times. Since the intensity of the projected laser is higher than that of the ambient light, when the ambient light cannot be accumulated under the synchronous turning-on scheme of the present invention, the structured light itself can be imaged extremely accurately. Since conventional image sensors usually perform row exposure, the image sensor that performs column-by-column (or multiple columns simultaneously) exposure used in the present invention can be obtained by transposing a conventional image sensor by 90°. After transposition, control for simultaneous exposure of the entire column needs to be added to it.

[0080] It should be understood that Figure 6 and Figure 9The pixel matrix shown is merely an example given to illustrate the synchronization principle of the present invention. In actual applications, the pixel matrix of an image sensor often has a much higher magnitude (e.g., 1000x1000), and the number of pixel columns simultaneously turned on each time can also have different ranges according to calibration (e.g., turning on 3 columns each time, or turning on different numbers of columns at different positions of the shooting area, etc.). Additionally, the turning on of pixel columns in the image sensor can be related only to the scanning position of the projection structure in the projection device, and has nothing to do with whether stripe light is actually projected at present. In other words, the turning off and on of the laser emitter based on the distribution of bright and dark stripes of the projection structure will not affect the scanning projection action of the projection structure, nor will it affect the action of turning on the pixel columns of the image sensor synchronized with the above scanning projection action.

[0081] The projection device as described above may include a galvanometer that vibrates reciprocally at a predetermined frequency, such as a MEMS galvanometer or a mechanical galvanometer, for scanning a linear laser beam to the shooting area at a predetermined frequency. Since the galvanometer can achieve an extremely high vibration frequency, for example, 2k per second, which is equivalent to scanning out a complete projection structured light in 250 μs, extremely precise synchronization of the reflected light position is required. This precision makes it impossible to directly use the start signal of the MEMS galvanometer for synchronization (because the delay is unreliable). Therefore, considering the characteristics of the phase vibration of the micromirror device, a measuring device for real-time measuring the vibration phase of the galvanometer can be included in the synchronization device, and based on the measurement results of the measuring device, the synchronous turning on of pixel column imaging is performed. Thus, ensuring the synchronization of scanning and imaging at an extremely high frequency. In the implementation of multiple groups of coaxial binoculars of the present invention, it can be required that the galvanometer scans out a complete projection structured light within one millisecond.

[0082] In one embodiment, the above measurement can be based on the emitted light itself. Then, the above measuring device can be one or more photoelectric sensors (e.g., two photodiodes PD), and the two photoelectric sensors are arranged in any of the following ways: arranged on different emission paths of the projection device; arranged on different reflection paths within the projection device; and respectively arranged on the emission and reflection paths inside and outside the projection device. The arrangement mode of the photoelectric sensors can be reasonably selected so that while accurately measuring the phase, it does not affect the normal projection of the structured light. As Figure 5As shown, the PD can be installed inside the projection device, and the instantaneous vibration phase can be determined by measuring the reflection angle when the laser exits the light window. Since the vibration phase of the MEMS galvanometer is sinusoidally distributed, one PD can determine the sinusoidal distribution information, and more PDs help to measure the phase more accurately. In other embodiments, the PD can also be installed outside the projection device, for example, on the light window, such as near the edge of the light window to prevent the influence on the projection within the shooting area. In other embodiments, other methods can also be used for phase measurement, such as capacitance measurement.

[0083] In one embodiment, each image sensor completes the imaging of one image frame after each scan projection of the projection device. For example, after the galvanometer completes half a cycle of vibration to scan the stripe light in the x direction from one side of the shooting area to the other side, the imaging of one image frame (for example, one of the patterns in FIG. 1 or Figure 2 is completed. When the projection power of the projection device is limited, or the object to be measured is far away from the measurement head, the charge amount obtained by the image sensor after a single scan usually cannot be imaged, and multiple scans for imaging are required. Thus, each image sensor completes the imaging of one image frame after the projection device performs a predetermined number of scan projections. For example, the DMD can scan the same structured light pattern within 5 consecutive vibration cycles, so that the image sensor obtains enough charge amount for imaging, and then scans the same next structured light pattern within the next 5 vibration cycles, and so on.

[0084] Although Figure 8 shows a galvanometer used as a reflection device. However, in other embodiments, the reflection device can be a mechanically rotating mirror that rotates unidirectionally and is used to scan and project the linear light generated by the laser generator onto the shooting area at the predetermined frequency. Correspondingly, the measuring device included in the synchronization device can be an angle measuring device for real-time measuring the rotation angle of the motor of the reflection device. The synchronization device can then synchronously start the pixel column imaging based on the measurement result of the angle measuring device.

[0085] In the above embodiments, the synchronization between the scan line and the column pixel exposure is achieved by controlling the exposure of the image sensor. This can be used when the light source scan is controllable (for example, the angle and speed of the mechanical galvanometer can be controlled by voltage and current), and is especially suitable for the case where the phase and speed of the light source scan are uncontrollable (for example, for MEMS galvanometers or mechanical rotating mirrors). Thus, the MEMS galvanometer can use a PD or capacitance to detect the angle, and the mechanical rotating mirror can also achieve position detection through voltage detection or photoelectric coding.

[0086] Whether it is a galvanometer scanner with a controllable rotation speed and reciprocating motion, or a rotating mirror with an uncontrollable rotation speed and unidirectional uniform motion, since its angular velocity can match the line-by-line exposure speed of the rolling shutter image sensor, a line-by-line exposure scheme based on line laser scanning can be achieved.

[0087] According to another embodiment of the present invention, it can also be implemented as a depth data calculation device, including: the depth data measurement head as described above; and a processor connected to the depth data measurement head, configured to determine the depth data of the object to be photographed in the photographing area according to the reference image of the image sensor and the set of image frames obtained by imaging the structured light in a monocular scheme; or determine the depth data of the object to be photographed in the photographing area according to the predetermined relative positions of the first and second image sensors and the set of image frame pairs obtained by imaging the structured light in a binocular scheme.

[0088] According to another embodiment of the present invention, it can also be implemented as a depth data measurement method. Figure 10 The schematic flowchart of the depth measurement method according to an embodiment of the present invention is shown. This method can be implemented using the depth data measurement head of the present invention.

[0089] In step S1010, project structured light (first pattern) onto the photographing area by scanning. In step S1020, use the first sub-image sensor (for example, sub-image sensor 223, or sub-image sensors 323 and 333) to photograph the photographing area to obtain a first image frame under the illumination of the structured light. In step S1030, project second structured light (second pattern) with a different pattern onto the photographing area by scanning. In step S1040, use the second sub-image sensor (for example, sub-image sensor 224, or sub-image sensors 324 and 334) to photograph the photographing area to obtain a second image frame under the illumination of the second structured light, wherein the first and second sub-image sensors share at least part of the optical path and form an image sensor, and the first and second image frames are used for single-time depth data calculation of the photographing area.

[0090] In binocular implementation, step S1020 may include using a first pair of sub-image sensors (e.g., 323 and 333) with a predetermined relative position relationship to capture the shooting area to obtain a first pair of image frames under structured light illumination. Step S1040 may then include using a second pair of sub-image sensors (e.g., 324 and 334) to capture the shooting area to obtain a second pair of image frames under a second structured light illumination, wherein the first and second pairs of sub-image sensors are coaxial. That is, one sub-image sensor in each of the first and second pairs of sub-image sensors shares at least part of the optical path and forms a first image sensor, and the other sub-image sensor in each of the first and second pairs of sub-image sensors shares at least part of the optical path and forms a second image sensor. The first and second pairs of image frames are used for single-shot depth data calculation of the shooting area.

[0091] Scanning and projecting different patterns of the second structured light onto the shooting area includes: projecting the second structured light at a first interval that is less than the frame imaging interval of the sub-image sensor, and the time interval between the first sub-image sensor (pair) capturing the first image frame (pair) and the second sub-image sensor (pair) capturing the second image frame (pair) is less than the frame imaging interval of the sub-image sensor.

[0092] In one embodiment, the method further includes: scanning and projecting a third structured light onto the shooting area, the pattern of the third structured light being different from that of the structured light and the second structured light; using the first or second sub-image sensor (pair) to capture the shooting area to obtain a third pair of image frames under the third structured light illumination, wherein the third pair of image frames is used for single-shot depth data calculation of the shooting area, or using a third pair of sub-image sensors to capture the shooting area to obtain a third pair of image frames under the third structured light illumination.

[0093] Wherein, the third sub-image sensor belongs to the image sensor and shares at least part of the optical path with other sub-image sensors therein. Specifically, in the binocular solution, one sub-image sensor in the third pair of sub-image sensors belongs to the first image sensor and shares at least part of the optical path with other sub-image sensors therein, and the other sub-image sensor in the third pair of sub-image sensors belongs to the second image sensor and shares at least part of the optical path with other sub-image sensors therein, and the third pair of image frames is used for single-shot depth data calculation of the shooting area.

[0094] Scanning and projecting the third structured light onto the shooting area includes: projecting the third structured light at a second interval that is not less than the frame imaging interval of the sub-image sensor from the projection of the structured light, and using the first sub-image sensor (pair) to shoot the shooting area to obtain the third image frame (pair) under the illumination of the third structured light.

[0095] To achieve coaxiality, the first sub-image sensor (pair) and the second sub-image sensor (pair) can each obtain the split light beams of the structured light and the second structured light, and selectively turn on the first sub-image sensor (pair) or the second sub-image sensor (pair) for shooting; or control the optical path of the incident light so that only the first sub-image sensor pair obtains the structured light and shoots, and only the second sub-image sensor pair obtains the second structured light and shoots.

[0096] The present invention can also be implemented as a depth data calculation method, including: obtaining the first and second image frames (pairs), or the first, second, and third image frames (pairs) according to the depth data measurement method as described above; and determining the depth data of the shooting object in the shooting area according to the predetermined relative position and the first and second image frames (pairs), or the first, second, and third image frames (pairs).

[0097] The multi-group coaxial monocular and binocular solutions according to the present invention have been described in detail above with reference to the drawings. By introducing multiple groups of coaxial photosensitive units, this solution can reduce the imaging time required for calculating depth data by multi-frame merging and improve the frame rate.

[0098] Those skilled in the art will also understand that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0099] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems and methods according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0100] The embodiments of the present invention have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A depth data measurement device, comprising a depth data measurement head and a processor connected to the depth data measurement head, The depth data measurement head includes: A projection device for scanning and projecting a set of structured light with different patterns onto a shooting area, where the set of structured light includes structured light with at least two different patterns of stripe coding; An image sensor for shooting the shooting area to obtain a set of image frames under the irradiation of the set of structured light for single-depth data calculation of the shooting area. Wherein, the image sensor includes at least two sub-image sensors sharing at least part of the optical path, and the at least two sub-image sensors are respectively used for imaging the structured light with different patterns successively projected by the projection device; and A synchronization device for, while the projection device projects at least two different patterns of structured light at a first interval less than the frame imaging interval of the sub-image sensor, enabling the at least two sub-image sensors to synchronously image the at least two different patterns of structured light successively at the first interval. Wherein, the first interval is the imaging required time of the sub-image sensor, and is used to enable each sub-image sensor to perform the next frame imaging of itself at a second interval not less than the frame imaging interval of the sub-image sensor and synchronize with the projection of the projection device, And, the processor is used to determine the depth data of the shooting object in the shooting area according to the set of image frames obtained by imaging the structured light.

2. The depth data measurement device according to claim 1, wherein, The image sensor includes: A first and a second image sensor with a predetermined relative position relationship for shooting the shooting area to obtain a set of image frame pairs under the irradiation of the set of structured light for single-depth data calculation of the shooting area, Wherein, each of the first and second image sensors includes at least two sub-image sensors sharing at least part of the optical path, and the at least two sub-image sensors are respectively used for imaging the structured light with different patterns successively projected by the projection device.

3. The depth data measurement device according to claim 1 or 2, wherein The image sensor includes: A lens unit for receiving the incident returned structured light; A beam splitting device for splitting the incident returned structured light into at least a first beam and a second beam; A first sub-image sensor for imaging the first beam; A second sub-image sensor for imaging the second beam corresponding to the returned structured light with different patterns.

4. The depth data measurement device according to claim 1 or 2, wherein The image sensor includes: A lens unit for receiving the incident returned structured light; An optical path conversion device for delivering the incident returned structured light to at least a first sub-path and a second sub-path; A first sub-image sensor for imaging the returned structured light on the first sub-path; A second sub-image sensor for imaging the returned structured light corresponding to different patterns on the second sub-path.

5. The depth data measurement device according to claim 1 or 2, wherein At least two sub-image sensors sharing at least part of the optical path have the same length of optical path.

6. The depth data measurement device according to claim 5, wherein, At least two sub-image sensors sharing at least part of the optical path are pixel-level aligned.

7. The depth data measurement device according to claim 1 or 2, wherein At least two sub-image sensors sharing at least part of the optical path are infrared light sensors; And / or The image sensor includes: A visible light image sensor is used to image incident structured light, wherein the visible light image sensor shares at least part of the optical path with the first and / or second image sub-sensors.

8. The depth data measurement device according to claim 1, wherein, A set of structured light with different patterns projected by the projection device is a set of structured light with different encoded stripes.

9. The depth data measurement device according to claim 1, wherein, The projection device includes: A laser generator for generating line-type and / or dot lasers, and the laser generator performs high-speed switching to scan and project the light and dark alternating structured light corresponding to the stripe encoding.

10. The depth data measurement device according to claim 9, wherein, The projection device includes: A light-emitting device for generating line-type light; and A reflection device for reflecting the line-type light to project the line-type light moving in the direction perpendicular to the stripe direction onto the shooting area.

11. The depth data measurement device according to claim 9, wherein, The at least two sub-image sensors sharing at least part of the optical path are global image sensors; Or The at least two sub-image sensors are rolling shutter image sensors, and the depth data measurement head further includes: A column synchronization device for synchronously turning on the pixel columns corresponding to the current sub-image sensor for imaging and the stripe direction at the current scanning position based on the scanning position of the projection device for imaging.

12. The depth data measurement device according to claim 11, wherein, The at least two sub-image sensors are installed upside down with respect to each other.

13. A depth data measurement method includes: Scanning and projecting structured light onto the shooting area; Using a first sub-image sensor to shoot the shooting area to obtain a first image frame under the illumination of the structured light; Scanning and projecting a second structured light with different patterns onto the shooting area; Using a second sub-image sensor to shoot the shooting area to obtain a second image frame under the illumination of the second structured light, wherein the first and second sub-image sensors share at least part of the optical path and form an image sensor, and the first and second image frames are used for single-depth data calculation of the shooting area; and Determining the depth data of the shooting object in the shooting area according to the first image frame and the second image frame, wherein a set of structured light with different patterns is scanned and projected onto the shooting area, and the set of structured light includes the structured light with stripe encoding and the second structured light, wherein scanning and projecting a second structured light with different patterns onto the shooting area includes: Projecting the second structured light at a first interval less than the frame imaging interval of the sub-image sensor for projecting the structured light, and The time interval between the first sub-image sensor shooting the first image frame pair and the second sub-image sensor pair shooting the second image frame is the imaging time required by the sub-image sensor and less than the frame imaging interval of the sub-image sensor.

14. The method according to claim 13, wherein, Using a first sub-image sensor to shoot the shooting area to obtain a first image frame under the illumination of the structured light includes: Using a pair of first sub-image sensors with a predetermined relative position relationship to shoot the shooting area to obtain a first image frame pair under the illumination of the structured light, and using a second sub-image sensor to shoot the shooting area to obtain a second image frame under the illumination of the second structured light includes: Use a second pair of sub-image sensors to capture the captured area to obtain a second pair of image frames under the illumination of the second structured light. Among them, one sub-image sensor in each of the first and second pairs of sub-image sensors shares at least part of the optical path and forms a first image sensor, and the other sub-image sensor in each of the first and second pairs of sub-image sensors shares at least part of the optical path and forms a second image sensor. The first and second pairs of image frames are used for single-shot depth data calculation of the captured area.

15. The method according to claim 13, further comprising: Scanning and projecting a third structured light onto the captured area, wherein the pattern of the third structured light is different from that of the structured light and the second structured light; Using the first or second pair of sub-image sensors to capture the captured area to obtain a third image frame under the illumination of the third structured light. Among them, the third image frame is used for single-shot depth data calculation of the captured area, or Using a third sub-image sensor to capture the captured area to obtain a third image frame under the illumination of the third structured light. Among them, the third sub-image sensor belongs to the image sensor and shares at least part of the optical path with other sub-image sensors therein, and the third image frame is used for single-shot depth data calculation of the captured area.

16. The method according to claim 15, wherein, Scanning and projecting a third structured light onto the captured area includes: Projecting the third structured light at a second interval that is not less than the frame imaging interval of the sub-image sensor from the projection of the structured light, and Using the first pair of sub-image sensors to capture the captured area to obtain the third pair of image frames under the illumination of the third structured light.

17. The method according to claim 13, wherein, The first sub-image sensor and the second pair of sub-image sensors each acquire split beams of the structured light and the second structured light, and selectively turn on the first sub-image sensor or the second sub-image sensor for capturing; Or Controlling the optical path of the incident light so that only the first pair of sub-image sensors acquire the structured light and perform capturing, and only the second pair of sub-image sensors acquire the second structured light and perform capturing.

Images