Structured light projection device, depth data measurement head, computing device and measurement method

Abstract

Disclosed are a structured light projection device, a depth data measuring head, a computing device, and a measuring method. The structured light projection device includes: a light source module for generating and emitting a textured light beam; a steering projection module including: a reflecting device arranged on the emission path of the light beam, for reflecting the incident light beam so as to emit the light beam; and a driving device connected to the reflecting device, for changing the angle of the reflecting device relative to the incident light beam so as to change the emission direction of the light beam. The present invention uses an improved structured light projection device that can reflect the structured light generated by the light source module at different angles, thereby achieving faster, more economical, and less faulty multi-pattern projection. Furthermore, the structured light projection device can cooperate with multiple pairs of binocular sensors sharing a common optical path, thereby further shortening the frame interval and improving the quality of depth fusion data.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional detection technology, and in particular to a structured light projection device, a depth data measuring head, a computing device and a measuring method. Background Art

[0002] In recent years, three-dimensional imaging technology has flourished. Currently, a binocular detection solution based on structured light can perform three-dimensional measurement of object surfaces in real time. Simply put, the solution first projects a two-dimensional laser texture pattern with coded information, such as a discretized speckle pattern, onto the surface of a natural object. Two image acquisition devices with relatively fixed positions continuously acquire the laser texture. The processing unit uses a sampling window to sample the two images simultaneously acquired by the two image acquisition devices, determines the matching laser texture patterns within the sampling window, and calculates the depth distance of each laser texture sequence segment projected on the surface of the natural object based on the difference between the matching texture patterns. The three-dimensional data of the surface of the object to be measured is then obtained by further measurement.

[0003] In the matching process, the larger the sampling window, the more pattern information is contained in a single sample, making it easier to match, but this results in greater granularity in the resulting depth image. Correspondingly, the smaller the sampling window, the finer the image granularity, but also the greater the mismatch rate. While the sampling window can be reduced by continuously capturing multiple sets of different images, this introduces additional system complexity and reduces the frame rate.

[0004] To this end, an improved structured light projection device and a depth data measurement solution using the device are needed. Summary of the Invention

[0005] One technical problem addressed by this disclosure is to provide a depth data measurement solution that utilizes an improved structured light projection device capable of reflecting structured light generated by a light source module at different angles, thereby enabling faster, more economical, and less error-prone multi-pattern projection. Furthermore, this structured light projection device can be used in conjunction with multiple pairs of binocular sensors sharing a common optical path, further shortening the frame interval and improving the quality of depth fusion data.

[0006] According to a first aspect of the present disclosure, a structured light projection device is provided, comprising: a light source module for generating and emitting a textured light beam; a steering projection module, comprising: a reflecting device arranged on an emission path of the light beam, for reflecting the incident light beam so as to emit the light beam; and a driving device connected to the reflecting device, for changing the angle of the reflecting device relative to the incident light beam so as to change the emission direction of the light beam.

[0007] According to a second aspect of the present disclosure, a depth data measuring head is provided, comprising: a structured light projection device as described in the first aspect of the present invention, configured to project a light beam with a texture into a measured space when driven by a driving device at different projection angles, so as to form different textures on an object to be detected in the measured space; and first and second image sensors respectively arranged on either side of the structured light projection device, the first and second image sensors having a predetermined relative spatial position relationship, and imaging the measured space at least twice during the movement of the reflecting device to obtain at least two sets of images with different texture distributions, wherein the at least two sets of images are used to obtain single-time measured depth data of the object to be detected.

[0008] According to a third aspect of the present disclosure, a depth data calculation device is provided, comprising: the depth data measurement head as described in the second aspect of the present invention, and a processor for acquiring the at least two sets of images, and determining the depth data of the texture in each set of images based on a predetermined relative spatial position relationship between the first and second image sensors, and fusing the depth data determined based on the at least two sets of images to obtain new depth data as single-measurement depth data of the object to be detected.

[0009] According to a fourth aspect of the present disclosure, a depth data measurement method is provided, comprising: performing rotational reflection on a light beam with a speckle pattern emitted by a light source module; imaging a measured space at least twice using first and second image sensors fixed in relative positions to obtain at least two sets of images, wherein different speckle patterns resulting from the rotational reflection are projected onto the measured space in the at least two imaging sessions; and obtaining depth data from the at least two sets of images and performing depth data fusion.

[0010] Thus, by rotating the reflected structured light, the projection flexibility of the structured light projection device is improved. This device can be further combined with a coaxial binocular solution to further improve the accuracy and imaging speed of the multi-frame fusion solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0012] Figure 1 A schematic diagram of the composition of a structured light projection device according to an embodiment of the present invention is shown.

[0013] Figure 2 A schematic diagram showing the changing of the projection direction of the present invention is shown.

[0014] Figure 3FIG. 1 is a perspective view of a structured light projection device according to an embodiment of the present invention.

[0015] Figure 4A -B shows different perspective angles Figure 3 An example of a structured light projection device projecting structured light.

[0016] Figure 5 FIG. 4 is a schematic diagram showing the composition of a depth data measurement head according to an embodiment of the present invention.

[0017] Figure 6 The comparative timing diagrams of coaxial two-group imaging and single-group imaging are shown.

[0018] Figure 7 The timing diagram of three groups of coaxial binocular imaging is shown.

[0019] Figure 8 A schematic flow chart of a depth data measurement method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0021] As mentioned earlier, in binocular imaging matching, a larger sampling window means more pattern information is contained in a single sample, making matching easier. However, this results in greater granularity in the resulting depth image. Conversely, a smaller sampling window results in finer granularity, but also a higher mismatch rate. Therefore, the sampling window can be reduced by capturing multiple sets of different images in succession.

[0022] For example, the same light source module can be driven to rotate by a driving device to achieve projection from different angles. At this time, even if the pattern projected by the light source module is the same, it will still appear as a different pattern in the field of view of the imaging device (image sensor) due to the different projection angles. However, due to the weight of the light source module itself and the need for wiring and power supply, the driving function of the driving device is limited. In addition, multi-frame-based deep fusion will reduce the frame rate and reduce the shooting performance of dynamic objects.

[0023] To this end, the present invention provides a depth data measurement solution that utilizes an improved structured light projection device capable of reflecting structured light generated by a light source module at different angles, enabling faster, more economical, and less error-prone multi-pattern projection. Furthermore, this structured light projection device can be used in conjunction with multiple pairs of binocular sensors sharing a common optical path, further shortening the frame interval and improving the quality of depth fusion data.

[0024] first, Figure 1 A schematic diagram of the composition of a structured light projection device according to an embodiment of the present invention is shown.

[0025] As shown in the figure, the structured light projection device 110 may include a light source module I located above the dotted line and a steering projection module II located below the dotted line. The light source module is used to generate a light beam to be projected, and the steering projection module is used to redirect and emit the light beam.

[0026] Specifically, the light source module is used to generate and emit a textured light beam. Usually, the light source module does not directly project the light beam emitted by the light emitting device, but performs certain optical processing on the light beam projected by the light emitting device to make it present the desired distribution, brightness or pattern. To this end,

[0027] like Figure 1 As shown, the light source module 1 may include a laser generator 111 and a diffractive optical element (DOE) 112. The laser generator 111 is used to emit a laser beam (as indicated by a single arrow in the figure). The DOE 112, arranged on the outgoing optical path of the laser beam, can modulate the incident laser light, for example, diffracting the incident laser light and modulating it into a discrete light spot with a specific projection rule (indicated by a double arrow in the figure to indicate that the diffracted light beam has a certain width).

[0028] In other embodiments, the light source module 1 may also be implemented using other schemes besides laser + diffraction. For example, the light source module may include a floodlight source for generating floodlight, and a mask disposed on the floodlight source for converting the floodlight into discrete light spots with specific projection rules.

[0029] Furthermore, the steering projection module is a module for reflecting the light beam (e.g., speckle pattern) generated by the light source module. However, unlike conventional reflection modules, the steering projection module of the present invention can drive the reflection module to move, thereby causing the light beam reflected by the reflection module to change.

[0030] Specifically, if Figure 1As shown, the steering projection module II may include a reflecting device 113 and a driving device 114. The reflecting device 113, which is arranged on the emission path of the light beam, may be used to reflect the incident light beam so that the light beam is emitted, while the driving device 114 connected to the reflecting device may be used to change the angle of the reflecting device relative to the incident light beam so as to change the emission direction of the light beam.

[0031] In the embodiment of the present invention, for the convenience of description, the direction in which the light exits the measuring head can be designated as the z direction, the horizontal direction of the shooting plane as the x direction, and the vertical direction as the y direction. Figure 1 and as described below Figure 2 、 Figure 3 as well as Figure 4A In Figure 2-B, the example in which the light source module projects structured light downward (in the y-direction) and the deflection projection module redirects the emission direction to the z-direction (actually, a direction slightly offset from the z-direction). It should be understood that in other embodiments, the structured light projection device of the present invention can be arranged or placed in other directions according to imaging requirements.

[0032] For ease of understanding, Figure 2 The figure shows a schematic diagram of changing the projection direction of the present invention. The light emitted by the light source is projected downward along the y-axis to the reflecting device, such as a reflector, and is reflected by the reflecting device to be projected to the measured space, and can form a light spot on the imaging plane perpendicular to the z-axis. The reflecting device can rotate axially along the x-axis, for example, as shown in the figure, within the AB angle range, thereby correspondingly obtaining a light spot moving within the range of A'-B' on the imaging plane. When Figure 1 When the light source module I shown projects a discrete light spot with a two-dimensional distribution pattern, although the pattern projected by the light source module I onto the reflector is the same (because the relative positions of the laser emitter 111 and the DOE 112 are fixed and do not move themselves), the rotation of the reflector causes the projected pattern to have an angular offset. Therefore, when used in combination with binocular imaging as described below, the patterns captured by the image sensor at different projection angles can be regarded as different patterns.

[0033] Although Figure 2 In order to illustrate the principles of the present invention, a reflector that can rotate within a larger AB angle range is shown, but it should be understood that in actual application scenarios, the angle difference of the projection of the same pattern can be very small, for example, 1°, thereby ensuring that the imaging patterns are different while ensuring that the imaging ranges roughly overlap.

[0034] Further, Figure 3 FIG. 1 is a perspective view of a structured light projection device according to an embodiment of the present invention. Figure 4A-B shows different perspective angles Figure 3 An example of a structured light projection device projecting structured light.

[0035] As shown in the figure, the laser generator 311 can be arranged inside the housing (or fixed structure) 315. The light source generated by it can be diffracted by the DOE 312 arranged on the output light path to obtain a diffraction pattern with a certain two-dimensional distribution. The diffraction pattern is transmitted to the reflector 313, and the reflector 313 reflects the diffraction pattern so that it is (roughly) emitted along the z direction and projected on a projection plane perpendicular to the z direction (such as Figure 4A and Figure 4B As shown, Figure 4B Can be seen as Figure 3 and Figure 4A A speckle pattern is formed on the device shown in the view rotated 90° along the y-axis. Figure 4A and Figure 4B The projection plane can be regarded as Figure 1 Three-dimensional representation of the imaging plane shown.

[0036] Furthermore, the driving device can control the reflecting device to move axially, wherein the light beam emitted from the light source module is incident on the reflecting device in a direction perpendicular to the axial direction (x direction), and the emission direction is changed based on the axial movement of the reflecting device.

[0037] As shown in the figure, a rotating shaft extending from a drive device (which may be a motor) 314 is fixedly connected to a reflector 313. When the motor is operating, the rotating shaft drives the reflector 313 to move axially. This creates projection patterns with a certain angular offset on the projection plane. These projection patterns can be different for the image sensor performing the projection capture. Thus, the projection device of the present invention can conveniently project "different" patterns by rotating the reflected structured light (e.g., a diffraction pattern with a two-dimensional distribution).

[0038] In some embodiments, the steering projection module may be a galvanometer, and the motor may drive the reflector to move back and forth within a certain range. In other embodiments, the steering projection module may be a rotating mirror, and the motor may only move in one direction along the axial direction.

[0039] Specifically, the steering projection module can be a mechanical galvanometer that reciprocates at a predetermined frequency, thereby projecting structured light at the predetermined frequency toward the measured area, thereby presenting a two-dimensional diffraction pattern (discrete light spot) that moves up and down along the y-direction on the projection plane. Due to the controllability of the mechanical galvanometer, the steering projection module can also remain stationary during a predetermined window period during the continuous movement.

[0040] For example, the mechanical galvanometer can have a range of motion of ±1° along the z-direction and can have an extremely high vibration frequency of up to 2k per second. When used in conjunction with an image sensor, for example, when synthesizing three frames into one image, the mechanical galvanometer can be made to pause for a period of time when moving to -1°, 0°, and 1°, for example, to remain stationary within the exposure time required by the photosensitive unit (e.g., 1ms). This allows the projected pattern to remain unchanged during the exposure period of the image sensor, thereby improving imaging accuracy.

[0041] In the embodiment using a rotating mirror, although the rotating mirror can only rotate in one direction and is difficult to control at a variable speed, the movement angle of the rotating mirror can be sensed by devices such as photodiodes, and structured light projection and dynamic imaging can be performed within a suitable angle, such as projection and corresponding imaging within a range of ±1° along the z direction in 360°.

[0042] In addition, it should be understood that although Figures 1-4B In the examples given, the projected structured light is projected along the z direction and varies in the vertical direction (y direction), but in other embodiments, the projected structured light may be projected along the z direction and varies in the horizontal direction (x direction), or may be projected with angle changes in both the x and y directions.

[0043] The structured light projection device of the present invention can be used for measuring depth data. Figure 5 FIG2 is a schematic diagram showing the composition of a depth data measurement head according to an embodiment of the present invention. For the sake of simplicity, the figure shows an example of the composition of an image sensor (520) in more detail.

[0044] like Figure 5 As shown, the depth data measuring head 500 based on the binocular principle includes a projection device 510 and a first image sensor 520 and a second image sensor 530 with a predetermined relative position relationship. The projection device 510 can be a combination of the above Figure 1 and Figure 3 A structured light projection device is described.

[0045] Although not shown in the figure for the convenience of explanation, the measuring head 500 may further include a housing for enclosing the above-mentioned device, and Figure 5 The illustrated connection structure 540 can be considered a mechanism for securing the aforementioned devices and connecting them to the housing. In some embodiments, the connection structure 540 can be a circuit board containing control circuitry. It should be understood that in other implementations, the aforementioned devices 510-530 can be connected to the housing in other ways and perform corresponding data transmission and command reception operations.

[0046] Here, the projection device 510 is used to project structured light onto the capture area. For example, the same pattern diffracted by the DOE can be projected at different angles due to the presence of a steering mechanism. This allows the first image sensor 520 and the second image sensor 530, which have a predetermined relative positional relationship, to capture the capture area and obtain a set of image frame pairs with different patterns. This set of image frame pairs can then be used to calculate a single depth data for the capture area.

[0047] Specifically, the first image sensor 520 and the second image sensor 530 can be arranged on either side of the structured light projection device 510. The first and second image sensors have a predetermined relative spatial position relationship, and image the measured space at least twice during the movement of the reflective device to obtain at least two sets of images with different texture distributions, wherein the at least two sets of images are used to obtain single-shot depth data of the object to be detected.

[0048] For example, the projection device 510 can be driven by its driving device to project a textured light beam into the measured space at a continuously changing projection angle to form different textures on the object to be detected in the measured space. The image sensor can perform multiple imaging during the steering process, for example, performing an imaging once each when moving to -1°, 0°, and 1°, thereby obtaining a set of image frame pairs including three pairs (6 frames). These 6 frames of images are used together to calculate the depth data of the shooting area, that is, a depth image of one frame can be calculated.

[0049] In some embodiments, the light source module in the projection device 510 can remain constantly lit during operation, and the image sensor can perform multiple imaging operations at a specific angle or any angle (or any angle within a predetermined range of motion) of the steering device.

[0050] For example, in scenarios where a controllable mechanical rotating mirror is used, corresponding imaging can be performed based on the mirror's rotation angle, such as performing an image at each of -1°, 0°, and 1°. Alternatively, corresponding exposure can be performed based on the measured rotation angle.

[0051] In some embodiments, synchronization between the rotation angle and the image sensor exposure is not necessary. For example, when the mechanical mirror is set to rotate within ±1°, the image sensor can capture a desired set of images at any time and at any interval. As long as the intervals do not completely overlap with the rotation frequency of the mirror, different images can be captured. In other words, since in a binocular scenario, the comparison is between the differences between the same set of images, rather than comparing them with a reference pattern, the specific pattern projected does not need to be specified.

[0052] In other embodiments, the light source module in the projection device 510 can be synchronized with the exposure of the image sensor. In this case, the measurement head 500 can also include a controller for controlling the light source module to light up synchronously when the first and second image sensors are exposed.

[0053] In addition, the controller can also be used to control the driving device to remain stationary during the exposure of the first and second image sensors. Thus, compared with imaging when the driving device is in motion, a clearer projection pattern, such as a discrete light spot, can be obtained.

[0054] In some embodiments of the present invention, the first and second image sensors may be conventional image sensors. However, in other embodiments, the first and second image sensors each include at least two sub-image sensors that share at least a portion of an optical path, and the at least two sub-image sensors are each used to perform one of the at least two imaging operations.

[0055] like Figure 5 As shown, the first image sensor 520 and the second image sensor 530 each include only one photosensitive unit, and each photosensitive unit performs three imaging operations to obtain a set of three pairs (6 frames) of image frame pairs. In the present invention, the first and second image sensors each include at least two sub-image sensors that share at least part of the optical path, and the at least two sub-image sensors are respectively used to image the structured light of different patterns projected successively by the projection device.

[0056] Figure 5An example is shown in which the first and second image sensors each include two sub-image sensors (photosensitive units). As shown, the first image sensor 520 includes sub-image sensors 523 and 554, while the second image sensor 330 includes sub-image sensors 533 and 534. Sub-image sensors 523 and 524 share a common optical path up to the beam-splitting surface of the beam-splitting device 522 and are equidistant from the aforementioned beam-splitting region. Similarly, sub-image sensors 533 and 534 share a common optical path up to the beam-splitting surface of the beam-splitting device 532 and are equidistant from the aforementioned beam-splitting region. In other words, the present invention introduces multiple coaxial binocular structures. Here, sub-image sensors 523 and 533 can be considered as the first group of image sensors (the first binocular group), used to image structured light at one projection angle. Subsequently, sub-image sensors 524 and 534, considered as the second group of image sensors (the second binocular group), can be used to image structured light at another projection angle. In other words, it can be considered that sub-image sensors 524 and 534, which are coaxial with sub-image sensors 523 and 533, respectively, are in place (i.e., have equivalent optical paths) and perform imaging of the latter pattern structured light instead of sub-image sensors 523 and 533. As a result, the imaging interval between two adjacent frames can be independent of the frame interval of each image sensor, and imaging can be performed at a smaller interval.

[0057] To this end, the measurement head 500 may further include a synchronization device configured to cause the first and second image sensors 520 and 530, each including at least two sub-image sensors, to synchronously and sequentially image the at least two different patterns of structured light at a first interval less than the frame imaging interval of the sub-image sensors, while the projection device projects the at least two different patterns of structured light at a first interval less than the frame imaging interval of the sub-image sensors. Accordingly, each sub-image sensor continues to image its next frame at a second interval no less than the frame imaging interval of the sub-image sensor (e.g., imaging at its own frame interval), and the above-described imaging operations can be synchronized with the projection of the projection device under the synchronization of the synchronization device.

[0058] Figure 6 The timing diagram for comparing coaxial two-group imaging and single-group imaging is shown. For ease of explanation, the frame rate of each photosensitive unit (sub-image sensor) can be set to 100 frames / s, the frame interval is 10ms, and the exposure time required for each photosensitive unit can be set to 1ms.

[0059] If the first and second image sensors 520 and 530 are conventional image sensors including only a single photosensitive unit, Figure 1 When the three patterns shown are used to calculate the depth data, Figure 6As shown in the lower part, three imaging operations are required at milliseconds 0, 10, and 20. To synthesize each depth data image, the subject must remain still for 21 milliseconds (making it more difficult to capture moving objects), and the frame rate drops from 100 frames per second to 33.3 frames per second.

[0060] In contrast, if the first and second image sensors 520 and 530 are image sensors of the present invention including two photosensitive units (for example, the first and second image sensors 520 and 530 respectively include sub-image sensors 523 and 524, and sub-image sensors 533 and 534), when depth data calculation is to be performed using three patterns, then Figure 6 As shown in the upper part, the first group of photosensitive units images pattern 1 (for example, the pattern at the first projection angle) at the 0th millisecond, and then the second group of photosensitive units images pattern 2 (for example, the pattern at the second projection angle) at the 1st millisecond. Then, after an interval of 10ms, the first group of photosensitive units images pattern 3 (for example, the pattern at the third projection angle) at the 10th millisecond, thus completing the three imaging required for a depth data image. Subsequently, at the 11th millisecond, the second group of photosensitive units can start the next round of imaging for pattern 1. At the 20th millisecond, the first group of photosensitive units images pattern 2. At the 21st millisecond, the second group of photosensitive units images pattern 3 again. In this way, the interval between imaging of different groups of photosensitive units only needs the time required for interval imaging (for example, 1ms), and the interval between imaging again of the same group of photosensitive units still follows the frame interval time corresponding to the frame rate (for example, 10ms). At this time, by introducing two sets of coaxial binoculars, the synthesis of each depth data image only requires the subject to remain motionless for 11ms (therefore it is easier to capture moving objects), and the frame rate can be maintained at close to 66.6 frames / s.

[0061] Although combined Figure 5 and Figure 6 An example of having two groups of coaxial (co-optical axis) photosensitive units is described, but in other embodiments, each of the first and second image sensors may further include more photosensitive units. Figure 7 The timing diagram of three coaxial binocular imaging groups is shown. In this case, the first and second image sensors can each include three coaxial photosensitive units (sub-image sensors). Figure 7As shown, the first group of photosensitive units images pattern 1 at 0 milliseconds, followed by the second group of photosensitive units imaging pattern 2 at 1 millisecond, and then the third group of photosensitive units imaging pattern 3 at 2 milliseconds. Subsequently, the next round of three-group imaging begins at 10 milliseconds, and the next round of three-group imaging begins at 20 milliseconds, and so on. At this point, by introducing three groups of coaxial binoculars, it only takes 3ms to acquire the three groups (6 frames) of images required to synthesize a depth data image, that is, the subject only needs to remain motionless for 3ms, thus greatly improving the shooting level for moving objects, and the frame rate can be maintained at close to 100 frames / s (in this example, it takes 1003ms, or 1.003 seconds, to shoot 100 frames).

[0062] Therefore, it should be understood that, by simply introducing an additional set of coaxial binocular structures (or monocular structures), the depth data frame rate based on multi-frame synthesis can be doubled and the imaging time of each frame can be shortened. In theory, the same number of coaxial binocular structures as the number of images projected by the projection device can be arranged, so that the framing time of each depth frame and the frame interval of the sensor are only related to the multiple of the exposure time (when the frame interval is greater than the exposure time x the number of coaxial structure groups). For example, in the case of synthesizing depth frames based on four patterns, if the following is used: Figure 5 With two coaxial binoculars, the time required to acquire four frames increases slightly to 12ms, but the frame rate drops to nearly 50 frames per second. However, using four coaxial binoculars reduces the time required to acquire four frames to just 4ms, while maintaining a frame rate close to 100 frames per second. However, the introduction of excessive coaxial structures increases the complexity of image sensor construction, necessitating a compromise between cost, feasibility, and imaging speed.

[0063] In addition, it should be understood that Figure 6 and Figure 7 To illustrate the performance of coaxial imaging, an example is given in which the second sub-image sensor 523 immediately images 1 ms after the first sub-image sensor 522 images. However, in actual applications, the imaging interval between the first and second sub-image sensors also needs to take into account the driving device. Specifically, if imaging is performed directly during the driving process and there is no requirement for the exact projection angle, then the first sub-image sensor can be directly imaged after the first sub-image sensor completes imaging. Figure 6 and Figure 7The second sub-image sensor is shown. However, if a precise projection angle is required, or if the drive device needs to remain stationary during exposure, after the first sub-image sensor completes imaging, it is necessary to wait for the drive device to move to the appropriate position (and / or enter the appropriate motion state, such as being completely stationary) before exposing the second sub-image sensor. However, due to the high speed of the rotating and galvanometer mirrors, the waiting time is relatively short, for example, tens of μs.

[0064] In order to achieve the coaxial configuration of different photosensitive units in the same image sensor, the optical path needs to be designed. Figure 5 In the example of FIG, a coaxial arrangement based on beam splitting is shown. Taking the first image sensor 520 as an example, it may include: a lens unit 521 for receiving incident return structured light; a beam splitting device 522 for splitting the incident return structured light into at least a first beam and a second beam; a first sub-image sensor 523 for imaging the first beam; and a second sub-image sensor 524 for imaging the second beam corresponding to a different pattern of return structured light.

[0065] In one embodiment, the beam splitting device 522 is an optical prism, such as a square prism or a triangular prism, so that the reflected infrared light in the incident light reaches the second sub-image sensor 524, while the unreflected visible light in the incident light can travel in a straight line to the first sub-image sensor 523.

[0066] As shown in the figure, the beam splitter 522 in the form of a prism can split the incident light into two beams with mutually perpendicular propagation directions. Accordingly, the first sub-image sensor 523 and the second sub-image sensor 524 can also be arranged vertically so as to receive the incident visible light and infrared light beams at perpendicular angles.

[0067] To eliminate parallax and achieve pixel-level alignment, the components of the incident light must have the same optical path length. To this end, when a quadrangular prism is used as the beam splitting device 522, the first sub-image sensor 523 and the second sub-image sensor 524 can be placed at equal distances from the beam splitting region of the beam splitting device 522. When a triangular prism is used as the beam splitting device 522, the distances between the two photosensitive units and the beam splitting device 522, particularly the beam splitting region, can be flexibly adjusted based on the refractive index ratio between air and the prism material.

[0068] Pixel-level alignment between the first sub-image sensor 523 and the second sub-image sensor 524 can theoretically be achieved by ensuring that the incident light shares a majority of the optical path and has the same optical length. However, during the actual manufacturing process of image sensors, the actual arrangement of the first sub-image sensor 523 and the second sub-image sensor 524 may not be ideally perpendicular and equidistant, resulting in deviations between the two images. In this case, forced software correction can be performed on the manufactured image sensors. For example, by introducing a calibration target and aligning the images of the first sub-image sensor 523 and the second sub-image sensor 524 with the calibration target, true pixel-level correction can be achieved. In other words, the pixel-level alignment between the first sub-image sensor 523 and the second sub-image sensor 524 can be precise pixel-level alignment or alignment with a few pixel differences achieved through calibration.

[0069] As shown in the figure, the image sensor 520 of the present invention can be implemented as a separate module. To this end, the image sensor 520 may also include a housing for fixing the relative positions of the lens unit, the beam splitting device, and the two photosensitive units. Preferably, the housing can be combined with the lens unit 521 to form a sealed body to prevent contamination of the contained components by the external environment. In other embodiments, the image sensor 520 of the present invention can be part of a larger module (e.g., a depth data measurement head), and the housing of the larger module can be used to fix the various components.

[0070] Preferably, the image sensor 520 may further include cables connected to the first sub-image sensor 523 and the second sub-image sensor 524. The housing has an opening for accessing the cables. In one embodiment, the cables may be flexible cables, such as FPC (flexible printed circuit) cables.

[0071] In one embodiment, the light beam may also pass through a filter before entering the first sub-image sensor 523 and the second sub-image sensor 524 to further filter out the influence of light of other wavelengths. In one embodiment, the projection device may project an infrared laser, so the filter arranged in the image sensor may be a corresponding infrared light transmission unit for transmitting infrared light in a specific frequency range, such as infrared light with a wavelength of 780-1100nm used in the present invention. In other embodiments, the projection device may also project visible light, such as a red laser or a blue laser, such as 635nm red light or 450nm blue light. Although the ambient light may also include red light or blue light, due to the short exposure time and the strong instantaneous light intensity of the laser, high signal-to-noise ratio imaging can also be performed with the help of the corresponding filter that projects red light or blue light.

[0072] Preferably, when the beam splitting device is a square prism, one side of the filter can be in direct physical contact with the square prism, and the other side can be in physical contact with the photosensitive unit, and the photosensitive unit and the square prism are snapped into the housing, thereby ensuring the high invariance of the relative positions of each component.

[0073] In some embodiments, especially when the first and second sub-image sensors are infrared light sensors for receiving the projected infrared pattern, an additional visible light sensing unit (not shown in the figure) may be arranged in the image sensor to capture image information of the object being measured, so that the image captured by the image sensor contains both image information and depth information of the object being measured. The visible light sensing unit may be a grayscale sensor or a color sensor. The grayscale sensor only captures brightness information, while the color sensor can be used to capture color information of the object being measured. In this case, the visible light sensing unit may be composed of three primary color sensing units, where the three primary colors may be red, green, and blue (RGB) or cyan, red, and yellow (CMY).

[0074] It should be understood that although based on Figure 5 The structure of the first image sensor 520 is specifically described, but the second image sensor 530 can also have the same structure. In addition, it should be understood that 523 and 533 can be regarded as the first set of binoculars, and 524 and 534 as the second set of binoculars. However, 523 and 534 can also be regarded as the first set of binoculars, and 524 and 533 as the second set of binoculars, as long as the corresponding pattern is incident and imaging is turned on.

[0075] In such Figure 5 In the case of using beam splitting to achieve light path sharing, since the light received by each photosensitive unit will be reduced, the imaging sensitivity or effective distance range can be ensured by increasing the projection brightness or expanding the incident aperture.

[0076] To this end, as an alternative, optical path sharing can also be achieved based on optical path conversion. In this case, the first and second image sensors 520 and 530 can each include: a lens unit for receiving the incident return structured light; an optical path conversion device for transmitting the incident return structured light to at least a first sub-path and a second sub-path; a first sub-image sensor for imaging the return structured light on the first sub-path; and a second sub-image sensor for imaging the return structured light corresponding to different patterns on the second sub-path. In one embodiment, the optical path conversion device can be a rotating mirror, which can, for example, reflect the incident light to the photosensitive unit 523 at the 0th millisecond, reflect the incident light to the photosensitive unit 524 at the 1st millisecond, and so on. In other embodiments, the optical path conversion device can also be a device that performs optical path conversion based on other mechanical, chemical, or electrical principles.

[0077] In the case where the projection device performs full pattern projection (rather than scanning projection), although the image sensor can be a rolling shutter image sensor, it is preferably implemented as a global image sensor (ie, all pixels perform imaging simultaneously).

[0078] As mentioned above, the projection device may include a galvanometer that vibrates back and forth at a predetermined frequency, such as a MEMS galvanometer or a mechanical galvanometer, for scanning and projecting structured light toward the shooting area at a predetermined frequency and range of motion. Since the galvanometer can achieve an extremely high vibration frequency, for example, 2k per second, it is impossible to directly use the start signal of the MEMS galvanometer for synchronization (because the delay is unreliable). Therefore, in scenarios where synchronization is required (for example, to know the rotation angle), considering the characteristics of the phase vibration of the micromirror device, a measuring device for real-time measurement of the vibration phase of the galvanometer may be included in the synchronization device.

[0079] In one embodiment, the above-mentioned measurement can be based on the outgoing light itself. Thus, the above-mentioned measurement device can be one or more photosensors (e.g., two photodiodes PD), and the two photosensors can be arranged in any of the following ways: on different outgoing paths of the projection device; on different reflection paths within the projection device; or on the outgoing and reflection paths inside and outside the projection device. The arrangement of the photosensors can be appropriately selected so that they accurately measure the phase while not affecting the normal projection of the structured light. The PDs can be installed within the projection device, and the instantaneous vibration phase can be determined by measuring the reflection angle of the laser light when it exits the light window. Because the vibration phase of the MEMS galvanometer has a sinusoidal distribution, a single PD can determine the sinusoidal distribution information, and more PDs can facilitate more accurate phase measurement. In other embodiments, the PDs can also be installed outside the projection device, for example, on the light window, such as near the edge of the light window to prevent interference with the projection within the capture area. In other embodiments, other methods can be used for phase measurement, such as capacitance measurement.

[0080] However, in other embodiments, the projection device may include a unidirectionally rotating mechanical mirror. Accordingly, when synchronization is required, the measuring device included in the synchronization device may be an angle meter for measuring the rotation angle of the motor of the reflection device in real time.

[0081] In the above embodiment, synchronization between projection and exposure is achieved by controlling the exposure of the image sensor. This can be used when the projection angle of the light source is controllable (for example, the angle and speed of the mechanical galvanometer can be controlled by voltage and current), and is particularly suitable for situations where the phase and speed of the light source scanning are uncontrollable (for example, for MEMS galvanometers or mechanical rotating mirrors). Therefore, MEMS galvanometers can use PD or capacitance to detect angles, and mechanical rotating mirrors can also achieve position detection through voltage detection or photoelectric encoding.

[0082] According to another embodiment of the present invention, it can also be implemented as a depth data calculation device, including: the depth data measurement head as described above; and a processor connected to the depth data measurement head, for determining the depth data of the object in the shooting area based on the predetermined relative positions of the first and second image sensors and the set of image frame pairs obtained by their structured light imaging under a binocular scheme.

[0083] The present invention can also be implemented as a depth data measurement method. Figure 8 FIG2 is a schematic flow chart of a depth data measurement method according to an embodiment of the present invention. The method can be implemented in combination with the structured light projection device, measurement head, and computing device of the present invention.

[0084] In step S810 , the light beam with the speckle pattern emitted by the light source module is rotated and reflected.

[0085] In step S820, the measured space is imaged at least twice using first and second image sensors fixed relative to each other to obtain at least two sets of images. In the at least two imaging steps, different speckle patterns resulting from the rotational reflection are projected onto the measured space. In step S830, depth data is obtained from the at least two sets of images and depth data fusion is performed.

[0086] In the case of using a coaxial binocular structure, imaging the measured space at least twice using first and second infrared light image sensors with fixed relative positions includes: using a first sub-image sensor pair with a predetermined relative position relationship to perform a first imaging to obtain a first image frame pair; using a second sub-image sensor pair to perform a second imaging to obtain a second image frame pair, wherein one sub-image sensor in each of the first and second sub-image sensor pairs shares at least a portion of the optical path and constitutes the first image sensor, and the other sub-image sensor in each of the first and second sub-image sensor pairs shares at least a portion of the optical path and constitutes the second image sensor, and the first and second image frame pairs are used for single depth data calculation of the shooting area.

[0087] The structured light projection device, depth data measurement head, computing device, and measurement method according to the present invention have been described in detail above with reference to the accompanying drawings. The depth measurement solution of the present invention uses an improved structured light projection device that can reflect the structured light generated by the light source module at different angles, thereby achieving faster, more economical, and less faulty multi-pattern projection. Furthermore, the structured light projection device can be used in conjunction with multiple pairs of binocular sensors sharing a common optical path, thereby further shortening the frame interval and improving the quality of the depth fusion data.

[0088] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0089] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A depth data measuring head, comprising: A structured light projection device, used to rotate and reflect the light beam with a speckle pattern emitted by the light source module to form different discrete light spots on the object to be detected in the measured space; a first and a second image sensor respectively arranged on either side of the structured light projection device, the first and second image sensors having a predetermined relative spatial positional relationship with each other, and imaging the measured space at least twice during the movement of the reflective device to obtain at least two sets of images with different discrete light spots, wherein the at least two sets of images are used to obtain single-shot measured depth data of the object to be detected, wherein the first and second image sensors each include at least two sub-image sensors that share at least a portion of the optical path, and the at least two sub-image sensors are each used to perform one of the at least two imaging steps; and a controller configured to cause each of the first and second image sensors, including at least two sub-image sensors, to synchronously and sequentially perform imaging at a first interval that is smaller than a frame imaging interval of the sub-image sensors, and to cause each sub-image sensor to perform its own next frame imaging at a second interval that is not smaller than the frame imaging interval of the sub-image sensor; Wherein, the structured light projection device includes: A light source module, used for generating and emitting a light beam with texture; Steering projection module, including: The reflecting device is arranged on the emission path of the light beam, and is used to reflect the incident light beam so that the light beam can be emitted; A driving device connected to the reflecting device is used to change the angle of the reflecting device relative to the incident light beam to change the emission direction of the light beam.

2. The depth data measuring head according to claim 1, wherein: The driving device controls the reflecting device to move along the axial direction, wherein the light beam emitted from the light source module is incident on the reflecting device in a direction perpendicular to the axial direction and changes the emission direction based on the axial movement of the reflecting device.

3. The depth data measuring head according to claim 1, wherein: The steering projection module is a mechanical galvanometer, and the reflecting device reciprocates along the axial direction; or The steering projection module is a mechanical rotating mirror, and the reflecting device performs unidirectional movement along the axial direction.

4. The depth data measuring head according to claim 1, wherein: The steering projection module remains stationary during a predetermined window period during the continuous movement.

5. The depth data measuring head according to claim 1, further comprising: The controller is used to control the light source module to light up synchronously when the first and second image sensors are exposed.

6. The depth data measuring head according to claim 1, further comprising: The controller is configured to control the driving device to remain stationary during the exposure of the first and second image sensors.

7. The depth data measuring head according to claim 1, wherein: The first and second image sensors each include: a lens unit, configured to receive incident return structured light; a beam splitting device for splitting the incident return structured light into at least a first light beam and a second light beam; a first sub-image sensor, configured to image the first light beam; The second sub-image sensor is used to image a second light beam corresponding to the returned structured light of a different pattern.

8. The depth data measuring head according to claim 7, wherein: The first sub-image sensor and the second sub-image sensor are at equal distances from the beam splitting region of the beam splitting device.

9. The depth data measuring head according to claim 1, wherein: The first and second image sensors each include: a lens unit, configured to receive incident return structured light; an optical path conversion device for transmitting the incident return structured light to at least a first subpath and a second subpath; a first sub-image sensor, configured to image the returned structured light on a first sub-path; The second sub-image sensor is used to image the returned structured light corresponding to the different pattern on the second sub-path.

10. The depth data measuring head according to claim 9, wherein: The first sub-image sensor and the second sub-image sensor are equidistant from the optical path conversion region of the optical path conversion device.

11. The depth data measuring head according to claim 7 or 9, wherein: The first sub-image sensor and the second sub-image sensor are aligned at a pixel level.

12. The depth data measuring head according to claim 7 or 9, wherein: The at least two sub-image sensors included in each of the first and second image sensors and sharing at least a portion of the optical path are infrared light sensors; and / or The first and second image sensors each include: A visible light image sensor is configured to image incident structured light, wherein the visible light image sensor shares at least a portion of an optical path with the first and / or second image sub-sensor.

13. A depth data calculation device, comprising: The depth data measuring head according to any one of claims 1 to 12, and A processor is configured to acquire the at least two sets of images, determine depth data of the texture in each set of images based on a predetermined relative spatial position relationship between the first and second image sensors, and fuse the depth data determined based on the at least two sets of images to obtain new depth data as single-measurement depth data of the object to be detected.

14. A depth data measurement method, comprising: Rotating and reflecting the light beam with the speckle pattern emitted by the light source module; Imaging the measured space at least twice using first and second image sensors fixed in relative positions to obtain at least two sets of images, wherein different speckle patterns exhibited by the rotational reflection are projected onto the measured space in the at least two imaging steps; Obtaining depth data from the at least two sets of images and performing depth data fusion, The method of imaging the measured space at least twice using first and second infrared light image sensors fixed in relative positions includes: Performing a first imaging operation using a first pair of sub-image sensors having a predetermined relative positional relationship to acquire a first pair of image frames; A second imaging is performed using the second sub-image sensor pair to acquire a second image frame pair, wherein one sub-image sensor in each of the first and second sub-image sensor pairs shares at least a portion of the optical path and constitutes a first image sensor, and the other sub-image sensor in each of the first and second sub-image sensor pairs shares at least a portion of the optical path and constitutes a second image sensor, and the first and second image frame pairs are used for single depth data calculation of the captured area. In which, the first sub-image sensor and the second sub-image sensor perform imaging successively at a first interval, which is smaller than the frame imaging interval of the sub-image sensor, and each sub-image sensor performs its own next frame imaging at a second interval that is not smaller than the frame imaging interval of the sub-image sensor.

Images