Coordinate measurement device with an indirect time of flight sensor

Abstract

Examples described herein provide a system that includes a housing, a projector, an indirect time-of-flight (ToF) camera, and one or more processors operably coupled to the housing, the projector, and the indirect ToF camera. The one or more processors are configured to determine dimensional (3D) coordinates of a point on the object by performing operations that include: capturing, while the housing is moving, a number n of phase images from varying poses, where n is a positive integer; determining for each of the number n of phase images a pixel of the array of pixels corresponding to a defined viewing direction in space; and computing a depth value of a defined viewing angle based at least in part on the determined pixels of the array of pixels in the number n of phase images, wherein the depth value represents a depth coordinate of the point on the object.

Description

COORDINATE MEASUREMENT DEVICE WITH AN INDIRECT TIME OFFLIGHT SENSORCROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application is related to the Provisional Patent Application Serial No. 63 / 519092 entitled, “COORDINATE MEASUREMENT DEVICE WITH AN INDIRECT TIME OF FLIGHT SENSOR” filed August 11, 2023, the contents of which are incorporated herein by reference.BACKGROUND

[0002] The subject matter disclosed herein relates to a system for measuring three-dimensional (3D) coordinates in an environment, and in particular to a system and method for measuring a pattern of light using a time of flight sensor.

[0003] A traditional time-of-flight (ToF) scanner is a scanner in which the distance to a target point is determined based on the speed of light in air of a beam of light traveling between the scanner and a target point. Traditional ToF scanners are typically used for scanning closed or open spaces such as interior areas of buildings, industrial installations and tunnels. They are used, for example, in industrial applications and accident reconstruction applications. A laser scanner optically scans and measures objects in a volume around the scanner through the acquisition of data points representing object surfaces within the volume. Such data points are obtained by transmitting a beam of light onto the objects and collecting the reflected or scattered light to determine the distance, two-angles (i.e., an azimuth and a zenith angle), and optionally a gray-scale value. This raw scan data is collected, stored and sent to a processor or processors to generate a 3D image representing the scanned area or object. For the case in which the light source within a scanner is a laser, such a scanner is often referred to as a laser scanner. The term laser scanner is often also used for scanners that use light sources that are not lasers, such as light sources using superluminescent diodes for example.

[0004] ToF measuring systems such as those used in laser scanners are typically one of two types: a phased-based ToF scanner or a pulsed ToF scanner. In a typical phase-based ToF scanner, a beam of light is modulated at a plurality of frequencies before being launched to a target. After the modulated beam of light has completed a round trip to and from the target, it is demodulated to determine the returning phase of the each of the plurality of frequencies. A processor within the ToF scanner uses the demodulated frequencies and the speed of light in air to determine a distance from the scanner to the target. In contrast, a pulsed ToF scanner typically emits a short pulse of light and measures the elapsed time between launch of the pulse and return of the pulse after having completed a round trip to the target. A processor within the pulsed ToF scanner determines the distance from the scanner to the target based at based at least in part on the measured elapsed time and the speed of light in air. The ToF scanners used in laser scanners today typically include a single optical detector that measures the signal returned from the target. Such optical detectors typically measure up to frequencies of several hundred MHz or down to pulse widths of a few picoseconds to nanoseconds.

[0005] More recently, ToF methods are being employed in camera sensors having a collection or array of photosensitive elements. Each of the photosensors in the array serves the same function as the single optical detector in a traditional ToF laser scanner, but the photosensors typically are more limited in the speed of their response and their optical bandwidths. On the other hand, arrays of photosensors are relatively inexpensive, thereby offering advantages where the range and accuracy requirements are not as stringent as for traditional laser scanners.

[0006] A device that uses an array of sensors to measure a stream of modulated light may be said to be an indirect ToF (or iToF) device, while a device that uses an array of sensors to measure pulsed light may be said to be a direct ToF (or dToF) device. If an array of pixels using a ToF is included within a camera having a camera lens, then both distances to the target points may be determined based on the signals received by the array of pixels. One difficulty faced by an iToF-based 3D measuring system is determining the number of complete phase cycles that have elapsed before a projectedbeam of light reflects off an object and travels to the iToF camera. Another difficulty faced by an iToF-based 3D measuring system is in sorting out the different projected pattern elements received by pixels of the iToF camera.

[0007] While existing systems for measuring distance and angles to an object are suitable for their intended purposes, the need for improvement remains, particularly in providing 3D measurement system having the features described herein.BRIEF DESCRIPTION OF THE DISCLOSURE

[0008] In one embodiment, a system is provided. The system includes a housing and a projector. The projector is operably coupled to the housing, the projector configured to project onto an object a light that includes discrete pattern elements having a first temporal modulation, the first temporal modulation including at least a whole number of adjoined modulation cycles, each adjoined modulation cycle having a first period. The system further includes an indirect time-of-flight (ToF) camera coupled to the housing, the indirect ToF camera having a lens and a two-dimensional array of pixels, each pixel configured to convert reflected light into an electrical signal. The system further includes one or more processors operably coupled to the housing, the projector and the indirect ToF camera, the one or more processors configured to determine three-dimensional (3D) coordinates of a point on the object by performing operations. The operations include capturing, while the housing is moving, a number n of phase images from varying poses, where n is a positive integer. The operations further include determining for each of the number n of phase images a pixel of the array of pixels corresponding to a defined viewing direction in space. The operations further include computing a depth value of a defined viewing angle based at least in part on the determined pixels of the array of pixels in the number n of phase images, wherein the depth value represents a depth coordinate of the point on the object.

[0009] In another embodiment, a computer-implemented method for determining a depth value for a three-dimensional (3D) coordinate of an object is provided. The method includes acquiring, using a measuring device, a first phase frame at a first position of the measuring device within an environment, the measurement device comprising an indirect time of flight (iToF) sensor. The method further includes acquiring, using the measuring device, a second phase frame at a second position of the measuring device within the environment. The method further includes determining whether a pixel in the second frame corresponds to a location that is detected in the first frame. The method further includes, responsive to determining that the pixel in the second frame corresponds to the location that is the same in the first frame, computing the depth value.

[0010] In another embodiment, a system is provided. The system includes a projector operably coupled to the housing, the projector configured to project onto an object a light that includes discrete pattern elements having a first temporal modulation, the first temporal modulation including at least a whole number of adjoined modulation cycles, each adjoined modulation cycle having a first period. The system further includes a first indirect time-of-flight (ToF) camera coupled to the housing, the first indirect ToF camera having a lens and a two-dimensional array of pixels, each pixel configured to convert reflected light into a first electrical signal. The system includes a second indirect ToF camera coupled to the housing, the second indirect ToF camera having a lens and a two-dimensional array of pixels, each pixel configured to convert reflected light into a second electrical signal. The system includes one or more processors operably coupled to the housing, the projector and the indirect ToF camera, the one or more processors configured to determine three-dimensional (3D) coordinates of a point on the object by performing operations. The operations include capturing a first phase image using the first indirect ToF camera. The operations further include capturing a second phase image using the second indirect ToF camera. The operations further include computing a depth value based at least in part on the first phase image and the second phase image.

[0011] These and other advantages and features will become more apparent from the following description taken in conjunction with the drawings.BRIEF DESCRIPTION OF DRAWINGS

[0012] The subject matter, which is regarded as the disclosure, is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features, and advantages of the disclosure are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:

[0013] FIG. l is a block diagram showing elements of a system for measuring three-dimensional coordinates in accordance with an embodiment;

[0014] FIG. 2A is a schematic illustration of one type of projector for use in an indirect time-of-flight (iToF) measurement system according to an embodiment;

[0015] FIG. 2B is a schematic illustration of a collection of spots projected onto a surface according to an embodiment;

[0016] FIG. 3 A is a schematic representation of a system having a projector that projects onto a surface a pattern captured by an iToF distance measuring device according to an embodiment;

[0017] FIG. 3B illustrates a method for using triangulation to eliminate range ambiguity in distance measurements of the system of FIG. 3 A;

[0018] FIG. 3C illustrates a method for using triangulation to reduce correspondence ambiguity in measurement by the system of FIG. 3 A;

[0019] FIG. 3D is a schematic illustration showing a projected spot of light displayed in relation to multiple pixels of an iToF sensor;

[0020] FIG. 4 A is a schematic representation of a system having a projector that projects onto a surface a pattern captured by two iToF distance measuring devices according to an embodiment;

[0021] FIG. 4B illustrates a method for using triangulation to eliminate range ambiguity in distance measurements of the system of FIG. 4 A;

[0022] FIG. 4C illustrates a method for using triangulation to reduce correspondence ambiguity in a measurement by the system of FIG. 4 A;

[0023] FIG. 5A illustrate a triangulation method for determining 3D coordinates of an object surface placed relatively near a 3D measuring device having a single projector and a single TOF camera according to an embodiment;

[0024] FIG. 5B illustrates a triangulation method for determining 3D coordinates of an object surface placed near a 3D measuring device having a single projector and two TOF cameras according to an embodiment;

[0025] FIGs. 6A and 6B are graphical illustrations of a waveform that might be emitted by a projector and received by an iToF sensor, respectively, according to an embodiment;

[0026] FIG. 6C is a graphical representation of signals produced by returning light for a first phase condition of zero degrees, and FIG. 6D is a graphical representation of signals produced by returning light for a second phase condition of 90 degrees, according to an embodiment;

[0027] FIG. 7A is a graphical representation of the FOV of a single color camera added to a 3D measuring instrument having two TOF devices according to an embodiment;

[0028] FIGs. 7B, 7C are illustrations of arrangements for two color cameras for placement in an TOF measuring device according to an embodiment;

[0029] FIGs. 8A, 8B, 8C are graphical representations showing the overlap in coverage of two two-dimensional (2D) image sensors when rotating around a vertical axis and stopping four, five, and six times, respectively, in covering 360 degrees according to an embodiment;

[0030] FIG. 9A shows an image captured by a camera without correction for lens aberrations and other camera parameter errors according to an embodiment;

[0031] FIG. 9B shows an image captured by a camera with correction to remove lens aberrations and other camera parameter errors according to an embodiment;

[0032] FIG. 10 is an isometric view of a 3D measuring system according to an embodiment;

[0033] FIG. 11 is a view of a 3D measuring system according to another embodiment;

[0034] FIG. 12 is a block diagram of elements within a system processing unit according to another embodiment;

[0035] FIG. 13 depicts a flow diagram of a method for computing a depth measurement for a moving measuring device according to one or more embodiments described herein; and

[0036] FIG. 14 depicts a flow diagram of a method for determining a depth value for a 3D coordinate of an object using a moving measuring device according to one or more embodiments described herein.

[0037] The detailed description explains embodiments of the disclosure, together with advantages and features, by way of example with reference to the drawings.DETAILED DESCRIPTION OF THE DISCLOSURE

[0038] Embodiments of the present disclosure provide for a low cost system for measuring three-dimensional (3D) coordinates and generating a dense 3D point cloud. Embodiments of the present disclosure provide for a sensor array that allows for measurement of 3D coordinates over an area based at least in part on the speed of light as the sensor array is rotated about an axis. Still further embodiments of the present disclosure provide for emitting a pattern of light onto one or more surfaces in the environment and measuring 3D coordinates of elements of the pattern using an iToF or dToF sensor.

[0039] Embodiments of the present disclosure further provide for a low-cost system that uses triangulation to reduce ambiguities in measured distances and also to reduce ambiguities among imaged points and measured distances. Other embodiments provide for fast and cost-effective 3D scanning measurements.

[0040] Embodiments described herein further provide for performing an iToF measurement using a number of image frames that are not recorded with the same viewing direction but cover different fields of view.

[0041] Referring now to FIG. 1, an embodiment of a system 100 is shown for measuring 3D coordinates on objects, also referred to as measuring 3D coordinates of surfaces in an environment. The system 100 includes a housing 102 configured to rotate about one or more axes. The housing 102 can be a structural member, such as a structural frame, configured to house and / or mount one or more components. One or more rotation mechanisms rotate the housing 102 about a corresponding axis. An example is the rotation mechanism 104 that rotates the system 100 about a vertical axis (e.g. an axis perpendicular to a plane of the floor in the environment), for example. Each rotation mechanism 104 optionally includes an angular transducer 106 configured to read the angle of rotation of the housing 102 about a particular axis. An example of an angular transducer 106 is an angular encoder. In an embodiment, the rotation mechanism 104 is optionally coupled to a support structure such as a tripod (not shown), for example. In an embodiment, the rotation mechanism includes a motor (not shown)that allows for selective rotation of the housing 102. In an embodiment, the housing 102 is selectively rotated in incremental steps (e.g., a predetermined angular rotation) and paused for predetermined amounts of time. In still another embodiment, the housing 102 is continuously rotated at a predetermined speed.

[0042] Within or coupled to the housing 102 is a measurement device 110. In an embodiment, the measurement device 110 includes a projector 112 that emits a patterned light beam onto an object surface. The patterned light beam reflects off the object surface and is received by the ToF camera 114, which measures distances from the measurement device 110 to illuminated points on the object surface, the measured distance values being partly based on timing signals shared by the ToF camera 114, the clock / timer 116, the projector 112, and the processor 118, the rotation mechanism 104, and the optional angular transducer 106.

[0043] The ToF camera 114 further includes a lens 118 and a ToF sensor 120. It should be appreciated that while a single lens element is illustrated, this is for example purposes and the claims should not be so limited. In other embodiments, the lens 118 may be a set of optical components. In some cases, the ToF sensor 120 is used in either an iToF or dToF mode. For example, FRAMOS offers an FSM-IMX570 development kit for Jetson AGX Orin / Xavier. This development kit demonstrates the iToF mode using the Sony IMX570PLR-C ToF sensor. The demonstration kit includes drives and a demonstration application for NVIDIA® Jetson AGX Orin™ and AGX Xavier™. Sony Semiconductor Solutions Corporation is located in the Kanagawa Prefecture of Japan. FRAMOS has its headquarters in Taufkirchen, Germany.

[0044] Referring now to FIG. 2 A, an embodiment is shown of projector 200, schematically represented in a cross-sectional view. In an embodiment, the projector 200 is a projector comprising a diffractive optical element (DOE) assembly. The projector 200 includes a vertical cavity surface-emitting laser (VCSEL) array 202. Light from laser emitter elements 204 of the VCSEL array 202 pass through a lens 208, which in an embodiment is an aspherical lens. The light continues on to a mask element 212 having a mask pattern 214 and passes onto a detector plane 220. In an embodiment,a front view of the detector plane 220 is shown in FIG. 2B. The resulting pattern 230 produced by the action of the DOE 212 comprises a collection of spots 232 that form a larger pattern 236. In other embodiments, other patterns are produced at the detector plane 220 according to the mask pattern 214 without deviating from the teachings herein. The light source is chosen to be a VCSEL array 202 because of the relatively high spatial coherence of every single VCSEL and relatively low temporal coherence of the VCSEL array. Further, the use of a VCSEL array provides further advantages in providing for high optical power at relatively low cost. These properties of the VCSEL array enable the bundles of light emitted by the VCSEL arrays to be relatively narrow while still reducing or minimizing the effects of laser speckle.

[0045] In other embodiments, other DOE-type projectors are used. For example, Digigram (headquarters in Saint Martin, France), Himax (headquarters in Tainan City, Taiwan) and AMS-Osram (headquarters in Premstatten, Pennsylvania) offer VCSEL based projectors that are commercially available. In other embodiments, projectors are made using metalenses, which are flat lenses that use metasurfaces to focus light. Metalenses comprise collections of artificial antennae that manipulate the amplitude phase and polarization of the incident light. Examples of companies that make metalenses are Coherent (headquarters in Saxonburg, Pennsylvania) and Metalenz (headquarters in Boston, Massachusetts). Popular VCSEL arrays for use in laser projectors are manufactured by Lumentum Operations (headquarters in San Jose, California).

[0046] FIG. 3A shows a device 300 that measures distance and angles to an object having a surface 342 shown in FIG. 3B and FIG. 3C. The device 300 includes a projector 302, a clock / timer 320, a processor 330, and a TOF camera 310. In some embodiments, the TOF camera 310 includes the clock / timer 320 having timing signals sent to other components in the device 300. The TOF camera 310 includes a lens and a TOF sensor, which is used in an iTOF or dTOF mode. In an embodiment, the projector 302 projects a pattern 312 of light, enabling the light to be concentrated in spots 314 or other patterns having higher power per unit area than would be possible were the lightprojected homogeneously over a wide angular region. This higher power per unit area enables more accurate measurements to be made at a given limited measurement rate.

[0047] In an embodiment, the projector 302 is spaced apart from the camera 310 by a baseline distance. Triangulation calculations are carried out by the processor 330, which include one or more computers external to the device 300 as well as processing elements internal to the TOF camera 310. A geometries representative of the system 300 is shown in FIG. 3B and FIG. 3C. In an embodiment, a compensation measurement is made prior to measuring in the field. Such compensation measurements may be carried out in a factory during device assembly or in the field at a preliminary stage using compensation aids such as calibrated dot plates. In other embodiments, the compensation measurements are performed in the field (e.g. outside of a controlled manufacturing setting) using a standard scanning process or a predetermined- altemative scanning process without the use of compensation aids.

[0048] Referring now to FIG. 3C, in an embodiment, compensation measurements assist in determining several characteristics of the measuring device 300. These characteristics include a baseline distance LBof a baseline 390 that extends from a perspective center 304 of the projector 302 to a perspective center 316 of the ToF camera 310. The projector 302 sends bundles of light 385 at angles aPrelative to the baseline 390, and the ToF camera 310 receives the imaged bundles of light 385 at angles acrelative to the baseline 390. Compensation measurements are performed using the device 300 to measure known positions on a reference artifact, which might be positions of dots on a calibration dot plate, for instance. The ToF camera 310 measures a total trip distance LT0Fof light emitted from the perspective center 304 of the projector 302, traveling to the object intersection point 388, and then traveling to the perspective center 316 of the camera 310.

[0049] In an embodiment, the relative positions of features on a calibration artifact are found based at least in part on the measured distance LT0F, a baseline distance LB, projection angles aP, and camera angles ac. The measured distance LT0Fdepends on the sum of the distance from the perspective center 304 to the intersectionpoint 388 and the distance from the intersection point 388 to the perspective center 316. The latter three parameters, LB, aP, and ac, are adjusted numerically along with the relative orientation of the measuring device 300 until errors in the measured 3D coordinates of features on the reference artifact are reduced or minimized. After such a compensation procedure is completed, either in the factory or before measurements in the field, the device 300 determines 3D coordinates of points such as the points 386 on the surface 342.

[0050] A triangulation calculation illustrated in FIG. 3B provides additional advantages if the ToF camera 310 is used in the iToF mode. In this case, it is possible to use the triangulation calculation to help resolve ambiguity in the number of full cycles of the modulated waveform that have elapsed before light reflects off the object and is detected by an iTOF sensor such as the sensor 120. One way to eliminate such ambiguities is to sequentially modulate the projector 302 at two or more modulation frequencies. By measuring a given distance at this plurality of modulation frequencies, it is possible using mathematical techniques well known in the art to determine the integer number of complete cycles used at each of the modulation frequencies. Although this approach is viable, triangulation often offers a faster way to eliminate ambiguity. It does this by calculating distance to the intersection point 388 accurately enough to eliminate the need for modulation at multiple frequencies or at least to reduce or minimize the number of required modulation frequencies. FIG. 3B shows the projected light 306 marked with circles 307 and lines 308. The distance between any two circles or any two lines represents the distance traveled by one complete modulation period, as explained further herein with reference to FIGs. 6A-6D.

[0051] Another advantage of spacing the projector 302 and the camera 310 apart by the baseline distance LBis to perform triangulation calculations that assist in distinguishing the measured spot 314 from other spots 314 in the pattern 312. FIG. 3C shows that the bundles of light 385 intersect the surface 342 at a collection of points 386. However, only the intersection point 388 is close to the position obtained from the time-of-flight calculations.

[0052] FIG. 3D shows a typical situation in which a spot of light 354 is captured on a TOF sensor array 350 having a collection of pixels 352. The exemplary spot 354 encompasses a width of 2 or 3 pixels and a height of 2 or 3 pixels, although more or fewer pixels are encompassed. The iTOF sensor array 350 provides a distance reading for each pixel 352. For triangulation calculations, these distance readings are used directly or distances found in centroids of a projected spots or other feature. Other mathematical techniques can alternatively be used to map out features of the surface 342.

[0053] FIG. 4A shows a 3D measuring device 400 that measures distance and angles to an object. The device 400 includes a projector 402, a clock / timer 420, a first ToF camera 440, and a second ToF camera 470. Each of the ToF cameras 440, 470 includes a lens and a ToF sensor, which are used in an iToF or dToF mode. In an embodiment, the projector 402 emits light picked up by the ToF camera 440 as pattern 412, enabling the light to be concentrated in spots 444 or other pattern elements having higher power per unit area than would be possible were the light projected homogeneously over a wide angular region. This higher power per unit enables more accurate measurements to be made for a given limited measurement rate. In a similar manner, the pattern 472 includes spots 474 or other concentrated pattern elements. FIG. 4C illustrates one triangulation geometry that is used with the device 400 to find the 3D coordinates of a point 488. In some embodiments, the functionality of the clock / timer 420 is incorporated into one TOF camera 440 or 470 as a primary clock / modulator and the projector 402 and other TOF camera acting as a secondary clock / modulator.

[0054] The geometry of the measuring device 400 enables three triangulation calculations to be carried out simultaneously if desired. In the case illustrated in FIG. 4C, one triangulation calculation uses a baseline 490 drawn between the perspective centers 446, 476 of the two cameras. The length of the baseline 490 is LB1and the intersection angles to the ToF cameras 440, 470 are acland aC2, respectively. In two other cases the triangulation calculations use baselines drawn between the projector 402 and one of the cameras 440, 470. If three such triangulation calculations are performed, the results should closely match in each of the three cases. If the results do not matchwell, then a position or orientation of at least one of the projector 402, TOF camera 440, and TOF camera 470 has probably moved. This result indicates that measurements are not reliable until another compensation of the device 400 is performed. It is also the case that certain of the three triangulation calculations are found to be more reliable than the others, perhaps because of the mechanical construction of the device 400. For any of these situations, the two-camera device 400 provides advantages over the one- camera device 300.

[0055] Measurements of the distances to multiple projected spots of light with the iToF camera enables the one or more processors to determine correspondences among projected pattern elements. In this case, the correspondences are determined based at least in part on the calculated path distances to the projected pattern elements. If the one or more processors determines that the identified correspondences are not consistent with the measured distances and further actions are taken by the system to eliminate the problem. Such actions might include dropping inconsistent measurements, performing additional measurements, or requesting a further compensation step by the operator.

[0056] FIG. 5A depicts the case in which the surface 542 to be measured is relatively close to the 3D measuring device 300. This contrasts to the case shown in FIG. 3 A in which the surface 342 is relatively far from the 3D measuring device 300. When an object 342 is relatively far from the 3D measuring device 300, it is generally advantageous to include iToF phase calculations in calculating distances to the object 342. On the other hand, when an object 542 is relatively close to the 3D measuring device 300, measurements are in some cases more accurately and quickly determined based entirely on triangulation calculations.

[0057] FIG. 5B depicts the case of FIG. 5A but with a second TOF camera 470. In the case depicted in FIG. 5B, the surface 542 is relatively close to the 3D measuring device 300. For this case, measurements are in some cases more accurately and quickly determined based entirely on triangulation calculations. On the other hand, for the surface 442 relatively far from the measuring device 400, it is generally advantageousto use the iToF sensor in a modulated mode to calculate 3D coordinates of points on the surface 542.

[0058] FIGs. 6A-6D illustrate the principle of operation of ToF sensor used in an iToF mode. FIG. 6A shows a temporal modulation pattern 600 for light emitted by the projector 112 in response to a signal generated by the clock / timer 116. The period of the emitted light pulses is shown to be T in this example. The modulation frequency is the reciprocal of this value: fM0D= 1 / T. FIG. 6B shows a temporal pattern of the modulated light of FIG. 6A received by the TOF sensor 120. The light has traveled a whole number n of complete periods T plus a measured portion of the period T, which is a phase shift 612. The total phase in radians accumulated by the modulated light in traveling from the projector 112 to the object and then on to a pixel of the ToF sensor 120 is equal to (ptot= 2nn + A< >, where A corresponds to the phase shift 612 measured by the pixel in units of radians.

[0059] An exemplary TOF sensor, such as the model IMX556 manufactured by Sony Corporation, converts photons to electrons and then creates an alternating drift field that divides electrons between two detection junctions synchronized to the modulation frequency also applied to the projector 112 by the clock / timer 116, which are at least partly contained within the TOF camera 114. The two detection junctions are 180 degrees out of phase, as illustrated in FIG. 6C by the timing diagrams 620, 630 of the two detection junctions. The signal generated by each of the two detection junctions is shown in the hatched regions 622, 632 of FIG. 6C. The exemplary sensor further samples incoming light at the additional phases 0, 90, 180, 270 degrees for each given depth frame. Other sampling strategies are possible and possibly desirable, such as if measurements are made continuously for a rotating system as further exemplary systems discussed herein below. The phase step of 90 degrees is illustrated in the waveforms of FIG. 6D. As can be seen from the line 645, the rising edge of the waveform 640 is 90 degrees of phase with respect to the waveform 620, and the waveform 650 is 90 degrees of phase with respect to 630. For this case, the electrons received by the first junction and the second junction are indicated in the highlighted regions 622, 632, respectively. The integrated electrical signals generated at the regions622, 632, 642, 652 can be used to determine the distance traveled by a beam of light over a region corresponding to the phase shift A marked 612 in FIGs. 6A, 6B. The additional phase step at 180 degrees is a reversal of the waveforms of 0 degree waveforms of FIGs. 6C and can be combined with the results of FIG. 6C to reduce or minimize signal noise. Likewise, the phase step at 270 degrees is a reversal of the waveforms at 90 degrees and can be combined with the results of FIG. 6D to reduce or minimize signal noise. The method described with respect to the FIGs. 6A, 6B, 6C, 6D provides a way to determine the phase shift A marked 612 in FIGs. 6 A, 6B. However, this still leaves undetermined the integer / ? complete 2TT radians phase shift of complete cycles in the distance travelled before the arrival of the reflected bundle of light at the iTOF detector 120. As explained herein above, multiple modulation frequencies can be launched sequentially, but an alternative embodiment performs a triangulation calculation as described herein above in reference to FIGs. 3A-4C. The triangulation calculation, possibly supplemented by one or more additional phase measurements, can be used to determine the whole number n of complete modulation cycles.

[0060] Mathematically, the temporal modulation pattern 600 includes a whole number of adjoined modulation cycles, each adjoined modulation cycle having a period T. The -temporal modulation of the optical signal is detected by the iToF sensor and appears after a delay as the detected waveform 610. The phase shift from the launch of the modulated light from the projector to the reception of the modulated light by an iToF pixel is equal to a whole number of modulation cycles plus the part of a modulation cycle. The iToF sensor is configured to determine the phase shift 612, also referred to as the phase shift A or phase offset. As indicated in FIG. 4B, triangulation calculations are used in many cases to directly determine the whole number n of modulation cycles, each of period T.

[0061] In an embodiment illustrated in FIG. 7A, a 3D measuring device 700 includes a housing 705, a rotation stage 710, a first measuring device 720, a second measuring device 730, and a two-dimensional (2D) camera 740. The rotation stage includes a motor 712 operable to rotate the housing 705 about an axis 714. Optionally, the rotation stage 710 further includes an angle measuring device such as an angularencoder 715. In an embodiment, the rotation stage 710 is mounted on a housing or stand 702, which might be a tripod or instrument stand, for example. In an embodiment, the first measuring device 720 and the second measuring device 730 each include a projector 112, a ToF camera 114, a clock / timer 116, and a processor 118 (FIG. 1). In other embodiments, the first measuring device 720 and the second measuring device 730 share a clock / timer 116, a processor 118, or both. The 3D measuring device 700 further includes the 2D camera 740, which in an embodiment is a color camera having many more pixels than the ToF camera 120 of the first measuring device 720 and further having many more pixels than the ToF camera of the second measuring device 730. In an embodiment, the color camera has about 40 times more pixels than the ToF camera. In an embodiment, the first measuring device 720 is aligned to enable its ToF camera 114 to cover angles up to vertical (e.g. a direction perpendicular to the floor the device 700 is mounted on) as the rotation stage 710 rotates the housing 705 about the axis 714. In an embodiment, the second measuring device 730 is aligned to enable the upper edge of the field of view of ToF camera 114 to slightly overlap the lower edge of the field of view of ToF camera 114 in the first measuring device 720. In consequence of this alignment, in this embodiment the first measuring device 720 and the second measuring device 730 cover all directions around the 3D measuring device 700 except for a conical region below the stand 702. In an embodiment, the 2D camera has a field of view covering nearly the same region, at least in the vertical direction. In an embodiment, the camera 740 has a portrait orientation 742.

[0062] In some embodiments, it is desired to replace the single 2D camera in the 3D measuring device 700 with two 2D cameras, each pointing in a different direction and each having a landscape orientation. It should be appreciated that while embodiments herein refer to imaging devices having a particular orientation (e.g. a landscape orientation), this is for example purposes and the claims should not be so limited. In other embodiments, one or more of the imaging devices are configured with a portrait orientation. Two possible arrangement for two such single 2D cameras are shown in FIG. 7B and FIG. 7C. In FIG. 7B, two 2D cameras 750, 760 are placed one above the other in a common housing 758. The top camera 750 includes a lens assembly752 and a circuit board 754 onto which is soldered a 2D camera chip (not shown) to produce images in a landscape orientation 756. The bottom camera 760 includes a lens assembly 762 and a circuit board 764 onto which is soldered a 2D camera chip (not shown) to produce images in a landscape orientation 766. In FIG. 7C, two 2D cameras 770, 780 are placed one to the side of the other. The left camera 770 includes a lens assembly 772 and a circuit board 774 onto which is soldered a 2D camera chip (not shown) to produce images in a landscape orientation 776. The right camera 780 includes a lens assembly 782 and a circuit board 784 onto which is soldered a 2D camera chip (not shown) to produce images in a landscape orientation 786.

[0063] With a measuring device 100 including 2D cameras 120 (referred to as optional color cameras 120 in FIG. 1), it is possible to use the 2D images from the cameras 120 to register images taken with the measuring device 100, capturing images from a plurality of orientations. For example, the 2D camera 740 in FIG. 7A are used to identify feature points in the environment. Such techniques, such as scale-invariant feature transform (SIFT) and other techniques well known in the art, are sometimes referred to as photogrammetry techniques. By extracting from 2D images a plurality of feature points in each of several angles of rotation about the axis 714, a 3D image can be reconstructed over 3D space, even without the assistance of an angle measuring device (e.g., the angular decoder 715). For this registration technique to be effective, some overlap is needed among adjacent images so that common feature points can be matched. In general, a greater amount of overlap will help ensure a greater likelihood that a satisfactory registration can be obtained. For the case in which a camera assembly includes two 2D cameras, such as those assemblies shown in FIGs. 7B, 7C, it is particularly useful to have a fairly large horizontal overlap as the cameras in FIGs. 7B, 7C are rotated about a vertical axis such as the axis 714.

[0064] FIGs. 8A, 8B, 8C illustrate three different rotation strategies for a system having two 2D cameras as shown in FIG. 7B or FIG. 7C. In each case, the 2D cameras 750, 760 are assumed to have a field-of-view (FOV) of 120 degrees (horizontal) x 85 degrees (vertical). In each case, there is a relatively small overlap in the vertical direction so that together the two cameras cover an angular area of around 120 degrees(horizontal) x 160 degrees (vertical) at a single position of the axis 714. For the case in which there are four horizontal rotations about the axis 714, the total overlap between any two successive captures is ((4 ■ 120) — 360°) / 4 = 30°, as shown in FIG. 8A. The cases for five and six overlapping rotations are illustrated in FIG. 8B and FIG. 8C, respectively. The overlap shown in FIG. 8B for five successive rotations is 48 degrees (horizontal), and the overlap shown in FIG. 8C for six successive rotations is 60 degrees (horizontal). Having a greater overlap makes it easier to register 2D images captured in successive rotations.

[0065] One method to estimate rotation angle is by using a stepper motor to drive rotation about an axis, which might be a vertical axis, also referred to as a pan axis. For example, a stepper motor might have a step resolution of 0.9 degree or 1.8 degrees. Smaller resolutions are available with micro-stepping motors. For most applications, stepper motors will not on their own provide enough accuracy. A significant improvement in angular accuracy can be obtained by matching feature points obtained on 2D cameras in overlapping regions on 2D cameras obtained in successive rotations, as illustrated in FIGs. 8A, 8B, 8C. By matching feature points and possibly imposing a loop closure requirement for rotation about 360 degrees, it is possible to obtain relatively high angular accuracy, even without an angular encoder.

[0066] It is possible to use stepper motors or direct drive motors in combination with angular encoders such as magnetic angular encoders or optical angular encoders. Very high accuracy optical angular encoders are used to obtain angular accuracies up to around one arcsecond, which is equal to approximately 5 microradians. Even when such higher accuracy angular encoders are used, it can still be useful to include one or more 2D cameras in a 3D measuring system such as the camera 740 in the system 700. For example, such 2D cameras help resolve ambiguities occurring when features change such as when a person walks in front of a rotating 3D measuring device 700. When angular encoders are of the lower accuracy variety such as magnetic angular encoders and some optical angular encoders, the photogrammetric matching of feature points with 2D cameras such as the 2D cameras 740, 750, 760, 770, and 780 can helpimprove accuracy of 3D measurement. Further advantages provided by 2D cameras are discussed herein below.

[0067] In an embodiment, one or more 2D cameras are compensated at the factory and possibly further compensated on site before making a measurement. The term calibration is sometimes used in place of the term compensation. In this document, both terms are used to mean obtaining parameters to improve accuracy of the cameras. Such parameters include aberration coefficients for the camera as well as coefficients angles and orientations for the projector and camera in the 3D measuring device.

[0068] FIGs. 9A, 9B illustrates one method of correcting images by using compensation parameters for a camera system such as the camera system 742 of FIG. 7A. In an embodiment, the compensation parameters are obtained in a factory compensation procedure. An uncorrected image captured by a 2D camera is shown in FIG. 9A. The corrected image is shown in FIG. 9B. Note that the proportions of the corrected images in FIG. 9B appear to follow the expected rules of perspective. Also, notice that lines are mostly straight in FIG. 9B, as expected, but often curved in FIG. 9A.

[0069] FIG. 10 shows a perspective view of an embodiment of a 3D measuring device 1000 that incorporates some of the elements described herein above. Two projectors 1012, 1014 project a pattern of light onto an object. In some embodiments, the projectors 1012, 1014 include small, motorized prisms 1016, 1018, respectively, operable to rotate each of the projected beams of light in a small cone angle. The motorized prisms 1016 and 1018 are low inertia elements and hence permit very rapid movement of the projected beams of light. Descriptions of representative projectors 1012, 1014 are given in reference to elements 112, 200, 302, 402. An exemplary modulation pattern appropriate for use with an iToF sensor is illustrated in FIG. 6A. It is also possible to modulate, either sequentially or simultaneously, at a plurality of modulation frequencies as a way of eliminating ambiguity in the number of complete modulation cycles that have elapsed before the iToF sensor detects reflected light. An alternative way to remove or reduce this ambiguity is through the use of triangulationas discussed herein above, especially in reference to FIGs. 3B, 4B, 5A, 5B. As illustrated in FIGs. 5 A, 5B, when an object being measured is within a predetermined distance to the 3D measuring device, triangulation measurements of distance to object points are more accurate than distance measurement that rely on an iTOF sensor. In an embodiment, the predetermined distance is between 2 - 3 meters. In this case, it is possible to turn off modulation of the projected light, thereby increasing the average optical power received by the optical detector and further reducing the measurement time.

[0070] In an embodiment, at least some of the patterned light projected by the projector 1012 onto an object is received by each of the iToF cameras 1020, 1022, and at least some of the patterned light projected by the projector 1014 onto the object is received by each of the iToF cameras 1030, 1032. A schematic representation of patterns of projected light 444, 474 picked up by iToF sensors 440, 470, respectively, is shown in FIG. 4A. FIG. 1 shows how timing of image processing by a TOF camera 114 is interconnected through a clock / timer 1167 and processor 118 to the projector 112.

[0071] In an embodiment, elements of the 3D measuring device 1000 are housed in a housing 1002 coupled to a motorized pan-axis rotation assembly 1040. In embodiments, the pan-axis rotation assembly 1040 includes a motor 1042 operable to rotate the housing 1002 about a motor axis 1044. In an embodiment, the 3D measuring device 1000 is affixed to a stand such as an instrument stand or tripod. In an embodiment, the rotation assembly 1040 further includes an angle measuring device such as an optical angular encoder 1046. In an embodiment, the optical angular encoder 1046 includes a read head 1047 and an encoder disk 1048 having a multiplicity of marks. In an embodiment, the angular encoder is designed to operate in a reflection mode, with light sources on the read head projecting light reflected by the encoder disk onto detectors on the read head. The angle of rotation of the disk is determined based on the reflected light received by the read-head detectors. In other embodiments, a different type of angular measuring device or no angular measuring device is used.

[0072] In an embodiment, the 3D measuring system 1000 further comprises an electronic system 1200 that includes an embedded personal computer (PC) 1060. FIG. 12 shows one way to implement system processing within the 3D measuring instrument 1000. Electronics 1200 within the 3D measuring system 1000 includes the embedded PC 1060 that has a control and processing unit 1212, a first MIPI aggregator 1214, a second MIPI aggregator 1216, a solid state drive (SSD) 1216, a wide local area network (WLAN) interface 1218, and a speaker 1219. In an embodiment, the control and processing unit 1212 is a Jetson Orin System on a Module (SoM) manufactured by NVIDIA Corporation. The first and second MIPI aggregators 1214, 1216 provide communication links between the control and processing unit 1212 and the iToF camera.

[0073] In an embodiment, additional components in communication with the control and processing unit 1212 include the motorized pan-axis rotation assembly 1240, buttons 1242, light emitting diodes (LEDs) 1244, an inertial measurement unit (IMU) 1246, one or more fans 1248, temperature sensors 1250, and a microcontroller 1256. In an embodiment shown in FIG. 2, connections are made with a Mobile Industry Processor Interface (MIPI), an Inter-Integrated Circuit (I2C) interface, a Serial Peripheral Interface (SPI), a Peripheral Component Interconnect express (PCIe) interface, a Controller Area Network (CAN) bus, and a General Purpose Input Output (GPIO) bus. Additional components include a system field programmable gate array (FPGA) 1257 or other processors. Mobile apps 1253 are supported. As shown in FIG. 10, the 3D measuring instrument 1000 further includes a power supply circuit board 1070, a heat sink 1072, and a battery 1074.

[0074] The 3D measuring device 1000 further includes 2D color cameras 1050, 1052. In an embodiment, the field-of-view (FOV) of the 2D color camera 1050 substantially overlaps the coverage of the 3D measuring subsystem that includes the projector 1012 and the TOF cameras 1020, 1030. Further, the FOV of the 2D color camera 1052 substantially overlaps the coverage of the 3D measuring subsystem that includes the projector 1014 and the TOF cameras 1030, 1032. Elements taken together to include the first projector 1016, the first and second TOF cameras 1020, 1022, andthe color camera 1050 all together is taken to comprise a first unit 1080. Elements taken together to include the second projector 1014, the third and fourth TOF cameras 1030, 1032, and the color camera 1052 are taken together to comprise a second unit 1082. The first unit 1080 and the second unit 1082 is considered to be subunits held within the housing 1002.

[0075] Advantages of 2D color cameras already discussed herein include (1) ability to determine angles to feature points to relatively high accuracy even without the availability of one or more high accuracy angular encoders; (2) obtaining a count of the number of modulation cycles completed before modulated light launched by a projector reaches the object and then travels to the ToF camera; (3) obtaining a more accurate understanding of the position of the projected spots in relation to nearby spots; (4) noting capture on unwanted image features such as the presence of people walking through abeam launched by the 3D measuring instrument 1000; (5) identifying possible errors in 3D (iToF) measurement at edges based at least partly on changes in depth or texture as seen in color images; and (6) colorizing an image. In addition, another important advantage of 2D color cameras has been described in commonly owned U.S. Non-Provisional Patent Application No. 18 / 364,699 (hereafter ‘699) on August 3, 2023, the contents of which are incorporated herein by reference. The disclosure of the ‘699 patent enables a high-resolution point cloud to be obtained from the combination of a low-resolution point cloud and a high-resolution image for the case in which each (A) pixel coordinate has at least a color and (B) the color is the correct color in the sense that no parallax errors are visible in the resulting point cloud.

[0076] FIG. 11 shows a front view of an embodiment of a 3D measuring device 1100 that incorporates some of the elements described herein above. The device 1100 is functionally similar to the device 1000. Two projectors 1112, 1114 project a pattern of light onto an object. In an embodiment, at least some of the patterned light projected by the projector 1112 onto an object is received by each of the iToF cameras 1130, 1122, and at least some of the patterned light projected by the projector 1114 onto the object is received by each of the iToF cameras 1120, 1132.

[0077] In an embodiment, elements of the 3D measuring device 1100 are housed in a housing 1102 coupled to a motorized pan-axis rotation assembly 1140. In embodiments, the pan-axis rotation assembly 1140 includes a motor 1142 operable to rotate the housing 1102 about a motor axis 1144. In an embodiment, the rotation assembly 1140 further includes an angle measuring device such as an optical angular encoder 1146. In an embodiment, the optical angular encoder 1146 includes a read head 1147 and an encoder disk 1148, the encoder disk having a multiplicity of marks. In an embodiment, the angular encoder is designed to operate in a reflection mode, with light sources on the read head projecting light reflected by the encoder disk onto detectors on the read head. The angle of rotation of the disk is determined based on the reflected light received by the read-head detectors. In other embodiments, a different type of angular measuring device or no angular measuring device is used.

[0078] The 3D measuring device 1100 further includes 2D color cameras 1150, 1152. In an embodiment, the field-of-view (FOV) the 2D color camera 1150 substantially overlaps the coverage of the 3D measuring subsystem that includes the projector 1112 and the TOF cameras 1122, 1130. Further, the FOV of the 2D color camera 1152 substantially overlaps the coverage of the 3D measuring subsystem that includes the projector 1114 and the TOF cameras 1120, 1132. The 3D measuring device is considered to comprise a first unit that includes the projector 1112, the TOF cameras 1122, 1130, and the color camera 1150, and a second unit that includes the projector 1114, the TOF cameras 1120, 1132, and the color camera 1152.

[0079] In an embodiment, a method is provided where the color image of a given frame is acquired. The method then proceeds to perform a method on the image that detects features in the color image. Such features can be e.g. lines, comers, areas where a sharp depth increase / decrease (due to prior knowledge on similar features), or other. In the areas of the detected features, one can run additional methods for depth computation (e.g. combining a different set of phase images, check with triangulation) to identify and correct potential errors that occur in these areas. When multiple measurements yield the same distance result, the method determines / concludes that there is no issue and simply use the measurement. Alternatively, the method simplydisregards the measurement in areas where different measurement techniques yield different results.

[0080] In another embodiment, multiple depth measurements of a single point in space is performed with the system described herein , i.e. triangulation, iToF with camera 1, iToF with camera 2. In one embodiment, these measurements can be averaged to give a more precise depth measurement. If these measurements yield distance values that are significantly (i.e. larger than the inherent error) different from one another, an issue is detected and either this data point is deleted or refined by other depth measurement techniques (e.g. use a different combination of phase images or scan the region again with a different rotation speed of the scanner).

[0081] In still another embodiment, depth measurements from different techniques (iToF camera 1, iToF camera 2, triangulation) are used to detect issues, depth measurements from different combination of iToF phase images are also used to make separate depth measurements, these are compared and issues detected in the image.

[0082] Embodiments described herein further provide for performing an iToF measurement using a number of image frames that are not recorded with the same viewing direction but cover different fields of view. Such embodiments are useful for iToF measurements of a moving system, such as a rotating laser scanner (e.g., the system 100 for measuring 3D coordinates on objects, the device 300, the device 400, the 3D measuring device 700, the 3D measuring device 1000, the 3D measuring device 1100, and / or the like including combinations and / or multiples thereof).

[0083] An iToF measurement uses a modulated detector chip (e.g., a sensor) of a camera (e.g., a camera, such as the ToF camera 114, and / or the like including combinations and / or multiples thereof) in combination with a modulated light source (e.g., VCSEL array 202 (see also FIG. 6)).

[0084] According to an embodiment, when a measuring device (e.g., the measuring device 100) rotates (or moves in some other way), the same point in spaceis covered by different pixels of a camera (e.g., the iToF camera 114). For example, at a first position and orientation (e.g., a first pose), a point is imaged into a first pixel (e.g., with phase 0 degrees); at a second position and orientation (e.g., a second pose), the same point is imaged onto a second pixel (e.g., with phase 90 degrees); and so forth for additional positions and orientations. One can then combine the measurements taken for the first pixel and the second pixel into a single depth measurement of the point in space. As used herein, a “pose” refers to the position and orientation of the measuring device. A first pose indicates a particular position and orientation, while a second pose indicates a different position and / or a different orientation relative to the first pose. That is, a change in pose is a change in position, a change in orientation, or a change in position and a change in orientation.

[0085] As described herein, a measuring device (e.g., the measuring device 100) that uses an iToF sensor (e.g., the iToF sensors 440, 470) detects multiple phases of light. More specifically, the measuring device emits a light with a defined phase. The light is then detected by detectors that are modulated at the same frequency but with varying phases. That is, the detectors have varying phase abilities. The measuring device then performs a distance measurement based on receiving the emitted light back at the iToF sensor. The iToF sensor includes pixels, and each of the pixels can include multiple “wells” for receiving each phase of light. According to one or more embodiments described herein, each well can be associated with multiple detectors. According to one or more embodiments described herein, each of the wells is associated with one of the phases of light. For example, a first well is associated with a 0 degree phase, a second well is associated with a 90 degree phase, a third well is associated with a 180 degree phase, and a fourth well is associated with a 270 degree phase. It should be appreciated that fewer or more wells can be implemented in other embodiments and that wells can be associated with phases other than the examples set forth herein.

[0086] The depth measurement uses a set of “phase frames” where each of the phase frames includes a number of images, known as “phase images,” that are recorded at the same time. For example, multiple images can be recorded with different phases at the same time. As an example, a single detector module can record, at the same item,a first image and a second image that are 180 degrees apart in phase. Other numbers of images and phase offsets can be used in other examples (e.g., a phase frame can include four images that are 90 degrees apart in phase (e.g., 0 degree phase, 90 degree phase, 180 degree phase, 270 degree phase). A phase image indicates the relative phase between the modulated light and the modulated camera and is acquired with each phase frame.

[0087] Conventionally, an iToF measurement is performed in a static situation where the measuring device does not move between phase frames used to make a depth measurement. In a conventional iToF depth measurement, each pixel of the sensor receives data as part of multiple phase images. From these data, a depth measurement can be computed for each pixel. This approach works because a pixel “sees” the same point in space for each of the phase images that are used to reconstruct the depth value. For measuring devices that record depth data during movement, conventional approaches are insufficient because the pixel does not “see” the same point in space for each of the phase images.

[0088] To address these and other shortcomings, one or more embodiments described herein provide for enabling depth measurements for a measuring device in motion where the measuring device uses an iToF device to perform measurements. Instead of using the same pixel in multiple phases as in conventional iToF-based measuring devices, a depth value can be reconstructed from multiple pixels for measuring devices that record depth data during movement. The movement of the measuring device can be determined as described herein. For example, for a rotating measuring device, an angle measuring device, such as an encoder (e.g., the angular encoder 715) or optical tracking system, can measure the movement of the measuring device. From the movement of the measuring device, the viewing direction and the position of each pixel at a given point in time is known. Then, the depth measurement can be computed using various techniques as now described.

[0089] For example, FIG. 13 depicts a flow diagram of a method 1300 for computing a depth measurement for a moving measuring device according to one ormore embodiments described herein. According to one or more embodiments described herein, the moving measuring device can be any one of the systems or devices described herein, such as the system 100 for measuring 3D coordinates on objects, the device 300, the device 400, the 3D measuring device 700, the 3D measuring device 1000, the 3D measuring device 1100, and / or the like including combinations and / or multiples thereof. The method 1300 can be implemented by any suitable device or system as described herein. For example, the method 1300 can be implemented by the measuring device 100 using the processor 118. As another example, the method 1300 can be implemented by the electronics 1200 within the 3D measuring system 1000. Other examples of devices and systems to implement the method 1300 are also possible.

[0090] At block 1302, the measuring device captures a number n of adjacent phase images, where n is a positive integer. Adjacent phase images are phase images that are taken in sequence as the measuring device moves, where two adjacent phase images have some overlap between the two phase images. The amount of overlap between two phase images can depend on various factors, such as a speed of movement of the measuring device (e.g., how quickly the measuring device is rotating), the frequency with which the phase images are captured, the field of view of the camera, and / or the like including combinations and / or multiples thereof.

[0091] At block 1304, the measuring device determines a pixel corresponding to a defined viewing direction in space for each of the subsequent phase images. The subsequent phase images are the images captured at subsequent phases. For example, for a pair of phase images are captured at the 0 degree phase and the 180 degree phase at a first point in time, and a subsequent pair of phase image are then captured at the 90 degree phase and the 270 degree phase at another point in time.

[0092] At block 1306, the measuring device uses the determined pixel in the n adjacent phase images to compute a distance (also referred to as a “depth value”). In doing so, it is not straightforward to ensure that the point in space along the defined viewing direction is illuminated by part of the pattern (as described herein) in each of the phase images used for the depth image. For example, because the laser moves withthe camera, the light pattern at a location in physical space is not in the same location from a first position to an / / th position. To address this, the space can be divided into voxels, and the measuring device can wait to calculate the distance until each of the four phase images is received for a voxel.

[0093] According to another embodiment, the measuring device can cycle through different phase images periodically and divide the viewing angle of the measuring device into defined viewing angles. The measuring device can then determine whether a viewing angle is illuminated by part of the projection pattern for at least one of each of the phase images used. Whenever a viewing angle is illuminated by part of the projection pattern (e.g., discrete pattern elements) for at least two of each of the phase images used, a depth measurement is stored for that viewing angle. In such an embodiment, the measurement rate and pattern density for collecting the images can be set such that, for a given viewing direction, the probability of being illuminated by the projection pattern in each of the phase images is sufficiently high.

[0094] According to one or more embodiments described herein, due to changing view angles, it is possible that measured reflectivities are different from one phase to another. This can cause reduced accuracy in the depth value. The measuring device can compensate for the reduced accuracy in the measured depth value caused by differences in the measured reflectivities. For example, the information from the color image for the same point in space can be used to correct the measured reflectivity in the phase image. As another example, the information from one or more neighboring images can be used to correct for differing reflectivities on an edge. As yet another example, the measurements from one or more neighboring pixels can be compared and used for compensation. As yet another example, when measuring with two iToF cameras, information from both cameras can be used to detect reduced accuracy in the depth value, such as by computing depth using triangulation. The depth values can then be corrected using the computed depth.

[0095] According to one or more embodiments described herein, if the iToF sensor or the projector are not on the same axis of rotation, parallax effects can beintroduced. Different phase images used to compute the depth value can have different depths. This difference can be compensated by using known movement information, such as by using an angle measuring device, such as an encoder (e.g., the angular encoder 715), an optical tracking system, and / or the like, to measure the movement of the measuring device.

[0096] According to one or more embodiments described herein, tracking of the measuring device can be performed, for example, using 2D images, such as in the case of a rotating device on a tripod or other fixture, using 2D images combined with 3D measurements determined using triangulation, and / or the like including combinations and / or multiples thereof. According to one or more embodiments described herein, the tracking can be performed using the depth value determined using iToF as described herein. In other embodiments, the tracking can be performed without using the depth value determined using iToF.

[0097] According to one or more embodiments described herein, each pixel of sensor can have a slightly varying gain and offset. For example, in the case of the Sony IMX556 or IMX570, there are two nodes per pixel, thus two gains and offsets. One or more embodiments described herein provide for calibrating the varying gain and offset. For example, a calibration technique can be performed using a stationary measurement of a homogeneously illuminated scene, a stationary or moving dark measurement, a scene with known depth, and / or the like including combinations and / or multiples thereof.

[0098] Additional processes are also included, and it should be understood that the process depicted in FIG. 13 represents an illustration, and that other processes are added or existing processes are removed, modified, or rearranged without departing from the scope of the present disclosure.

[0099] FIG. 14 depicts a flow diagram of a method 1400 for determining a depth value for a 3D coordinate of an object using a moving measuring device according to one or more embodiments described herein. According to one or more embodiments described herein, the moving measuring device can be any one of thesystems or devices described herein, such as the system 100 for measuring 3D coordinates on objects, the device 300, the device 400, the 3D measuring device 700, the 3D measuring device 1000, the 3D measuring device 1100, and / or the like including combinations and / or multiples thereof. The method 1400 can be implemented by any suitable device or system as described herein. For example, the method 1400 can be implemented by the measuring device 100 using the processor 118. As another example, the method 1400 can be implemented by the electronics 1200 within the 3D measuring system 1000. Other examples of devices and systems to implement the method 1400 are also possible.

[0100] At block 1402, the measuring device acquires a first phase frame at a first position of a measuring device within an environment, the measurement device comprising an iToF sensor. At block 1404, the measuring device acquires a second phase frame at a second position of the measuring device within the environment. At block 1406, the measuring device determines whether a pixel in the second frame corresponds to a location that is the same in the first frame. At block 1408, the measuring device computes the depth value responsive to determining that the pixel in the second frame corresponds to the location that is the same in the first frame.

[0101] According to one or more embodiments described herein, three or more phase images are used to make depth measurements, where the three or more phase images provide an offset of a light signal (e.g., ambient light and reflectivity of the surface of the object), an amplitude of the light signal, and a phase of the light signal.

[0102] Additional processes are also included, and it should be understood that the process depicted in FIG. 14 represents an illustration, and that other processes are added or existing processes are removed, modified, or rearranged without departing from the scope of the present disclosure.

[0103] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include computing the depth value comprises dividing the space into voxels.

[0104] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include the measuring device computes the depth value responsive to each of multiple phase images being received for a voxel.

[0105] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include computing the depth value comprises cycling through the number n of images periodically.

[0106] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include computing the depth value further comprises dividing a viewing angle of the measuring device into defined viewing angles.

[0107] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include computing the depth value further comprises determining whether one of the defined viewing angles is illuminated by part of the discrete pattern elements for at least two phase images.

[0108] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include computing the depth value further comprises, responsive to determining that the one of the defined viewing angles is illuminated by part of the discrete pattern elements for the at least two phase images, storing the depth value for the one of the defined viewing angles.

[0109] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include the housing is moving comprises the housing rotating about an axis.

[0110] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include a motor configured to rotate the housing about the axis.

[0111] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include an angle measuring device configured to measure an angle of rotation of the housing about the axis.

[0112] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include that the housing is moving comprises the housing being moved through an environment by an operator.

[0113] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include that the number n of phase images are adjacent phase images.

[0114] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include that the operations further include correcting for a difference in a measure reflectivity.

[0115] In addition to one or more of the features described herein, or as an alternative, further embodiments of the method include computing the depth value comprises dividing the space into voxels.

[0116] In addition to one or more of the features described herein, or as an alternative, further embodiments of the method include the measuring device computes the depth value responsive to each of multiple phase images being received for a voxel.

[0117] In addition to one or more of the features described herein, or as an alternative, further embodiments of the method include computing the depth value comprises cycling through the number n of images periodically.

[0118] In addition to one or more of the features described herein, or as an alternative, further embodiments of the method include computing the depth value further comprises dividing a viewing angle of the measuring device into defined viewing angles.

[0119] In addition to one or more of the features described herein, or as an alternative, further embodiments of the system include capturing the first phase image comprises capturing a first pair of phase images using the first indirect ToF camera, wherein the first pair of phase images comprises the first phase image captured at a 0 degree phase and a third phase image captured at a 180 degree phase. The system further includes that capturing the second phase image comprises capturing a second pair of phase images using the second indirect ToF camera, wherein the second pair of phase images comprises the second phase image captured at a 90 degree phase and a fourth phase image captured at a 270 degree phase. The system further includes that computing the depth value is based at least in part on the first phase image, the second phase image, the third phase image, and the fourth phase image.

[0120] The term “about” is intended to include the degree of error associated with measurement of the particular quantity based upon the equipment available at the time of filing the application. For example, “about” can include a range of ± 8% or 5%, or 2% of a given value.

[0121] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" are understood to include any integer number greater than or equal to one, i.e. one, two, three, four, etc. The terms "a plurality" are understood to include any integer number greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include an indirect "connection" and a direct "connection." It should also be noted that the terms “first”, “second”, “third”, “upper”, “lower”, and the like are used herein to modify various elements. These modifiers do not imply a spatial, sequential, or hierarchical order to the modified elements unless specifically stated.

[0122] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein,the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, element components, and / or groups thereof.While the disclosure is provided in detail in connection with only a limited number of embodiments, it should be readily understood that the disclosure is not limited to such disclosed embodiments. Rather, the disclosure can be modified to incorporate any number of variations, alterations, substitutions or equivalent arrangements not heretofore described, but which are commensurate with the spirit and scope of the disclosure. Additionally, while various embodiments of the disclosure have been described, it is to be understood that the exemplary embodiment(s) include only some of the described exemplary aspects. Accordingly, the disclosure is not to be seen as limited by the foregoing description but is only limited by the scope of the appended claims.

Claims

CLAIMSWhat is claimed is:

1. A system comprising: a housing; a projector operably coupled to the housing, the projector configured to project onto an object a light that includes discrete pattern elements having a first temporal modulation, the first temporal modulation including at least a whole number of adj oined modulation cycles, each adjoined modulation cycle having a first period; an indirect time-of-flight (ToF) camera coupled to the housing, the indirect ToF camera having a lens and a two-dimensional array of pixels, each pixel configured to convert reflected light into an electrical signal; and at least one processor operably coupled to the housing, the projector and the indirect ToF camera, the at least one processor configured to determine three- dimensional (3D) coordinates of a point on the object by performing operations comprising: capturing, while the housing is moving, a number n of phase images from varying poses, where n is a positive integer; determining for each of the number n of phase images a pixel of the array of pixels corresponding to a defined viewing direction in space; and computing a depth value of a defined viewing angle based at least in part on the determined pixels of the array of pixels in the number n of phase images, wherein the depth value represents a depth coordinate of the point on the object.

2. The system of claim 1, wherein computing the depth value comprises dividing the space into voxels.

3. The system of claim 2, wherein the measuring device computes the depth value responsive to each of multiple phase images being received for a voxel.

4. The system of claim 1, wherein computing the depth value comprises cycling through the number n of images periodically.

5. The system of claim 4, wherein computing the depth value further comprises dividing a viewing angle of the measuring device into defined viewing angles.

6. The system of claim 5, wherein computing the depth value further comprises determining whether one of the defined viewing angles is illuminated by part of the discrete pattern elements for at least two phase images.

7. The system of claim 6, wherein computing the depth value further comprises, responsive to determining that the one of the defined viewing angles is illuminated by part of the discrete pattern elements for the at least two phase images, storing the depth value for the one of the defined viewing angles.

8. The system of claim 1, wherein the housing is moving comprises the housing rotating about an axis.

9. The system of claim 8, further comprising a motor configured to rotate the housing about the axis.

10. The system of claim 8, further comprising an angle measuring device configured to measure an angle of rotation of the housing about the axis.

11. The system of claim 1, wherein the number n of phase images are adjacent phase images.

12. The system of claim 1, wherein the operations further include correcting for a difference in a measure reflectivity.

13. A computer-implemented method for determining a depth value for a three- dimensional (3D) coordinate of an object, the method comprising:acquiring, using a measuring device, a first phase frame at a first position of the measuring device within an environment, the measurement device comprising an indirect time of flight (iToF) sensor; acquiring, using the measuring device, a second phase frame at a second position of the measuring device within the environment; determining whether a pixel in the second frame corresponds to a location that is detected in the first frame; responsive to determining that the pixel in the second frame corresponds to the location that is the same in the first frame, computing the depth value.

14. The computer-implemented method of claim 13, wherein computing the depth value comprises dividing the space into voxels.

15. The computer-implemented method of claim 14, wherein the measuring device computes the depth value responsive to each of multiple phase images being received for a voxel.

16. The computer-implemented method of claim 13, wherein computing the depth value comprises cycling through the number n of images periodically.

17. The computer-implemented method of claim 16, wherein computing the depth value further comprises dividing a viewing angle of the measuring device into defined viewing angles.

18. A system comprising: a projector operably coupled to the housing, the projector configured to project onto an object a light that includes discrete pattern elements having a first temporal modulation, the first temporal modulation including at least a whole number of adj oined modulation cycles, each adjoined modulation cycle having a first period;a first indirect time-of-flight (ToF) camera coupled to the housing, the first indirect ToF camera having a lens and a two-dimensional array of pixels, each pixel configured to convert reflected light into a first electrical signal; a second indirect ToF camera coupled to the housing, the second indirect ToF camera having a lens and a two-dimensional array of pixels, each pixel configured to convert reflected light into a second electrical signal; and at least one processor operably coupled to the housing, the projector and the indirect ToF camera, the at least one processor configured to determine three- dimensional (3D) coordinates of a point on the object by performing operations comprising: capturing a first phase image using the first indirect ToF camera; capturing a second phase image using the second indirect ToF camera; and computing a depth value based at least in part on the first phase image and the second phase image.

19. The system of claim 18, wherein capturing the first phase image comprises capturing a first pair of phase images using the first indirect ToF camera, wherein the first pair of phase images comprises the first phase image captured at a 0 degree phase and a third phase image captured at a 180 degree phase; wherein capturing the second phase image comprises capturing a second pair of phase images using the second indirect ToF camera, wherein the second pair of phase images comprises the second phase image captured at a 90 degree phase and a fourth phase image captured at a 270 degree phase; and wherein computing the depth value is based at least in part on the first phase image, the second phase image, the third phase image, and the fourth phase image.

Images