Locating and sizing 2d imaged features in 3D laser scanning coordinate system

Abstract

A method of determining a location, in a three-dimensional laser scanning coordinate system for a laser scanning system, of a target, the method comprising: engendering motion, relative to the laser scanning system, of an alignment target having a known geometry; determining a three-dimensional path of motion of the alignment target in the three-dimensional laser scanning coordinate system; capturing, with a camera system having a field of view in a two-dimensional vision coordinate system that is stationary with respect to the laser scanning system, an image sequence for tracking fiducials on a calibration surface of the alignment target responsive to the engendered motion; determining two-dimensional paths of motion of the fiducials tracked within the field of view of the image sequence; and mapping two-dimensional fiducial positions from the two-dimensional paths of motion to the three-dimensional path of motion.

Description

LOCATING AND SIZING 2D IMAGED FEATURES IN 3D LASER SCANNING COORDINATE SYSTEMTECHNICAL FIELD

[0001] This application relates generally to laser scanning and machine vision systems. In particular, this application relates to mapping features, observed with machine vision, from a 2D camera coordinate space to a 3D laser scanning coordinate space.BACKGROUND INFORMATION

[0002] French Patent No. 3,035,207 describes a device for non-contact measurement of an object or part of an object, such as a mechanical part. The device includes a first means for supporting said object, measuring means integral with a frame, said measuring means comprising at least one measuring module including a source of emission of a light beam on the object, and an optical sensor oriented towards the object and capable of detecting the light beam reflected by the object. The device also includes a second means for supporting said measuring module integral with said frame and comprising means for moving in a manner adjustable said measuring module relative to the object.BRIEF SUMMARY OF THE DISCLOSURE

[0003] In some embodiments, a method of determining a location, in a three-dimensional laser scanning coordinate system for a laser scanning system, of a target, wherein the method comprises engendering motion, relative to the laser scanning system, of an alignment target having a known geometry. The method also includes determining a three-dimensional path of motion of the alignment target in the three-dimensional laser scanning coordinate system. The method further includes capturing, with a camera system having a field of view in a two- dimensional vision coordinate system that is stationary with respect to the laser scanning system, an image sequence for tracking fiducials on a calibration surface of the alignment target responsive to the engendered motion and determining two-dimensional paths of motion of the fiducials tracked within the field of view of the image sequence. The method also includes mapping two-dimensional fiducial positions from the determined two-dimensional paths of motion to the determined three-dimensional path of motion.

[0004] Additional aspects and advantages will be apparent from the following detailed description of embodiments, which proceeds with reference to the accompanying drawings.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.

[0006] FIG. 1 is a screen capture of a point cloud of a part scanned by a laser scanning system, according to one embodiment.

[0007] FIG. 2 is a perspective view of a laser scanning system including a nest for multiple laser scanner modules and a camera system, according to one embodiment.

[0008] FIG. 3 is a top plan view of a part being rotatably scanned while a camera image field of view is shown both superimposed atop the part and enlarged in greater detail.

[0009] FIG. 4 is a top plan view of a two-dimensional calibration surface of an alignment target with a field of view shown in different locations relative to the alignment target as the alignment target is rotated.

[0010] FIG. 5 is a screen capture showing multiple fiducials used in tracking the rotation of the alignment target.

[0011] FIG. 6 is another top plan view of the two-dimensional calibration surface of the alignment target showing ambiguity in determining a location of a field of view relative to the alignment target.

[0012] FIG. 7 is a top plan view of a composite image used to resolve the ambiguity shown in FIG. 6.

[0013] FIG. 8A is a top plan view of an example of an alignment target.

[0014] FIG. 8B illustrates a perspective view of the alignment target shown in FIG. 8A.

[0015] FIG. 9A through FIG. 9C are pictorial views of an example of the camera system shown in FIG. 2, according to one embodiment.

[0016] FIG. 10A through FIG. 10C are pictorial views of another example of the camera system shown in FIG. 2, according to one embodiment.

[0017] FIG. 11 is a flowchart illustrating a method of determining a location, in a three- dimensional laser scanning coordinate system for a laser scanning system, of a target according to some embodiments.

[0018] FIG. 12 is a block diagram of circuitry that, in some embodiments, may be used to implement various functions, operations, acts, processes, and / or methods described herein.DETAILED DESCRIPTION OF EMBODIMENTS

[0019] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown, by way of illustration, specific examples of embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable a skilled person to practice the present disclosure. However, other embodiments enabled herein may be utilized, and structural, material, and process changes may be made without departing from the scope of the disclosure.

[0020] The illustrations presented herein are not meant to be actual views of any particular method, system, device, or structure, but are merely idealized representations that are employed to describe the embodiments of the present disclosure. In some instances, similar structures or components in the various drawings may retain the same or similar numbering for the convenience of the reader; however, the similarity in numbering does not necessarily mean that the structures or components are identical in size, composition, configuration, or any other property.

[0021] The following description may include examples to help enable one of ordinary skill in the art to practice the described embodiments. The use of the terms "exemplary," "by example," and "for example," means that the related description is explanatory, and though the scope of the disclosure is intended to encompass the examples and legal equivalents, the use of such terms is not intended to limit the scope of an embodiment or this disclosure to the specified components, steps, features, functions, or the like.

[0022] It will be readily understood that the components of the embodiments as generally described herein and illustrated in the drawings could be arranged and designed in a wide variety of different configurations. Thus, the following description of various embodiments is not intended to limit the scope of the present disclosure but is merely representative of various embodiments. While the various aspects of the embodiments may be presented in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0023] Furthermore, specific implementations shown and described are only examples and should not be construed as the only way to implement the present disclosure unless specified otherwise herein. Elements, circuits, and functions may be shown in block diagram form in order not to obscure the present disclosure in unnecessary detail. Conversely, specific implementations shown and described are exemplary only and should not be construed as the only way to implement the present disclosure unless specified otherwise herein. Additionally,block definitions and partitioning of logic between various blocks is exemplary of a specific implementation. It will be readily apparent to one of ordinary skill in the art that the present disclosure may be practiced by numerous other partitioning solutions. For the most part, details concerning timing considerations and the like have been omitted where such details are not necessary to obtain a complete understanding of the present disclosure and are within the abilities of persons of ordinary skill in the relevant art.

[0024] Those of ordinary skill in the art would understand that information and signals may be represented using any of a variety of different technologies and techniques. Some drawings may illustrate signals as a single signal for clarity of presentation and description. It will be understood by a skilled person that the signal may represent a bus of signals, wherein the bus may have a variety of bit widths and the present disclosure may be implemented on any number of data signals including a single data signal.

[0025] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments described herein may be implemented or performed with a general purpose processor, a special purpose processor, a digital signal processor (DSP), an Integrated Circuit (IC), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor (may also be referred to herein as a host processor or simply a host) may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. A general-purpose computer including a processor is considered a special-purpose computer while the general-purpose computer is configured to execute computing instructions (e.g., software code) related to embodiments of the present disclosure.

[0026] The embodiments may be described in terms of a process that is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operational acts as a sequential process, many of these acts can be performed in another sequence, in parallel, or substantially concurrently. In addition, the order of the acts may be re-arranged. A process may correspond to a method, a thread, a function, a procedure, asubroutine, a subprogram, other structure, or combinations thereof. Furthermore, the methods described herein may be implemented in hardware, software, or both. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on computer-readable media. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.

[0027] As used herein, the term "substantially" in reference to a given parameter, property, or condition means and includes to a degree that one of ordinary skill in the art would understand that the given parameter, property, or condition is met with a small degree of variance, such as, for example, within acceptable manufacturing tolerances. By way of example, depending on the particular parameter, property, or condition that is substantially met, the parameter, property, or condition may be at least 90% met, at least 95% met, or even at least 99% met.

[0028] FIG. 1 shows an example of a point cloud 100 of a workpiece 102 captured by a laser scanning system with a three-dimensional laser scanning coordinate system 104, wherein a resolution of point cloud 100 is insufficient to resolve geometry and position of workpiece features. For instance, point cloud 100 represents a plate surface 106 with small divots 108 in it, which are too small for accurately determining their center or diameter based on information from point cloud 100. This is so because the resolution of the pitch between data points is approximately in a range of around 10 pm to 15 pm. In this example, divots 108 are around 450 pm in diameter, so while they can be detected in point cloud 100, using point cloud 100 for measuring their diameters with sufficient repeatability for tight tolerances, i.e., ±10 pm, is challenging.

[0029] FIG. 2 shows a combined laser and camera system 200, according to some embodiments. Combined laser and camera system 200 includes a stage 202, which may be a rotary stage 202, and may comprise a nest 204 defined by various components. Combined laser and camera system 200 includes a laser scanning system, which may be mounted to a laser mounting bracket 206. The laser scanning system is configured to scan a workpiece on the stage 202 (e.g., during performance of manufacturing operations on the workpiece) and generate a three-dimensional point cloud of the workpiece. The point cloud of the workpiece in a three-dimensional laser scanning coordinate system of the laser scanning system, may be insufficient to resolve geometry and position of workpiece features (see, e.g., FIG. 1).

[0030] Combined laser and camera system 200 also includes a camera system having a field of view in a two-dimensional vision coordinate system that is stationary with respect to the laser scanning system, which may be mounted to a camera mounting bracket 208. Combined laser and camera system 200 also optionally includes a mirror mounting bracket configured to mount a reflective surface (e.g., a mirror) for directing light from the camera system downward toward the workpiece on rotary stage 202 in nest 204 of combined laser and camera system 200. The reflective surface may also be configured to reflect light back from the workpiece on rotary stage 202 in nest 204 to the camera system mounted to camera mounting bracket 208.

[0031] In some embodiments, no mirror mounting bracket is used, and light may travel between the workpiece and the camera system directly without intervention from a reflective surface. In other words, the camera system may be free of a reflective surface intervening between a light source and the camera system of combined laser and camera system 200. Two- dimensional images the camera system captures of the workpiece in a two-dimensional vision coordinate system may be mapped to the point cloud in the three-dimensional laser scanning coordinate system to resolve geometry and position of the workpiece features.

[0032] An alignment target 212 on rotary stage 202 may be used to calibrate a mapping between the two-dimensional vision coordinate system of the camera with the point cloud in the three-dimensional laser scanning coordinate system of the laser scanning system.Alignment target 212 may include fiducials that may be detected and tracked by the camera system. In some embodiments, the fiducials are intersections of lines defining a grid pattern applied to alignment target 212. In some embodiments, the fiducials are holes that are drilled in alignment target 212.

[0033] During a calibration operation, rotary stage 202 engenders motion, relative to camera mounting bracket 208 and laser mounting bracket 206, of alignment target 212. For example, rotary stage 202 may rotate relative to camera mounting bracket 208 and laser mounting bracket 206. Accordingly, alignment target 212 may move relative to a camera mounted to camera mounting bracket 208 and a laser scanning system mounted to laser mounting bracket 206. The laser scanning system may be used (e.g., by a processor) to determine a three- dimensional path of motion of alignment target 212 in the three-dimensional laser scanning coordinate system. The camera may capture an image sequence for tracking the fiducials on a calibration surface of alignment target 212 responsive to the engendered motion. The image sequence may be used (e.g., by a processor) to determine two-dimensional paths of motion ofthe fiducials tracked within the field of view of the image sequence. A processor may map two- dimensional fiducial positions from the determined two-dimensional paths of motion to the determined three-dimensional path of motion. Accordingly, the two-dimensional fiducial positions shown in the field of view are mapped to coordinates in the three-dimensional laser scanning coordinate system.

[0034] In combined laser and camera system 200 illustrated in FIG. 2 the motion engendered by rotary stage 202 may be rotational motion of alignment target 212 about a rotation axis. In such embodiments, mapping the two-dimensional fiducial positions to the determined three- dimensional path of motion may include mapping the two-dimensional fiducial positions to the determined three-dimensional path of motion responsive to a detected center of rotation and a detected rotational position of the alignment target. The rotational position may be detected with an encoder of the laser scanning system. It should be understood, however, that in some embodiments the engendered motion may be a linear translation motion.

[0035] In some embodiments, the image sequence may be processed to develop a composite image of the calibration surface of the alignment target. For example, perpendicularity of light from the camera system relative to the calibration surface may be adjusted based on uneven lighting of the composite image. As another example, a perpendicularity of the camera system relative to the calibration surface may be adjusted based on maximum brightness lighting of the composite image. As a further example, the composite image of the calibration surface of the alignment target may be a three hundred and sixty degree (360°) composite image of the calibration surface.

[0036] In some embodiments combined laser and camera system 200 includes an optional robot arm 214 configured to place alignment target 212 or other workpiece on rotary stage 202, adjust alignment target 212 or other workpiece, and remove alignment target 212 or other workpiece from rotary stage 202. Robot arm 214, in some embodiments, is used to load alignment target 212 after a predetermined time, on demand, or in response to ambient temperature changes so that the coordinate system mapping may be re-calibrated.

[0037] In some embodiments, the combined laser and camera system 200 comprises multiple cameras, positioned to captured different portions of an alignment target and / or of a workpiece when it is positioned on the stage. In some embodiments, there are multiple cameras positioned such that images covering the entire alignment target may be captured during a 360 degree rotation of the alignment target.

[0038] FIG. 3 shows an example of how camera-based measurements may give relative dimensions of fiducials 302 to edges of a camera field of view (camera FOV 304). In terms of an entire measured part 306, the relative measurements from the FOV are transformed into the three-dimensional laser scanning coordinate system of the part / laser scan data. To do this, the position of camera FOV 402, as well as the encoder position (e.g., indicating a rotational position of an alignment target such as alignment target 212 of FIG. 2) at the moment of capture, would be known relative to an axis of rotation 308. In some embodiments an angular offset 310 and a radial offset 312 from the axis of rotation 308 may be determined. It may be assumed in some embodiments that an optical axis of the camera system is parallel with the rotational axis.

[0039] FIG. 4 shows that a calibration surface 400 may be used to find a location of a camera FOV 402 relative to a part frame of reference 404. Part frame of reference 404 may be arbitrary and may be any portion of a workpiece or alignment target (not shown). In the example of FIG. 4, an origin 406 of part frame of reference 404 is at a lower left corner of calibration surface 400, with positive X (horizontal) values being to the right of origin 406 and positive Y (vertical) values being upward from origin 406 to define a two-dimensional vision coordinate system. This maintains as positive numbers all coordinate values in part frame of reference 404.

[0040] By knowing positions of visible features 408 (e.g., intersections between lines of part frame of reference 404) within camera FOV 402, the location and orientation of visible features 408 in terms of part frame of reference 404 may be determined. For example, locating camera FOV 402 in part frame of reference 404 may be accomplished by tracking multiple points (e.g., multiple intersections between lines, corresponding to visible features 408). In the example of FIG. 4, by using the location of at least two FOVs (i.e., camera FOV 402 and another camera FOV 410 at a different point in time responsive to rotational motion of part frame of reference 404 relative to a camera), and knowing an angle 412 between them (between camera FOV 402 and camera FOV 410), a center of rotation 414 may be located.

[0041] FIG. 5 shows a chrome-on-glass checkerboard pattern created with fiducials (points at intersections between lines) having an accuracy of plus or minus one micron (±1 pm). The fiducials may be measured using a camera system. These fiducials 502 on the checkerboard are designed to within the desired tolerance. It is possible to determine a position of a camera FOV 504 based on two fiducials. The position of the points, however, may not be determinedperfectly. The more fiducials that are identified within a camera FOV, the more accurately the camera FOV position may be determined. This is because the additional fiducials reduce a mismatch across all the fiducials. FIG. 5 shows that by tracking many fiducials, gaussian errors average closer to zero and the camera FOV location may be determined more accurately.

[0042] FIG. 6 shows a checkerboard pattern 602, which may be used on a calibration surface of an alignment target (e.g., alignment target 212 of FIG. 2), according to some embodiments. Since checkerboard pattern 602 is a repeating pattern, it may be necessary to distinguish which fiducials of a captured camera FOV 604 correspond to which fiducials of checkerboard pattern 602. For example, absent a second captured camera FOV, it may not be possible to determine whether fiducial 606 of camera FOV 604 corresponds to fiducial 608a, fiducial 608b, fiducial 608c, or fiducial 608d, and whether fiducial 610 of camera FOV 604 corresponds to fiducial 612a, fiducial 612b, fiducial 612c, or fiducial 612d. Accordingly, absent a second captured camera FOV, it may not be possible to determine whether camera FOV 604 corresponds to camera FOV 614a, camera FOV 614b, camera FOV 614c, camera FOV 614d, or some other camera FOV of checkerboard pattern 602. As discussed with reference to FIG. 4, however, with a second camera FOV it may be determined which portion of checkerboard pattern 602 corresponds to camera FOV 604.

[0043] FIG. 7 shows that when multiple images corresponding to different camera FOVs 702a-702b are taken at known encoder positions, the same fiducial may be followed (tracked) through the subsequent images. For example, some fiducials at intersections between lines defining a checkerboard pattern 704 are overlapped in more than one of camera FOVs 702a- 702b. The encoder correlated to a three-dimensional laser scanning system coordinate system provides information that may be used to determine where a specific fiducial should be in the workpiece or alignment target frame of reference, and the images (camera FOVs 702a-702b) provide information on where the specific fiducial actually is. By doing this for multiple fiducials, it is possible to calibrate a difference (e.g., a delta) between two cases for the degrees of freedom required. In this particular example, by identifying at least two fiducials in a FOV (preferably more than two fiducials to increase accuracy), and knowing where the fiducials are located on the workpiece or alignment target, it is possible to determine the position and orientation of that one FOV.

[0044] By taking enough images at known, relative angles, it is possible to construct a legend to identify the common fiducials in each image. In some embodiments, a composite image 700is constructed by positioning all the camera captures (camera FOVs 702a-702b) radially around a center, as illustrated in FIG. 7. By adjusting the radius and angle of each image, eventually the composite image 700 matches the actual workpiece or alignment target. In the example of FIG. 7, the composite image is created by ensuring the lines defining the checkerboard pattern 704 intersections line up, as well as that the total number of squares matches a known total number of squares for the workpiece or alignment target. It is then possible to identify the location of each line intersection (fiducial) relative to the workpiece or alignment target coordinate frame of reference.

[0045] FIG. 8A and FIG. 8B show an alignment target 800, which is an example of alignment target 212 of FIG. 2. As mentioned previously, a center point of rotation may be used as a link between data obtained by the laser scanning system and alignment target 800. The actual position of alignment target 800, however, is arbitrary since an angle 802 of a pattern edge 804 of a checkerboard pattern 806 of alignment target 800 grid relative to a target edge 808 (a laser detected feature) of alignment target 800 may be used to determine a position of a camera FOV relative to the three-dimensional laser scanning coordinate system.

[0046] For example, FIG. 8 A shows that since the center of rotation of alignment target 800 is known for the lasers, and the aforementioned method determines the center of rotation for the camera FOV, angle 802 between pattern edge 804 and target edge 808 is used for mapping camera positions to workpiece or alignment target positions. This angle 802 is measured for alignment target 800 using a separate optical measurement machine (OMM). An absolute position of features of angle 802 is not use, rather a relative angle corresponding to angle 802 is used, which makes it readily measurable. An OMM can look for features on checkerboard pattern 806 as well as features on the physical alignment target 800 (e.g., side “wings” of alignment target 800). This may be done using a top-down camera that identifies the edges of alignment target 800, as well as the edges of the checkerboard pattern 806.

[0047] For every camera FOV location there exists an affine transform that converts between the camera FOV and a point cloud coordinate frame of references. This may be described as a function F where F(x_FOV, y_FOV) -> x_pointcloud, y pointcloud, z pointcloud. Embodiments described herein may produce this function F as an output.

[0048] A center 810 of checkerboard pattern 806 may be cut away from alignment target 800 so that center 810 does not interfere with laser calibration features.

[0049] FIG. 9A shows that camera system 902 (e.g., mounted to camera mounting bracket 208 of FIG. 2) in an example portion 900 of a combined laser and camera system (e.g., combined laser and camera system 200 of FIG. 2) may be mounted horizontally, reflecting off a 45° reflective surface 904 (e.g., mounted to mirror mounting bracket of FIG. 2), to point straight down at alignment target 906 on rotary stage 908. Incident light 910 provided by camera system 902 is coaxial with the camera lens, and may be implemented with an integrated beam splitter. Camera system 902 is at a fixed position for a given configuration, assuring that features (e.g., fiducials, edges) to be measured are within the camera FOV.

[0050] Incident light 910 reflected from reflective surface 904 toward alignment target 906 may be colinear with a rotational axis 912 of alignment target 906. Rotary stage 908 and calibration surface 914 of alignment target 906, however, may be set at an angle different from 90° with rotational axis 912. Accordingly, return light 916 may be provided from alignment target 906 at a non-zero angle relative to rotational axis 912 and incident light 910, and may reflect off of reflective surface 904 back to camera system 902.

[0051] FIG. 9B shows that as alignment target 906 rotates about rotational axis 912, return light 916 returns to reflective surface 904 at an angle that sweeps to an opposite extreme from that shown in FIG. 9A. With calibration surface 914 of alignment target 906 and rotary stage 908 at an angle different from 90° from rotational axis 912, a resulting composite image (e.g., similar to composite image 700 of FIG. 7) of calibration surface 914 may manifest a gradient going from light on one side of calibration surface 914 to dark on another side of calibration surface 914. This gradient may indicate that calibration surface 914 is not level (i.e., not perpendicular to incident light 910 arriving at calibration surface 914).

[0052] Rotary stage 908 may in some embodiments be supported by a plurality of finger members that include adjustment mechanisms to change the height of each finger member (see, e.g., FIG. 2 showing a three-finger stand holding alignment target 212 atop rotary stage 202). After each adjustment a new composite image may be captured and analyzed to determine whether the adjustments have brought rotary stage 908 and calibration surface 914 closer to perpendicular to rotational axis 912 or further from perpendicular to rotational axis 912. A decreased gradient of a captured composite image may indicate that an adjustment to one or more finger members has brought calibration surface 914 closer to perpendicular with rotational axis 912. Iterating through adjustments may result in a composite image having atleast substantially uniform brightness, which may indicate that calibration surface 914 and rotary stage 908 is at least substantially perpendicular to rotational axis 912.

[0053] FIG. 9C shows rotary stage 908 and calibration surface 914 brought substantially to perpendicular to rotational axis 912. FIG. 9C also shows that during setup of the system, the perpendicularity of rotary stage 908 to rotational axis 912 may be determined using camera system 902. So long as alignment target 906 top and bottom surfaces are parallel, an angle of rotary stage 908 relative to rotational axis 912 will affect how much return light 916 is reflected back to camera system 902.

[0054] FIG. 9C also shows that once the composite image is captured with at least substantially uniform brightness, a similar process may be used to adjust a position of camera system 902 to cause incident light 910 provided to calibration surface 914 to be at least substantially perpendicular to calibration surface 914. Sweeping for a maximum brightness may be associated with perpendicularity of incident light 910 to rotary stage 908. Camera system 902 would be positioned intentionally (position also referred to herein as “tip”) angled relative to alignment target 906, then images would be captured as adjustments are made to the position of camera system 902 (e.g., sweeping incident light 910 delivered to calibration surface 914 towards vertical and past vertical). The composite image with maximum brightness may correspond to an ideal position of camera system 902. The same may be done for pointing angle of camera system 902 (also referred to herein as “tilt”), which may be 90° relative to the tip axis. This process may be completed several times to verify that an acceptable location has been found.

[0055] FIG. 10A, FIG. 10B, and FIG. 10C illustrate that a camera system 1002 of an example portion 1000 of a combined laser and camera system (e.g., combined laser and camera system 200 of FIG. 2) may be mounted vertically to point directly down at an alignment target 1004 without intervention from a reflective surface (e.g., reflective surface 904 of FIG. 9A through FIG. 9C). Example portion 1000 includes camera system 1002, alignment target 1004, calibration surface 1006, checkerboard pattern 1008, incident light 1010, rotational axis 1012, return light 1014, and rotary stage 1016, which are similar to camera system 902, alignment target 906, calibration surface 914, checkerboard pattern 918, incident light 910, rotational axis 912, return light 916, and rotary stage 908, respectively, of example portion 900 of FIG. 9 A.

[0056] Similarly as discussed with reference to FIG. 9A, incident light 1010 provided by camera system 1002 toward alignment target 1004 may be colinear with rotational axis 1012 ofalignment target 1004. Rotary stage 1016 and calibration surface 1006 of alignment target 1004, however, may be set at an angle different from 90° with rotational axis 1012. Accordingly, return light 1014 may be provided from alignment target 1004 at a non-zero angle relative to rotational axis 1012 and incident light 1010, and may return to camera system 1002.

[0057] Similarly as discussed above with reference to FIG. 9B, FIG. 10B shows that as alignment target 1004 rotates about rotational axis 1012, return light 1014 returns to camera system 1002 at an angle that sweeps to an opposite extreme from that shown in FIG. 10A. With calibration surface 1006 of alignment target 1004 and rotary stage 1016 at an angle different from 90° from rotational axis 1012, a resulting composite image (e.g., similar to composite image 700 of FIG. 7) of calibration surface 1006 may manifest a gradient going from light on one side of calibration surface 1006 to dark on another side of calibration surface 1006. This gradient may indicate that calibration surface 1006 is not level (i.e., not perpendicular to incident light 1010 arriving at calibration surface 1006). Adjustments may be made (e.g., iteratively) to bring calibration surface 1006 perpendicular to rotational axis 1012 (e.g., by iteratively reducing a gradient of brightness of a composite image corresponding to calibration surface 1006).

[0058] FIG. 10C shows rotary stage 1016 and calibration surface 1006 brought substantially to perpendicular to rotational axis 1012. FIG. 10C also shows that during setup of the system, the perpendicularity of rotary stage 1016 to rotational axis 1012 may be determined using camera system 1002. So long as alignment target 1004 top and bottom surfaces are parallel, an angle of rotary stage 1016 relative to rotational axis 1012 will affect how much return light 1014 is reflected back to camera system 1002.

[0059] FIG. 10C also shows that once the composite image is captured with at least substantially uniform brightness, a similar process may be used to adjust a position of camera system 1002 to cause incident light 1010 provided to calibration surface 1006 to be at least substantially perpendicular to calibration surface 1006. Sweeping for a maximum brightness may be associated with perpendicularity of incident light 1010 to rotary stage 1016. Camera system 1002 would be positioned (tipped) intentionally angled relative to alignment target 1004, then images would be captured as adjustments are made to the position of camera system 1002 (e.g., sweeping incident light 1010 delivered to calibration surface 1006 towards vertical and past vertical). The composite image with maximum brightness may correspond to an ideal position of camera system 1002. The same may be done for pointing angle (tilt) of camerasystem 1002, which may be 90° relative to the tip axis. This process may be completed several times to verify that an acceptable location has been found.

[0060] FIG. 11 is a flowchart illustrating a method 1100 of determining a location, in a three- dimensional laser scanning coordinate system of a laser scanning system, of a target. The target may in some embodiments be an alignment target used for calibrating the system. In some embodiments, the target may be a workpiece or a feature of a workpiece, wherein the feature may be a workpiece feature for which a resolution of samples in a point cloud obtained with the laser scanning system is insufficient to resolve geometry and position of the workpiece features. In some embodiments, determining a location may comprise determining a geometry and / or a position of the target.

[0061] At operation 1102, method 1100 includes engendering motion, relative to the laser scanning system, of an alignment target having a known geometry, in the three-dimensional laser scanning coordinate system. In some embodiments, the alignment target may also have known positions in the laser scanning coordinate system, which may have been obtained from prior measurements with a laser scanning system. In some embodiments, the engendered motion is rotational motion of the alignment target about a rotation axis. In some embodiments, the engendered motion is a linear translation motion. The engendering of motion may be performed by positioning the alignment target on a stage, and then engendering motion of the stage.

[0062] At operation 1104, method 1100 includes determining a three-dimensional path of motion of the alignment target in the three-dimensional laser scanning coordinate system. The determining of the motion may be performed by a laser scanner of the laser scanning system. The determining of the motion may comprise generating a three-dimensional point cloud of the alignment target as it moves in the three-dimensional laser scanning coordinate system.

[0063] At operation 1106, method 1100 includes capturing, with a camera system having a field of view in a two-dimensional vision coordinate system that is stationary with respect to the rest of the laser scanning system, an image sequence for tracking fiducials on a calibration surface of the alignment target responsive to the engendered motion. In some embodiments, the fiducials are intersections of lines defining a grid pattern (e.g., a checkerboard pattern) applied to the alignment target. In some embodiments, the fiducials are holes drilled in the alignment target. In some embodiments, operation 1106 and 1104 are performed simultaneously, such thattwo-dimensional data is obtained by the camera system at the same time as three-dimensional data is obtained by the laser scanning system.

[0064] At operation 1108, method 1100 includes determining two-dimensional paths of motion of the fiducials tracked within the field of view of the image sequence.

[0065] At operation 1110, method 1100 includes mapping two-dimensional fiducial positions from the determined two-dimensional paths of motion to the determined three-dimensional path of motion. In some embodiments the two-dimensional fiducial positions shown in the field of view are mapped to coordinates in the three-dimensional laser scanning coordinate system. In some embodiments, the engendered motion is rotational motion of the alignment target about a rotation axis and mapping the two-dimensional fiducial positions to the determined three- dimensional path of motion includes mapping the two-dimensional fiducial positions to the determined three-dimensional path of motion based on a detected center of motion and / or a detected rotational position of the alignment target. In some embodiments, the detected rotational position is detected with an encoder of the laser scanning system. In some embodiments, the mapping is performed by having a rotational position associated with each captured image and each captured point cloud, such that a specific image is mapped to a corresponding point cloud based on the detected rotational positions. In some embodiments, any image data from the camera system and any point cloud data from the laser scanning system comprises or is associated with rotational position data from the encoder.

[0066] At optional operation 1112, method 1100 may include processing the image sequence to develop a composite image of a surface (e.g., a calibration surface) of the alignment target. In some embodiments the method also includes adjusting, based on uneven lighting of the composite image, perpendicularity of light from the camera system relative to the surface. In some embodiments, based on maximum brightness lighting of the composite image, perpendicularity of the camera system relative to the surface may be adjusted. In some embodiments, the composite image of the surface of the alignment target may be a three hundred and sixty degree (360°) composite image of the surface.

[0067] In some embodiments, the method further comprises calibrating the laser scanning system, based on the performed mapping. By calibrating the system, more accurate measurements may be obtained for a workpiece is to be inspected with the system.

[0068] In some embodiments, the method further comprises, after calibrating the system, obtaining a point cloud of a workpiece feature, wherein a resolution of samples in the pointcloud is insufficient for determining geometry and / or position of the workpiece feature, capturing an image of the workpiece feature with the camera system, and determining a geometry and / or position of the workpiece feature based on the captured image.

[0069] In some embodiments, the alignment target is adapted in size based on a workpiece to be measuring after the system has been calibrated using the alignment target, such that the alignment target is approximately the same size as the workpiece. In some embodiments, the alignment target is adapted based on the features of the workpiece which are to be measured, such that the fiducials on the calibration surface of the alignment target are of a similar or approximately the same size as the features to be measured. In some embodiments, the method may comprise a step of choosing a suitable alignment target prior to step 1102.

[0070] All components used for performing the method may be configured and / or constructed as any of the embodiments described herein, such as the ones described in relation to Figs. 1- 10, including the camera, laser scanner, and alignment target.

[0071] The method may be performed in a combined camera and laser scanning system, such as the one described in relation to e.g. Figs. 2, 9 and 10. The system may further comprise processing circuitry and a memory, wherein the memory contains instructions executable by the processing circuitry, whereby the system is operative for performing steps of any method as described herein.

[0072] It will be appreciated by those of ordinary skill in the art that functional elements of embodiments described herein (e.g., functions, operations, acts, processes, and / or methods) may be implemented in any suitable hardware, software, firmware, or combinations thereof. FIG. 12 illustrates non-limiting examples of implementations of functional elements described herein. In some embodiments, some or all portions of the functional elements described herein may be performed by hardware specially configured for carrying out the functional elements.

[0073] FIG. 12 is a block diagram of circuitry 1200 that, in some embodiments, may be used to implement various functions, operations, acts, processes, and / or methods described herein. Circuitry 1200 includes one or more processors 1202 (sometimes referred to herein as “processors 1202”) operably coupled to one or more data storage devices (sometimes referred to herein as “storage 1204”). Storage 1204 includes machine executable code 1206 stored thereon and the processors 1202 include logic circuitry 1208. Machine executable code 1206 includes information describing functional elements that may be implemented by (e.g., performed by) logic circuitry 1208. Logic circuitry 1208 is adapted to implement (e.g.,perform) the functional elements described by machine executable code 1206. Circuitry 1200, when executing the functional elements described by machine executable code 1206, should be considered as special purpose hardware configured for carrying out functional elements described herein. In some embodiments processors 1202 may be configured to perform the functional elements described by machine executable code 1206 sequentially, concurrently (e.g., on one or more different hardware platforms), or in one or more parallel process streams.

[0074] When implemented by logic circuitry 1208 of processors 1202, machine executable code 1206 is configured to adapt processors 1202 to perform operations of embodiments described herein. For example, machine executable code 1206 may be configured to adapt processors 1202 to perform at least a portion of method 1100 of FIG. 11. As specific, nonlimiting examples, machine executable code 1206 may be configured to adapt processors 1202 to trigger operation 1102, perform operation 1104, trigger operation 1106, and perform operations 1108, 1110, and 1112.

[0075] Processors 1202 may include a general purpose processor, a special purpose processor, a central processing unit (CPU), a microcontroller, a programmable logic controller (PLC), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field- programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, other programmable device, or any combination thereof designed to perform the functions described herein. A general-purpose computer including a processor is considered a special-purpose computer while the general-purpose computer is configured to execute functional elements corresponding to machine executable code 1206 (e.g., software code, firmware code, hardware descriptions) related to embodiments of the present disclosure. It is noted that a general-purpose processor (may also be referred to herein as a host processor or simply a host) may be a microprocessor, but in the alternative, processors 1202 may include any conventional processor, controller, microcontroller, or state machine. Processors 1202 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0076] In some embodiments storage 1204 includes volatile data storage (e.g., random-access memory (RAM)), non-volatile data storage (e.g., Flash memory, a hard disc drive, a solid state drive, erasable programmable read-only memory (EPROM), etc.). In some embodiments processors 1202 and storage 1204 may be implemented into a single device (e.g., asemiconductor device product, a system on chip (SOC), etc.). In some embodiments the processors 1202 and storage 1204 may be implemented into separate devices.

[0077] In some embodiments machine executable code 1206 may include computer-readable instructions (e.g., software code, firmware code). Computer-readable instructions may be stored by storage 1204, accessed directly by processors 1202, and executed by processors 1202 using at least logic circuitry 1208. Also, computer-readable instructions may be stored on storage 1204, transferred to a memory device (not shown) for execution, and executed by processors 1202 using at least logic circuitry 1208. Accordingly, in some embodiments logic circuitry 1208 includes electrically configurable logic circuitry 1208.

[0078] In some embodiments machine executable code 1206 may describe hardware (e.g., circuitry) to be implemented in logic circuitry 1208 to perform the functional elements. This hardware may be described at any of a variety of levels of abstraction, from low-level transistor layouts to high-level description languages. At a high-level of abstraction, a hardware description language (HDL) such as an IEEE Standard hardware description language (HDL) may be used. Verilog™, SystemVerilog™ or very large scale integration (VLSI) hardware description language (VHDL™) may be used.

[0079] HDL descriptions may be converted into descriptions at any of numerous other levels of abstraction as desired. As a non-limiting example, a high-level description can be converted to a logic-level description such as a register-transfer language (RTL), a gate-level (GL) description, a layout-level description, or a mask-level description. As a non-limiting example, micro-operations to be performed by hardware logic circuits (e.g., gates, flip-flops, registers, without limitation) of logic circuitry 1208 may be described in a RTL and then converted by a synthesis tool into a GL description, and the GL description may be converted by a placement and routing tool into a layout-level description that corresponds to a physical layout of an integrated circuit of a programmable logic device, discrete gate or transistor logic, discrete hardware components, or combinations thereof. Accordingly, in some embodiments machine executable code 1206 may include an HDL, an RTL, a GL description, a mask level description, other hardware description, or any combination thereof.

[0080] In embodiments where machine executable code 1206 includes a hardware description (at any level of abstraction), a system (not shown, but including storage 1204) may be configured to implement the hardware description described by machine executable code 1206. Processors 1202 may include a programmable logic device (e.g., an LPGA or a PLC) and logiccircuitry 1208 may be electrically controlled to implement circuitry corresponding to the hardware description into logic circuitry 1208. Also, logic circuitry 1208 may include hardwired logic manufactured by a manufacturing system (not shown, but including storage 1204) according to the hardware description of machine executable code 1206.

[0081] Regardless of whether machine executable code 1206 includes computer-readable instructions or a hardware description, logic circuitry 1208 is adapted to perform the functional elements described by machine executable code 1206 when implementing the functional elements of machine executable code 1206. It is noted that although a hardware description may not directly describe functional elements, a hardware description indirectly describes functional elements that the hardware elements described by the hardware description are capable of performing.

[0082] Skilled persons will appreciate that many changes may be made to the details of the above-described embodiments without departing from the underlying principles of the invention. The scope of the present invention should, therefore, be determined only by claims and equivalents thereof.

Claims

CLAIMSWhat is claimed is:

1. A method of determining a location, in a three-dimensional laser scanning coordinate system of a laser scanning system, of a target, the method comprising: engendering motion, relative to the laser scanning system, of an alignment target having a known geometry; determining a three-dimensional path of motion of the alignment target in the three- dimensional laser scanning coordinate system; capturing, with a camera system having a field of view in a two-dimensional vision coordinate system that is stationary with respect to the laser scanning system, an image sequence for tracking fiducials on a calibration surface of the alignment target responsive to the engendered motion; determining two-dimensional paths of motion of the fiducials tracked within the field of view of the image sequence; and mapping two-dimensional fiducial positions from the determined two-dimensional paths of motion to the determined three-dimensional path of motion.

2. The method of claim 1, in which: the engendered motion is rotational motion of the alignment target about a rotation axis; and mapping the two-dimensional fiducial positions to the determined three-dimensional path of motion comprises mapping the two-dimensional fiducial positions to the determined three-dimensional path of motion responsive to a detected center of rotation and a detected rotational position of the alignment target.

3. The method of claim 2, in which the detected rotational position is detected with an encoder of the laser scanning system.

4. The method of claim 1, in which the camera system is free of a reflective surface intervening between a light source and a nest of the laser scanning system.

5. The method of claim 1, in which the camera system includes a reflective surface for directing light downward into a nest of the laser scanning system.

6. The method of claim 1, in which the fiducials are intersections of lines defining a grid pattern applied to the alignment target.

7. The method of claim 1, in which the fiducials are holes drilled in the alignment target.

8. The method of claim 1, in which the two-dimensional fiducial positions shown in the field of view are mapped to coordinates in the three-dimensional laser scanning coordinate system.

9. The method of claim 1, further comprising processing the image sequence to develop a composite image of the calibration surface of the alignment target.

10. The method of claim 9, further comprising adjusting, based on uneven lighting of the composite image, perpendicularity of light from the camera system relative to the calibration surface.

11. The method of claim 9, further comprising adjusting, based on maximum brightness lighting of the composite image, perpendicularity of the camera system relative to the calibration surface.

12. The method of claim 9, in which the composite image of the calibration surface of the alignment target is a three hundred and sixty degree (360°) composite image of the calibration surface.

13. The method according to claim 1, further comprising: calibrating the laser scanning system based on the mapping.

14. The method according to claim 13, further comprising, after the calibrating: obtaining a three-dimensional point cloud of a workpiece feature, capturing an image of the workpiece feature with the camera system; and determining geometry and / or position of the workpiece feature based on the captured image.

15. The method according to claim 14, wherein a resolution of samples in the point cloud is insufficient to determining geometry and / or position of the workpiece feature.

Images