Fiducial markers and systems and methods including same

Abstract

The present disclosure provides a method including applying a fiducial marker to a workpiece and performing a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker. The method further includes performing a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker and performing at least one process based on the measurement of the geometric feature(s) of the workpiece. A fiducial marker is provided including information providing instructions for a process to modify a workpiece that is readable by at least one of an image capture equipment or a sensor. A system includes a fiducial marker and at least one non-transitory computer-readable medium. Additional methods are provided, including linking to a data store at least one process based on the measurement of information present on a fiducial marker or performing a process to modify a workpiece based on a measurement to determine the location of a fiducial marker.

Description

[0001] FIDUCIAL MARKERS AND SYSTEMS AND METHODS INCLUDING SAME

[0002] BACKGROUND

[0003] Robots are an interesting technology in that they effectively bridge the gap between the digital and physical, providing (physical) manipulation capability guided by (digital) computer programs and sensors. In the domain of industrial manufacturing, these machines enable flexible, repeatable automation well suited for applications that are dull, dirty, and / or dangerous. Robotics, however, are not without cost and, in general, require significant investment and operational expertise for a typical installation. In order to bridge the gap and enable wide-spread adoption of robots across more than just large manufacturing firms, the significant cost and operational expertise required must be addressed.

[0004] SUMMARY OF THE DISCLOSURE

[0005] In a first aspect, a method is provided. The method comprises a) applying a fiducial marker to a workpiece and b) performing a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker. The method further comprises c) performing a second scan of at least one geometric feature of the workpiece identified by the fiducial marker and d) performing at least one process based on the measurement of the at least one geometric feature of the workpiece.

[0006] In a second aspect, another method is provided. The method comprises a) applying a fiducial marker to a workpiece and b) performing a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker. The method further comprises c) performing a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker; and linking to a data store at least one process based on the measurement of information present on the fiducial marker.

[0007] In a third aspect, a fiducial marker is provided. The fiducial marker comprises a tape, a sticker, a sticky note, a tool, a magnet, a pin, a material that is consumable by a process to modify the workpiece, or a geometric object. The fiducial marker further comprises information providing instructions for a process to modify a workpiece. The information is readable by at least one of an image capture equipment or a sensor.

[0008] In a fourth aspect, a system is provided. The system comprises a fiducial marker of any embodiment according to the third aspect and at least one non-transitory computer-readable medium. The at least one non-transitory computer-readable medium is encoded with instructions that, upon execution, configure a processor for: a) receiving a measurement of a workpiece located in a robotic workspace to determine a location of the fiducial marker; and b) at least one of: 1) linking to a data store at least one process based on the measurement of information present on the fiducial marker; or 2) providing an output of instructions to perform at least one process based on the measurement of information present on the fiducial marker.

[0009] In a fifth aspect, an additional method is provided. The method comprises a) applying a fiducial marker to a workpiece; b) performing a measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; and c) performing at least one process to modify the workpiece based on the measurement.

[0010] The above summary of the present disclosure is not intended to describe each disclosed embodiment or every implementation of the present disclosure. The description that follows more particularly exemplifies illustrative embodiments. In several places throughout the application, guidance is provided through lists of examples, which examples may be used in various combinations. In each instance, the recited list serves only as a representative group and should not be interpreted as an exclusive list. Thus, the scope of the present disclosure should not be limited to the specific illustrative structures described herein, but rather extends at least to the structures described by the language of the claims, and the equivalents of those structures. Any of the elements that are positively recited in this specification as alternatives may be explicitly included in the claims or excluded from the claims, in any combination as desired. Although various theories and possible mechanisms may have been discussed herein, in no event should such discussions serve to limit the claimable subject matter.

[0011] BRIEF DESCRIPTION OF FIGURES

[0012] FIG. 1A illustrates a system in which example embodiments can be implemented.

[0013] FIG. IB illustrates an example system in accordance with embodiments herein.

[0014] FIG. 2 illustrates a method for performing a process in which example embodiments can be implemented.

[0015] FIG. 3 illustrates a method for linking a process to a data store in accordance with embodiments herein.

[0016] FIG. 4 illustrates a system including linking a process to a data store and / or performing a process, in which example embodiments can be implemented.

[0017] FIG. 5 illustrates a method for performing a process to modify a workpiece in which example embodiments can be implemented.

[0018] FIG. 6A illustrates one example sensor payload for use in sensing fiducial markers according to embodiments herein.

[0019] FIG. 6B illustrates one example sensor payload for use in robot tool calibration according to embodiments herein.

[0020] FIG. 7 illustrates a schematic diagram of one suitable deployment hardware setup for use according to embodiments herein.

[0021] FIG. 8 illustrates a method for calibrating a camera sensor mounted to the robot according to embodiments herein.

[0022] FIG. 9 illustrates a method for calibrating a laser scanner sensor mounted to the robot according to embodiments herein.

[0023] FIG. 10 illustrates a method for calibrating a touch probe mounted to the robot using a laser scanner according to embodiments herein. FIG. 11 illustrates a method for registration of a workpiece using a camera according to embodiments herein.

[0024] FIG. 12 illustrates a method for registration of a workpiece using a laser scanner according to embodiments herein.

[0025] FIG. 13 illustrates a method for computing a pose of a feature on a workpiece using each of a camera and a laser scanner according to embodiments herein.

[0026] FIG. 14 illustrates a method for programming a process to modify a workpiece according to embodiments herein.

[0027] FIG. 15 illustrates a method for executing a program of a mapped process to modify a workpiece according to embodiments herein.

[0028] FIG. 16 illustrates a high-level method of modifying a workpiece according to embodiments herein.

[0029] FIG. 17 is a photograph of an example of numerous fiducial markers applied to a workpiece according to embodiments herein.

[0030] FIG. 18 is a photograph of an example of an outer region of an area of a workpiece marked off with a tape fiducial marker according to embodiments herein.

[0031] FIG. 19 is a photograph of an example of a workpiece having a fiducial marker attached where a defect is located in a center hold of the sticky note fiducial marker according to embodiments herein.

[0032] FIG. 20A illustrates use of an abrasive disc fiducial marker as a consumable tool for a workpiece according to embodiments herein.

[0033] FIG. 20B illustrates modification of the workpiece of FIG. 20A with the abrasive disc fiducial marker.

[0034] FIG. 21 A illustrates use of a polishing tape fiducial marker as a consumable tool for a workpiece according to embodiments herein.

[0035] FIG. 2 IB illustrates modification of the workpiece of FIG. 21 A with the polishing tape fiducial marker.

[0036] FIG. 22A illustrates use of a tape fiducial marker to indicate where to look for an edge of a workpiece according to embodiments herein.

[0037] FIG. 22B illustrates dispensing a bead of material along the edge of the workpiece according to embodiments herein.

[0038] FIG. 23 illustrates a system in an example network architecture.

[0039] FIGS. 24-26 illustrate example computing devices that can be used in embodiments herein.

[0040] In the drawings, like reference numerals indicate like elements. While the above-identified drawings, which may not be drawn to scale, set forth various embodiments of the present disclosure, other embodiments are also contemplated, as noted in the Detailed Description. In all cases, this disclosure describes the presently disclosed disclosure by way of representation of exemplary embodiments and not by express limitations. It should be understood that numerous other modifications and embodiments can be devised by those skilled in the art, which fall within the scope and spirit of this disclosure.

[0041] DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0042] For the following Glossary of defined terms, these definitions shall be applied for the entire application, unless a different definition is provided in the claims or elsewhere in the specification.

[0043] Glossary

[0044] Certain terms are used throughout the description and the claims that, while for the most part are well known, may require some explanation. It should be understood that:

[0045] “workpiece” refers to any object being worked on with a tool or machine, and the term may be used interchangeably with “part” herein;

[0046] “workspace” refers to any area in which a robot operates;

[0047] “fiducial marker” refers to an object that provides a measurable feature to a workpiece; “measurement” refers to ascertaining information from a point, a line, and / or an area, using equipment or a user’s vision, and may include information from one or more measurements (e.g., a set of measurements); and

[0048] “geometric object” refers to an item having a shape (e.g., an edge, a plane, a sphere, a cube, a cylinder, etc.) that functions solely as a fiducial marker. This is opposed to an object that has at least one additional function, such as a tool, a tape, an abrasive disc, etc.

[0049] Methods and systems according to at least certain embodiments of the present disclosure advantageously provide a no / low-code approach to robotics programming that uses easily adhered (and often removable) fiducial markers as the primary system interface. Such an interface stands to provide an easy and intuitive way to program robots (e.g., on the shop floor) that greatly relaxes or removes the requirement of having expert programmers on site and thus alleviates one of the major inhibiting factors of widespread robotics adoption (e.g., to small and medium-sized manufacturing companies). One such use case is for weld grinding in job shops where part variety (and variance) is high and quantity relatively low (i.e., low-volume / high-mix). This is a representative example of where many businesses stand to benefit from robotics and automation but cannot justify the technical burden.

[0050] Additionally, robotic workspace (e.g., workcell) calibration tools are provided that satisfy the constraints of a typical industrial controls environment and tend to be easy to use, reliable, hardwareagnostic, and low-cost. In some embodiments, the calibration enables native offline frame-centric programming approach where all motions are programmed relative to (calibrated) reference frames in the workspace. Such relative moves are unique in that they move with the reference frame and are thus independent of the exact workspace configuration. This makes debugging and maintenance simpler as the problematic areas are isolated to a single source of truth, that is, the numerical identification of the reference frames as measured by calibration tools disclosed herein. Additionally, the issue of workspace replication, code reuse, and fleet management may be made simpler as at least some of the existing code can be reused with only the workspace-specific calibration / alignment data needing to change. As fiducial markers are easily identifiable and add structure to a workpiece, algorithms can be used to locate features with low-cost sensors and measure only the parts of interest with high precision (e.g., laser line profilometer or similar). By only measuring the parts that are required for processing, much time can be saved while simultaneously relaxing the need for accurate models (e.g., CAD models). With respect to part registration, the in-situ measuring natively solves the issue by not requiring any predefined (e.g., precision) jigs and / or fixturing. The only, and easily satisfied, requirement is that the workpiece must not move after imaging has begun. In some cases, a set of base fiducial markers may be provided that identify workpiece origins and / or datum features from which measured features and information are tracked. This solves the aforementioned registration issue, but also allows for process transfer from a “master” marked workpiece to identical (or same within tolerance) workpieces (e.g., all processing is saved with respect to some identified datum feature and is applied to later workpieces needing only the identified datum of the replicate workpieces).

[0051] FIG. 1A illustrates a system in which example embodiments of the present disclosure can be implemented. The system 100 includes main components of a controller 110, a fiducial marker 120, a measurement system 130, and an optional modifier 140. System 100 is illustrated in FIG. 1A as in communication with a data store 150. However, it is expressly contemplated that, in some embodiments, data store 150 may be local to, or integrated into system 100. Similarly, system 100 is illustrated as projecting to a display 10. However, it is expressly contemplated that system 100 may be integrated into a processor of a device that includes display 10. The system 100 may be implemented by one or more suitable computing devices in communication with each of these main components.

[0052] The controller 110 comprises a processor 111. A single processor 111 is depicted in system 100; however, it is expressly contemplated that processing for any operations described herein may be performed by any number of one or more processors in one or more locations. The processor 111 receives as inputs one or more of the following: at least one measurement of a workpiece in a robotic workspace (optionally two different measurements of the workpiece), a location of a fiducial marker on a workpiece, an orientation of a fiducial marker on a workpiece, and / or information present on a fiducial marker.

[0053] In some embodiments, when the processor 111 receives as an input a first measurement (of at least two measurements) of a workpiece in a robotic workspace to determine a location (and optionally an orientation) of a fiducial marker, the processor 111 outputs instructions to a measurement equipment controller 112 to perform a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker. In select embodiments, the first measurement is an optical measurement performed a user (with the user’s vision), then the user moves a robot to a position to perform the second measurement based on the location of the fiducial marker. Next, when the processor 111 receives as an input the second measurement including information present on the fiducial marker, the processor 111 outputs instructions to perform at least one process based on the second measurement and / or the processor 111 links the at least one process to a data store 150. In certain cases, a user may manually link information to instructions for a process to modify the workpiece using a robot. In some embodiments, when the processor 111 receives as an input a single measurement of a workpiece in a robotic workspace to determine a location of a fiducial marker, the processor 111 outputs instructions to the workpiece modification controller 114 to perform at least one process to modify the workpiece based on the measurement.

[0054] The controller 110 further comprises a measurement equipment controller 112. The measurement equipment controller 112 is sometimes included in the hardware of measurement equipment. There may be software pairing the measurement equipment controller 112 to a driver that sends commands to a measurement equipment controller 112 in the hardware and receives telemetry back. The measurement equipment controller 112 is configured to control operation of at least one measurement equipment, such as a first measurement equipment 131 and an optional second measurement equipment 132. The measurement equipment controller 112 receives as inputs instructions from the processor 111 to perform a measurement and outputs instructions to either the first measurement equipment 131 or the second measurement equipment 132 to implement the instructions and perform the measurement.

[0055] The controller 110 additionally comprises a robot motion controller 113. The robot controller 113 is sometimes included in a robot’s hardware. There may be software pairing the robot controller 113 to a driver that sends commands to a robot controller 113 in the hardware and receives telemetry back. The robot motion controller 113 receives as inputs instructions from the processor 111 and in response controls an articulated arm of a robot according to the instructions. In some embodiments, the robot motion controller 113 controls the articulated arm of the robot to move a workpiece. In some embodiments, the robot motion controller 113 controls the articulated arm to move a workpiece modifying equipment 141.

[0056] The controller 110 optionally comprises a modification equipment controller 114. The modification equipment controller 114 receives as inputs instructions from the processor 111 and in response controls a workpiece modifying equipment 141 to perform a modification of a workpiece. This is generally in sync with robot motion as well (i.e., the robot moves during modifications).

[0057] The controller 110 optionally also comprises a graphical user interface (GUI) generator 115, which may be configured to send information to a display 10. The GUI generator 115 may generate a graphical user interface for display on a display component 10 based on some or all of the information gathered or generated by the controller 110. Suitable displays include for instance and without limitation, a computer screen, a smart phone, or some other user device. Other units 116 may further be included in the controller 110. The controller 110 is described as having the functionality of receiving and sending communicable information to and from other devices. This may be done through an application program interface, for example, such that the controller 110 can receive and communicate with any units and / or models within each of the measurement system 130, the (optional) modifier 140, and the data store 150.

[0058] The system 100 also comprises a fiducial marker 120, which is an object that provides a reference point on a workpiece. The fiducial marker 120 gets adhered to a workpiece, typically by a user, although it is expressly contemplated that mechanical equipment such as a robot could attach the fiducial marker 120 to a workpiece. Advantageously, having a user affix the fiducial marker 120 to a workpiece provides a simple way to provide information about the workpiece. The object used as a fiducial marker 120 is not particularly limited. In some embodiments, the fiducial marker 120 comprises a tape, a sticker, a sticky note (e.g., a Post-it® Note from 3M Company (St. Paul, MN), a magnet, a pin, a material that is consumable by a process to modify the workpiece, a tool (e.g., an abrasive disc stuck to the workpiece by polish), or a geometric object. As noted above, a geometric object as used herein refers to an item having a three-dimensional shape that functions solely as a fiducial marker. Information provided by a fiducial marker 120 may include instructions for a process to modify the workpiece using a robot. Information can be included on the fiducial marker 120 by various means. For instance, in some embodiments the fiducial marker 120 comprises at least one of a color, a symbol, a geometric shape, a pattern, a matrix barcode (e.g., a quick-response (QR) code or an ArUco code), a word, an angle dependent dichroic or color shifting feature, a color calibration feature, a retroreflective feature, or a number. Optionally, the fiducial marker 120 is configured to be written on, for example to have a writable surface on at least a portion of the fiducial marker 120 (e.g., a tape that has a nonwoven backing and an acrylic adhesive layer). Advantageously, having a writable surface allows a user to apply a handwritten marking on the fiducial marker 120 to provide custom information to the fiducial marker 120. Often, the fiducial marker 120 is removable from the workpiece, although in some cases it is not necessary to remove the fiducial marker 120.

[0059] The system 100 further comprises a measurement system 130. The measurement system 130 comprises a first measurement equipment 131 and optionally a second measurement equipment 132. Other units 133 may further be included in the scanner system 130. The measurement output may vary significantly depending on the type of measurement equipment employed, e.g., one or more static images, a series of (e.g., height) maps, information from an air tag or a radio-frequency identification (RFID) tag, etc.

[0060] The first measurement equipment 131 is in some embodiments the only measurement equipment. The first measurement equipment 131 is not particularly limited but can include equipment that employs an optical measurement (either as equipment or as a user’s eyes), a touch measurement, or other frequency measurements. In select embodiments, a touch measurement is performed by a touch sensor. Various frequency measurements may include the output from an air tag or an RFID tag. In embodiments in which the first measurement equipment 131 is optical equipment, an optical measurement may be performed by an image capture equipment selected from the group consisting of a red, blue, and green (RBG) camera; a red, blue, green, and depth (RBGD) camera; a black and white (B&W) camera; and a three-dimensional (3D) image sensor. Alternatively, an optical measurement may be performed by a sensor selected from the group consisting of a laser profilometer, an area snapshot sensor, a triangulation-based sensor, a time-of-flight sensor, a laser point sensor, an optical coherence tomography sensor, a confocal sensor, and a dynamic vision sensor.

[0061] The optional second measurement equipment 132 equipment includes the image capture equipment and sensors mentioned above with respect to the first measurement equipment 131. When measurement is performed by equipment (not a user’s eyes), the first measurement equipment 131 and / or the optional second measurement equipment 132 may be mounted on a surface in the workspace. Often, the first measurement equipment 131 and / or the optional second measurement equipment 132 is mounted on a robot in the robotic workspace. A surface-mounted configuration is generally used to calibrate the robot’s tooling to itself (e.g., robot-mounted modification equipment). A surface-mounted configuration is also useful for registration when the process involves the robot holding the workpiece and the modification equipment mounted on a surface in the workspace. When the measurement equipment is mounted on the robot, it is used to measure items in the workspace; when mounted in the workspace, it is used to measure items on the robot.

[0062] The system 100 optionally also comprises a modifier 140. The modifier 140 comprises workpiece modifying equipment 141. Workpiece modifying equipment 141 may be a part of a robot or an end effector (e.g., a tool) removably attached to an articulated arm of a robot. The workpiece modifying equipment 141 is not particularly limited and may be configured to perform any process on a workpiece, such as at least one of grinding, deburring, denibbing, polishing, welding, painting, imparting an abrasive pattern, applying a wrap, forming a hole, tapping, screwing, boring, riveting, masking, chamfering, cutting, bending the workpiece, attaching the workpiece to a part, attaching parts to the workpiece, disassembling the workpiece, assembling the workpiece, or dispensing a material on the workpiece. Such processes and suitable apparatus to perform the processes are well known to those of skill in the art. Other units 142 may further be included in the modifier 140.

[0063] The system 100 additionally comprises a data store 150. The data store 150 is configured to communicate with the controller 110, the measurement system 130, and the (optional) modifier 140. The data store 150 may be local to the controller 110 or may be accessible through a cloud-based network. Similarly, while the controller 110 is illustrated in FIG. 1 A as local to the system 100, it is expressly contemplated that the controller 110 may be remote from the system 100 and may receive signals, and send commands, using a wireless or cloud-based network. The data store 150 comprises a measurement information unit 151, an optional process instructions unit 152, an optional calibration unit 153, an optional non-transitory computer-readable medium 154, and an optional registration unit 155. Other units 156 may further be included in the data store 150.

[0064] The data store 150 comprises a measurement information unit 151. The measurement information unit 151 contains information (e.g., the output) from each of the first measurement equipment 131 and the optional second measurement equipment 132.

[0065] The data store 150 optionally comprises a process instructions unit 152. The optional process instructions unit 152 contains information about one or more processes to perform on a workpiece. For example, the information may be obtained from a fiducial marker 120 and manually linked or linked by the processor 111 to a library of processes to modify a workpiece.

[0066] The data store 150 optionally comprises a calibration unit 153. The calibration unit 153 contains information about a calibration of the robotic workspace in which the workpiece is located. Often, the data store 150 comprises a non-transitory computer-readable medium 154. In some embodiments, at least one non-transitory computer-readable medium 154 is encoded with instructions that, upon execution, configure the processor 111 for: a) receiving a measurement of a workpiece located in a robotic workspace to determine a location of a fiducial marker; and b) at least one of: 1) linking to a data store 150 at least one process based on a measurement of information present on the fiducial marker; or 2) providing an output of instructions to perform at least one process based on a measurement of information present on the fiducial marker. Optionally, the at least one non-transitory computer-readable medium is encoded with instructions that, upon execution, further configure the processor 111 for receiving a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker.

[0067] The data store 150 optionally comprises a registration unit 155. The registration unit 155 contains information with respect to aligning the digital / intemal representation of the workpiece in the actual workspace.

[0068] Referring to FIG. IB, an example system is depicted, for which some possible interactions between certain components are depicted. A workpiece 160 is depicted as interacting between each of a first measurement equipment 131, an optional second measurement equipment 132, a workpiece modification controller 114, a robot motion controller 113, and a fiducial marker 120. A measurement information unit 151 is depicted as interacting with each of the first measurement equipment 131, the optional second measurement equipment 132, and a processor 111. Each of the first measurement equipment 131 and the optional second measurement equipment 132 is depicted as interacting between each of a measurement equipment controller 112, the workpiece 160, the measurement information unit 151, and the processor 111. The measurement equipment controller 112 is depicted as interacting between each of the first measurement equipment 131, the optional second measurement equipment 132, and the processor 111. The processor 111 is depicted as interacting between each of the measurement equipment controller 112, the robot motion controller 113, the workpiece modification controller 114, the first measurement equipment 131, the optional second measurement equipment 132, the measurement information unit 151, a process instructions unit 152, a calibration unit 153, a non-transitory computer- readable medium unit 154, and a registration unit 155. The workpiece modification controller 114 is depicted as interacting between each of the processor 111, the robot motion controller 113, and the workpiece 160.

[0069] In a first aspect, a method comprises: a) applying a fiducial marker to a workpiece; b) performing a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; c) performing a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker; and d) performing at least one process based on the measurement of the at least one geometric feature of the workpiece. Referring to FIG. 2, a flow chart is provided of method embodiments according to various implementations of the present disclosure. The method includes operation 210 to apply a fiducial marker to a workpiece and operation 220 to perform a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker. In some cases, the method includes an optional operation 230 to move a robot to a position to perform a second measurement based on the location of the fiducial marker. The robot may be moved manually by a user, or by sending instructions to a robot motion controller. In some embodiments, the method includes operation 240 to perform a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker. If desired, the method may include the operation 250 to manually link information to instructions for a process to modify the workpiece using a robot. Additionally, the method may also include operation 260 to perform at least one process based on the measurement of the at least one geometric feature of the workpiece.

[0070] Optionally, the workpiece is a first workpiece and the method further comprises repeating operations a) through d), with a second workpiece and a second fiducial marker that contains less information than the fiducial marker applied to the first workpiece.

[0071] In a second aspect, another method comprises: a) applying a fiducial marker to a workpiece; b) performing a first optical measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; c) performing a second optical measurement of at least one geometric feature of the workpiece identified by the fiducial marker; and d) linking to a data store at least one process based on the measurement of information present on the fiducial marker.

[0072] Optionally, the process or processes are manually linked to the data store. This may be done by having a user enter data about the process(es), such as via a graphical user interface.

[0073] Referring to FIG. 3, a flow chart is provided of method embodiments according to various implementations of the present disclosure. The method includes operation 310 to apply a fiducial marker to a workpiece and operation 320 to perform a first optical measurement of the workpiece located in a robotic workspace to determine a location (and optionally an orientation) of the fiducial marker. The method further includes operation 330 to perform a second optical measurement of at least one geometric feature of the workpiece identified by the fiducial marker and operation 340 to link to a data store at least one process based on the measurement of information present on the fiducial marker.

[0074] Suitable processes include, for instance, a calibration of the robotic workspace in which the workpiece is located, registration of the workpiece within the robotic workspace, and / or modifying the workpiece using a robot.

[0075] Regarding calibration of the robotic workspace in which the workpiece is located, an improved approach is provided that is in-situ (i.e., robot-in-the-loop) while replacing prior human-in-the-loop (i.e., eyeball) calibration methods with a low-cost sensor package. In this respect, the standard approach’s simplicity is maintained while improving repeatability and precision (i.e., to the robot’s precision) extrinsic calibration of both frames in the workspace as well as at tool alignment (e.g., tool center frame (TCF)). Fiducial requirements are minimal and limited to simple geometric shapes and / or sets of distinguishing features on the object of interest (e.g., spheres, planes, edges, cube-comers, etc.).

[0076] In some cases, a hybrid mix of macro and micro scale measuring provides algorithm optimization, auto initialization, and safety (e.g., via sanity checks and collision avoidance). For instance, FIG. 6A shows one example sensor payload 610 comprised of an RBG-Depth camera 620 paired with a high-precision laser line profilometer 630. This provides coarse (e.g., macro) 3D depth measurements with color along with fine (e.g., micro) 2D line scans. The coarse measurements serve for initial alignment and optionally also information encoding via a fiducial marker 640, while the fine measurements are used for the actual calibration algorithm. In this case, the fiducial marker 640 is a first fiducial marker and there is a second fiducial marker 650 also present on a surface 660 in the workspace that provides information. Additionally, the coarse sensor may be useful for operator teleoperation / monitoring, collision avoidance, as well as for providing algorithmic initialization (e.g., numerical initial guesses for optimizers). Advantageously, the coarse (first) measurement is still effective even when it has an accuracy substantially lower than an accuracy of the second measurement. For instance, in some cases, the accuracy of fiducial marker placement is 50% or less than the accuracy of the second measurement (of the geometric feature), such as 10%, 1%, or 0.1% or less of the accuracy of the second measurement. Traditionally, fiducial markers used for localization are measured directly and assumed to be perfectly placed (at least to the same accuracy as the measurements and subsequent processing). Methods according to at least certain embodiments of the present disclosure, with their second measurement, use the fiducial marker as a means to find a (subsequently measured) geometric feature and thus has to only be accurate enough to enable the detection of said feature for measurement. In this regard, the requirement on fiducial marker placement is much less strict; the fiducial marker only has to be near enough and with orientation enough to uniquely identify the geometric feature (for the second measurement) from the first measurement.

[0077] In this regard, the present disclosure provides capability that enables convenient human placement of fiducial markers while maintaining the accuracy / precision of the final process / system. As an example, it is often the case that one can uniquely identify a linear feature with a fiducial marker on the order of centimeters away from the feature, aligned such that an axis of the fiducial marker intersects the feature of interest. In this scenario, with a sensor / robot system capable of measuring and processing on the order of tens of microns, the requirements on fiducial marker placement are only 0.1% of the final measurement accuracy. Advantageously, such a method is effective even with fiducial marker placement accuracy substantially lower than the accuracy of the measurement system. In some cases, a first measurement may have an accuracy that is 50% or less than a second measurement accuracy, such as 10% or less, 1% or less, or 0.1% or less than the second measurement accuracy.

[0078] As shown in FIG. 6B, an example sensor pay load 610 comprised of an RBG-Depth camera 620 paired with a high-precision laser line profilometer 630 may be mounted to a surface 680 separately from a robot and used for robot tool 670 calibration by computing the tool and / or sensor frames relative to the robot (e.g., the tool center frame (TCF)).

[0079] One practical implementation of methods according to the present disclosure includes a hybrid personal computer (PC) - programmable logic controller (PLC) approach that handles all high-level sensor integration and algorithmic requirements in a PC while communicating with a traditional PLC running standard controls code. FIG. 7 depicts a schematic diagram of one suitable deployment hardware setup. As part of the architecture via pre-defined components, non-intrusive function blocks may be provided in the native language of the PLC that only run when granted permission. This greatly diminishes the integration burden for a controls engineer developing the main application while also ensuring the underlying safety (owned by the controls engineer) is upheld. For example, the PC portion of a system may include a PC 710, a first measurement equipment 720, and an optional second measurement equipment 730. The PLC portion of a system may include a PLC 740, a robot 750, a robot controller 760, and a fieldbus 770 as a communication protocol between the PLC 740 and the robot controller 760.

[0080] A variety of existing numerical optimization methods are available depending on the particular sensor modality / fiducial marker properties and the resulting numerical constraints. Examples are plentiful and include, for instance, point-plane, line-plane, line-sphere, line-edge, and the like. Suitable sensors include a touch sensor as well as the other equipment mentioned above with respect to the first measurement equipment 131.

[0081] Referring to FIG. 8, a flow chart is provided of a method for calibrating a camera sensor location relative to the robot’s mounting flange. The method assumes a camera is mounted rigidly to a flange of a robot with known intrinsic position from a previous standalone camera calibration. The method further assumes the presence of a calibration artifact in the robotic workspace with an approximately known position to ensure measurement via camera. In methods according to the present disclosure, the calibration artifact may be a fiducial marker, a set of fiducial markers, a checkerboard, or the like. The method includes operation 810 to move a robot to a unique pose with a calibration artifact in view of a camera sensor and operation 820 to image the calibration artifact. Next, the method includes operation 830 to compute an artifact pose in the camera frame and store as a pair with the current robot position (with the flange relative to the robot base). Operations 810, 820, and 830 are repeated n times. Typically, n is repeated enough times that an error residual reaches an acceptably low value for the particular application, as is known to those skilled in the art. Last, the method includes operation 840 to solve (simultaneously) for the camera pose (relative to the robot flange) and the artifact pose (relative to the robot base) using previously saved measurement-pose pairs. Optionally, after operation 840, the method returns to operation 810 to begin again. In some cases, the solved camera pose and / or artifact pose are saved in the calibration unit 153 of the data store 150 depicted in FIG. 1 A.

[0082] The solving operations for calibration are well-known and discussed in the literature. For calibration using a camera (e.g., in FIG. 8) there are several formulations including “AX=XB” and “AX=ZB”, articles describing these formulations and methods can be found below to provide additional details to those skilled in the art:

[0083] AX = XB:

[0084] - Daniilidis, Konstantinos. “Hand-Eye Calibration Using Dual Quaternions.” (1998).

[0085] - Horaud, Radu, and Fadi Domaika. “Hand-eye calibration.” The international journal of robotics research 14.3 (1995): 195-210.

[0086] - Park, Frank C., and Bryan J. Martin. “Robot sensor calibration: solving AX= XB on the Euclidean group.” IEEE Transactions on Robotics and Automation 10.5 (1994): 717-721.

[0087] - Tsai, Roger Y., and Reimar K. Lenz. “A new technique for fully autonomous and efficient 3 d robotics hand / eye calibration.” IEEE Transactions on robotics and automation 5.3 (1989): 345-358.

[0088] AX = ZB:

[0089] - Tsai, Roger Y., and Reimar K. Lenz. “A new technique for fully autonomous and efficient 3 d robotics hand / eye calibration.” IEEE Transactions on robotics and automation 5.3 (1989): 345-358.

[0090] - Zhuang, Hanqi, Zvi S. Roth, and Kuanchih Wang. “Robot calibration by mobile camera systems.” Journal of Robotic Systems 11.3 (1994): 155-167.

[0091] Referring to FIG. 9, a flow chart is provided of a method for calibrating a laser scanner location relative to the robot’s mounting flange. The method assumes a laser scanner is mounted rigidly to a flange of a robot. The method further assumes the presence of a calibration artifact in the robotic workspace with an approximately known position to ensure measurement via camera. In methods according to the present disclosure, the calibration artifact may be a fiducial marker such as a sphere, cube, plane, etc. The method includes operation 910 to move a robot to a unique pose with a calibration artifact in view of a laser scanner and an operation 920 to take a line scan of the calibration artifact and store as a pair with the current robot position (with the flange relative to the robot base). Operations 910 and 920 are repeated n times. Last, the method includes operation 930 to solve (simultaneously) for the laser scanner pose (relative to the robot flange) and the artifact pose (relative to the robot base) using previously saved measurement-pose pairs. Optionally, after operation 930, the method returns to operation 910 to begin again. In some cases, the solved laser scanner pose and / or artifact pose are saved in the calibration unit 153 of the data store 150 depicted in FIG. 1 A.

[0092] Calibration of robot-mounted laser scanners is a special form of the eye-hand problem where the sensor has lower dimensionality than the commonly-studied camera form. This area has its own focus of research in the literature. Solutions / approaches can be found below, broken up by configuration, to provide additional details to those skilled in the art.

[0093] 2D Eye-in-hand:

[0094] - Xu, Jing, et al. “Hand-eye calibration for 2D laser profile scanners using straight edges of common objects.” Robotics and Computer-Integrated Manufacturing 73 (2022): 102221.

[0095] - Li, Mingyang, et al. “A robot hand-eye calibration method of line laser sensor based on 3D reconstruction.” Robotics and Computer-Integrated Manufacturing 71 (2021): 102136. - Sharifzadeh, Sara, Istvan Biro, and Peter Kinnell. “Robust hand-eye calibration of 2D laser sensors using a single-plane calibration artefact.” Robotics and Computer-Integrated Manufacturing 61 (2020): 101823.

[0096] - Chen, Wenyu, et al. “A noise-tolerant algorithm for robot-sensor calibration using a planar disc of arbitrary 3-D orientation.” IEEE Transactions on Automation Science and Engineering 15.1 (2016): 251-263.

[0097] - Carlson, Fredrik Bagge, Rolf Johansson, and Anders Robertsson. “Six DOF eye-to-hand calibration from 2D measurements using planar constraints.” 2015 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2015.

[0098] - Hu, Jwu-Sheng, and Yung- Jung Chang. “Automatic calibration of hand-eye-workspace and camera using hand-mounted line laser.” IEEE / ASME Transactions on Mechatronics 18.6 (2012): 1778- 1786.

[0099] 2D Eye-to-hand:

[0100] - Xie, He, et al. “Hand-eye calibration and its accuracy analysis in robotic grinding.” 2015 IEEE International Conference on Automation Science and Engineering (CASE). IEEE, 2015.

[0101] Referring to FIG. 10, a flow chart is provided of a method for calibrating a touch probe and a laser scanner relative to the robot’s mounting flange. The method assumes a laser scanner is mounted rigidly in a robot’s workspace. The method also assumes a touch probe is mounted rigidly to a flange of the robot and that the probe geometry is known. Also, the method assumes that a pose of the touch probe is known enough to ensure measurement via the laser scanner. The method includes operation 1010 to move a robot to a unique pose (with a touch probe in view of a laser scanner) and operation 1020 to take a line scan of the touch probe and store as a pair with the current robot position (with the flange relative to the robot base). Operations 1010 and 1020 are repeated n times. Last, the method includes operation 1030 to solve (simultaneously) for the touch probe pose (relative to the robot flange) and the laser scanner pose (relative to the robot base) using previously saved measurement-pose pairs. Optionally, after operation 1030, the method returns to operation 1010 to begin again. In some cases, the solved touch probe pose and / or laser scanner pose are saved in the calibration unit 153 of the data store 150 depicted in FIG. 1A.

[0102] Robot-mounted touch probe calibration can be considered yet another (even lower dimension) variation of the traditional eye-hand calibration problem. Here the sensor signal is a binary response depending on if it is in contact with another physical object. The literature presents several works that focus on using touch probes as a way to calibrate the intrinsics of the robot. The touch probe calibration by itself (i.e., pose relative to the robot’s flange) is less studied. Works include the following, to provide additional details to those skilled in the art, related to robot intrinsic calibration:

[0103] - Messay-Kebede, Temesguen, George Sutton, and Ouboti Djaneye-Boundjou. “Geometry based self kinematic calibration method for industrial robots.” 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018. - Joubair, Ahmed, and Ilian A. Bonev. “Non-kinematic calibration of a six-axis serial robot using planar constraints.” Precision Engineering 40 (2015): 325-333.

[0104] - Joubair, Ahmed, and Ilian A. Bonev. “Kinematic calibration of a six-axis serial robot using distance and sphere constraints.” The International Journal of Advanced Manufacturing Technology 'll (2015): 515-523.

[0105] - Ikits, Milan, and John M. Hollerbach. “Kinematic calibration using a plane constraint.” Proceedings of International Conference on Robotics and Automation. Vol. 4. IEEE, 1997.

[0106] - Zhong, Xiao-Lin, JohnM. Lewis, and LN-Nagy Francis. “Autonomous robot calibration using a trigger probe.” Robotics and Autonomous Systems 18.4 (1996): 395-410.

[0107] - Zhong, Xiao-Lin, and John M. Lewis. “A new method for autonomous robot calibration.” Proceedings of 1995 IEEE international conference on robotics and automation. Vol. 2. IEEE, 1995.

[0108] Another possible process to perform according to methods of the present disclosure is registration of the workpiece within the robotic workspace. Registration refers to information aligning the digital / intemal representation of the workpiece in the actual workspace.

[0109] Referring to FIG. 11, a flow chart is provided of a method for registration of a workpiece using a camera. The method assumes a camera is mounted rigidly to a flange of a robot with both a known pose (extrinsics) and intrinsics via calibration, as previously mentioned with respect to FIGS. 8-10. The method also assumes that each workpiece is tagged with a unique fiducial marker. The method includes operation 1110 to move a robot to a unique pose and operation 1120 to take a camera image. The method further includes operation 1130 to find a fiducial marker in the image space and operation 1140 to, for each found marker, compute a work pose (using the robot / camera pose and calibration). Operations 1110, 1120, 1130, and 1140 are repeated n times. Last, the method includes operation 1150 to store the workpiece id and pose (relative to the robot base) for later use, e.g., in the registration unit 155 of the data store 150 depicted in FIG. 1A.

[0110] Referring to FIG. 12, a flow chart is provided of a method for registration of a workpiece using a laser scanner. The method assumes a laser scanner is mounted rigidly to a flange of a robot with a known pose (via calibration), as previously mentioned with respect to FIGS. 8-10. The method includes operation 1210 to move a robot to a unique pose (with a workpiece in view of a laser scanner) and operation 1220 to take a line scan of the workpiece and store as a pair with the current robot position (the flange relative to the robot base). Operations 1210 and 1220 are repeated n times. Last, the method includes operation 1230 to solve for the workpiece pose (relative to the robot base). Optionally, after operation 1230, the method returns to operation 1210 to begin again. The workpiece pose may be stored in the registration unit 155 of the data store 150 depicted in FIG. 1A.

[0111] In certain embodiments, a method according to FIG. 11 may be performed to provide a macro localization followed by performing a method according to FIG. 12 to provide a more precise micro localization.

[0112] Referring to FIG. 13, a flow chart is provided of a method for computing a pose of a feature on a workpiece using each of a camera and a laser scanner. The method assumes that the camera and the laser scanner have been calibrated as previously mentioned in FIG. 8 and FIG. 9, respectively. The method includes operation 1310 to compute a gross feature pose from a camera image (using fiducial marker information) and operation 1320 to, from an approximate (macro) pose, compute laser scan poses. Next, the method includes operation 1330 to move a robot to a scan pose and operation 1340 to take a laser scan (relative to a robot flange) and save as a pair with the robot pose (relative to the robot base).

[0113] Operations 1330 and 1340 are repeated n times. Last, the method includes operation 1350 to solve for the feature pose (relative to a frame of interest; i.e., the robot base and / or a workpiece frame). Optionally, after operation 1350, the method returns to operation 1330 and repeats operations 1330, 1340, and 1350.

[0114] Referring to FIG. 14, a flow chart is provided of a method for programming a process to modify a workpiece. The method includes operation 1410 to measure a workpiece for fiducial marker(s), operation 1420 to compute feature pose(s), and operation 1430 to identify reference features. The method further includes operation 1440 to compute feature poses (relative to the workpiece reference feature), operation 1450 to map processes to features (relative to the reference features), and operation 1460 to store a generated program of the mapped processes.

[0115] Referring to FIG. 15, a flow chart is provided of a method for executing a program of a mapped process to modify a workpiece. The method includes operation 1510 to measure a workpiece for fiducial marker(s) identifying reference feature(s) and operation f520 to localize the reference feature(s). The method further includes operation 1530 to load a (previously generated) program and operation 1540 to execute the program to modify a workpiece.

[0116] Referring to FfG. f6, a flow chart is provided of a high-level method of modifying a workpiece. The method includes operation 1610 to measure a workpiece for fiducial marker(s) and operation 1620 to compute feature pose(s). The method further includes operation 1630 to map a process to the feature pose(s) and operation 1640 to execute the process to modify the workpiece.

[0117] Advantageously, methods according to the present disclosure enable a variety of modifications of workpieces by generally including at least one of a fiducial marker-driven programming operation or a fiducial marker-driven operation for each different workpiece.

[0118] Regarding modifying the workpiece using a robot, any process for modifying a workpiece may be performed, such as at least one of grinding, deburring, denibbing, polishing, welding, painting, imparting an abrasive pattern, applying a wrap, forming a hole, tapping, screwing, boring, riveting, masking, chamfering, cutting, bending the workpiece, attaching the workpiece to a part, attaching a part to the workpiece, disassembling the workpiece, assembling the workpiece, or dispensing a material on the workpiece. As noted above, such processes and suitable apparatus to perform the processes are well known to those of skill in the art.

[0119] In select cases, the fiducial marker includes an object adhered to the workpiece that does not necessarily contain any information, but rather acts as a typical fiducial marker to provide a reference point on the workpiece. Such an option may be particularly useful when the process is a calibration of the robotic workspace and / or registration of the workpiece within the robotic workspace. In each of these cases identifying a location, an orientation, or both, of the workpiece is a main purpose of applying the fiducial marker to the workpiece. An object can be as simple as an edge, a plane, or a three-dimensional shape like a sphere attached to the workpiece.

[0120] In some cases, an applied fiducial marker marks a boundary of an area of the workpiece. Another example of base fiducial marker functionality is in the form of “keep-out” or “keep-away” fiducial markers to indicate regions that should be avoided during processing. Along these lines, generic fiducial markers can be used to add contrast for imaging to overcome any sensor limitations for highly specular / reflective, transparent, thin / small, especially fragile, etc. features on the workpiece to be processed.

[0121] Fiducial markers can be global in function (i.e., fiducial markings are locked to a particular functionality) or can be custom-mapped by the user for customer-specific use cases and functionality. For an example of the latter, a set of uniquely identifiable fiducial markers can be purchased and mapped (associated) to those processing functions that the user is interested in. This may greatly reduce the number of unique fiducial markers that must be produced.

[0122] In some embodiments of methods described herein, the first measurement, the second measurement, or both, may further identify any information present on the fiducial marker. The information can be used to specify any process, for instance a finishing / processing pattern, abrasive type / grit / etc., force, size, time, polish, etc. In select embodiments, a plurality of fiducial markers is used in a method, in which at least two of the fiducial markers provide different information than each other.

[0123] All process instructions are computed and stored relative to the workpiece’s base frame (defined by fiducial marker-identified datum features) and thus only require successful identification of the same datum features on subsequent parts for execution on a new part. These are identified on the original (e.g., master) workpiece by a fiducial marker. Subsequent parts, if close enough, can be automatically localized or, in the case of large displacements, localized requiring only fiducial markers needed to locate one or more of the original datum references. Advantageously, after information for a single workpiece has been computed and stored, methods according to certain embodiments accommodate transfer to other workpieces requiring only a designated reference. This can be in the form of a comer, set of edges, set of points, etc.; any set of features that can be used to unambiguously identify the spatial position and orientation (e.g., pose) of the workpiece.

[0124] In a third aspect, a fiducial marker is provided. The fiducial marker comprises a tape, a sticker, a sticky note, a tool, a magnet, a pin, a material that is consumable by a process to modify the workpiece, or a geometric object, wherein the fiducial marker further comprises information providing instructions for a process to modify a workpiece, the information readable by at least one of an image capture equipment or a sensor.

[0125] Information can be provided by a fiducial marker in numerous different ways. For example, via any one or more of a color, a symbol, a geometric shape, a pattern, a matrix barcode, a word, an angle dependent dichroic or color shifting feature, a color calibration feature, a retroreflective feature, or a number. Methods for finding fiducial markers in a measurement (e.g., by image capture equipment or a sensor) are openly available for various fiducial marker designs. For finding Aruco or RUNEtag in particular, solutions exist within the OpenCV computer vision library and the RUNEtag repository, respectively. These are released as open source under Apache and MIT licenses respectively. For proprietary tag identification, a user can write their own detection algorithms using more general approaches using known tools in the field of computer vision (e.g., combining various functions from the OpenCV library, etc.).

[0126] The fiducial markers may be used to denote a region of interest for processing while also providing process specification. For example, a line of “edge” tape may be applied along an edge of a workpiece along with modifiers that specify a “chamfering” operation as well as a “with radius equal 1 mm”. Base function fiducial markers can be used to specify point features (e.g., a weldment comer, etc.), linear features (e.g., a weldment edge, a weld bead, etc.), surface features (a planar area, etc.), area features (e.g., a region on a car door), etc. In this regard, the fiducial markers are used as part of a fiducial-assisted feature extraction.

[0127] A tape may be a convenient form factor for a fiducial marker. Suitable tapes include pressure sensitive adhesive tapes that are easily applied (and removed) by hand. Tapes are often sufficiently flexible to follow curved features and contours. Tapes optionally have printed markings and / or colors that denote their functionality and can be read at process time by the sensor on the robot. Additionally, they are also human-readable in a way that makes identification easy and intuitive for the operator. Some suitable tapes may include those commercially available under the trade designations “3M Masking Tape 2308”, “SCOTCH Printable Flatback Paper Tape 256”, “3M Vinyl Tape 4712”, or “3M General Purpose Vinyl Tape 764”, all from 3M Company (St. Paul, MN).

[0128] A sticker or a sticky note may also be a convenient form factor for a fiducial marker. Similar to a tape, a sticker may be easily applied (and removed) by hand. Additionally, stickers and sticky notes are provided as discrete items instead of having to be cut from a roll like most tapes. Sticky notes are also specifically designed to be written on, which may be useful for certain methods. Some suitable sticky notes may include those commercially available under the trade designations “POST-IT Notes” or “POST-IT Super Sticky Notes” from 3M Company (St. Paul, MN).

[0129] A magnet may be a convenient form factor for a fiducial marker for a ferromagnetic metal workpiece, while a pin may be a suitable fiducial marker for a workpiece having a puncturable surface or existing holes, bores, or the like.

[0130] In some embodiments, a fiducial marker may have additional functionality beyond marking and providing information. For instance, a consumable material or a tool may be a useful form factor for a fiducial marker. A suitable consumable material is a polish, which gets used up during the robot polishing process on a surface of the workpiece, or a fiducial marker made of a material that dissolves during the process. Similarly, a suitable tool is an abrasive disc, which may be adhered to a workpiece, e.g., by polish. In these cases, it may not be necessary to use a fiducial marker that only provides information as the tool may have information encoded directly on its surface. After a measurement, a robot may approach, attach to the tool, and process the workpiece. Another option would be to use a consumable material and / or a tool as a fiducial marker that indicates a geographic location on a workpiece, in addition to a separate fiducial marker (e.g., a tape) that provides information about the process to be performed with the consumable material and / or using the tool. For instance, information on a fiducial marker can specify details such as grit, polish, number of stages of sanding, etc.

[0131] Referring to FIG. 17, a photograph is provided of an exemplary workpiece 1700 to which several fiducial markers have been applied. Two edge tape fiducial markers 1710 having the letter “v” marked along the tapes are attached along opposing edges 1701 and 1702 of the workpiece 1700. The physical locations of the tapes indicate where an edge grind weld should be performed, for instance. Two round sticker fiducial markers 1720 are placed one each overlapping the edge tapes 1710, to indicate a specific property of the edge grind welding process. Additionally, two tape fiducial markers 1730 are located in the center of the workpiece 1700 outer surfaces 1703 as well as 1704, 1705, and 1706 to indicate that a planar surface finish is to be performed on those surfaces. Further, two sticker fiducial markers 1740 having the letters “DA” on each overlap the tapes 1730 to indicate a specific property of the planar surface finish.

[0132] The types of information provided by a fiducial marker are wide-ranging. A few non-limiting examples of information include the following. Instructing a robot to follow to the right of a tape, to the left of a tape, or the center of a tape; using multiple fiducial markers to define a path gradient; defining a straight path in the direction of an arrow on a fiducial marker; instructing a robot to follow a polygon shape of a workpiece in a certain direction and to round the comers; using many separate fiducial markers to instruct a robot to follow a shape in a certain direction; using several fiducial markers to instruct a robot to perform a process multiple times and optionally using different materials (e.g., three passes of sanding a workpiece using a succession of finer grit abrasives); providing a conditional instruction (e.g., continue processing until a specified surface finish is achieved); defining an area of specific shape and dimensions on a workpiece to be processed; instructing a robot to process the entire surface of a workpiece; instructing a robot to fill an area of a workpiece with a particular pattern (e.g., an artistic grind / sand pattern); and draping a fiducial marker over an edge of a workpiece to instruct a robot to fillet or chamfer the comer.

[0133] In a fourth aspect, a system is provided. The system comprises: a fiducial marker (according to any embodiment of the third aspect); and at least one non- transitory computer-readable medium encoded with instructions that, upon execution, configure a processor for: a) receiving a measurement of a workpiece located in a robotic workspace to determine a location of the fiducial marker; and b) at least one of:

[0134] 1) linking to a data store at least one process based on the measurement of information present on the fiducial marker; or 2) providing an output of instructions to perform at least one process based on the measurement of information present on the fiducial marker.

[0135] Referring to FIG. 4, a flow chart is provided of system embodiments according to various implementations of the present disclosure. The system includes a fiducial marker and at least one non- transitory computer-readable medium encoded with instructions that, upon execution, configure a processor for at least one non-transitory computer-readable medium encoded with instructions that, upon execution, configure a processor for operation 410 to receive a measurement of a workpiece located in a robotic workspace to determine a location of a fiducial marker, and at least one of operation 420 or operation 430. Operation 420 is to link to a data store at least one process based on the measurement of information present on the fiducial marker. Operation 430 is to provide an output of instructions to perform at least one process based on the measurement of information present on the fiducial marker. Further optionally, the at least one non-transitory computer-readable medium is encoded with instructions that, upon execution, further configure a processor for operation 440 to receive a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker. In some cases, the system contains a data store comprising a library of processes to modify a workpiece. Optionally, the library also includes objects to process as further information in the library. The non-transitory computer-readable medium may be installed on a PC that communicates with a PLC.

[0136] In a fifth aspect, another method is provided. The method comprises: a) applying a fiducial marker to a workpiece; b) performing a measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; and c) performing at least one process to modify the workpiece based on the measurement. The process to modify the workpiece may be any of the processes mentioned above.

[0137] Referring to FfG. 5, a flow chart is provided of method embodiments according to various implementations of the present disclosure. The method includes operation 510 to apply a fiducial marker to a workpiece; operation 520 to perform a measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; and operation 530 to perform at least one process to modify the workpiece based on the measurement, in some cases, the measurement is an optical measurement and the optical measurement is performed by an image capture equipment selected from the following: a red, blue, and green (RBG) camera; a red, blue, green, and depth (RBGD) camera; a black and white (B&W) camera; or a three-dimensional (3D) image sensor. In this method, only a single measurement is required. This may be useful when the same operation is going to be performed on numerous individual workpieces and the information regarding the process does not change.

[0138] In certain embodiments, the applied fiducial marker marks a boundary of an area of the workpiece. For instance, a worker may mark an outline on one or more particular areas of a car (e.g., doors, hoods, bumpers, etc.) and the method sands / scuffs the surface without requiring further action from the worker. This particular application is well-suited due to its “mass-individualized” nature in that the process is similar fundamentally, but it is every time on a different car, panel, color, etc. Of particular note is the general problem of digitally viewing highly specular / reflective parts (i.e., car paint). Methods according to the present disclosure may help greatly in this area. Referring to FIG. 18, a photograph is provided of a portion of a car door 1800, with the outer region of an area 1810 requiring polishing marked off with a tape fiducial marker 1820. An example workflow might be as follows:

[0139] 1. Tape off outer region to be processed.

[0140] 2. Take a RGBD camera measurement, as specular regions can be invisible to the camera, but the tape features are correctly identified and located.

[0141] 3. Spray process additive (e.g., polish onto measured region), which is non-contact and from a distance, so the partial measurement (with missing information) is sufficient as the tape was correctly registered.

[0142] 4. Take a high-fidelity surface measurement because viewing after applying the process additive results in sufficient contrast for an accurate precision measurement.

[0143] 5. Process the part based on information from the precision measurement.

[0144] Referring to FIG. 19, another example of a useful method includes applying to a car door workpiece 1900 a sticky note fiducial marker 1910 that has holes, with paint defects 1920 centered in a hole 1912 of the sticky note 1910. In this case, the defect size / type is encoded by color and / or fiducial on the chosen marking sticky note (or could be encoded by another fiducial marker such as a sticker). The car is measured and then processed. The sticky notes can be moved manually or at process time.

[0145] Referring to FIGS. 20A and 20B, an additional example of a useful method includes applying to a workpiece 2020 a fiducial marker 2010 that is a consumable tool. In this case, the consumable tool is an abrasive disc that is encoded with information including at least one symbol. The workpiece 2020 is measured by an image capture equipment 2030 (e.g., a laser line profilometer). Information contained on the fiducial marker 2010 provides instruction for a robot 2050 to attach to the fiducial marker 2010 and modify (e.g., sand) the workpiece 2020 using the abrasive disc fiducial marker 2010. Optionally, a polish (not shown) is employed to apply the fiducial marker 2010 to the workpiece 2020, and the polish is consumed during the modification of the workpiece 2020. At the completion of the workpiece 2020 modification, the fiducial marker 2010 may be removed from the robot 2050 and discarded.

[0146] Referring to FIGS. 21 A and 2 IB, an additional example of a useful method includes applying to a workpiece 2120 a fiducial marker 2110 that is a consumable material. In this case, the consumable material is polishing tape that is encoded with information including a color. The workpiece 2120 is measured by an image capture equipment 2130 (e.g., a laser line profilometer). Information contained on the fiducial marker 2110 provides instruction for a robot 2150 to modify (i.e., polish) the workpiece 2120 using the polishing tape fiducial marker 2110. The polishing tape fiducial marker 2110 is consumed during the modification of the workpiece 2120.

[0147] Referring to FIGS. 22A and 22B, a further example of a useful method includes applying to a workspace surface 2200 a tape fiducial marker 2210 that indicates a direction to look for a rising edge of a workpiece 2220 in a measurement. A user then just needs to specify the edge offset and measurement height. A scanning plane 2230 is shown and travels in the direction of the arrow D. Following the measurement, information from the tape 2210 informs where the robot 2240 should dispense a bead of material 2250 along the edge of the workpiece 2220. Advantageously, this method eliminates the need for a CAD model that provides workpiece topology information.

[0148] FIG. 23 illustrates a system architecture. Architecture 2300 illustrates one embodiment of an implementation of a system 2310. As an example, architecture 2300 can provide computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various embodiments, remote servers can deliver the services over a wide area network, such as the internet, using appropriate protocols. For instance, remote servers can deliver applications over a wide area network and they can be accessed through a web browser or any other computing component. Software or components shown or described in FIGS. 1-22 as well as the corresponding data, can be stored on servers at a remote location. The computing resources in a remote server environment can be consolidated at a remote data center location or they can be dispersed. Remote server infrastructures can deliver services through shared data centers, even though they appear as a single point of access for the user. Thus, the components and functions described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, they can be provided by a conventional server, installed on client devices directly, or in other ways.

[0149] In the example shown in FIG. 23, some items are similar to those shown in earlier figures. FIG. 23 specifically shows that a controller 2310 can be located at a remote server location 2302. Therefore, a computing device 2320 accesses the controller 2310 through the remote server location 2302. A user 2350 can use the computing device 2320 to access user interfaces 2322 as well. For example, a user 2350 may be a user wanting to check on the progress of processing a workpiece while sitting in a parking lot, and interacting with an application on the user interface 2322 of their smartphone 2320, or laptop 2320, or other computing device 2320, e.g., an augmented reality (AR) device such as AR glasses.

[0150] FIG. 23 shows that it is also contemplated that some elements of systems described herein are disposed at a remote server location 2302 while others are not. By way of example, each of a data store 2330, a measurement system 2360, and the modifier 2370 can be disposed at a location separate from the location 2302 and accessed through the remote server at location 2302. Regardless of where it is located, the data store 2330 can be accessed directly by a computing device 2320, through a network (either a wide area network or a local area network), hosted at a remote site by a service, provided as a service, or accessed by a connection service that resides in a remote location. Also, the data can be stored in substantially any location and intermittently accessed by, or forwarded to, interested parties. For instance, physical carriers can be used instead of, or in addition to, electromagnetic wave carriers. This may allow a user 2350 to interact with the controller 2310 through their computing device 2320.

[0151] It will also be noted that the elements of systems described herein, or portions of them, can be disposed on a wide variety of different devices. Some of those devices include servers, desktop computers, laptop computers, imbedded computer, industrial controllers, tablet computers, or other mobile devices, such as palm top computers, cell phones, smart phones, multimedia players, personal digital assistants, etc.

[0152] FIGS. 24-25 illustrate example devices that can be used in the embodiments shown in previous Figures. FIG. 24 illustrates an example mobile device that can be used in the embodiments shown in previous Figures. FIG. 24 is a simplified block diagram of one illustrative example of a handheld or mobile computing device that can be used as either a worker’s device or a supervisor / safety officer device, for example, in which the present system (or parts of it) can be deployed. For instance, a mobile device can be deployed in the operator compartment of computing device for use in generating, processing, or displaying the data.

[0153] FIG. 24 provides a general block diagram of the components of a mobile cellular device 2416 that can mn some components shown and described herein. The mobile cellular device 2416 interacts with them or runs some and interacts with some. In the device 2416, a communications link 2413 is provided that allows the handheld device to communicate with other computing devices and under some embodiments provides a channel for receiving information automatically, such as by scanning. Examples of communications link 2413 include allowing communication though one or more communication protocols, such as wireless services used to provide cellular access to a network, as well as protocols that provide local wireless connections to networks.

[0154] In other examples, applications can be received on a removable Secure Digital (SD) card that is connected to an interface 2415. The interface 2415 and communication links 2413 communicate with a processor 2417 (which can also embody a processor) along a bus 2419 that is also connected to a memory 2421 and input / output (I / O) components 2423, as well as clock 2425 and location system 2427.

[0155] I / O components 2423, in one embodiment, are provided to facilitate input and output operations and the device 2416 can include input components such as buttons, touch sensors, optical sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors and output components such as a display device, a speaker, and or a printer port. Other I / O components 2423 can be used as well.

[0156] The clock 2425 illustratively comprises a real time clock component that outputs a time and ate. It can also provide timing functions for the processor 2417.

[0157] Illustratively, the location system 2427 includes a component that outputs a current geographical location of the device 2416. This can include, for instance, a global positioning system (GPS) receiver, a LORAN system, a dead reckoning system, a cellular triangulation system, or other positioning system. It can also include, for example, mapping software or navigation software that generates desired maps, navigation routes and other geographic functions.

[0158] A memory 2421 stores operating system 2429, network settings 2431, applications 2433, application configuration settings 2435, data store 2437, communication drivers 2439, and communication configuration settings 2441. The memory 2421 can include all types of tangible volatile and non-volatile computer-readable memory devices, ft can also include computer storage media (described below). Memory 2421 stores computer readable instructions that, when executed by the processor 2417, cause the processor to perform computer-implemented steps or functions according to the instructions. Processor 2417 can be activated by other components to facilitate their functionality as well. It is expressly contemplated that, while a physical memory store 2421 is illustrated as part of a device, that cloud computing options, where some data and / or processing is done using a remote service, are available.

[0159] FIG. 25 shows that the device can also be a smart phone 2571. The smart phone 2571 has a touch sensitive display 2573 that displays icons or tiles or other user input mechanisms 2575. Mechanisms 2575 can be used by a user to run applications, make calls, perform data transfer operations, etc. In general, the smart phone 2571 is built on a mobile operating system and offers more advanced computing capability and connectivity than a feature phone. Note that other forms of the devices are possible. However, while FIG. 25 illustrates an embodiment where a device 2500 is a smart phone 2571, it is expressly contemplated that a display may be presented on another computing device.

[0160] FIG. 26 is one example of a computing environment in which elements of systems and methods described herein, or parts of them (for example), can be deployed. With reference to FIG. 26, an example system for implementing some embodiments includes a general-purpose computing device in the form of a computer 2610. Components of the computer 2610 may include, but are not limited to, a processing unit 2620 (which can comprise a processor), a system memory 2630, and a system bus 2621 that couples various system components including the system memory to the processing unit 2620. The system bus 2621 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Memory and programs described with respect to systems and methods described herein can be deployed in corresponding portions of FIG. 26.

[0161] The computer 2610 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by the computer 2610 and includes both volatile / nonvolatile media and removable / non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media is different from, and does not include, a modulated data signal or carrier wave. It includes hardware storage media including both volatile / nonvolatile and removable / non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile discs (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disc storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer 2610. Communication media may embody computer readable instructions, data structures, program modules or other data in a transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The system memory 2630 includes computer storage media in the form of volatile and / or nonvolatile memory such as read only memory (ROM) 2631 and random-access memory (RAM) 2632. A basic input / output system 2633 (BIOS) containing the basic routines that help to transfer information between elements within the computer 2610, such as during start-up, is typically stored in ROM 2631. RAM 2632 typically contains data and / or program modules that are immediately accessible to and / or presently being operated on by processing unit 2620. By way of example, and not limitation, FIG. 26 illustrates an operating system 2634, application programs 2635, other program modules 2636, and program data 2637.

[0162] The computer 2610 may also include other removable / non-removable and volatile / nonvolatile computer storage media. By way of example only, FIG. 26 illustrates a hard disc drive 2641 that reads from or writes to non-removable, nonvolatile magnetic media, nonvolatile magnetic disc 2652, an optical disc drive 2655, and nonvolatile optical disc 2656. The hard disc drive 2641 is typically connected to the system bus 2621 through a non-removable memory interface such as interface 2640, and optical disc drive 2655 is typically connected to the system bus 2621 by a removable memory interface, such as interface 2650.

[0163] Alternatively, or in addition, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (e.g., ASICs), Application-specific Standard Products (e.g., ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0164] The drives and their associated computer storage media discussed above and illustrated in FIG. 26, provide storage of computer readable instructions, data structures, program modules and other data for the computer 2610. In FIG. 26, for example, a hard disc drive 2641 is illustrated as storing operating system 2644, application programs 2645, other program modules 2646, and program data 2647. Note that these components can either be the same as or different from operating system 2634, application programs 2635, other program modules 2636, and program data 2637.

[0165] A user may enter commands and information into the computer 2610 through input devices such as a keyboard 2662, a microphone 2663, and a pointing device 2661, such as a mouse, trackball or touch pad. Other input devices (not shown) may include a joystick, game pad, satellite receiver, scanner, or the like. These and other input devices are often connected to the processing unit 2620 through a user input interface 2660 that is coupled to the system bus but may be connected by other interface and bus stmctures. A visual display 2691 or other type of display device is also connected to the system bus 2621 via an interface, such as a video interface 2690. In addition to the monitor, computers may also include other peripheral output devices such as speakers 2697 and printer 2696, which may be connected through an output peripheral interface 2695.

[0166] The computer 2610 is operated in a networked environment using logical connections, such as a Local Area Network (LAN) or Wide Area Network (WAN) to one or more remote computers, such as a remote computer 2680. When used in a LAN networking environment, the computer 2610 is connected to the LAN 2671 through a network interface or adapter 2670. When used in a WAN networking environment, the computer 2610 typically includes a modem 2672 or other means for establishing communications over the WAN 2673, such as the Internet. In a networked environment, program modules may be stored in a remote memory storage device. FIG. 26 illustrates, for example, that remote application programs 2685 can reside on a remote computer 2680.

[0167] In the present detailed description of the preferred embodiments, reference is made to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. The illustrated embodiments are not intended to be exhaustive of all embodiments according to the invention. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0168] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood as being modified in all instances by the term “about.” Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein.

[0169] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” encompass embodiments having plural referents, unless the content clearly dictates otherwise. As used in this specification and the appended claims, the term “or” is generally employed in its sense including "and / or" unless the content clearly dictates otherwise.

[0170] As used herein, when an element, component, or layer for example is described as forming a “coincident interface” with, or being “on,” “connected to,” “coupled with,” “stacked on” or “in contact with” another element, component, or layer, it can be directly on, directly connected to, directly coupled with, directly stacked on, in direct contact with, or intervening elements, components or layers may be on, connected, coupled or in contact with the particular element, component, or layer, for example. When an element, component, or layer for example is referred to as being “directly on,” “directly connected to,” “directly coupled with,” or “directly in contact with” another element, there are no intervening elements, components or layers for example. The techniques of this disclosure may be implemented in a wide variety of computer devices, such as servers, laptop computers, desktop computers, notebook computers, tablet computers, hand-held computers, smart phones, and the like. Any components, modules or units have been described to emphasize functional aspects and do not necessarily require realization by different hardware units. The techniques described herein may also be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules, units or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. In some cases, various features may be implemented as an integrated circuit device, such as an integrated circuit chip or chipset. Additionally, although a number of distinct modules have been described throughout this description, many of which perform unique functions, all the functions of all of the modules may be combined into a single module, or even split into further additional modules. The modules described herein are only exemplary and have been described as such for better ease of understanding.

[0171] If implemented in software, the techniques may be realized at least in part by a computer- readable medium comprising instructions that, when executed in a processor, performs one or more of the methods described above. The computer-readable medium may comprise a tangible computer-readable storage medium and may form part of a computer program product, which may include packaging materials. The computer-readable storage medium may comprise random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The computer-readable storage medium may also comprise a non-volatile storage device, such as a hard-disc, magnetic tape, a compact disc (CD), digital versatile disc (DVD), Blu-ray disc, holographic data storage media, or other nonvolatile storage device.

[0172] The term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured for performing the techniques of this disclosure. Even if implemented in software, the techniques may use hardware such as a processor to execute the software, and a memory to store the software. In any such cases, the computers described herein may define a specific machine that is capable of executing the specific functions described herein. Also, the techniques could be fully implemented in one or more circuits or logic elements, which could also be considered a processor.

[0173] Listing of Exemplary Embodiments

[0174] In a first embodiment, the present disclosure provides a method. The method comprises a) applying a fiducial marker to a workpiece and b) performing a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker. The method further comprises c) performing a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker and d) performing at least one process based on the measurement of the at least one geometric feature of the workpiece.

[0175] In a second embodiment, the present disclosure provides a method according to the first embodiment, wherein the at least one process comprises a calibration of the robotic workspace in which the workpiece is located.

[0176] In a third embodiment, the present disclosure provides a method according to the first embodiment or the second embodiment, wherein the at least one process comprises registration of the workpiece within the robotic workspace. In a fourth embodiment, the present disclosure provides a method according to any of the first through third embodiments, wherein the at least one process comprises modifying the workpiece using a robot.

[0177] In a fifth embodiment, the present disclosure provides a method according to the fourth embodiment, wherein the modifying comprises at least one of grinding, deburring, denibbing, polishing, welding, painting, imparting an abrasive pattern, applying a wrap, forming a hole, tapping, screwing, boring, riveting, masking, chamfering, cutting, bending the workpiece, attaching the workpiece to a part, disassembling the workpiece, or dispensing a material on the workpiece.

[0178] In a sixth embodiment, the present disclosure provides a method according to any of the first through fifth embodiments, wherein the first measurement is an optical measurement and the first optical measurement is performed by an image capture equipment selected from the group consisting of a red, blue, and green (RBG) camera; a red, blue, green, and depth (RBGD) camera; a black and white (B&W) camera; and a three-dimensional (3D) image sensor.

[0179] In a seventh embodiment, the present disclosure provides a method according to the sixth embodiment, wherein the image capture equipment is mounted on a robot in the robotic workspace.

[0180] In an eighth embodiment, the present disclosure provides a method according to any of the first through seventh embodiments, wherein the first measurement is an optical measurement performed by a user and the method further comprises operation e) performed after operation b) and before operation c), wherein operation e) comprises moving a robot to a position to perform the second measurement based on the location of the fiducial marker.

[0181] In a ninth embodiment, the present disclosure provides a method according to any of the first through fifth embodiments, wherein the first measurement is a touch measurement and the touch measurement is performed by a touch sensor mounted on a robot in the robotic workspace.

[0182] In a tenth embodiment, the present disclosure provides a method according to any of the first through ninth embodiments, wherein the second measurement is performed by a sensor selected from the group consisting of a laser profilometer, an area snapshot sensor, a triangulation-based sensor, a time-of- flight sensor, a laser point sensor, an optical coherence tomography sensor, a confocal sensor, and a dynamic vision sensor.

[0183] In an eleventh embodiment, the present disclosure provides a method according to the tenth embodiment, wherein the sensor is mounted on a robot in the robotic workspace.

[0184] In a twelfth embodiment, the present disclosure provides a method according to any of the first through eleventh embodiments, wherein at least one of the first measurement or the second measurement further identifies any information present on the fiducial marker.

[0185] In a thirteenth embodiment, the present disclosure provides a method according to the twelfth embodiment, wherein the information includes instructions for a process to modify the workpiece using a robot. In a fourteenth embodiment, the present disclosure provides a method according to the twelfth embodiment, wherein, after measuring, the information is manually linked to instructions for a process to modify the workpiece using a robot.

[0186] In a fifteenth embodiment, the present disclosure provides a method according to any of the first through fourteenth embodiments, wherein the fiducial marker comprises an object adhered to the workpiece.

[0187] In a sixteenth embodiment, the present disclosure provides a method according to any of the first through fourteenth embodiments, wherein the fiducial marker comprises a tape, a sticker, a sticky note, a magnet, a pin, a material that is consumable by a process to modify the workpiece, a tool, or a geometric object.

[0188] In a seventeenth embodiment, the present disclosure provides a method according to any of the first through sixteenth embodiments, wherein the fiducial marker comprises at least one of a color, a symbol, a geometric shape, a pattern, a matrix barcode, a word, an angle dependent dichroic or color shifting feature, a color calibration feature, a retroreflective feature, or a number.

[0189] In an eighteenth embodiment, the present disclosure provides a method according to any of the first through seventeenth embodiments, wherein the fiducial marker comprises a handwritten marking.

[0190] In a nineteenth embodiment, the present disclosure provides a method according to any of the first through eighteenth embodiments, wherein the applied fiducial marker marks a boundary of an area of the workpiece.

[0191] In a twentieth embodiment, the present disclosure provides a method according to any of the first through nineteenth embodiments, wherein the fiducial marker comprises a plurality of fiducial markers, wherein at least two of the fiducial markers provide different information than each other.

[0192] In a twenty -first embodiment, the present disclosure provides a method according to any of the first through twentieth embodiments, wherein the workpiece is a first workpiece and the method further comprises repeating operations a) through d), with a second workpiece and a second fiducial marker that contains less information than the fiducial marker applied to the first workpiece.

[0193] In a twenty-second embodiment, the present disclosure provides another method. The method comprises a) applying a fiducial marker to a workpiece and b) performing a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker. The method further comprises c) performing a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker; and linking to a data store at least one process based on the measurement of information present on the fiducial marker.

[0194] In a twenty -third embodiment, the present disclosure provides a method according to the twenty- second embodiment, wherein the at least one process is manually linked to the data store.

[0195] In a twenty -fourth embodiment, the present disclosure provides a method according to the twenty-second embodiment or the twenty -third embodiment, wherein the performing a first measurement of the workpiece located in a robotic workspace further determines an orientation of the fiducial marker. In a twenty -fifth embodiment, the present disclosure provides a fiducial marker. The fiducial marker comprises a tape, a sticker, a sticky note, a tool, a magnet, a pin, a material that is consumable by a process to modify the workpiece, or a geometric object. The fiducial marker further comprises information providing instructions for a process to modify a workpiece. The information is readable by at least one of an image capture equipment or a sensor.

[0196] In a twenty-sixth embodiment, the present disclosure provides a fiducial marker according to the twenty -fifth embodiment, wherein the process to modify a workpiece comprises at least one of grinding, deburring, denibbing, polishing, welding, painting, imparting an abrasive pattern, applying a wrap, forming a hole, tapping, screwing, boring, riveting, masking, chamfering, cutting, bending the workpiece, attaching the workpiece to a part, disassembling the workpiece, or dispensing a material on the workpiece.

[0197] In a twenty-seventh embodiment, the present disclosure provides a fiducial marker according to the twenty -fifth embodiment or the twenty-sixth embodiment, wherein the fiducial marker comprises at least one of a color, a symbol, a geometric shape, a pattern, a matrix barcode, a word, an angle dependent dichroic or color shifting feature, a color calibration feature, a retroreflective feature, or a number.

[0198] In a twenty -eighth embodiment, the present disclosure provides a system. The system comprises a fiducial marker according to any of the twenty -fifth through twenty-seventh embodiments and at least one non-transitory computer-readable medium. The at least one non-transitory computer-readable medium is encoded with instructions that, upon execution, configure a processor for: a) receiving a measurement of a workpiece located in a robotic workspace to determine a location of the fiducial marker; and b) at least one of: 1) linking to a data store at least one process based on the measurement of information present on the fiducial marker; or 2) providing an output of instructions to perform at least one process based on the measurement of information present on the fiducial marker.

[0199] In a twenty -ninth embodiment, the present disclosure provides a system according to the twentyeighth embodiment, wherein the at least one non-transitory computer-readable medium is encoded with instructions that, upon execution, further configure a processor for receiving a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker.

[0200] In a thirtieth embodiment, the present disclosure provides a system according to the twentyeighth embodiment or the twenty -ninth embodiment, further comprising a data store comprising a library of processes to modify a workpiece.

[0201] In a thirty -first embodiment, the present disclosure provides an additional method. The method comprises a) applying a fiducial marker to a workpiece; b) performing a measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; and c) performing at least one process to modify the workpiece based on the measurement.

[0202] In a thirty-second embodiment, the present disclosure provides a method according to the thirty- first embodiment, wherein the process to modify the workpiece comprises at least one of grinding, deburring, denibbing, polishing, welding, painting, imparting an abrasive pattern, applying a wrap, forming a hole, tapping, screwing, boring, riveting, masking, chamfering, cutting, bending the workpiece, attaching the workpiece to a part, disassembling the workpiece, or dispensing a material on the workpiece.

[0203] In a thirty -third embodiment, the present disclosure provides a method according to the thirty- first embodiment or the thirty-second embodiment, wherein the measurement is an optical measurement and the optical measurement is performed by an image capture equipment selected from the group consisting of a red, blue, and green (RBG) camera; a red, blue, green, and depth (RBGD) camera; a black and white (B&W) camera; and a three-dimensional (3D) image sensor.

[0204] In a thirty -fourth embodiment, the present disclosure provides a method according to any of the first through twenty -first embodiments, wherein an accuracy of placement of the fiducial marker is 50% or less than an accuracy of the second measurement, 10%, 1%, or 0.1% or less than the accuracy of the second measurement.

[0205] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations can be substituted for the specific embodiments shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this disclosure be limited only by the claims and the equivalents thereof.

[0206] Furthermore, all publications and patents referenced herein are incorporated by reference in their entirety to the same extent as if each individual publication or patent was specifically and individually indicated to be incorporated by reference. In the event of inconsistencies or contradictions between portions of the incorporated references and this application, the information in the preceding description prevails.

Claims

What Is Claimed Is:

1. A method comprising: a) applying a fiducial marker to a workpiece; b) performing a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; c) performing a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker; and d) performing at least one process based on the measurement of the at least one geometric feature of the workpiece.

2. The method of claim 1, wherein the at least one process comprises a calibration of the robotic workspace in which the workpiece is located.

3. The method of claim 1 or claim 2, wherein the at least one process comprises registration of the workpiece within the robotic workspace.

4. The method of any of claims 1 to 3, wherein the at least one process comprises modifying the workpiece using a robot.

5. The method of claim 4, wherein the modifying comprises at least one of grinding, deburring, denibbing, polishing, welding, painting, imparting an abrasive pattern, applying a wrap, forming a hole, tapping, screwing, boring, riveting, masking, chamfering, cutting, bending the workpiece, attaching the workpiece to a part, disassembling the workpiece, or dispensing a material on the workpiece.

6. The method of any of claims 1 to 5, wherein the first measurement is an optical measurement and the first optical measurement is performed by an image capture equipment selected from the group consisting of a red, blue, and green (RBG) camera; a red, blue, green, and depth (RBGD) camera; a black and white (B&W) camera; and a three-dimensional (3D) image sensor.

7. The method of any of claims 1 to 6, wherein the first measurement is an optical measurement performed by a user and the method further comprises operation e) performed after operation b) and before operation c), wherein operation e) comprises moving a robot to a position to perform the second measurement based on the location of the fiducial marker.

8. The method of any of claims 1 to 7, wherein the second measurement is performed by a sensor selected from the group consisting of a laser profilometer, an area snapshot sensor, a triangulation-based sensor, a time-of-flight sensor, a laser point sensor, an optical coherence tomography sensor, a confocal sensor, a touch sensor, and a dynamic vision sensor.

9. The method of any of claims 1 to 8, wherein at least one of the first measurement or the second measurement further identifies any information present on the fiducial marker.

10. The method of claim 9, wherein the information includes instructions for a process to modify the workpiece using a robot.

11. The method of claim 9, wherein, after measuring, the information is manually linked to instructions for a process to modify the workpiece using a robot. f2. The method of any of claims 1 to 11, wherein the fiducial marker comprises an object adhered to the workpiece.

13. The method of any of claims 1 to 11, wherein the fiducial marker comprises a tape, a sticker, a sticky note, a magnet, a pin, a material that is consumable by a process to modify the workpiece, a tool, or a geometric object.

14. The method of any of claims 1 to 13, wherein the fiducial marker comprises at least one of a color, a symbol, a geometric shape, a pattern, a matrix barcode, a word, an angle dependent dichroic or color shifting feature, a color calibration feature, a retroreflective feature, or a number.

15. The method of any of claims 1 to 14, wherein the fiducial marker comprises a handwritten marking.

16. The method of any of claims 1 to 15, wherein the applied fiducial marker marks a boundary of an area of the workpiece.

17. The method of any of claims 1 to 16, wherein the fiducial marker comprises a plurality of fiducial markers, wherein at least two of the fiducial markers provide different information than each other.

18. The method of any of claims 1 to 17, wherein the workpiece is a first workpiece and the method further comprises repeating operations a) through d), with a second workpiece and a second fiducial marker that contains less information than the fiducial marker applied to the first workpiece.

19. The method of any of claims 1 to 18, wherein an accuracy of placement of the fiducial marker is 50% or less than an accuracy of the second measurement, 10%, 1%, or 0.1% or less than the accuracy of the second measurement.

20. A method comprising: a) applying a fiducial marker to a workpiece;b) performing a first measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; c) performing a second measurement of at least one geometric feature of the workpiece identified by the fiducial marker; and d) linking to a data store at least one process based on the measurement of information present on the fiducial marker.

21. The method of claim 20, wherein the at least one process is manually linked to the data store.

22. The method of claim 20 or claim 21, wherein the performing a first measurement of the workpiece located in a robotic workspace further determines an orientation of the fiducial marker.

23. A fiducial marker comprising a tape, a sticker, a sticky note, a tool, a magnet, a pin, a material that is consumable by a process to modify the workpiece, or a geometric object, wherein the fiducial marker further comprises information providing instructions for a process to modify a workpiece, the information readable by at least one of an image capture equipment or a sensor.

24. The fiducial marker of claim 23, wherein the process to modify a workpiece comprises at least one of grinding, deburring, denibbing, polishing, welding, painting, imparting an abrasive pattern, applying a wrap, forming a hole, tapping, screwing, boring, riveting, masking, chamfering, cutting, bending the workpiece, attaching the workpiece to a part, disassembling the workpiece, or dispensing a material on the workpiece.

25. The fiducial marker of claim 23 or claim 24, wherein the fiducial marker comprises at least one of a color, a symbol, a geometric shape, a pattern, a matrix barcode, a word, an angle dependent dichroic or color shifting feature, a color calibration feature, a retroreflective feature, or a number.

26. A system comprising a fiducial marker of any of claims 23 to 25 and at least one non-transitory computer-readable medium encoded with instructions that, upon execution, configure a processor for: a) receiving a measurement of a workpiece located in a robotic workspace to determine a location of the fiducial marker; and b) at least one of: i. linking to a data store at least one process based on the measurement of information present on the fiducial marker; or ii. providing an output of instructions to perform at least one process based on the measurement of information present on the fiducial marker.

27. The system of claim 26, wherein the at least one non-transitory computer-readable medium is encoded with instructions that, upon execution, further configure a processor for receiving asecond measurement of at least one geometric feature of the workpiece identified by the fiducial marker.

28. The system of claim 26 or 27, further comprising a data store comprising a library of processes to modify a workpiece.

29. A method comprising: a) applying a fiducial marker to a workpiece; b) performing a measurement of the workpiece located in a robotic workspace to determine a location of the fiducial marker; and c) performing at least one process to modify the workpiece based on the measurement.

30. The method of claim 29, wherein the process to modify the workpiece comprises at least one of grinding, deburring, denibbing, polishing, welding, painting, imparting an abrasive pattern, applying a wrap, forming a hole, tapping, screwing, boring, riveting, masking, chamfering, cutting, bending the workpiece, attaching the workpiece to a part, disassembling the workpiece, or dispensing a material on the workpiece.

31. The method of claim 29 or claim 30, wherein the measurement is an optical measurement and the optical measurement is performed by an image capture equipment selected from the group consisting of a red, blue, and green (RBG) camera; a red, blue, green, and depth (RBGD) camera; a black and white (B&W) camera; and a three-dimensional (3D) image sensor.

Images