Annotation projections for workpieces

Abstract

A method, a system, and a computer program product for generating annotation projections for workpieces. A projection file defining one or more datums associated with one or more features on a surface of a workpiece and including one or more annotations associated with one or more datums is received. One or more annotations define one or more actions to be performed on the surface of the workpiece. The surface of the workpiece is scanned using one or more sensors and sensor data is generated. One or more scanned datums are extracted from the sensor data and combined into one or more synthesized datums. One or more synthesized datums are aligned with one or more datums in the projection file. A projection command for transmission to a projection device is generated using aligned at least one datum and at least one synthesized datum.

Description

Docket No. 4150.0001WOANNOTATION PROJECTIONS FOR WORKPIECESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority to U.S. Provisional Patent Appl. No. 63 / 682,469 to Strat et al., filed August 13, 2024, and entitled “Device and Method for Accurate Annotation Projection on As-Built Workpieces’', and incorporates its disclosure herein by reference in its entirety.GOVERNMENT LICENSE RIGHTS

[0002] This invention was made with government support under FA8650-17-D- 5650 awarded by United States Air Force. The government has certain rights in the invention.BACKGROUND

[0003] In many manufacturing processes, there is a need to mark or otherwise annotate a position on the surface of a workpiece relative to specified datums defined by relevant workpiece geometry'. There are many methods to measure and annotate but existing methods become increasingly more cumbersome as workpiece geometry becomes more complex and relevant datums less accessible. In the inspiring manufacturing process, operators take a transparent overlay of datums and annotations, align the datum reference markings to corresponding workpiece geometry, and then transpose the annotated markings to the workpiece. This method is often cumbersome, time consuming, and prone to operator error, often leading to misaligned markings, especially when applying transparencies to non-trivial workpiece geometry' or when the workpiece is part of an as-built assembly which precludes access to convenient datums. These misalignments can negatively impact the execution of subsequent processes resulting in rework or failed workpieces.SUMMARY

[0004] The following summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key or essential features ofDocket No. 4I5Q.OOOIWO the claimed subject matter, nor is it intended as an aid in determining the scope of the claimed subject matter.

[0005] In some implementations, the current subject matter relates to device and method that may be used to detect a potential set of datums from provided geometry, guide a user through the selection of a subset of potential datums, which properly constrain an annotation, automatically detect user specified datums on corresponding workpiece geometry, and to annotate the workpiece with visible light in the appropriate place relative to specified datums. By removing the need for any operator placement of the annotation and / or precise location of an annotating device, the annotations can be placed with greater consistency, reducing the risk of rework or failed parts. The annotations may guide the installation of precision bracketry and other material to an otherwise featureless surface.

[0006] In some implementations, the current subject matter relates to a computer-implemented method for generation and applying of annotation projection on workpieces. The method may include receiving, using at least one processor, a projection file defining one or more datums associated with one or more features on a surface of a workpiece and including one or more annotations associated with one or more datums. One or more annotations may define one or more actions to be performed on the surface of the workpiece. The method may also include scanning, using one or more sensors, the surface of the workpiece and generating sensor data, extracting one or more scanned datums from the sensor data and combining one or more scanned datums into one or more synthesized datums, aligning the one or more synthesized datums with one or more datums in the projection file, and generating a projection command for transmission to a projection device using aligned at least one datum in one or more datums and at least one synthesized datum in one or more synthesized datums.

[0007] In some implementations, the current subject matter may include one or more of the following optional features. The method may also include causing, using the projection command, the projection device to execute one or more actions in accordance with one or more annotations. One or more actions may include projecting one or more projections defined by one or more annotationsDocket No. 4150.0001WO onto the surface of the workpiece. Receiving of the projection file may include receiving a selection of a reference datum in one or more datums, where the reference datum may define a reference location for annotating the surface of the workpiece, and generating one or more annotations based on the receiving of the selection. One or more datums may include at least one of: one or more edges on the surface, one or more intersections of two or more planes on the surface, one or more vertices on the surface, one or more patterns on the surface, or any combinations thereof. One or more datums may be associated with one or more constraints configured to restrict operation of the projection device.

[0008] In some implementations, aligning may include matching one or more datums with one or more synthesized datums, and determining a relative position of one or more sensors relative to the surface of the workpiece. One or more actions may be based on the relative position.

[0009] In some implementations, the current subject matter relates to a system for generation and applying of annotation projection on workpieces. The system may include at least one processor and at least one non-transitory storage media storing instructions, that when executed by the at least one processor, cause the processor to perform one or more of the operations described herein.

[0010] In some implementations, the current subject matter relates to a non- transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the programmable processor to perform one or more of the operations described herein.

[0011] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, show certain aspects of the subject matter disclosed herein and, together with the description, help explain some of the principles associated with the disclosed implementations. In the drawings,Docket No. 4150.0001WOBRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0013] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.

[0014] FIG. 1 illustrates an example projection system, according to some implementations of the current subject matter.

[0015] FIGS. 2A-2D illustrate examples of surface features and associated datums, according to some implementations of the current subject matter.

[0016] FIG. 3A illustrates an example process for generating an annotation projection using sensing / proj ection device, according to some implementations of the current subj ect matter.

[0017] FIG. 3B illustrates further details of the process shown in FIG. 3 A related to projecting the generated annotation projection onto the surface of the workpiece, according to some implementations of the current subject matter.

[0018] FIG. 4 illustrates further details of the process shown in FIG. 3B related to projecting the generated annotation projection onto the surface of the workpiece, according to some implementations of the current subject matter.

[0019] FIG. 5 illustrates an example process for generation of annotation projection, according to some implementations of the current subject matter.

[0020] FIG. 6 illustrates an example of such processing system that may be configured to perform one or more operations of the current subject matter.

[0021] FIG. 7 illustrates a computer-readable storage medium, according to some embodiments of the current subject matter.

[0022] FIG. 8 illustrates a computing architecture, according to some embodiments of the current subject matter.

[0023] FIG. 9 illustrates a communications architecture, according to some embodiments of the current subject matter.

[0024] The drawings are not necessarily to scale. The drawings are merely representations, not intended to portray specific parameters of the disclosure. The drawings are intended to depict exemplary implementations of the current subjectDocket No. 4150.00Q1WO matter, and therefore, are not to be considered as limiting in scope. In the drawings, like numbering represents like elements.

[0025] Further, certain elements in some of the figures may be omitted, and / or illustrated not-to-scale, for illustrative clarity. Cross-sectional views may be in the form of “slices'; and / or “near-sighted” cross-sectional views, omitting certain background lines otherwise visible in a “true” cross-sectional view, for illustrative clarity. Additionally, for clarity, some reference numbers may be omitted in certain drawings.DETAILED DESCRIPTION

[0026] Various approaches in accordance with the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, where implementations of a system and method are shown. The devices, system(s), component(s), etc., may be embodied in many different forms and are not to be construed as being limited to the example implementations set forth herein. Instead, these example implementations are provided so this disclosure will be thorough and complete, and will fully convey the scope of the current subject matter to those skilled in the art.

[0027] To address these and potentially other deficiencies of currently available solutions, one or more implementations of the current subject matter relate to methods, systems, articles of manufacture, and the like that can, among other possible advantages, provide devices and methods for accurate annotation projection on workpieces.

[0028] Computer-aided design (CAD) based systems are used for analysis, scanning, and / or application of various treatments to surfaces of workpieces (e.g., objects that may be worked on by a tool, a machine, a laser, a robot, etc.). The surfaces can be metal, polymer, wood, plastic, and / or any other types of surfaces and / or any combinations of surfaces. Such systems employ various sensors / devices for obtaining information about a workpiece. The sensors / devices can be cameras, proximity sensors, scanners, and / or any other types of sensors / devices, and / or any combination thereof. The systems also use projection devices (e.g.. standalone projection devices, etc.) that can be configured to project certain information / data (which may also be referred to as annotations) onto theDocket No. 4150.0001WO workpiece. Standalone projection devices use localization markers that are placed within the operating environment, e.g.. on a workpiece, and such projection devices use ideal CAD positions of target locations on the workpiece for localization of projection devices relative to the workpiece. Some examples existing standalone projection devices include Virtek Iris 3D, as available from Virtek Vision International, Waterloo, Ontario, Canada; FARO® Tracer Laser Projectors for Manufacturing, as available from FARO Technologies. Inc., Lake Mary, Florida, USA. CAD based systems also typically use CAD software to generate visualizations of a workpiece for further analysis. In addition to visualization, CAD software may be used to determine and / or define various geometry aspects of the workpiece (including ideal geometries, some of which might not align with a physical workpiece), fixtures or features of the workpiece (e.g., curves, vertices, lines, protrusions, dents, etc.), and / or any other elements of the workpiece.

[0029] Some existing systems utilize workpiece fixtures and / or attempt to match as-built workpiece geometries to ideal CAD visualizations. This limits systems' applicability when workpiece fixtures are unclear, impractical, etc. and / or workpieces do not match ideal CAD visualizations. Further, such systems heavily rely on the entire physical workpiece geometry matching ideal geometries in generated CAD visualizations for performing further operations by any associated equipment, e.g., alignment of projector devices vis-a-vis the workpiece. For example, some existing system, e.g., augmented reality / virtual reality (AR / VR) systems, use a point cloud fitting and average fit approaches to align its projector devices to workpiece geometry. AR localization and projection devices have limited abi 1 i ty to accurately localize themselves within the operating environment with no projection onto the workpiece. Also, annotations are only visible through a transparent screen that the operator may use to view the workpiece. No user input for selection of datums defining annotation alignment is available. Some existing AR systems include HoloLens, as available from Microsoft Corporation, Redmond, Washington, USA, and MagicLeap 2, as available from Magic Leap, Inc., Plantation, Florida. USA.

[0030] Additionally, the existing systems also use physical reference markers that are positioned on the workpiece for alignment of projections whenDocket No. 4150.0001WO discrepancies between physical workpiece and ideal CAD visualizations exist. Moreover, existing systems do not provide an ability to generate and align annotations relative to a workpiece in accordance with one or more user specified datums. Existing systems also have a relatively narrow field of view and depth of field.

[0031] To resolve these issues, the current subject matter, as described herein, may be configured to provide accurate annotations on workpieces using components having different fields of view, depths of field, etc. In some example, non-limiting implementations, the current subject matter may be configured to locate and proj ect a visible annotation in a precise location on a workpiece relative to defined datum(s) (e.g., user-defined datums, machine-defined datums, software-defined datums, etc.) that may be matched to one or more corresponding geometry features on the workpiece that may include one or more features (e.g., curves, vertices, lines, protrusions, dents, etc.).

[0032] In some implementations, the current subject matter may be configured to provide control of a projector through use of a projection file and automatically detect a set of potential datums from a three-dimensional (3D) model. The current subject matter may then guide the user through both the specification of an annotation and the selection of datums which constrain the annotation (e.g., location(s) of annotation(s); orientation(s) of annotation(s); type(s) of annotation(s), location(s). orientation(s), etc. of one or more sensing device(s) configured to sense one or more surface(s) of a workpiece; location(s). orientation(s), etc. of one or more projection device(s) configured to project one or more annotation(s) onto one or more surface(s) of the workpiece; and / or any other factors associated with the annotation(s) and / or sensing and / or projection device(s); and / or any combinations thereof). This information may be packaged into a projection file. This projection file may then be provided to a device which is capable of detecting the datums and projecting the annotation with light onto the appropriate location.

[0033] In some implementations, the current subject matter device may include calibrated sensor(s) that may detect workpiece features, the sensor(s) report to a controller. The controller may synthesize the sensed features into datum features. The datum features may then be digitally aligned with a projection file, whichDocket No. 4150.0001WO was defined prior to execution and includes information on the relevant ideal datums and annotation relative to those ideal datums. Given the alignment of the ideal to as built datums, the annotations may then be packaged into a projection command. A calibrated visible light projector may then be used to project the annotation onto the same workpiece that was sensed, mirroring the virtual aligned projection on the actual workpiece.

[0034] The present disclosure will now be described with reference to the attached drawing figures, wherein like reference numerals are used to refer to like elements throughout, and wherein the illustrated structures and devices are not necessarily drawn to scale. As utilized herein, terms “component,” “system,” “interface,” and the like are intended to refer to a computer-related entity, hardware, software (e.g., in execution), and / or firmware. For example, a component can be a processor (e.g., a microprocessor, a controller, or other processing device), a process running on a processor, a controller, an object, an executable, a program, a storage device, a computer, a tablet PC and / or a user equipment (e.g., mobile phone, etc.) with a processing device. By way of illustration, an application running on a server and the server can also be a component. One or more components can reside within a process, and a component can be localized on one computer and / or distributed between two or more computers. A set of elements or a set of other components can be described herein, in which the term “set” can be interpreted as “one or more.”

[0035] Further, these components can execute from various computer readable storage media having various data structures stored thereon such as with a module, for example. The components can communicate via local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network, such as, the Internet, a local area network, a wide area network, or similar network with other systems via the signal).

[0036] As another example, a component can be an apparatus with specific functionality provided by mechanical parts operated by electric or electronic circuitry, in which the electric or electronic circuitry can be operated by a software application or a firmware application executed by one or more processors. TheDocket No. 4150.0001WO one or more processors can be internal or external to the apparatus and can execute at least a part of the software or firmware application. As yet another example, a component can be an apparatus that provides specific functionality through electronic components without mechanical parts; the electronic components can include one or more processors therein to execute software and / or firmware that confer(s). at least in part, the functionality of the electronic components.

[0037] Use of the word exemplary is intended to present concepts in a concrete fashion. As used in this application, the term “or"’ is intended to mean an inclusive “‘of’ rather than an exclusive “or”. That is, unless specified otherwise, or clear from context. “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form. Furthermore, to the extent that the terms “including”, “includes”, “having”, “has”, “with”, or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising.” Additionally, in situations wherein one or more numbered items are discussed (e g., a “‘first X”, a “second X”, etc ), in general the one or more numbered items may be distinct or they may be the same, although in some situations the context may indicate that they are distinct or that they are the same.

[0038] As used herein, the term “circuitry” may refer to, be part of, or include a circuit, an integrated circuit (IC), a monolithic IC, a discrete circuit, a hybrid integrated circuit (HIC), an Application Specific Integrated Circuit (ASIC), an electronic circuit, a logic circuit, a microcircuit, a hybrid circuit, a microchip, a chip, a chiplet, a chipset, a multi-chip module (MCM), a semiconductor die, a system on a chip (SoC), a processor (shared, dedicated, or group), a processor circuit, a processing circuit, or associated memory' (shared, dedicated, or group) operably coupled to the circuitry that execute one or more software or firmware programs, a combinational logic circuit, or other suitable hardware components that provide the described functionality7. In some embodiments, the circuitry7may be implemented in, or functions associated with the circuitry7may be implementedDocket No. 4150.0001WO by, one or more software or firmware modules. In some embodiments, circuitry may include logic, at least partially operable in hardware.

[0039] FIG. 1 illustrates an example projection system 100, according to some implementations of the current subject matter. The system 100 may be configured to be used for generation of and applying projections on one or more surfaces (e.g., surfaces 120) of one or more workpieces, e.g., workpieces 118. The workpieces 118 may be as built or any type of workpieces, e.g., not specifically designed for operation of the system 100.

[0040] The system 100 may include a sensing / proj ection device 102 that may include a perception sensor 104, a projection device 106, and, optionally, a frame 108 that may be configured to hold the perception sensor 104 and the projection device 106. The system 100 may also include a computing device 112 that may be communicatively coupled to the sensing / proj ection device 102 using a connection 110. The connection 110 may be wired, wireless and / or any combinations thereof.

[0041] The perception sensor 104 may be used for sensing the surface 120 of the workpiece 118. The perception sensor 104 may be any type of sensor, e g., a camera, an infrared sensor, a position sensor, a scanner, a proximity sensor, a 3D sensor, a LiDAR sensor, a structured-light 3D scanner, and / or any other sensor, and / or any combination thereof. The perception sensor 104 may be configured to have a perception sensor field 114, which may be a field within which the perception sensor 104 may be configured to view the surface 120 of the workpiece 118. The perception sensor field 114 may encompass one or more portions of the surface 120 of the 118 and / or an entirety of the surface 120.

[0042] The projection device 106 may be used for projecting a projection device field 116 onto the surface 120 of the workpiece 118. The projection device field 116 may be defined by a projection data, e.g., a projection file. The projection file may be generated by the computing device 112 and provided to the projection device 106 for projecting the projection device field 116. The perception sensor field 114 may be a laser projection device, and / or any other device.

[0043] The computing device 112 may be configured to serve as a controller and / or as a storage device and / or may be configured to provide instructions (e.g.,Docket No. 4150.00Q1WO via connection 110) to the sensing / proj ection device 102 for performing sensing of the surface 120 of the workpiece 118 using perception sensor 104 and / or projection on the surface 120 of the workpiece 118 using projection device 106. In some implementations, the perception sensor 104 and projection device 106 may be separate devices that may be held by the frame 108. Alternatively, or in addition, the perception sensor 104 and projection device 106 may form a unitary device that may be configured to perform sensing and projection.

[0044] One or more components of the system 100 shown in FIG. 1 may be communicatively coupled using one or more communications networks. The communications networks may include one or more of the following: a wired network, a wireless network, a metropolitan area network ("MAN"), a local area network ("LAN"), a wide area network ("WAN"), a virtual local area network ("VLAN"), an internet, an extranet, an intranet, and / or any other type of network and / or any combination thereof.

[0045] Further, one or more components of the system 100 may include any combination of hardware and / or software. In some implementations, one or more components of the system may be disposed on one or more computing devices, such as, server(s), database(s), personal computer(s), laptop(s), cellular telephone(s), smartphone(s), tablet computer(s), virtual reality devices, and / or any other computing devices and / or any combination thereof. In some example implementations, one or more components of the system may be disposed on a single computing device and / or may be part of a single communications network. Alternatively, or in addition to, such devices may be separately located from one another. A device may be a computing processor, a memory, a software functionality, a routine, a procedure, a call, and / or any combination thereof that may be configured to execute a particular function disclosed herein.

[0046] In some implementations, one or more components of the system 100 may include network-enabled computers. As referred to herein, a network- enabled computer may include, but is not limited to a computer device, or communications device including, e g., a server, a network appliance, a personal computer, a w orkstation, a phone, a smartphone, a handheld PC, a personal digital assistant, a thin client, a fat client, an Internet browser, or other device. One or more components of the system also may be mobile computing devices, forDocket No. 4150.0001WO example, an iPhone. iPod, iPad from Apple® and / or any other suitable device running Apple’s iOS® operating system, any device running Microsoft's Windows®. Mobile operating system, any device running Google's Android® operating system, and / or any other suitable mobile computing device, such as a smartphone, a tablet, or like wearable mobile device.

[0047] One or more components of the system 100 may include a processor and a memory, and it is understood that the processing circuitry may contain additional components, including processors, memories, error and parity / CRC checkers, data encoders, anti-collision algorithms, controllers, command decoders, security primitives and tamper-proofing hardware, as necessary to perform the functions described herein. One or more components of the system may further include one or more displays and / or one or more input devices. The displays may be any type of devices for presenting visual information such as a computer monitor, a flat panel display, and a mobile device screen, including liquid crystal displays, light-emitting diode displays, plasma panels, and cathode ray tube displays. The input devices may include any device for entering information into the user's device that is available and supported by the user's device, such as a touchscreen, keyboard, mouse, cursor-control device, touchscreen, microphone, digital camera, video recorder or camcorder. These devices may be used to enter information and interact with the software and other devices described herein.

[0048] In some example implementations, one or more components of the system 100 may execute one or more applications, such as software applications, that enable, for example, network communications with one or more components of system and transmit and / or receive data.

[0049] One or more components of the system 100 may include and / or be in communication with one or more servers via one or more networks and may operate as a respective front-end to back-end pair with one or more servers. One or more components of the system may transmit, for example from a mobile device application (e g., executing on one or more user devices (e g., computing device 112, etc.), one or more requests to one or more servers. The requests may be associated with retrieving data from servers (e.g., retrieving one or more electronic documents, etc.). The servers may receive the requests from theDocket No. 4150.0001WO components of the system. Based on the requests, servers may be configured to retrieve the requested data from one or more storage locations. Based on receipt of the requested data from the databases, the servers may be configured to transmit the received data to one or more components of the system, where the received data may be responsive to one or more requests.

[0050] The system 100 may include one or more networks, such as, for example, networks that may be communicatively coupling the computing device 112, the sensing / proj ection device 102, and / or any other computing components. In some implementations, networks may be one or more of a wireless network, a wired network or any combination of wireless network and wired network and may be configured to connect the components of the system and / or the components of the system to one or more servers. For example, the networks may include one or more of a fiber optics network, a passive optical network, a cable network, an Internet network, a satellite network, a wireless local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual local area network (VLAN), an extranet, an intranet, a Global System for Mobile Communication, a Personal Communication Service, a Personal Area Network, Wireless Application Protocol, Multimedia Messaging Service, Enhanced Messaging Service, Short Message Service. Time Division Multiplexing based systems, Code Division Multiple Access based systems, D-AMPS, Wi-Fi, Fixed Wireless Data, IEEE 802.11b, 802.15.1, 802. l ln and 802.11g, Bluetooth, NFC, Radio Frequency Identification (RFID), Wi-Fi, and / or any other type of network and / or any combination thereof.

[0051] In addition, the networks may include, without limitation, telephone lines, fiber optics, IEEE Ethernet 802.3, a wide area network, a wireless personal area network, a LAN, or a global network such as the Internet. Further, the networks may support an Internet network, a wireless communication network, a cellular network, or the like, or any combination thereof. The networks may further include one network, or any number of the exemplary types of networks mentioned above, operating as a stand-alone network or in cooperation with each other. The networks may utilize one or more protocols of one or more network elements to which they are communicatively coupled. The networks may translate to or from other protocols to one or more protocols of network devices. TheDocket No. 4150.0001WG networks may include a plurality of interconnected networks, such as, for example, the Internet, a service provider's network, a cable television network, corporate networks, such as credit card association networks, and home networks.

[0052] The system 100 may include one or more servers, which may include one or more processors that may be coupled to memory. Servers may be configured as a central system, server or platform to control and call various data at different times to execute a plurality7of workflow actions. Servers may be configured to connect to the one or more databases. Servers may be incorporated into and / or communicatively coupled to at least one of the components of the system.

[0053] Further, one or more components of the system 100 may be configured to execute one or more actions using one or more containers. In some implementations, each action may be executed using its own container. A container may refer to a standard unit of software that may be configured to include the code that may be needed to execute the action along with all its dependencies. This may allow execution of actions to run quickly and reliably.

[0054] In some implementations, the system 100, and in particular, sensing / proj ection device 102 and / or computing device 112, may be configured in such a way that the perception sensor 104 and / or projection device 106 used can be configured on a per implementation basis to accommodate different physical sizes of the surface 120 and / or workpiece 118 and / or perception sensor 104, and / or projection device 106 as well as various sensing and / or projection accuracy requirements. In some implementations, the computing device 112 may be configured to support a multitude of communication protocols and / or data formats. As long as the content, format, etc. of communications exchanged between the computing device 112 and components of the sensing / proj ection device 102 are supported, different types of components may be supported for performing different functions (e.g., sensing, projecting, processing, etc.) of the system 100.

[0055] The system 100 may be configured to perform automatic detection of the workpiece 118 and one or more features disposed on its surface 120. The detection of the workpiece 118 and / or the features disposed on the surface 120 may be performed by the sensing / proj ection device 102, and in particular, its perception sensor 104, upon receiving one or more commands from the computing deviceDocket No. 4150.0001WG112 via the connection 110. The features may correspond to datums that may be used for generation of an annotation projection by the computing device 112.

[0056] In some implementations, the datums may correspond to geometric, visible fiducials, and / or other detectable elements on the surface 120 of the workpiece 118 that may be detected by the perception sensor 104. A datum may include a collection of data, such as, for example, but not limited to, spatial coordinates, pixel data, surface element data, graphical data, and / or any other data and / or any combination of data, that may be a representation of one or more and / or a combination of features on the surface 120. The features may, for example, include, but are not limited to edges, corners, holes, edges next to other elements (e g., holes, etc.), intersections of planes, vertices, curvatures, etc. and / or any combination of features. Further, datums may correspond to workpiece 118's geometry defined features (e.g., as being relative and / or defined by local geometries on the surface 120 of the workpiece 118), which may be any features that may be measurable and / or detectable using three-dimensional data associated the workpiece 118. Moreover, datums, either on their own and / or in various combinations, may be used to constrain and / or measure a relative position of a particular feature and / or features of the workpiece 118. One or more measurements associated with datums may be between a target position of the sensing / proj ection device 102 (the perception sensor 104 and / or projection device 106) and a datum or set of datums. One or more descriptors associated with one or more datums may be determined for the purposes of identify ing the datum(s).

[0057] The perception sensor 104 may provide the detected features to the computing device 112 via the connection 110. The computing device 112 may then extract features as datums and generate, using such datums, an annotation projection that may be provided to the projection device 106 for generating projection device field 116.

[0058] In some implementations, the computing device 112 may execute a process for automatically identifying one or more datums within an ideal representation (e g., a CAD file) of the surface 120 of workpiece 118). The datums and / or sets of datums in such representation of the surface 120 of the workpiece 118 may be selected and that may be used to set a reference for an annotation on the surface 120 of the workpiece 118. The selection may be performed by theDocket No. 4150.0001WG computing device 112 and / or one or more users of the computing device 112. The computing device 112 may also be configured to determine one or more selectable datums for presentation to the user, where the datums may be configured to constrain the annotation to detectable geometry associated with the surface 120 of the workpiece 118.

[0059] The computing device 112 may then combine selected datums with one or more annotations (e.g., user-defined annotations) to generate a projection file. The computing device 112 may then send the projection file to the sensing / proj ection device 102. The projection file may be used by the perception sensor 104 for scanning the surface 120 of the workpiece 118 and collecting sensor data that may contain data representing features on the surface 120, e.g., edges, vertices, planes, intersections of planes, curves, dents, protrusions, markings, etc. The projection file may also be used by the sensing / proj ection device 102 for positioning of one or both of the perception sensor 104 and / or projection device 106 for the purposes of performing their respective functions (e.g., scanning and obtaining sensor data related to the surface 120, and projecting annotation(s) onto the surface 120). In some implementations, the projection file may also include one or more reference markers that may assist with alignment of the projection device 106 with the surface 120 and / or one or more of its features. The alignment of the projection device may be automatic and / or manual (e.g., by the user).

[0060] FIGS. 2A-2D illustrate examples of surface features and associated datums, according to some implementations of the current subject matter. As can be understood, the examples shown in FIGS. 2A-2D are not limiting and other features / datums are possible.

[0061] Referring to FIG. 2A, a workpiece 202 may include a surface 206 that may include one or more features 204 (a, b, c, d). The surface 206 may be a two- dimensional surface. The features 204 (a, b, c, d) may be holes 204 (a, b, c, d) that may be disposed on the surface 206. The holes 204 (a. b, c, d) may be disposed in any desired way.

[0062] Each hole 204 (a, b. c, d) may be associated with various geometries (e.g., whether the hole is round, oval, irregular, etc.) and / or data. e.g.. coordinates indicating location, data indicating size of the hole (e.g., diameter, etc.), dataDocket No. 4150.0001WG indicating hole's position vis-a-vis other elements (e.g., other holes), and / or any other data. The perception sensor 104 may detect such geometries / data and provide it to the computing device 112 that may use it to determine one or more datums that may be associated with each hole 204 (a, b, c, d).

[0063] As shown in FIG. 2B, a workpiece 208 may be a three-dimensional workpiece. The surface of the workpiece 208 may include various features, such as, for example, an edge 210a, an edge 210b, and a corner 212. The edges 210a and 210b may be configured to intersect at the corner 212. Moreover, each edge 210a, 210b may be configured to define an intersection between two or more planes. For example, edge 210a may define an intersection between plane 214 and plane 216, and edge 210b may define an intersection between plane 216 and plane 218. The comer 212 may likewise define an intersection of all three planes 214, 216, and 218.

[0064] Similarly, each edge 210 (a, b), the corner 212, and / or each plane 214, 216, 218 may be associated with various geometries (e.g., angular orientation of a plane, etc.) and / or data, e.g., coordinates indicating location, data indicating size of the plane (e.g.. area, etc.), data indicating each element's position vis-a-vis other elements (e.g., other planes, edges, comers, etc.), and / or any other data. Again, the perception sensor 104 may detect such geometries / data and provide it to the computing device 112 for determining one or more datums associated with each edge 210 (a, b), the corner 212, and / or each plane 214, 216. 218.

[0065] FIG. 2C illustrates a workpiece 220 having a surface 222. The surface 222 may include a feature of an elevated surface element 224, e.g., a curved surface, a protrusion, etc. (as can be understood, the element 224 can be an indentation, etc.). As shown in FIG. 2C, the elevated surface element 224 may extend from the surface 222.

[0066] The elevated surface element 224 may, similar to the elements shown in FIGS. 2A-2B, be associated with specific geometries (e.g., elevation height, shapes associated with the protrusion, etc.) and / or data, e.g., coordinates indicating location of the elevated surface element 224 on the surface 222, data indicating elevated surface element 224 position with respect to other elements (e.g., other protrusions, planes, surfaces, etc.), and / or any other data. The perception sensor 104 may likewise detect such geometries / data and provide it toDocket No. 4150.0001WO the computing device 112 for determining one or more datums associated with elevated surface element 224.

[0067] FIG. 2D illustrates a workpiece 226 having a surface 230. The workpiece 226 may be a three-dimensional workpiece, similar to the workpiece 208 shown in FIG. 2B. The surface 230 may include a visible feature 228. In this case, the visible feature 228 may be a QR code. As can be understood, the visible feature 228 may be any desired feature that may be visible on the surface (whether or not visible to a naked eye). The elevated surface element 224 may, likewise, detect geometry and / or other data (e.g., coordinates, location, type of feature, etc.) and provide the detected data / information to the computing device 112 for processing, e g., determining one or more datums.

[0068] FIG. 3A illustrates an example process 300 for generating an annotation projection using sensing / proj ection device 102. according to some implementations of the current subject matter. In particular, one or more of the operations may be performed by the computing device 112 based on the data and / information that may be obtained by the sensing / proj ection device 102, and / or provided by a user of the computing device 112 and / or retrieved / obtained from any other source.

[0069] At 302, the computing device 112 may be configured to execute operations associated generating an annotation projection file that may be used by the projection device 106 of the sensing / proj ection device 102 to generate and / or apply one or more projections onto the surface 120 of the workpiece 118, as shown in FIG. 1. To generate the projection file, the computing device 112 maybe provided with data, e.g., a three-dimensional model of the surface 120 of the workpiece 118. The model may be configured to be generated from data provided to the computing device 112. The model of the surface 120 of the workpiece 118 may be provided by the user of the computing device 112. For example, the model may be a computer-aided design (CAD) model generated using CAD software that may be executing on the computing device 112 and / or on any other computing devices. By way of anon-limiting example, the CAD model may be in any desired format (e.g., .SldPrt, .acd, .STEP, .OBJ, etc.).

[0070] Using the CAD model, the computing device 112 may be configured to identify7one or more detectable datums. As discussed herein, the detectableDocket No. 4150.0001WG datums may be various data / information that may be representative of various physical features (e.g.. edges, curves, intersections of planes, vertices, etc. (as for example shown in FIGS. 2A-2D)) that may be present on the surface 120 of the workpiece 118. The computing device 112 may, using the data contained in the CAD model, be configured to identify specific geometries, which may include, but are not limited to, size parameters, shape(s). location(s), location(s) vis-a-vis other features that may be present on the surface 120 of the workpiece 118 (and / or across other surfaces of the workpiece 118).

[0071] In some implementations, the computing device 112 may be configured to automatically identify datums in the model. Alternatively, or in addition, the computing device 112 may be configured to guide the user in identifying the datums. For instance, the computing device 112 may provide suggestions to the user about specific datums as representing particular physical features on the surface 120 of the workpiece 118.

[0072] At 304, the computing device 112 may be configured to receive selections of one or more identified datums. The selections may be performed automatically, e.g., using information associated with each datum. Alternatively, or in addition, the selections may be received from the user of the computing device 112. For instance, the user may select a datum that may represent a particular edge located on the surface 120 of the workpiece 118. As can be understood, any selections of datums may be performed.

[0073] Once one or more datums are selected, the computing device 112 may be configured to determine one or more constraints associated with each selected datum and corresponding physical features. The constraints may define a geometric mapping that may need to be determined between projection device 106 and one or more features of and / or the entire workpiece 118 prior to application of annotation projection by the projection device 106. The geometric mapping may be configured to constrain one or more projections annotations applied by the projection device 106 onto the workpiece 118. For example, the computing device 112 may determine that a selection of a single datum representing a vertex on the surface 120 of the workpiece 118 may not sufficiently constrain the projection device 106 to be able to apply the annotation projection onto the surface 120 of the workpiece 118. However, selection of the vertex andDocket No. 4150.0001WG three edges representing intersections of planes may be sufficient for constraining the projection device 106 in applying an appropriate annotation projection onto the surface 120. Alternatively, or in addition, selection of too many datums may also prevent the projection device 106 from application of the annotation projection onto the surface 120, as the projection device 106 may be too constrained from doing so (e.g., selection of several hundred datums).

[0074] In some implementations, the computing device 112 may, upon receiving datum selections, be configured to continuously analyze constraints associated with each selected datum and generate indications whether a particular datum insufficiently constrains, correctly constrains, and / or overly constrains the projection device 106 from applying the annotation projection onto the surface 120 of the workpiece 118. Further, the computing device 112 may be configured to receive selections of any number of datums. The datums may be continuously selected. Alternatively, or in addition, the computing device 112 may limit the number of datums that can be selected. For example, the computing device 112 analyze the particular workpiece 118 and, based on such analysis, indicate that only a certain number of datums may be selected for that workpiece. Moreover, the computing device 112 may be configured to perform a continuous real-time analysis of selected datums. For instance, upon selection of a specific datum, the computing device 112 may determine that other datums cannot be selected, as either overly constraining or under-constraining the projection device 106, and generate an appropriate indication to be displayed on a graphical user interface of the computing device 112.

[0075] After selections of datums that sufficiently constrain the projection device 106 are completed, the computing device 112 may be configured to define the annotation projection, at 306. The annotation projection defines the annotation that is to be projected and / or applied by the projection device 106 onto the surface 120 of the workpiece 118. The annotation may include an indication of one or more features, e.g., edge(s), line(s), pattem(s), etc., that are to be projected by the projection device 106. The annotation may be contained in the CAD data and may be interpreted by the computing device 112 as a geometric and / or special annotation feature and / or generated by the computing device 112 as an annotation.Docket No. 4150.0001WG

[0076] Once the datum requirements for a constrained relationship are met and one or more annotation projections are generated, the computing device 112 may be configured to combine the annotation projections into a data projection file, at 308. The data projection file may then be provided to the projection device 106 for application of the combination of annotation projections contained in the projection file on the surface 120 of the workpiece 118. as shown in FIG. 3B.

[0077] FIG. 3B and FIG. 4 illustrate further details of the process 300 related to projecting the generated annotation projection onto the surface 120 of the workpiece 118, according to some implementations of the current subject matter. The operations 310 - 318 of the process 300 may be associated with a rough alignment of the projection device and the annotation projection(s) contained in the projection file. The operations 320 - 328 may be associated with a refinement the rough alignment in the prior operations. As shown in FIG. 3B and FIG. 4, the alignment relies on the annotation projection(s) contained in the data projection file generated by the computing device 112, at 308, as shown in FIG. 3A.

[0078] In some implementations, the perception sensor 104 of the sensing / proj ection device 102 may be include one or more calibrated sensor(s) that may detect one or more physical features present on the surface 120 of the workpiece 118. The perception sensor 104 may provide (e.g., via connection 110) data / information associated with the detected features to the computing device 112. The computing device 112 may then synthesize the sensed features into datum features. The datum features may then be digitally aligned with datums in the projection file as generated, at 308, and representing ideal datums, where the projection file includes annotation information relative to such ideal datums. Once detected datums and ideal datums from the projection file are compared, the computing device 112 may use this information to generate a projection command for transmission to the projection device 106 of the sensing / projection device 102. The projection device 106 may include a calibrated visible light projector that may then project the annotation onto the surface 120 of the workpiece 118, thereby mirroring the virtually aligned projection (as contained in the projection file) onto the actual workpiece.

[0079] Referring to FIG. 3B and FIG. 4, as discussed herein, the perception sensor 104 of the sensing / projection device 102 may be configured to collectDocket No. 4150.0001WG various data associated with physical features disposed on the surface 120 of the workpiece 118, at 310. The physical features may include any types of features, e.g., edges, intersections of planes, curvatures, vertices, indentations, protrusions, markings, bents, dents, openings / holes, barriers, and / or any other physical features, and / or any combination of features. The data detected by the perception sensor 104 may be any type of data, including, for example, three-dimensional data, two-dimensional data. etc.

[0080] In some example, non-limiting implementations, the perception sensor 104 may be configured to capture one or more point clouds (e.g., three- dimensional point clouds) of one or more areas of interest (e.g., an edge, etc., as discussed herein) on the surface 120 of the workpiece 118. A depth of one or more frames may also be captured, at 402 (as shown in FIG. 4). The capture assumes that the perception sensor 104 and the projection device 106 may be positioned in such a way that they are nominally pointed at and / or observing the area of interest (e.g., an edge, etc.), where the area of interest and / or a particular physical feature may include a corresponding datum in the generated projection file. Optionally, the perception sensor 104 may be a structured light projector that may enable collection of additional position information (e.g., three-dimensional position information) surrounding the physical feature.

[0081] At 312, the perception sensor 104 of the sensing / proj ection device 102 may be configured to send the data related to the detected physical features to the computing device 112 via the connection 110, as shown in FIG. 1. Any desired supported communication protocol may be used for sending of the data from the perception sensor 104 to the computing device 112.

[0082] At 314, the computing device 112, upon receipt of the data associated with the detected features, may be configured to extract datum features from the data. To do so, the computing device 112 may be configured to execute one or more algorithms that may be configured to extract one or more geometric feature data from the received data, where the geometric feature data may correspond to one or more physical features detected by the perception sensor 104. For example, at 404, the computing device 112 may be configured to extract one or more three- dimensional datum points from the depth frame captured, at 402. The computing device 112 may then be configured to analyze the detected datums. The computingDocket No. 4150.0001WO device 112 may use one or more rules for analyzing datums. For example, the datums may be analyzed based on their type, e.g., detected edge datums may be compared to other detected edge datums. The computing device 112 may then combine and / or synthesize the detected datums to generate one or more synthesized datums.

[0083] In some implementations, to generate synthesized datums, the computing device 112 may be configured to synthesize one or more surface elements (surfels) from subsets of neighboring points on the surface 120. Surfels may, for example, be conceptually represented as discs resting on a plane, where the discs may or may not have radii, but may be characterized using an (x. y, z) center and a normal vector. The surfels may be constructed, at 406, from one or more three- dimensional datum points that were extracted, at 404.

[0084] The computing device 112 may then cluster surfels, at 408, based on. for example, their one or more of their positions and / or orientations, to generate one or more features, e.g., planes, etc. The clustered surfels may then be used to generate one or more surfaces, at 410. Using this information, the computing device 112 may perform datum synthesis, where features may be combined into one or more candidate datums and descriptors may be generated from the candidate datum(s). The descriptor(s) may be used for identifying a particular datum. In some example, non-limiting implementations, the features (or locating features) may be detected and / or determined, at 412, on the surfaces generated from clustered surfels and / or at intersections of such surfaces.

[0085] The computing device 112 may be configured to align, at 316, one or more synthesized datums with one or more datums in the generated projection file, which was generated, at 308, as shown in FIG. 3A. In some example, nonlimiting implementations, the computing device 112 may be configured to execute a best fit and / or any other process to determine one or more matches between one or more synthesized datums and one or more datums in the projection file. Synthesizing of datums may include constructing, at 414, locating features that were detected / determined, at 412. Further, one or more structured light projections may, optionally, be used to extract additional geometric information.

[0086] Moreover, the computing device 112 may be configured to compare the synthesized datum(s) to one or more geometric features and / or any other dataDocket No. 4150.0001WG associated with datums and / or any other data contained in the projection file. As part of this, the computing device 112 may be configured to determine one or more positions of the perception sensor 104 relative to the workpiece 118 in one or more reference frames in the projection file. This may result in an understanding of where a projection may need to be applied to the surface 120 of the workpiece 118 by the projection device 106 relative to the perception sensor 104.

[0087] At 318, the computing device 112 may then generate one or more projection commands based on the determined alignment of the synthesized datums and the datums contained in the projection file and send the projection command(s) to the projection device 106 of the sensing / proj ection device 102 to apply annotation projection to the surface 120 of the workpiece 118. At 416, as shown in FIG. 4, one or more annotation projection parameters may be determined using depth frame data (as determined at 402) and annotation projection file (as generated at 308). At 418, one or more desired locations of annotation in one or more perception sensor 104 (e.g., RGB camera) frame may be determined from one or more perception sensor 104 properties as well as the annotation projection file (as generated at 308). One or more fiducials representative of the annotation may then be projected, at 420. This information may then be provided for refining of alignment processes.

[0088] To apply annotation projection, the projection device 106 may be configured to transmit one or more light rays onto a specific location on the surface 120 relative to the detected datums and / or features, as defined by the projection file. In some implementations, the projection commands may be formatted so that they are able to be interpreted by the specific projection device 106. The computing device 112 may further be configured to determine a specific orientation / position of the projection device 106 relative to the surface 120 of the workpiece 118 and the desired annotation location, and use that information for transmission of instructions to the projection device 106 to apply projection (e.g., illuminate) that desired annotation location.

[0089] As discussed herein, the current subject matter may be configured to perform refinement of the alignment of datums determined in 310 - 318. In particular, in some implementations, the current subject matter may also includeDocket No. 4150.0001WG another sensor(s) (and / or potentially another mode of sensing that may be integrated with the perception sensor 104) that may capture an annotation as projected relative to the workpiece 118 features that represent one or more datums. The annotation and datum features may be extracted from the data as detected by such sensor and relative position(s) between annotation and datum(s) may then be determine by the computing device 112 and compared to one or more pre-planned annotation positions. If a difference is detected (e.g., more than a predetermined configurable threshold), the projection of the annotation may be updated to reduce the error in the projection. This refinement of annotation position may be iterated until the control floor of the hardware is reached. This may result in a metric which the system 100 may self-assess and report performance (in the form of annotation accuracy) of an annotation.

[0090] In some implementations, for the purposes of executing refining of the alignment processes, the system 100 may include another sensor(s) and / or, the perception sensor 104 may include another mode of sensing integrated therein, that may be configured to capture the annotation as projected relative to the workpiece 118 features representing one or more datums. The annotation and one or more datum features may be extracted from the perception sensor 104 data and the relative position between annotation and datum may be then determined by the computing device 112 and compared to one or more pre-determined annotation positions. If a difference over a configurable threshold is detected, the projection of the annotation may be updated to reduce errors in the projection. This refinement of annotation position may be iterated until a predetermined hardware “control floor” is reached. This process may also result in a metric which the system 100 may use to self-assess and / or report its performance (e.g., in the form of annotation accuracy) of an annotation.

[0091] Referring to FIG. 3B and FIG. 4, at 320, the perception sensor 104 and / or another sensor may be configured to detect (similar to 310) and / or collect further sensor data from the workpiece 118 and the projection as applied by the projection device 106. The collection of such additional data may be triggered by the computing device 112.Docket No. 4150.0001WO

[0092] At 322, the perception sensor 104 and / or another sensor may be configured to send the collected data to the computing device 112. This process may be similar to sending of data, at 312.

[0093] At 324, the computing device 112, upon receipt of the collected data, may be configured to extract datum features from the data. To do so. similar to 314, the computing device 112 may be configured to execute one or more algorithms that may be configured to extract one or more geometric feature data from the received data, where the geometric feature data may correspond to one or more physical features detected by the perception sensor 104. The computing device 112 may extract one or more three-dimensional datum points from the collected data. For example, the perception sensor 104 and / or another sensor may be configured to capture one or more images (e.g., RGB images) of the workpiece 118 and / or the applied projection, at 422, and locate one or more fiducials in the captured image(s). at 424. The computing device 112 may then be configured to analyze the detected datums using one or more rules.

[0094] At 326, the computing device 112 may be configured to determine error in the position of the projection device 106 and / or any corrections for correcting the projection device 106. In some example, non-limiting implementations, the computing device 112 may be configured to determine position of the projection device 106 and compare it to a predetermined target position of the projection device 106. The computing device 112 may then use the difference in positions to determine a correction transform, where the error in the position of the projection device 106 may correspond to the magnitude of the correction transform. In some example, non-limiting implementations, the computing device 112 may use the data associated with one or more fiducials determined at 424, to determine whether a fiducial appears in a predetermined location (e.g.. target location correspond to the target position of the projection device 106). If so, the fiducial is correctly positioned and the annotation may be projected on the target location, at 432. Otherwise, at 428, the computing device 112 may determine the correction transform using the difference between fiducials from the observed location and intended location.

[0095] The correction transform may then be applied to the fiducial, at 430, and the computing device 112 may generate and send a corrected project! on(s) to theDocket No. 4150.0001WG projection device 106 to generate a corrected projection, at 328, on the surface 120 of the workpiece 118. This process may be performed iteratively (e.g.. by proceeding to 318) until a desired projection is achieved and / or the hardware (e.g., projection device 106) is no longer able to perform such projection, thereby reaching ‘‘control floor”.

[0096] In some implementations, the fine alignment may be performed automatically by the components of the system 100. Alternatively, or in addition, fine alignment may be performed manually, e.g., in response to a command from the user. These decisions may be made based on a predetermined level of accuracy of the projection. Moreover, the computing device 112 may be configured to augment and / or restrict annotation projection based on a self-assessment of projection accuracy. If a certain annotation cannot be made accurately, it may be prevented from being shown to the user. The resulting measurements discussed herein may be used as inputs to adjust how projection command is generated and / or what it needs to include to update the projection and to improve accuracy of the projection. If computing device 112 fails to update and / or cannot update the projection command, the projection device 106 may be automatically shut off and an error may be disposed on a user interface of computing device 112. For instance, the computing device 112 may include a configurable parameter that may be compared to the determined measure for projection accuracy (at 326). If the projection accuracy does not meet and / or exceed such configured parameter, the computing device 112 may be configured to automatically iteratively perform the fine alignment procedure to either generate a refined projection command and / or generate an error indicating that the projection is not accurate. Different permissions may be set to allow the computing device 112 and / or the user to retry performing rough localization and / or continue to iterate on the fine localization. Examples of such permissions may include having an acceleration sensor that may perform monitoring of projections along with a process logic that may disallow projection until the projection device 106 is repositioned, thereby forcing a restart of the localization process.

[0097] In some implementations, the flexible architecture enabling variable sensing and projection performed by perception sensor 104 and projection device 106, respectively, may allow the system 100 to support variable accuracy,Docket No. 4150.0001WG physical standoff, and / or projection volumes with different hardware components. The datum extraction algorithms may be hardware agnostic and may use data from calibrated projection devices 106 that provide three-dimensional position information. The computing device 112 may also generate projection commands in a hardware agnostic format. Any calibrated projection hardware may be used with paired software.

[0098] FIG. 5 illustrates an example process 500 for generation of annotation projection, according to some implementations of the current subject matter. The process 500 may be executed by one or more components of the system 100 shown in FIG. I, and in particular by the computing device 112, the sensing / proj ection device 102 that includes perception sensor 104 and projection device 106.

[0099] At 502, the computing device 112 may be configured to receive a projection file that may define one or more datums associated with one or more features on a surface (e g., surface 120) of a workpiece (e.g., workpiece 118). The projection file may include one or more annotations associated with one or more datums. One or more annotations may define one or more actions to be performed on the surface of the workpiece.

[0100] At 504, the computing device 112 may trigger the perception sensor 104 to scan the surface of the workpiece and obtain and / or generate sensor data.

[0101] At 506, the computing device 112 may extract one or more scanned datums from the sensor data and combine one or more scanned datums into one or more synthesized datums.

[0102] At 508, the computing device 112 may align one or more synthesized datums with one or more datums in the projection file.

[0103] At 510m the 112 may generate a projection command for transmission to the projection device (e g., projection device 106) using aligned at least one datum and at least one synthesized datum.

[0104] FIG. 6 illustrates an example of such processing system 600 that may be configured to perform one or more operations of the current subject matter. The system 600 may include an input / output (I / O) device 602, a processor 604, a memory 606, a storage 608, and one or more communication components 610. Each of the components 602-610 may be interconnected using a system bus 612.Docket No. 4150.0001WOThe processor 604 may be configured to process instructions for execution within system 600. In some implementations, the processor 604 may be a single-threaded processor. Alternatively, or in addition, the processor 604 may be a multithreaded processor. The processor 604 may be further configured to process instructions stored in the memory 606 and / or in the storage 608, including, but not limited to, receiving and / or sending information through the I / O device 602. The memory 606 may store information within the system 600. In some implementations, the memory 606 may be a computer-readable medium. Alternatively, or in addition, the memory' 606 may be a volatile memory' unit. In yet some implementations, the memory' 606 may be a non-volatile memory unit. The storage 608 may be capable of providing mass storage for the system 600. In some implementations, the storage 608 may be a computer-readable medium. Alternatively, or in addition, the storage 608 may be a floppy disk device, a hard disk device, an optical disk device, a tape device, non-volatile solid-state memory', or any other type of storage device. The I / O device 602 may provide input / output operations for the system 600. In some implementations, the I / O device 602 may include a keyboard and / or pointing device. Alternatively, or in addition, the I / O device 602 may include a display unit for displaying graphical user interfaces.

[0105] In some example implementations, one or more components of the system 600 may include any combination of hardware and / or software. In some implementations, one or more components of the system 600 may be disposed on one or more computing devices, such as, server(s), database(s), personal computer(s), laptop(s), cellular telephone(s), smartphone(s). tablet computer(s), virtual reality devices, and / or any other computing devices and / or any combination thereof. In some example implementations, one or more components of the system 600 may be disposed on a single computing device and / or may be part of a single communications network. Alternatively, or in addition to, such services may be separately located from one another.

[0106] In some implementations, the system 600's one or more components may include network-enabled computers. As referred to herein, a network-enabled computer may include, but is not limited to a computer device, or communications device including, e.g., a server, a network appliance, a personal computer, a workstation, a phone, a smartphone, a handheld PC, a personal digital assistant, aDocket No. 4I5Q.OOOIWO thin client, a fat client, an Internet browser, or other device. One or more components of the system 600 also may be mobile computing devices, for example, an iPhone, iPod, iPad from Apple® and / or any other suitable device running Apple’s iOS® operating system, any device running Microsoft's Windows®. Mobile operating system, any device running Google's Android® operating system, and / or any other suitable mobile computing device, such as a smartphone, a tablet, or like wearable mobile device.

[0107] FIG. 7 illustrates an apparatus 700. Apparatus 700 may comprise any non-transitory computer-readable storage medium 702 or machine-readable storage medium, such as an optical, magnetic or semiconductor storage medium. In various embodiments, apparatus 700 may comprise an article of manufacture or a product. In some embodiments, the computer-readable storage medium 702 may store computer executable instructions with which circuitry can execute. For example, computer executable instructions 704 can include instructions to implement operations described with respect to any logic flows described herein. Examples of computer- readable storage medium 702 or machine-readable storage medium may include any tangible media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. Examples of computer executable instructions 704 may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like.

[0108] FIG. 8 illustrates an embodiment of a computing architecture 800. Computing architecture 800 is a computer system with multiple processor cores such as a distributed computing system, supercomputer, high-performance computing system, computing cluster, mainframe computer, mini -computer, client-server system, personal computer (PC), workstation, server, portable computer, laptop computer, tablet computer, handheld device such as a personal digital assistant (PDA), or other device for processing, displaying, or transmitting information. Similar embodiments may comprise, e.g., entertainment devices such as a portable music player or a portable video player, a smart phone or other cellular phone, a telephone, a digital video camera, a digital still camera, anDocket No. 4150.0001WG external storage device, or the like. Further embodiments implement larger scale server configurations. In other embodiments, the computing architecture 800 may have a single processor with one core or more than one processor. Note that the term “processor’’ refers to a processor with a single core or a processor package with multiple processor cores. In at least one embodiment, the computing architecture 800 is representative of the components of the system 100. More generally, the computing architecture 800 is configured to implement all logic, systems, logic flows, methods, apparatuses, and functionality described herein with reference to previous figures.

[0109] As used in this application, the terms “system” and “component” and “module” are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution, examples of which are provided by the exemplary computing architecture 800. For example, a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical and / or magnetic storage medium), an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and / or thread of execution, and a component can be localized on one computer and / or distributed between two or more computers. Further, components may be communicatively coupled to each other by various types of communications media to coordinate operations. The coordination may involve the uni-directional or bi-directional exchange of information. For instance, the components may communicate information in the form of signals communicated over the communications media. The information can be implemented as signals allocated to various signal lines. In such allocations, each message is a signal. Further embodiments, however, may alternatively employ data messages. Such data messages may be sent across various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.

[0110] As shown in FIG. 8, computing architecture 800 comprises a system-on- chip (SoC) 802 for mounting platform components. System-on-chip (SoC) 802 is a point-to-point (P2P) interconnect platform that includes a first processor 804Docket No. 4150.0001WG and a second processor 806 coupled via a point-to-point interconnect 870 such as an Ultra Path Interconnect (UPI). In other embodiments, the computing architecture 800 may be of another bus architecture, such as a multi-drop bus. Furthermore, each of processor 804 and processor 806 may be processor packages with multiple processor cores including core(s) 808 and core(s) 810, respectively. While the computing architecture 800 is an example of a two-socket (2S) platform, other embodiments may include more than two sockets or one socket. For example, some embodiments may include a four-socket (4 S) platform or an eight-socket (8S) platform. Each socket is a mount for a processor and may have a socket identifier. Note that the term platform may refers to a motherboard with certain components mounted such as the processor 804 and chipset 832. Some platforms may include additional components, and some platforms may only include sockets to mount the processors and / or the chipset. Furthermore, some platforms may not have sockets (e.g. SoC, or the like). Although depicted as a SoC 802. one or more of the components of the SoC 802 may also be included in a single die package, a multi-chip module (MCM), a multi-die package, a chiplet, a bridge, and / or an interposer. Therefore, embodiments are not limited to a SoC.

[0111] The processor 804 and processor 806 can be any of various commercially available processors, including without limitation an Intel® Celeron®, Core®, Core (2) Duo®, Itanium®, Pentium®, Xeon®, and XScale® processors; AMD® Athlon®, Duron® and Opteron® processors; ARM® application, embedded and secure processors; IBM® and Motorola® DragonBall® and PowerPC® processors; IBM and Sony® Cell processors; and similar processors. Dual microprocessors, multi-core processors, and other multi-processor architectures may also be employed as the processor 804 and / or processor 806. Additionally, the processor 804 need not be identical to processor 806.

[0112] Processor 804 includes an integrated memory controller (IMC) 820 and point-to-point (P2P) interface 824 and P2P interface 828. Similarly, the processor 806 includes an IMC 822 as well as P2P interface 826 and P2P interface 830. IMC 820 and IMC 822 couple the processor 804 and processor 806, respectively, to respective memories (e.g., memory 816 and memory 818). Memory 816 and memory 818 may be portions of the main memory7(e.g., a dynamic random-accessDocket No. 415Q.OOQ1WO memory (DRAM)) for the platform such as double data rate type 4 (DDR4) or type 5 (DDR5) synchronous DRAM (SDRAM). In the present embodiment, the memory 816 and the memory 818 locally attach to the respective processors (i.e., processor 804 and processor 806). In other embodiments, the main memory may couple with the processors via a bus and shared memory hub. Processor 804 includes registers 812 and processor 806 includes registers 814.

[0113] Computing architecture 800 includes chipset 832 coupled to processor 804 and processor 806. Furthermore, chipset 832 can be coupled to storage device 850, for example, via an interface (I / F) 838. The I / F 838 may be, for example, a Peripheral Component Interconnect-enhanced (PCIe) interface, a Compute Express Link ® (CXL) interface, or a Universal Chiplet Interconnect Express (UCIe) interface. Storage device 850 can store instructions executable by circuitry' of computing architecture 800 (e.g., processor 804, processor 806, GPU 848, accelerator 854. vision processing unit 856, or the like). For example, storage device 850 can store instructions for sensing / proj ection device 102, computing device 112, or the like.

[0114] Processor 804 couples to the chipset 832 via P2P interface 828 and P2P 834 while processor 806 couples to the chipset 832 via P2P interface 830 and P2P 836. Direct media interface (DMI) 876 and DMI 878 may couple the P2P interface 828 and the P2P 834 and the P2P interface 830 and P2P 836. respectively . DMI 876 and DMI 878 may be a high-speed interconnect that facilitates, e.g., eight Giga Transfers per second (GT / s) such as DMI 3.0. In other embodiments, the processor 804 and processor 806 may interconnect via a bus.

[0115] The chipset 832 may comprise a controller hub such as a platform controller hub (PCH). The chipset 832 may include a system clock to perform clocking functions and include interfaces for an I / O bus such as a universal serial bus (USB), peripheral component interconnects (PCIs), CXL interconnects, UCIe interconnects, interface serial peripheral interconnects (SPIs), integrated interconnects (I2Cs), and the like, to facilitate connection of peripheral devices on the platform. In other embodiments, the chipset 832 may comprise more than one controller hub such as a chipset with a memory' controller hub, a graphics controller hub, and an input / output (I / O) controller hub.Docket No. 4I5Q.OOOIWO

[0116] In the depicted example, chipset 832 couples with a trusted platform module (TPM) 844 and UEFI. BIOS, FLASH circuitry 846 via I / F 842. The TPM 844 is a dedicated microcontroller designed to secure hardware by integrating cryptographic keys into devices. The UEFI, BIOS, FLASH circuitry 846 may provide pre-boot code. The I / F 842 may also be coupled to a network interface circuit (NIC) 880 for connections off-chip.

[0117] Furthermore, chipset 832 includes the I / F 838 to couple chipset 832 with a high-performance graphics engine, such as, graphics processing circuitry or a graphics processing unit (GPU) 848. In other embodiments, the computing architecture 800 may include a flexible display interface (FDI) (not shown) between the processor 804 and / or the processor 806 and the chipset 832. The FDI interconnects a graphics processor core in one or more of processor 804 and / or processor 806 with the chipset 832.

[0118] The computing architecture 800 is operable to communicate with wired and wireless devices or entities via the network interface (NIC) 180 using the IEEE 802 family of standards, such as wireless devices operatively disposed in wireless communication (e.g., IEEE 802.11 over-the-air modulation techniques). This includes at least Wi-Fi (or Wireless Fidelity ), WiMax, and Bluetooth™ wireless technologies, 3G, 4G, LTE wireless technologies, among others. Thus, the communication can be a predefined structure as with a conventional network or simply an ad hoc communication between at least two devices. Wi-Fi networks use radio technologies called IEEE 802.1 lx (a. b, g. n, ac, ax. etc.) to provide secure, reliable, fast wireless connectivity. A Wi-Fi network can be used to connect computers to each other, to the Internet, and to wired networks (which use IEEE 802.3-related media and functions).

[0119] Additionally, accelerator 854 and / or vision processing unit 856 can be coupled to chipset 832 via I / F 838. The accelerator 854 is representative of any type of accelerator device (e.g., a data streaming accelerator, cryptographic accelerator, cryptographic co-processor, an offload engine, etc.). One example of an accelerator 854 is the Intel® Data Streaming Accelerator (DSA). The accelerator 854 may be a device including circuitry to accelerate copy operations, data encry ption, hash value computation, data comparison operations (including comparison of data in memory 816 and / or memory 818). and / or data compression.Docket No. 4150.00Q1WOFor example, the accelerator 854 may be a USB device, PCI device, PCIe device, CXL device, UCIe device, and / or an SPI device. The accelerator 854 can also include circuitry arranged to execute machine learning (ML) related operations (e.g., training, inference, etc.) for ML models. Generally, the accelerator 854 may be specially designed to perform computationally intensive operations, such as hash value computations, comparison operations, cryptographic operations, and / or compression operations, in a manner that is more efficient than when performed by the processor 804 or processor 806. Because the load of the computing architecture 800 may include hash value computations, comparison operations, cryptographic operations, and / or compression operations, the accelerator 854 can greatly increase performance of the computing architecture 800 for these operations.

[0120] The accelerator 854 may include one or more dedicated work queues and one or more shared work queues (each not pictured). Generally, a shared work queue is configured to store descriptors submitted by multiple software entities. The software may be any type of executable code, such as a process, a thread, an application, a virtual machine, a container, a microservice, etc., that share the accelerator 854. For example, the accelerator 854 may be shared according to the Single Root I / O virtualization (SR-IOV) architecture and / or the Scalable I / O virtualization (S-IOV) architecture. Embodiments are not limited in these contexts. In some embodiments, software uses an instruction to atomically submit the descriptor to the accelerator 854 via a non-posted write (e.g., a deferred memory write (DMWr)). One example of an instruction that atomically submits a work descriptor to the shared work queue of the accelerator 854 is the ENQCMD command or instruction (which may be referred to as “ENQCMD” herein) supported by the Intel® Instruction Set Architecture (ISA). However, any instruction having a descriptor that includes indications of the operation to be performed, a source virtual address for the descriptor, a destination virtual address for a device-specific register of the shared work queue, virtual addresses of parameters, a virtual address of a completion record, and an identifier of an address space of the submitting process is representative of an instruction that atomically submits a work descriptor to the shared work queue of the acceleratorDocket No. 4150.0001WG854. The dedicated work queue may accept job submissions via commands such as the movdir64b instruction.

[0121] Various I / O devices 860 and display 852 couple to the bus 872, along with a bus bridge 858 which couples the bus 872 to a second bus 874 and an I / F 840 that connects the bus 872 with the chipset 832. In one embodiment, the second bus 874 may be a low pin count (LPC) bus. Various devices may couple to the second bus 874 including, for example, a keyboard 862, a mouse 864 and communication devices 866.

[0122] Furthermore, an audio I / O 868 may couple to second bus 874. Many of the I / O devices 860 and communication devices 866 may reside on the systemon-chip (SoC) 802 while the keyboard 862 and the mouse 864 may be add-on peripherals. In other embodiments, some or all the I / O devices 860 and communication devices 866 are add-on peripherals and do not reside on the system-on-chip (SoC) 802.

[0123] FIG. 9 illustrates a block diagram of an exemplary communications architecture 900 suitable for implementing various embodiments as previously described. The communications architecture 900 includes various common communications elements, such as a transmitter, receiver, transceiver, radio, network interface, baseband processor, antenna, amplifiers, filters, power supplies, and so forth. The embodiments, however, are not limited to implementation by the communications architecture 900.

[0124] As shown in FIG. 9, the communications architecture 900 includes one or more clients 902 and servers 904. The clients 902 may implement a client version of the computing device 112, for example. The servers 904 may implement a server version of the computing device 112, for example. The clients 902 and the servers 904 are operatively connected to one or more respective client data stores 908 and server data stores 910 that can be employed to store information local to the respective clients 902 and servers 904, such as cookies and / or associated contextual information.

[0125] The clients 902 and the servers 904 may communicate information between each other using a communication framework 906. The communications communication framework 906 may implement any well-known communicationsDocket No. 4150.0001WO techniques and protocols. The communications communication framework 906 may be implemented as a packet-switched network (e.g., public networks such as the Internet, private networks such as an enterprise intranet, and so forth), a circuit-switched network (e.g., the public switched telephone network), or a combination of a packet-switched network and a circuit-switched network (with suitable gateways and translators).

[0126] The communication framework 906 may implement various network interfaces arranged to accept, communicate, and connect to a communications network. A network interface may be regarded as a specialized form of an input output interface. Network interfaces may employ connection protocols including without limitation direct connect, Ethernet (e.g., thick, thin, twisted pair 10 / 700 / 1000 Base T, and the like), token ring, wireless network interfaces, cellular network interfaces, IEEE 802.11 network interfaces, IEEE 802.16 network interfaces, IEEE 802.20 network interfaces, and the like. Further, multiple network interfaces may be used to engage with various communications network types. For example, multiple network interfaces may be employed to allow for the communication over broadcast, multicast, and unicast networks. Should processing requirements dictate a greater amount speed and capacity, distributed network controller architectures may similarly be employed to pool, load balance, and otherwise increase the communicative bandwidth required by clients 902 and the servers 904. A communications network may be any one and the combination of wired and / or wireless networks including without limitation a direct interconnection, a secured custom connection, a private network (e.g.. an enterprise intranet), a public network (e.g., the Internet), a Personal AreaNetwork (PAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN), an Operating Missions as Nodes on the Internet (OMNI), a Wide Area Network (WAN), a wireless network, a cellular network, and other communications networks.

[0127] It will be appreciated that the exemplary devices shown in the block diagrams described above may represent one functionally descriptive example of many potential implementations. Accordingly, division, omission or inclusion of block functions depicted in the accompanying figures does not infer that theDocket No. 4I5Q.OOOIWO hardware components, circuits, software and / or elements for implementing these functions would necessarily be divided, omitted, or included in embodiments.

[0128] Some implementations and / or embodiments may be described using the expression “one embodiment” or “an embodiment” or “one implementation” or “an implementation” along with their derivatives. These terms mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment”, “in some implementations” (or derivatives thereof) in various places in the specification are not necessarily all referring to the same embodiment. Moreover, unless otherwise noted the features described above are recognized to be usable together in any combination. Thus, any features discussed separately may be employed in combination with each other unless it is noted that the features are incompatible with each other.

[0129] It is emphasized that the abstract of the disclosure is provided to allow a reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing detailed description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment. In the appended claims, the terms “including” and “in which” are used as the plain- English equivalents of the respective terms “comprising” and “wherein,” respectively. Moreover, the terms “first,” “second,” “third,” and so forth, are used merely as labels, and are not intended to impose numerical requirements on their objects. Further, the use of “including,” “comprising,” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Accordingly, the terms “including,” “comprising,” or “having” and variations thereof are open-ended expressions and can be used interchangeably herein.Docket No. 4I5Q.OOOIWO

[0130] What has been described above includes examples of the disclosed architecture. It is, of course, not possible to describe every conceivable combination of components and / or methodologies, but one of ordinary skill in the art may recognize that many further combinations and permutations are possible. Accordingly, the novel architecture is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims.

[0131] In one aspect, a computer-implemented method may include receiving, using at least one processor, a projection file defining one or more datums associated with one or more features on a surface of a workpiece and including one or more annotations associated with one or more datums, wherein the one or more annotations define one or more actions to be performed on the surface of the workpiece; scanning, using the at least one processor, using one or more sensors, the surface of the workpiece and generating sensor data; extracting, using the at least one processor, one or more scanned datums from the sensor data and combining the one or more scanned datums into one or more synthesized datums; and aligning, using the at least one processor, the one or more synthesized datums with one or more datums in the projection file; and generating, using the at least one processor, a projection command for transmission to a projection device using aligned at least one datum in the one or more datums and at least one synthesized datum in the one or more synthesized datums.

[0132] The method may include causing, using the at least one processor, using the projection command, the projection device to execute the one or more actions in accordance with the one or more annotations.

[0133] The method may include wherein the one or more actions includes projecting one or more projections defined by the one or more annotations onto the surface of the workpiece.

[0134] The method may include wherein the receiving includes receiving, using the at least one processor, a selection of a reference datum in the one or more datums, the reference datum defining a reference location for annotating the surface of the workpiece; and generating, using the at least one processor, the one or more annotations based on the receiving of the selection.Docket No. 4150.0001WO

[0135] The method may include wherein the one or more datums include at least one of: one or more edges on the surface, one or more intersections of two or more planes on the surface, one or more vertices on the surface, one or more patterns on the surface, or any combinations thereof.

[0136] The method may include wherein the one or more datums are associated with one or more constraints configured to restrict operation of the projection device.

[0137] The method may include wherein the aligning includes matching, using the at least one processor, the one or more datums with the one or more synthesized datums; and determining, using the at least one processor, a relative position of the one or more sensors relative to the surface of the workpiece, wherein the one or more actions are based on the relative position.

[0138] In one aspect, a system may include at least one processor; and at least one non-transitory storage media storing instructions, that when executed by the at least one processor, cause the at least one processor to: receive a projection file defining one or more datums associated with one or more features on a surface of a workpiece and including one or more annotations associated with one or more datums, wherein the one or more annotations define one or more actions to be performed on the surface of the workpiece; scan, using one or more sensors, the surface of the workpiece and generating sensor data; extract one or more scanned datums from the sensor data and combining the one or more scanned datums into one or more synthesized datums; and align the one or more synthesized datums with one or more datums in the proj ection file; and generate a proj ection command for transmission to a projection device using aligned at least one datum in the one or more datums and at least one synthesized datum in the one or more synthesized datums.

[0139] The system may include wherein the at least one processor is configured to cause, using the projection command, the projection device to execute the one or more actions in accordance with the one or more annotations.

[0140] The system may include wherein the one or more actions includes projecting one or more projections defined by the one or more annotations onto the surface of the workpiece.Docket No. 4150.0001WO

[0141] The system may include wherein receiving includes receiving a selection of a reference datum in the one or more datums, the reference datum defining a reference location for annotating the surface of the workpiece; and generating the one or more annotations based on the receiving of the selection.

[0142] The system may include wherein the one or more datums include at least one of: one or more edges on the surface, one or more intersections of two or more planes on the surface, one or more vertices on the surface, one or more patterns on the surface, or any combinations thereof.

[0143] The system may include wherein the one or more datums are associated with one or more constraints configured to restrict operation of the projection device.

[0144] The system may include wherein aligning includes matching, using the at least one processor, the one or more datums with the one or more synthesized datums; and determining, using the at least one processor, a relative position of the one or more sensors relative to the surface of the workpiece, wherein the one or more actions are based on the relative position.

[0145] In one aspect, computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to: receive a projection file defining one or more datums associated with one or more features on a surface of a workpiece and including one or more annotations associated with one or more datums, wherein the one or more annotations define one or more actions to be performed on the surface of the workpiece; scan, using one or more sensors, the surface of the workpiece and generating sensor data; extract one or more scanned datums from the sensor data and combining the one or more scanned datums into one or more synthesized datums; and align the one or more synthesized datums with one or more datums in the projection file; and generate a projection command for transmission to a projection device using aligned at least one datum in the one or more datums and at least one synthesized datum in the one or more synthesized datums.

[0146] The computer program product may include wherein the at least one processor is configured to cause, using the projection command, the projectionDocket No. 4150.00Q1WO device to execute the one or more actions in accordance with the one or more annotations.

[0147] The computer program product may include wherein the one or more actions includes projecting one or more projections defined by the one or more annotations onto the surface of the workpiece.

[0148] The computer program product may include wherein receiving includes receiving a selection of a reference datum in the one or more datums, the reference datum defining a reference location for annotating the surface of the workpiece; and generating the one or more annotations based on the receiving of the selection.

[0149] The computer program product may include wherein the one or more datums include at least one of: one or more edges on the surface, one or more intersections of two or more planes on the surface, one or more vertices on the surface, one or more patterns on the surface, or any combinations thereof; wherein the one or more datums are associated with one or more constraints configured to restrict operation of the projection device.

[0150] The computer program product may include wherein aligning includes matching, using the at least one processor, the one or more datums with the one or more synthesized datums; and determining, using the at least one processor, a relative position of the one or more sensors relative to the surface of the workpiece, wherein the one or more actions are based on the relative position.

[0151] All directional references (e.g., proximal, distal, upper, lower, upward, downward, left, right, lateral, longitudinal, front, back, top, bottom, above, below, vertical, horizontal, radial, axial, clockwise, and counterclockwise) are just used for identification purposes to aid the reader's understanding of the present disclosure, and do not create limitations, particularly as to the position, orientation, or use of this disclosure. Connection references (e.g., attached, coupled, connected, and joined) are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily infer that two elements are directly connected and in fixed relation to each other.Docket No. 4I5Q.OOOIWO

[0152] Further, identification references (e.g., primary, secondary, first, second, third, fourth, etc.) are not intended to connote importance or priority but are used to distinguish one feature from another. The drawings are for purposes of illustration only and the dimensions, positions, order and relative sizes reflected in the drawings attached hereto may vary.

[0153] The present disclosure is not to be limited in scope by the specific implementations described herein. Indeed, other various implementations of and modifications to the present disclosure, in addition to those described herein, will be apparent to those of ordinary skill in the art from the foregoing description and accompanying drawings. Thus, such other implementations and modifications are intended to fall within the scope of the present disclosure. Furthermore, the present disclosure has been described herein in the context of a particular implementation in a particular environment for a particular purpose. Those of ordinary skill in the art will recognize the usefulness is not limited thereto and the present disclosure may be beneficially implemented in any number of environments for any number of purposes. Thus, the claims set forth below are to be construed in view of the full breadth and spirit of the present disclosure as described herein.

[0154] At least one computer-readable storage medium may include instructions that, when executed, cause a system to perform any of the computer-implemented methods described herein.

[0155] With general reference to notations and nomenclature used herein, the detailed descriptions herein may be presented in terms of program procedures executed on a computer or network of computers. These procedural descriptions and representations are used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art.

[0156] A procedure is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It proves convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters,Docket No. 4150.00Q1WO terms, numbers, or the like. It should be noted, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to those quantities.

[0157] Further, the manipulations performed are often referred to in terms, such as adding or comparing, which are commonly associated with mental operations performed by a human operator. No such capability of a human operator is necessary, or desirable in most cases, in any of the operations described herein, which form part of one or more embodiments. Rather, the operations are machine operations. Useful machines for performing operations of various embodiments include general purpose digital computers or similar devices.

[0158] Some embodiments may be described using the expression "coupled" and "connected" along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, some embodiments may be described using the terms “connected” and / or “coupled” to indicate that two or more elements are in direct physical or electrical contact with each other. The term "coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other.

[0159] Various embodiments also relate to apparatus or systems for performing these operations. This apparatus may be specially constructed for the required purpose, or it may comprise a general-purpose computer as selectively activated or reconfigured by a computer program stored in the computer. The procedures presented herein are not inherently related to a particular computer or other apparatus. Various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these machines will appear from the description given.

[0160] The various elements of the devices as previously described with reference to FIGS. 1-5 may include various hardware elements, software elements, or a combination of both. Examples of hardware elements may include devices, logic devices, components, processors, microprocessors, circuits, processors, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), fieldDocket No. 4150.0001WO programmable gate array (FPGA), memory units, logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. Examples of software elements may include software components, programs, applications, computer programs, application programs, system programs, software development programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. However, determining whether an embodiment is implemented using hardware elements and / or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints, as desired for a given implementation.

[0161] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium which represents various logic within the processor, which when read by a machine causes the machine to fabricate logic to perform the techniques described herein. Such representations, known as “IP cores,’7may be stored on a tangible, machine readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that make the logic or processor. Some embodiments may be implemented, for example, using a machine-readable medium or article which may store an instruction or a set of instructions that, if executed by a machine, may cause the machine to perform a method and / or operations in accordance with the embodiments. Such a machine may include, for example, any suitable processing platform, computing platform, computing device, processing device, computing system, processing system, computer, processor, or the like, and may be implemented using any suitable combination of hardware and / or software. The machine-readable medium or article may include, for example, any suitable ty pe of memory unit, memory device, memory article, memory medium, storage device, storage article, storage medium and / or storage unit, for example, memory, removable or non-removable media, erasable or non-erasable media, writeable or re-writeable media, digital or analog media,Docket No. 4150.0001WO hard disk, floppy disk. Compact Disk Read Only Memory' (CD-ROM), Compact Disk Recordable (CD-R), Compact Disk Rewriteable (CD-RW), optical disk, magnetic media, magneto-optical media, removable memory cards or disks, various types of Digital Versatile Disk (DVD), a tape, a cassette, or the like. The instructions may include any suitable ty pe of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, and the like, implemented using any suitable high-level, low-level, object- oriented, visual, compiled and / or interpreted programming language.

[0162] At least one computer-readable storage medium may include instructions that, when executed, cause a system to perform any of the computer-implemented methods described herein.

[0163] Any of the computing apparatus examples given above may also be implemented as means plus function examples. Other technical features may be readily apparent to one skilled in the art from the following figures, descriptions, and claims.

[0164] The foregoing description of example embodiments has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Many modifications and variations are possible in light of this disclosure. It is intended that the scope of the present disclosure be limited not by this detailed description, but rather by the claims appended hereto. Future filed applications claiming priority' to this application may claim the disclosed subject matter in a different manner and may generally include any set of one or more limitations as variously disclosed or otherwise demonstrated herein.

Claims

Docket No. 4150.00Q1WOCLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: receiving, using at least one processor, a projection file defining one or more datums associated with one or more features on a surface of a workpiece and including one or more annotations associated with one or more datums, wherein the one or more annotations define one or more actions to be performed on the surface of the workpiece; scanning, using the at least one processor, using one or more sensors, the surface of the workpiece and generating sensor data; extracting, using the at least one processor, one or more scanned datums from the sensor data and combining the one or more scanned datums into one or more synthesized datums; and aligning, using the at least one processor, the one or more synthesized datums with one or more datums in the projection file; and generating, using the at least one processor, a projection command for transmission to a projection device using aligned at least one datum in the one or more datums and at least one synthesized datum in the one or more synthesized datums.

2. The method of claim 1, further comprising causing, using the at least one processor, using the projection command, the projection device to execute the one or more actions in accordance with the one or more annotations.

3. The method of claim 2, wherein the one or more actions includes projecting one or more projections defined by the one or more annotations onto the surface of the workpiece.

4. The method of claim 2, wherein the receiving includes receiving, using the at least one processor, a selection of a reference datum in the one or more datums, the reference datum defining a reference location for annotating the surface of the workpiece; andDocket No. 4150.00Q1WO generating, using the at least one processor, the one or more annotations based on the receiving of the selection.

5. The method of claim 4, wherein the one or more datums include at least one of: one or more edges on the surface, one or more intersections of two or more planes on the surface, one or more vertices on the surface, one or more patterns on the surface, or any combinations thereof.

6. The method of claim 5, wherein the one or more datums are associated with one or more constraints configured to restrict operation of the projection device.

7. The method of any of the preceding claims, wherein the aligning includes matching, using the at least one processor, the one or more datums with the one or more synthesized datums; and determining, using the at least one processor, a relative position of the one or more sensors relative to the surface of the workpiece, wherein the one or more actions are based on the relative position.

8. A system, comprising: at least one processor; and at least one non-transitory storage media storing instructions, that when executed by the at least one processor, cause the at least one processor to: receive a projection file defining one or more datums associated with one or more features on a surface of a workpiece and including one or more annotations associated with one or more datums, wherein the one or more annotations define one or more actions to be performed on the surface of the workpiece; scan, using one or more sensors, the surface of the workpiece and generating sensor data; extract one or more scanned datums from the sensor data and combining the one or more scanned datums into one or more synthesized datums; andDocket No. 4150.00Q1WO align the one or more synthesized datums with one or more datums in the projection file; and generate a projection command for transmission to a projection device using aligned at least one datum in the one or more datums and at least one synthesized datum in the one or more synthesized datums.

9. The system of claim 8, wherein the at least one processor is configured to cause, using the projection command, the projection device to execute the one or more actions in accordance with the one or more annotations.

10. The system of claim 9, wherein the one or more actions includes projecting one or more projections defined by the one or more annotations onto the surface of the workpiece.

11. The system of claim 9. wherein receiving includes receiving a selection of a reference datum in the one or more datums, the reference datum defining a reference location for annotating the surface of the workpiece; and generating the one or more annotations based on the receiving of the selection.

12. The system of claim 11, wherein the one or more datums include at least one of: one or more edges on the surface, one or more intersections of two or more planes on the surface, one or more vertices on the surface, one or more patterns on the surface, or any combinations thereof.

13. The system of claim 12, wherein the one or more datums are associated with one or more constraints configured to restrict operation of the projection device.

14. The system of any of the preceding claims 8-13, wherein aligning includes matching, using the at least one processor, the one or more datums with the one or more synthesized datums; andDocket No. 4150.00Q1WO determining, using the at least one processor, a relative position of the one or more sensors relative to the surface of the workpiece, wherein the one or more actions are based on the relative position.

15. A computer program product comprising a non-transitory machine- readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to: receive a projection file defining one or more datums associated with one or more features on a surface of a workpiece and including one or more annotations associated with one or more datums, wherein the one or more annotations define one or more actions to be performed on the surface of the workpiece; scan, using one or more sensors, the surface of the workpiece and generating sensor data; extract one or more scanned datums from the sensor data and combining the one or more scanned datums into one or more synthesized datums; and align the one or more synthesized datums with one or more datums in the projection file; and generate a projection command for transmission to a projection device using aligned at least one datum in the one or more datums and at least one synthesized datum in the one or more synthesized datums.

16. The computer program product of claim 15, wherein the at least one processor is configured to cause, using the projection command, the projection device to execute the one or more actions in accordance with the one or more annotations.

17. The computer program product of claim 16, wherein the one or more actions includes projecting one or more projections defined by the one or more annotations onto the surface of the workpiece.

18. The computer program product of claim 16. wherein receiving includesDocket No. 4150.00Q1WO receiving a selection of a reference datum in the one or more datums, the reference datum defining a reference location for annotating the surface of the workpiece; and generating the one or more annotations based on the receiving of the selection.

19. The computer program product of claim 18, wherein the one or more datums include at least one of: one or more edges on the surface, one or more intersections of two or more planes on the surface, one or more vertices on the surface, one or more patterns on the surface, or any combinations thereof; wherein the one or more datums are associated with one or more constraints configured to restrict operation of the projection device.

20. The computer program product of any of the preceding claims 15- 19, wherein aligning includes matching, using the at least one processor, the one or more datums with the one or more synthesized datums; and determining, using the at least one processor, a relative position of the one or more sensors relative to the surface of the workpiece, wherein the one or more actions are based on the relative position.

Images