Enhanced positioning system and method

Abstract

Systems and methods for machine positioning are provided herein. Exemplary embodiments include systems and methods that use external position information of an object under observation to compare with the programmed position of the object under observation. The comparison can be used in different ways, including, for example, course correction, future path planning, object avoidance, etc. The exemplary embodiments described herein may be used for a 7-degree-of-freedom real-time augmented positioning system. Exemplary embodiments may use up to 6 degrees of spatial positioning.

Description

[Background technology]

[0001] (Priority) This application claims priority to U.S. Provisional Patent Application No. 63 / 272,542, filed August 25, 2022, which is incorporated herein in its entirety.

[0002] Machining systems can now control their positioning and perform actions based on the position as tracked by the machine itself. However, in order to operate, such systems must first be calibrated and configured to track their own location over time. Once calibrated, conventional systems can drift or become misaligned over time. To correct the machine positioning, the machine must be recalibrated.

[0003] Calibration in these situations requires that the machine's motion process needs to stop. The machine tool is measured in some way and the information is provided to the machine. This can be done by moving the machine tip to a known or calibrated position and zeroing the machine. These calibrations require an interruption of the machine so that calibration can occur. Calibration can then improve position tracking over a period of time until the machine drifts again.

[0004] The precision of a machine is also limited by its unique tracking device. For these integrated machines, to maintain precision over time, as the precision device becomes less precise, or as the device ages and is no longer as precise as new precision equipment, new investments are required to replace the equipment to maintain precision installation. However, continually replacing equipment with tracking devices integrated into the machine to maintain precision tracking can be expensive.

[0005] Examples of location systems can be found, for example, in US Pat. Nos. 9,557,157, 10,545,014, 11,035,659, and 11,035,660, each of which is incorporated by reference in its entirety. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 9,557,157 [Patent Document 2] U.S. Patent No. 10,545,014 [Patent Document 3] U.S. Patent No. 11,035,659 [Patent Document 4] U.S. Patent No. 11,035,660 Summary of the Invention [Means for solving the problem]

[0007] The exemplary embodiments described herein may be used for a seven-degree-of-freedom real-time augmented positioning system. The exemplary embodiments may use up to six degrees of spatial positioning. Up to three degrees of spatial positioning may be translation along three perpendicular axes in three dimensions: forward / aft, i.e., surge; up / down, i.e., heave; and left / right, i.e., sway. Up to three degrees of spatial positioning may be in rotation about three perpendicular axes: rotation about a normal or up / down axis, i.e., yaw; rotation about a horizontal or left / right axis, i.e., pitch; and rotation about a longitudinal or forward / aft axis, i.e., roll. The final degree of freedom may be in time to enable real-time assessment and correction of the position of an augmented positioning system as described herein.

[0008] Exemplary embodiments described herein may include receiving information from multiple sensors, such as any combination of inertial measurement units, laser trackers, laser scanning devices, cameras, ranging systems, probing sensors, etc., and combining the information into a real-time data set for predictive machine path correction or machine installation. Exemplary embodiments include receiving precision metrology data from one, two, or more sources, and combining the information for improved accuracy of location information.

[0009] Due to mechanical limitations, some machines may never have sufficient precision for a desired application. Tracking devices integrated into the machine may be used to improve precision for an application. [Brief explanation of the drawings]

[0010] [Figure 1] 1-5 illustrate exemplary embodiments of a system for real-time location accuracy according to the present description. [Figure 2] 1-5 illustrate exemplary embodiments of a system for real-time location accuracy according to the present description. [Figure 3] 1-5 illustrate exemplary embodiments of a system for real-time location accuracy according to the present description. [Figure 4] 1-5 illustrate exemplary embodiments of a system for real-time location accuracy according to the present description. [Figure 5] 1-5 illustrate exemplary embodiments of a system for real-time location accuracy according to the present description.

[0011] [Figure 6] FIG. 6 illustrates an exemplary system diagram according to embodiments described herein.

[0012] [Figure 7] FIG. 7 illustrates an exemplary method according to an embodiment described herein.

[0013] [Figure 8] FIG. 8 illustrates an example architecture structure according to embodiments described herein.

[0014] [Figure 9] FIG. 9 illustrates an example architecture structure according to embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0015] explanation The following detailed description illustrates, by way of example, and not by way of limitation, the principles of the present invention. The description clearly enables one skilled in the art to make and use the invention and describes several embodiments, adaptations, variations, alternatives, and uses of the invention, including what is currently contemplated to be the best mode of carrying out the invention. It should be understood that the drawings are diagrammatic and schematic representations of exemplary embodiments of the invention and are not limiting of the invention, nor are they necessarily drawn to scale.

[0016] Exemplary embodiments described herein include systems and methods for multi-degree-of-freedom real-time precision positioning and correction. The exemplary embodiments described herein include multiple integrated sensors to provide real-time positioning that can be used in a feedback loop to provide position correction within the system.

[0017] Exemplary embodiments of the systems and methods described herein may operate outside of a Windows® (or other platform) operating system environment. Exemplary embodiments may use dedicated computing hardware and software or portions of the system architecture that are separate from the machine's operating system to provide high-speed processing of positioning data and loop correction information. Exemplary embodiments described herein may include a real-time operating system (Programmable Logic Controller (PLC) and Industrial Personal Computer (IPC))-based system architecture with precision metrology and inertial measurement unit (IMU) sensor data. An exemplary programmable controller may be a Beckhoff IPC.

[0018] Exemplary embodiments of the systems and methods described herein may use a feedback loop to enable real-time positioning information to be synchronized and analyzed to provide real-time updated positioning information. In exemplary embodiments, the feedback loop may provide estimated future positions of one or more objects in the system. The estimated future positions of multiple objects may be based on different time related position processing and / or sensor velocities, providing an updated position for each object so that the estimated position for each object in the system can be synchronized.

[0019] The exemplary embodiments described herein may provide fast, real-time, precise metrology-grade position correction.

[0020] While embodiments of the present invention may be described and illustrated herein in terms of seven degrees of freedom (7DoF), it should be understood that embodiments of the present invention are not so limited and are additionally applicable to fewer positional degrees of freedom. Furthermore, while embodiments of the present invention may be described and illustrated herein in terms of real-time positioning, it should be understood that embodiments of the present invention are also applicable to other slower positioning refinement systems and methods. For example, the time for correction may be extended, and / or the data sampling rate for processing may be less or more spaced apart due to slower processing rates. Such embodiments remain within the scope of the present disclosure, as calibration and / or position corrections may still occur while the system is in use. The exemplary embodiments described herein may be used to track machine paths, for example, to determine drift over time or across a path. The exemplary embodiments may then use the position corrections and offsets described herein to refine path corrections when the machine is intended to repeat the same or similar path.

[0021] Exemplary embodiments described herein may include systems and methods having multiple sensors, one or more processors, and one or more memories having machine-readable instructions configured to perform the functions described herein when executed by the one or more processors. The sensors are configured to generate sensor data. Precision metrology data is fed directly into a high-speed PLC processor. The data is then synchronized with an IMU data stream and fused together for position enhancement between the two data sets. A comparison of the actual object position to the desired object position is performed in real time, and kinematic corrections are calculated and sent to the machine to update the object position.

[0022] The exemplary embodiments described herein include a combination of sensors combined with an IMU and software for performing data analysis to synchronize and fuse data sets together for path correction analysis and implementation. The exemplary embodiments described herein may also, or alternatively, be used to monitor multiple objects within a single environment (cell).

[0023] The exemplary embodiments described herein may include sensors including laser trackers, photogrammetry, and inertial measurement units (IMUs), among others. Exemplary embodiments including photogrammetric sensors may be used to observe and analyze multiple objects in an environment. Exemplary embodiments including IMU sensors may be used regardless of line-of-sight limitations within the system environment to individual devices or objects for observation.

[0024] 1 illustrates an exemplary system 100 for providing real-time seven degrees of freedom (7DoF) positioning accuracy for position correction. Exemplary embodiments may use positioning information from component parts already integrated into the system components. Exemplary embodiments of the system described herein may use established capabilities from machine controllers, such as those from Leica, in combination with exemplary embodiments of the system components and methods described herein to provide a streamlined system that enables high-rate 7DoF feeds to the desired robot closed loop.

[0025] In an exemplary embodiment, the systems described herein may include an object 102 to be tracked. As shown, the object is a robot with an arm at its end for provisioning position manipulation of a tool or component part. While a robotic arm is provided as an example, the object to be tracked may be any object, including personnel, machinery, parts, components, etc. In the illustrated example, the robot may provide data output in machine coordinates based on the measured position of its own measurement system. The machine coordinates may be provided by the machine at a machine rate, such as, for example, 300 Hz.

[0026] In an exemplary embodiment, the systems described herein may include one or more sensors 104. A position tracker may be provided as one or more of the system sensors. As shown, a laser tracker is provided for absolute position tracking. In an exemplary embodiment, a Leica AT960 is used in conjunction with a real-time feature pack and a Leica T-Frame. The sensor may provide position data at a rate that is close to or better (faster) than the object position information. For example, if the robot is providing data output at 300 Hz, the sensor information is preferably greater than 300 Hz, such as 1,000 Hz.

[0027] In an exemplary embodiment, the system 100 described herein may include one or more processors 106 configured to execute instructions from a machine-readable memory to implement the control functions described herein. In an exemplary embodiment, the system may be configured to receive a first data stream from the object under observation 102 and at least one second data stream from one or more sensors 104. The first data stream may be position data provided by the robot. The second data stream may be the position of the robot as detected by one of the sensors 104. Multiple second data streams may be provided to receive a data stream from each of the sensors from the multiple sensors if more than one sensor is used in the system.

[0028] As shown, system 100 may include one or more sensors 104 (identified in FIG. 1 as laser trackers) external to object 102 to be tracked. In this embodiment, one or more sensors 104 may directly detect the location and / or orientation of the object to be tracked (together as the object's position). Additionally or alternatively, one or more sensors may be coupled to the object to be tracked and used to identify other objects or locations in the environment. The location and / or orientation of the object to be tracked may then be determined by the sensor information and / or relative to other objects or locations in the environment.

[0029] In an exemplary embodiment, the system may include an object controller. The object controller may be a controller configured to provide instructions to the object to be tracked. For example, the object controller 108 may be a controller of a robot. The object controller 108 may be integrated with one or more processors 106 that provide analysis and position corrections as described herein. The object controller 108 may be integrated with the object to be tracked 102, such as a robot. The object controller 108 may be a separate, stand-alone controller configured to receive instructions from the one or more processors 106 and, based on those instructions, transmit signals to control the movement of the object to be tracked. In an exemplary embodiment, robot position correction information is generated by the one or more processors and transmitted to the controller, and the controller is configured to implement path corrections or compensations based on the robot position correction information to improve real-time position accuracy of the object to be tracked, such as a robot.

[0030] In an exemplary embodiment, the processor and memory are configured to provide initial processing. The initial processing may include receiving and processing global metrology data. The metrology data may include any combination of position information as described herein. For example, the metrology data may include position data received from an object to be tracked (e.g., a robot self-identification coordinate location) and / or position data received from one or more sensors, such as a laser sensor, that provide position information of the object under observation.

[0031] In exemplary embodiments, the initial processing may include synchronization. Multiple measurement data streams may be received by a processor and synchronized to a master clock. Exemplary embodiments may also, or alternatively, extrapolate information between data points within the data streams to synchronize data streams having different sampling rates. For example, a first data stream may have a sampling rate of 60 Hz, while a second data stream may have a sampling rate of 120 Hz. Data points between the 60 Hz sampling rates may be extrapolated from the data streams so that the two data streams can be integrated, analyzed, and / or compared with corresponding data points as described herein.

[0032] In an exemplary embodiment, the initial processing may include linking. Multiple measurement data streams may be received by a processor and analyzed so that the data streams can be transformed into the same coordinate system. For example, a first data stream may comprise orientation information, while a second data stream may provide location information. The two data streams may be merged to provide a complete location and orientation description of the object to be tracked. However, if a third data stream is used that provides additional location information, the location information from the different data streams is first transferred to a universal coordinate system. The universal coordinate system may be the same as one or more of the data streams from one or more sensors, may be an environmental coordinate system, may be relative to the object to be tracked, etc. The system may therefore convert data streams that are not in a universal coordinate system to the universal coordinate system in order to integrate, analyze, and / or compare the data streams as described herein. In an exemplary embodiment, the initial processing may include data conversion. In an exemplary embodiment, one or more of the data streams may be converted into a common coordinate system. For example, the data may be converted into a machine coordinate system.

[0033] An exemplary embodiment may include a tracker positioned within the environment. The tracker may provide a known location within the environment from which a universal coordinate system can be determined. The tracker may be separate from the environment, positioned for coordinate calibration, and / or may be part of the environment, such as being or being included with one or more objects within the environment. Information from the tracker may be received from one or more processors and used to transform the data stream into the universal coordinate system. For example, a data stream received at a machine location may be transformed by knowing the machine location relative to the tracker. The system may therefore take the location measured from the machine coordinates, compare it to the tracker coordinates, and transform the location information into a universal coordinate system, such as the environment location.

[0034] In exemplary embodiments, the initial processing may include data offsetting. In exemplary embodiments, the data may be offset relative to a point of interest on the object under observation, such as a tool center point (TCP) frame. For example, if the location of a machine is detected by one or more sensors, an offset relative to the actual point of interest on the machine may be determined by knowing the relative position of the point of interest to a point measured by the sensors. The exemplary embodiments may therefore track the location of a portion of the object under observation when the point of interest is not observable by a sensor and still provide improved or confirmed location information for the point of interest.

[0035] In an exemplary embodiment, the initial processing may include error analysis. Exemplary embodiments may include error checking, data transmission monitoring, etc. to reduce errors in the system.

[0036] In exemplary embodiments, the processor and memory are configured to provide position processing. Position processing may include receiving and processing measurement data and / or initial processing data to provide position accuracy and / or updates as described herein. Any of the exemplary embodiments described herein may include analysis and / or processing within a pre-processing stage that analyzes the data and / or within a processing stage that analyzes the data. The steps are not necessarily separate or require a specific order. For example, data synchronization may occur in the pre-processing stage and / or in the processing stage of data analysis. The pre-processing and processing steps may occur in different parts of the system or may be implemented in the same processor.

[0037] In an exemplary embodiment, position processing includes receiving and synchronizing position data from an object to be tracked to a master controller. For example, as shown, robot 102 may provide information about its position to system 100, such as processor 106. The position data from the object may be where the machine is intended to be, but this may be imprecise or inaccurate due to machine drift during use. The position data from the object may therefore not be the actual position of the object to be tracked. The position data from the object to be tracked may be the object's position based on information from or to the object to control its position without the benefit of the systems and methods described herein. In other words, this may be position information where the machine is intended to be, regardless of the object's actual physical location.

[0038] In an exemplary embodiment, position processing includes comparing position data from object 102 with position data as determined by one or more sensors 104. An exemplary embodiment may therefore compare a location where object 102 to be tracked (e.g., robot position) is intended to be, based on information from the robot itself, with the actual real-world location of the robot, which may include variations or deviations from the intended location (e.g., through mechanical drift). The deviation between the actual location and the intended location of the object to be tracked is thus determined. In an exemplary embodiment, the deviation may traverse seven degrees of freedom (7 DoF).

[0039] In an exemplary embodiment, the position processing comprises validation: The correction values may be validated based on set correction tolerances.

[0040] In an exemplary embodiment, position processing includes determining expected deviations and adjusting the current position of the object under observation in real time for synchronization and real-time correction. In an exemplary embodiment, one or more data streams may be constructed from different data rates with different latencies. An exemplary embodiment may therefore generate a position prediction to provide current instructions that can be executed and implemented to account for latency from machine-implemented position calculations and corrections.

[0041] In an exemplary embodiment, the position commands may be sent to the robot controller 108 to be implemented in a closed loop to provide real-time position compensation.

[0042] 2 illustrates an example system 200 for real-time seven degrees of freedom (7DoF) positioning accuracy with data synchronization and fusion. Example embodiments of the system described herein may fuse data used to obtain high-accuracy / rate position data, such as that described with respect to FIG. 1, with high-accuracy / rate rotational or other position data provided by an inertial measurement unit (IMU), etc., to produce a 7DoF data stream. Example embodiments may use a rotational data stream in conjunction with a machine's positional data stream because the rotational data stream may not degrade or drift over distance.

[0043] The system of FIG. 2 may use any combination of system components and algorithms as described with respect to FIG.

[0044] In an exemplary embodiment, the systems described herein may include an object 202 to be tracked. As shown, the object is a robot with an arm at its end for provisioning a tool or component part position manipulation. The robot may provide data output in machine coordinates based on its current measured position. The machine coordinates may be provided by the machine at a machine rate, such as, for example, 300 Hz.

[0045] In exemplary embodiments, the systems described herein may include one or more sensors 204. As shown, a position tracker may be provided as one or more of the system sensors. As shown, a laser tracker is provided for absolute position tracking. In the exemplary embodiment, a Leica AT960 is used with a real-time feature pack, although other trackers may be used. In the exemplary system illustrated in FIG. 2, the system also includes one or more IMUs. Other trackers (not just laser trackers), such as OptiTrack high-speed photogrammetry and / or Emcore IMUs, may also be used in combination with or in place of the described Leica laser tracker, which may provide position data. In the exemplary embodiment, the robot provides data output at 300 Hz, while the sensor provides output at approximately 200 Hz.

[0046] In an exemplary embodiment, the systems described herein may include one or more processors 206 configured to execute instructions from a machine-readable memory to implement the control functions described herein. In an exemplary embodiment, the system may be configured to receive a first data stream from an object under observation and at least one second data stream from one or more sensors. The first data stream may be position data provided by the robot. The second data stream may be the position of the robot detected by one of the sensors. Multiple second data streams may be provided to receive a data stream from each sensor from the multiple sensors when more than one sensor is used in the system. In the configuration illustrated in FIG. 2, multiple second data streams are provided, including at least a data stream from a data tracker and one or more data streams from an IMU. In an exemplary embodiment, the IMU may provide three rotational degrees of freedom position information in one or more data streams.

[0047] In an exemplary embodiment, the system may include an object controller 208. The object controller may be a controller configured to provide instructions to the object under observation. For example, the object controller may be a controller of a robot. The object controller may be integrated with one or more processors that provide analysis and position correction as described herein. The object controller may be integrated into the object under observation, such as a robot. The object controller may be a separate, stand-alone controller configured to receive instructions from the one or more processors and, based on those instructions, send signals to control the movement of the object under observation. In an exemplary embodiment, robot position correction information is generated by the one or more processors and sent to the controller, which is configured to implement path corrections or compensations based on the robot position correction information to improve real-time position accuracy of the object under observation, such as a robot.

[0048] In an exemplary embodiment, the processor and memory are configured to provide initial processing. The initial processing may include receiving and processing global metrology data. The metrology data may include any combination of position information as described herein. For example, the metrology data may include position data received from the object under observation (e.g., robot self-identification coordinate location) and / or position data received from one or more sensors, such as a laser sensor and / or an IMU, that provide position information of the object under observation.

[0049] In an exemplary embodiment, the initial processing may include synchronization: multiple metrology data streams may be received by a processor and synchronized to a master clock.

[0050] In an exemplary embodiment, the initial processing may include binding. An exemplary embodiment of the initial processing with binding may be as described with respect to FIG. 1 , where measurement instruments and / or sensors generate data streams and transform one or more different coordinate systems into a universal coordinate system. In an exemplary embodiment, the initial processing may include data transformation. In an exemplary embodiment, one or more of the data streams may be transformed into a common coordinate system. For example, the data may be transformed into a machine coordinate system or an environmental coordinate system.

[0051] In an exemplary embodiment, the initial processing may include a data offset: In an exemplary embodiment, an IMU offset from NED (north-east-down) to TCP may be calculated and applied.

[0052] In an exemplary embodiment, initial processing may include fusing data streams. For example, data received from one or more sensors may provide information regarding one or more different positional degrees of freedom. Information from multiple sensors may therefore be synchronized and integrated to fuse the data to generate a single multi-degree-of-freedom determined position. For example, a laser sensor may provide three translational or coordinate degrees of freedom (location), while an inertial measurement unit (IMU) may provide three rotational degrees of freedom (orientation). Data streams from the laser sensor and IMU may be fused to generate six degrees of freedom (position) and a seventh degree of freedom in time. In an exemplary embodiment, translational and angular data may be fused using Kalman filtering to generate a 7DoF frame. Although Kalman filtering is described herein, other data fusion techniques may also be used. Sensors may include overlapping data on one or more degrees of freedom. For example, an IMU may also provide translational or coordinate degrees of freedom (location) in addition to rotational degrees of freedom. In this case, the translational degree-of-freedom information may be integrated.

[0053] In an exemplary embodiment, the initial processing may include error analysis. Exemplary embodiments may include error checking, data transmission monitoring, etc. to reduce errors in the system.

[0054] In an exemplary embodiment, the processor and memory are configured to provide position processing, which may include receiving and processing measurement data and / or initial processing data to provide position accuracy and / or updates as described herein.

[0055] In an exemplary embodiment, position processing includes receiving position data from the object under observation and synchronizing it to a master clock. For example, as shown, a robot may provide information about its position to the system. The position data from the object may be where the machine is intended to be, but this may be imprecise or inaccurate due to machine drift during use. The position data from the object may therefore not be the actual location of the object under observation. The position data from the object may be the position of the object based on information from or to the object for controlling its position without the benefit of the systems and methods described herein. In other words, this may be position information where the machine is intended to be, regardless of the actual location of the object.

[0056] In an exemplary embodiment, position processing includes comparing position data from the object with position data as determined by one or more sensors. An exemplary embodiment may therefore compare a position where the object under observation (e.g., the robot position) is intended to be, based on information from the robot itself, with the robot's actual real-world position, which may include variations or deviations from the intended position (e.g., through mechanical drift). The deviation between the actual position and the intended position of the object under observation is thus determined. In an exemplary embodiment, the deviation may traverse seven degrees of freedom (7DoF).

[0057] In an exemplary embodiment, the position processing comprises validation: The correction values may be validated based on set correction tolerances.

[0058] In an exemplary embodiment, position processing includes determining expected deviations for synchronization and real-time corrections and adjusting the current position of the object under observation in real time. In an exemplary embodiment, one or more data streams may be constructed from different data rates with different latencies. An exemplary embodiment may therefore generate a position prediction to provide current instructions that can be executed and implemented to account for latency from position calculations and corrections in position implementations in time.

[0059] In exemplary embodiments, position prediction may include providing a time-based offset and / or a position-based offset. Exemplary embodiments may therefore be used when a machine is used repeatedly and / or for the same duration of time along the same intended path. In these instances, exemplary embodiments may be used to monitor the actual path compared to the machine path to determine drift over time and / or drift associated with a specific direction and / or path. When the intended path is repeated for a redundant task, the machine may correct using position predictions from prior use and monitoring.

[0060] In an exemplary embodiment, the position commands may be sent to a robot controller to be implemented in a closed loop to provide position compensation, which may be done in real time for machine use or in a sequential loop.

[0061] In exemplary embodiments, pre-processing and / or processing of the data streams may include any combination of steps of synchronizing the data streams so that the data is provided in a single time frame, transforming the data so that the data is provided in a single coordinate system, determining corrections for the machine from its actual position as determined from the data streams from one or more sensors, and / or forecasting the data so that location corrections can be provided when the corrections can occur on the machine itself.

[0062] In an exemplary embodiment, determining the correction may include comparing an actual position of the object to be tracked from an integrated data stream from one or more sensors with a machine position based on a machine position data stream that provides the machine position as perceived by itself and / or a controller providing the machine position instructions. The comparison may be to determine an offset from the machine position and the actual position.

[0063] 3 illustrates an exemplary system for real-time seven degrees of freedom (7DoF) positioning accuracy for multiple object motion detection and / or tracking. Exemplary embodiments of the systems described herein may include a photogrammetric camera combined with high accuracy / rate position data, for example, as described with respect to FIG. 1. In exemplary embodiments of the systems and methods described herein, the system may improve position accuracy, improve position knowledge with limited or blocked line of sight, improve multiple object positioning and / or motion tracking, and / or improve relative position tracking of multiple objects.

[0064] The system of FIG. 3 may use any combination of system components and algorithms as described with respect to FIGS.

[0065] In an exemplary embodiment, a system 300 described herein may include an object 302 to be tracked. As shown, the object is a robot with an arm at its end for provisioning a tool or component part position manipulation. The robot may provide data output in machine coordinates based on its current measured position. The machine coordinates may be provided by the machine at a machine rate, such as, for example, 300 Hz.

[0066] In an exemplary embodiment, the system 300 described herein may include one or more sensors 304. As shown, a position tracker may be provided as one or more of the system sensors. As shown, a high-speed photogrammetry camera is provided to provide sequential images (frames) of an area under observation that may include an object under observation. The exemplary embodiments described herein may include sensors, including one or more IMUs. For example, the tracker may be a combination of OptiTrack high-speed photogrammetry and / or Emcore IMUs. The sensors may provide position data at a rate such as about 120 Hz. As described herein, the sensors have approximate rates, which are exemplary only. Other data rates are within the scope of this disclosure.

[0067] In an exemplary embodiment, the system 300 described herein may include one or more processors 306 configured to execute instructions from a machine-readable memory to implement the control functions described herein. In an exemplary embodiment, the system may be configured to receive a first data stream from an object under observation and at least one second data stream from one or more sensors. The first data stream may be position data provided by the robot. The second data stream may be the position of the robot detected by one of the sensors. Multiple second data streams may be provided to receive a data stream from each sensor from the multiple sensors if more than one sensor is used in the system. As shown, multiple second data streams are provided, including a data stream from a camera and at least one data stream from an IMU.

[0068] In an exemplary embodiment, system 300 may include an object controller 308. The object controller may be a controller configured to provide instructions to an object under observation. For example, the object controller may be a controller of a robot. The object controller may be integrated with one or more processors that provide analysis and position correction as described herein. The object controller may be integrated into the object under observation, such as a robot. The object controller may be a separate, stand-alone controller configured to receive instructions from one or more processors and, based on those instructions, send signals to control the movement of the object under observation. In an exemplary embodiment, robot position correction information is generated by the one or more processors and sent to the controller, which is configured to implement path corrections or compensations based on the robot position correction information to improve real-time position accuracy of the object under observation, such as a robot.

[0069] In an exemplary embodiment, the processor and memory are configured to provide initial processing. The initial processing may include receiving and processing global metrology data. The metrology data may include any combination of position information as described herein. For example, the metrology data may include position data received from the object under observation (e.g., robot self-identification coordinate location) and / or position data received from one or more sensors, such as a camera, that provide position information for the object under observation.

[0070] In an exemplary embodiment, the initial processing may comprise object recognition and / or position identification from images captured by the camera. For example, position information may be obtained by determining object point locations by recognizing features in image frames from the camera data stream. The camera data may then be transformed into a machine coordinate system.

[0071] In an exemplary embodiment, the initial processing may include synchronization: multiple metrology data streams may be received by a processor and synchronized to a master clock.

[0072] In an exemplary embodiment, the initial processing may include data transformation. In an exemplary embodiment, one or more of the data streams may be transformed to a common coordinate system. For example, the data may be transformed to a machine coordinate system.

[0073] In an exemplary embodiment, the initial processing may include a data offset, in an exemplary embodiment, an IMU offset from the NED to the TCP may be calculated and applied.

[0074] In an exemplary embodiment, the initial processing may include error analysis. Exemplary embodiments may include error checking, data transmission monitoring, etc. to reduce errors in the system.

[0075] In an exemplary embodiment, the initial processing may include fusing data streams. For example, data received from one or more sensors may provide information regarding one or more different positional degrees of freedom. Information from multiple sensors may therefore be synchronized and integrated to fuse the data to generate a single multi-degree-of-freedom determined position. For example, images from a camera may be analyzed to generate three translational or coordinate degrees of freedom, while an IMU may provide three rotational degrees of freedom. Data streams generated from the camera data stream and the data stream from the IMU may be fused to generate six degrees of freedom position and a seventh degree of freedom in time. In an exemplary embodiment, translational data and angular data may be fused using Kalman filtering to generate a 7DoF frame. Although Kalman filtering is described herein, other data fusion techniques may also be used.

[0076] In an exemplary embodiment, the processor and memory are configured to provide position processing, which may include receiving and processing measurement data and / or initial processing data to provide position accuracy and / or updates as described herein.

[0077] In an exemplary embodiment, position processing includes receiving position data from the object under observation and synchronizing it to a master clock. For example, as shown, a robot may provide information about its position to the system. The position data from the object may be where the machine is intended to be, but this may be imprecise or inaccurate due to machine drift during use. The position data from the object may therefore not be the actual location of the object under observation. The position data from the object may be the position of the object based on information from or to the object for controlling its position without the benefit of the systems and methods described herein. In other words, this may be position information where the machine is intended to be, regardless of the actual location of the object.

[0078] In an exemplary embodiment, position processing includes comparing position data from the object with position data as determined by one or more sensors. An exemplary embodiment may therefore compare a position where the object under observation (e.g., the robot position) is intended to be, based on information from the robot itself, with the robot's actual real-world position, which may include variations or deviations from the intended position (e.g., through mechanical drift). The deviation between the actual position and the intended position of the object under observation is thus determined. In an exemplary embodiment, the deviation may traverse seven degrees of freedom (7DoF).

[0079] In an exemplary embodiment, the position processing comprises validation: The correction values may be validated based on set correction tolerances.

[0080] In an exemplary embodiment, position processing includes determining expected deviations for synchronization and real-time corrections and adjusting the current position of the object under observation in real time. In an exemplary embodiment, one or more data streams may be constructed from different data rates with different latencies. An exemplary embodiment may therefore generate a position prediction to provide current instructions that can be executed and implemented to account for latency from position calculations and corrections in position implementations in time.

[0081] In an exemplary embodiment, the position commands may be sent to a robot controller to be implemented in a closed loop to provide real-time position compensation.

[0082] 4 illustrates an exemplary system for real-time seven degrees of freedom (7DoF) positioning accuracy for multiple object motion detection and / or tracking. Exemplary embodiments of the systems described herein may include a photogrammetric camera for generating position data, such as that described with respect to FIG. 3. In exemplary embodiments of the systems and methods described herein, the system may improve position accuracy, improve position knowledge with limited or blocked line of sight, improve multiple object positioning and / or motion tracking, and / or improve relative position tracking of multiple objects.

[0083] The system of FIG. 4 may use any combination of system components and algorithms as described with respect to any combination of FIGS. 1-3.

[0084] The exemplary embodiment of system 400 illustrated in Figure 4 may include system components similar to those of Figures 1-3, including an object to be tracked 402, a processor 406, and a controller 408. These component parts will not be described in full detail here, but may have the same or similar features as described in other embodiments presented herein.

[0085] In an exemplary embodiment, the system 400 described herein may include one or more sensors 404. As shown, the sensors comprise cameras. The sensors may provide position data at a rate of approximately 120 Hz.

[0086] In an exemplary embodiment, the system 400 described herein may include one or more processors configured to execute instructions from a machine-readable memory to implement the control functions described herein. In an exemplary embodiment, the system 400 may be configured to receive a first data stream from the object under observation 402 and at least one second data stream from one or more sensors 404. The first data stream may be position data provided by the robot. The second data stream may be the position of the robot as detected by one of the sensors.

[0087] In an exemplary embodiment, the processor and memory are configured to provide initial processing. The initial processing may include receiving and processing global metrology data. The metrology data may include any combination of position information as described herein. For example, the metrology data may include position data received from the object under observation (e.g., robot self-identification coordinate location) and / or position data received from one or more sensors, such as a camera, that provide position information for the object under observation.

[0088] In an exemplary embodiment, the initial processing may include object recognition and / or position identification from images captured by the camera. For example, position information may be obtained by determining object point locations by recognizing features in image frames from the camera data stream. The camera data may then be transformed into a machine coordinate system. In an exemplary embodiment, object recognition in the data stream from the camera may include generating seven degrees of freedom, including positional and rotational degrees of freedom, through object recognition, placement, and orientation. The object may include a marker to assist in object detection and / or orientation analysis. The marker may comprise any identifier that can be analyzed and recognized by the system, such as color, shape, etc. The marker may have a known position, orientation, size, etc., so that the location and orientation of the object under observation can be determined from images from the camera data stream. Other image recognition features may also or alternatively be used in object detection, etc.

[0089] In an exemplary embodiment, the initial processing may include synchronization: multiple metrology data streams may be received by a processor and synchronized to a master clock.

[0090] In an exemplary embodiment, the initial processing may include data transformation. In an exemplary embodiment, one or more of the data streams may be transformed to a common coordinate system. For example, the data may be transformed to a machine coordinate system.

[0091] In an exemplary embodiment, the initial processing may include a data offset: In an exemplary embodiment, the camera 7DoF rigid body is offset relative to the TCP frame.

[0092] In an exemplary embodiment, the initial processing may include error analysis. Exemplary embodiments may include error checking, data transmission monitoring, etc. to reduce errors in the system.

[0093] In an exemplary embodiment, the initial processing may include fusing data streams. For example, data received from one or more sensors may provide information regarding one or more different positional degrees of freedom. Information from multiple sensors may therefore be synchronized and integrated to fuse the data to generate a single multi-degree-of-freedom determined position. For example, images from a camera may be analyzed to generate three translational or coordinate degrees of freedom, while an IMU may provide three rotational degrees of freedom. Data streams generated from the camera data stream and the data stream from the IMU may be fused to generate six degrees of freedom position and a seventh degree of freedom in time. In an exemplary embodiment, translational data and angular data may be fused using Kalman filtering to generate an optimal 7DoF frame. Although Kalman filtering is described herein, other data fusion techniques may also be used.

[0094] In an exemplary embodiment, the processor and memory are configured to provide position processing. Position processing may include receiving and processing measurement data and / or initial processing data to provide position accuracy and / or updates as described herein. Position processing may comprise any processing algorithm as described herein. In an exemplary embodiment, various data streams are synchronized to a master clock, location data of the object under observation is compared against actual position data as determined by analysis of one or more data streams received from sensors, corrections are verified based on tolerances, and closed-loop predictive updates are provided to correct for deviations.

[0095] In an exemplary embodiment, the position processing may also include timing closed-loop updates to machine latency. This position processing may be used in any combination of the exemplary embodiments described herein, including, for example, FIGS. 1-3. The exemplary embodiments described herein may include estimating a predicted position of the object under observation. The exemplary embodiments described herein may also include estimating a correct predicted position and offset for the object under observation at a predicted time. An exemplary embodiment of predicted timing correction may be used to provide an offset to correct the predicted machine position at the time the machine executes the offset instruction. This may also, or alternatively, include modifying the data rate to match the transition rate of the machine instruction to the object under observation.

[0096] In an exemplary embodiment, the position commands may be sent to a robot controller to be implemented in a closed loop to provide real-time position compensation.

[0097] FIG. 5 illustrates an example system 500 according to embodiments described herein that includes multiple system components, including a sensor 504, an object under observation 502, a computer 516, a processor 506, memory, and a controller 508. The system of FIG. 5 may use any combination of system components, algorithms, and functions as described herein. As shown, the system components may be replicated such that multiple objects 502 may be tracked through the use of one or more sensors 504 to observe the objects. Exemplary embodiments of trackers described herein may include any combination of laser trackers, cameras, IMUs, etc. For example, the system may include a Leica AT960 and Leica T-Frame, OptiTrack high-speed photogrammetry, and / or an Emcore IMU. A first group of one or more sensors may be used to track a first group of one or more objects, while a second group of one or more sensors may be used to track a second group of one or more objects. A first group of one or more sensors may be the same as or different from a second group of one or more sensors. The groupings of sensors may overlap, such that some sensors are in both groups of sensors, while others are unique to a given group of sensors. Components as described herein may therefore be added, duplicated, merged, separated, or otherwise combined and remain within the scope of the present disclosure.

[0098] The exemplary embodiments described herein separate algorithms and processing steps as examples. However, the processing steps do not necessarily have to occur in any order or in any combination of steps. For example, steps from an initial processing algorithm may occur later and / or in combination with a location processing algorithm. The separation is provided for clarity and as an example only and does not require separate processing algorithms, calls, logical segments, etc. Instead, they may be combined, separated, duplicated, repeated, or otherwise recombined and remain within the scope of the present disclosure.

[0099] The exemplary embodiments described herein may not require stability of system components to obtain metrology-grade measurements or observations for system position correction. For example, system components including sensors (such as cameras) may drift as the object under observation may also drift. The exemplary embodiments described herein may account for system drift to still provide precise positioning information and corrections.

[0100] The exemplary embodiments described herein may combine different sensors together to generate an improved data set. The sensor combination may be implemented in different ways.

[0101] For example, two different sensors may monitor an object, and their information combined to increase the accuracy of the position and / or orientation of the object under observation, and / or the inclusion of the two data sets may provide additional position and / or orientation information about the object. As a specific exemplary embodiment, an inertial measurement unit 510 may be positioned on the object under observation 502, and a laser tracker 504 may be configured to track the distance to the object under observation. Fusion of the two data sets may obtain the position, including the orientation, including yaw, pitch, and roll, from the inertial measurement unit, and the Cartesian (or some other system) coordinate location (x, y, z), to obtain complete location information for the object under observation.

[0102] As another example, one sensor may be positioned on another sensor to provide information about itself, which in turn provides information about the object under observation. For example, inertial measurement unit 510 may be positioned on laser tracker 514. The laser tracker may be configured to track object under observation 502 (also described herein as the object to be tracked) and provide its location. By using a sensor on another sensor, additional location information can be obtained. For example, using a sensor on a detection sensor to monitor an object allows the location of the detection sensor to be known and updated based on information received from the sensor. Thus, the accuracy of the information detected from the detection sensor is improved because its unique location is known to a more precise extent. Additionally, the exemplary embodiments may be used to reduce feedback time because the sensor can be used to predict changes in the detection sensor and thereby update its positioning information more quickly than by tracking the movement of the detection sensor alone.

[0103] In an exemplary embodiment, a first sensor may be positioned on another sensor configured to monitor the object under observation. The sensor on the other sensor may be used for maintenance, external calibration, predictive localization, faster processing, or a combination thereof. To enhance the maintenance of external calibration of sensors within the array, an angle sensor fit may be installed on the dimensional measurement device.

[0104] For example, an inertial measurement unit may be positioned on the laser tracker. To eliminate sensor position drift, the adaptation of the inertial measurement unit can dynamically correct the external calibration of the sensor used to observe the object, ensuring that the relationship between the machine frame and the sensor array is maintained and reducing unnecessary noise in the resulting data. Thus, the 7DoF output produced by the sensor can be greatly improved and maintained over time. Furthermore, the correction of the external calibration can be performed dynamically without interrupting machine operation.

[0105] In another embodiment, an inertial measurement unit may be positioned on the photogrammetric camera. To eliminate sensor position drift, the inertial measurement unit can dynamically correct the external calibration of the sensor used to observe the object, ensuring that the relationship between the machine frame and the sensor array is maintained and reducing unnecessary noise in the resulting data. Thus, the 7DoF output produced by the sensor array is greatly improved / maintained over time. Furthermore, the correction of the external calibration can be performed dynamically without interrupting machine operation.

[0106] As described herein, exemplary embodiments include generating predictive position information for an object. The predictive position information may be used in different ways within the exemplary embodiments described herein. For example, the predictive position information may be used to provide opportunity commands to operate an object and correct the object's position in real time, such that the correction occurs along with or is synchronized with the actual implementation of instructions to control the machine. The predictive position information may be used for faster analysis of position information. For example, if a position is already predicted, the system may be used to correct or work from the prediction instead of determining a position based solely on the data stream. Starting from a predictive position may improve processing speed. The predictive position information may be used for error correction or validation of determined position corrections. The predictive position information may be used to improve position information received from one or more sensors, such as by predicting the position of the sensor itself.

[0107] 6 illustrates an exemplary system diagram according to embodiments described herein. The system may include different combinations of computing devices, such as a mobile device 1004, a laptop 1001, a computer 1002, a server 1003, a programmable logic controller 1020, a processor, an industrial personal computer, etc. The system may include different combinations of sensors, such as a camera 1014, a video, a laser tracker 1016, an inertial measurement unit 1012, etc. One or more sensors may be configured to communicate with a local processing unit 1020 as described herein. Example embodiments may perform pre-processing in the individual sensors 1012, 1014, 1016, etc. and / or in the local processor 1020. In an exemplary embodiment, the one or more sensors may communicate with one or more remote electronic devices 1004, 1003, 1002, 1001, etc., which may be used to analyze data, store data, control and / or send instructions to the one or more sensors 1012, 1014, 1016, and assist in observing the one or more sensors, and / or device controller 1018, and / or the environment with the machine 1010. Location processing as described herein may occur in the local processing 1020 and / or in the remote electronic devices 1004, 1003, 1002, 1001. The system may also communicate with one or more databases or memory locations 1005 (or as contained within one or more electronic devices 1001, 1002, 1003, 1004, 1020). After position processing, the position control information may be provided to a machine controller 1018 to provide positioning instructions to the machine 1010 and / or may provide position instructions directly to the machine 1010.

[0108] In exemplary embodiments, the processor and memory are configured to provide position processing. Position processing may include receiving and processing measurement data and / or initial processing data to provide position accuracy and / or updates as described herein. The exemplary embodiments described herein may include analysis and / or processing within a pre-processing stage that analyzes the data and / or within a processing stage that analyzes the data. The steps are not necessarily separate or require a specific order. For example, data synchronization may occur in the pre-processing stage and / or in the processing stage of data analysis. The pre-processing and processing steps may occur in different parts of a system as described herein or may be performed in the same processor. For example, some pre-processing may occur in the sensor and / or one or more local processors. Processing may occur in one or more local processors (either the same or different from those for pre-processing) and / or in a remote electronic device, in a machine controller, or other combinations as described herein.

[0109] The processor and memory may be configured to provide processing of data streams received from one or more sensors. Initial processing may include receiving and processing global metrology data. The metrology data may include any combination of position information as described herein. For example, the metrology data may include position data received from an object to be tracked (e.g., a robot's self-identifying coordinate location) and / or position data received from one or more sensors, such as a laser sensor, that provide position information for the object under observation. The processors described herein may be configured through the use of machine-readable instructions stored in memory and executed by the processor to perform the analysis of the metrology data described herein and provide position corrections as described herein. Position corrections may also, or alternatively, include machine or obstacle avoidance.

[0110] Figure 8 illustrates an example architecture structure according to embodiments described herein. The measurement data illustrated from Figure 8 may be from one or more sensors described herein, such as sensors 1012, 1014, or 1016 of Figure 6. The position calculation and / or receipt of the position data may be performed, for example, by a local processor 1020 as described herein. The robot interface may be an object under direct observation, such as object 1010, or a controller thereto, such as controller 1018.

[0111] FIG. 7 illustrates an example method 700 of position correction according to embodiments described herein.

[0112] In step 702, a method according to embodiments described herein may include providing an object under observation and a sensor for generating a data stream. The sensor may be configured to provide information about the object under observation.

[0113] An exemplary method may include providing one or more sensors for observing an object under observation (otherwise described herein as an object to be tracked). Exemplary embodiments described herein may include providing a plurality of sensors, one or more processors, and one or more memories having machine-readable instructions configured to perform the functions described herein when executed by the one or more processors. The sensors may be configured to generate sensor data. The sensor data may be precision measurement data.

[0114] In step 704, sensors may be used to generate data streams, and data may be communicated from the sensors to one or more processors. In an exemplary embodiment, one or more sensors may be used to generate a data stream from each of the one or more sensors. The data streams may correspond to an object under observation. The data streams may provide location information about the object under observation. The data streams may provide information about another sensor, the combination of which may provide location information about the object under observation.

[0115] In step 706, the method may include processing the received data stream corresponding to the object under observation. The processing may include any combination of processing as described herein.

[0116] In an exemplary embodiment, the method may include integrating data streams of one or more sensors. Integrating the data streams from one or more sensors may generate a single integrated data stream. Integrating the data streams from one or more sensors may include synchronizing the data streams from the one or more data streams to a master time. Integrating the data streams from one or more sensors may include transforming the data streams to a universal coordinate system. Integrating the data streams may include extrapolating data points in one or more of the data streams. Integrating the data streams may include filtering the data streams. In an exemplary embodiment, integrating the data streams may use Kalman filtering to produce the integrated data stream. The integration may include additional or alternative methods of integrating data. For example, statistical theory for estimation of data sets over time may be used. The method may account for imprecision, statistical noise, etc. to produce estimates of unknown variables. Exemplary embodiments may use multiple measurements to provide a probability distribution for a variable over one or more time frames.

[0117] In step 708, the method may include providing an object under observation and receiving information from the object under observation. In an exemplary embodiment, the method may include obtaining position data from the object under observation. The position data from the object under observation may be from a controller or the object itself. The position data from the object may be based on a position command where the object believes its position should be. The position data from the object may be from one or more sensors used to control the position of the object.

[0118] The exemplary embodiments described herein include providing precision metrology data directly into a high-speed PLC processor. The data is then synchronized with an integrated data stream from one or more sensors. A comparison is made in which the actual object position based on processing from the one or more sensors is compared to a desired object position based on a position generated by the object itself. The comparison may be made in real time, and kinematic corrections may be calculated. The updated kinematic corrections may be sent to the object to update the object position.

[0119] In step 710, the information from the integrated data stream from the sensors and the information from the object under observation are compared. In an exemplary embodiment, the comparison is used to determine an offset or position discrepancy between the actual position of the object under observation and the intended position of the object under observation.

[0120] As illustrated herein, the method may include using the updated kinematic corrections to reposition or update the position information of the object under observation. Thus, the object under observation may have a corrected position such that the actual position is closer to the desired position as intended by the user of the object. Other exemplary embodiments may use the updated kinematic corrections to avoid collisions between obstacles or objects by observing the actual position relative to the programmed position to avoid collisions when the programmed position deviates from the actual position.

[0121] In step 712, the method may include correcting the position of the object under observation using the position information. The correction may be performed in any combination of manners. For example, the system may determine an offset to move the object from its actual position to a desired position, accounting for drift. The correction may be performed by calibrating the object under observation so that the object's controller is updated to correct its intended position to its actual position. In an exemplary embodiment, the correction may be performed on subsequent paths of repeated actions of the object under observation to adjust the path to account for drift over time.

[0122] In an exemplary embodiment, the method may include providing the position information to an object controller to correct location or position commands to the object.

[0123] An exemplary embodiment may include a tracker positioned within the environment. The tracker may provide a known location within the environment from which a universal coordinate system can be determined. The tracker may be separate from the environment, positioned for coordinate calibration, and / or may be part of the environment, such as being or being included with one or more objects within the environment. Thus, the method may include providing a tracker at a known location within the environment. The tracker location is used to determine coordinate information in one or more of the data streams.

[0124] In an exemplary embodiment, the method may include data offsetting, in an exemplary embodiment, the data may be offset relative to a point of interest on the object under observation, such as a TCP frame.

[0125] In an exemplary embodiment, the method may include receiving and synchronizing position data from an object under observation. For example, as shown, a robot may provide information about its position to the system. The position data from the object may be where the machine is intended to be, but this may be imprecise or inaccurate due to machine drift during use. The position data from the object may therefore not be the actual position of the object under observation. The position data from the object may be the position of the object based on information from or to the object for controlling its position without the benefit of the systems and methods described herein. In other words, this may be position information where the machine is intended to be, regardless of the actual location of the object.

[0126] In an exemplary embodiment, the method may include comparing position data from the object with position data as determined by one or more sensors. An exemplary embodiment may therefore compare a position where the object under observation (e.g., the robot position) is intended to be, based on information from the robot itself, with the robot's actual real-world position, which may include variations or deviations from the intended position (e.g., through mechanical drift). A deviation between the actual position and the intended position of the object under observation is thus determined. In an exemplary embodiment, the deviation may traverse seven degrees of freedom (7DoF).

[0127] In an exemplary embodiment, the method comprises data validation: The correction values may be validated based on set correction tolerances.

[0128] In exemplary embodiments, the method may include determining an expected deviation for synchronization and real-time correction and adjusting the current position of the object under observation in real time. In exemplary embodiments, one or more data streams may be constructed from different data rates with different latencies. Exemplary embodiments may therefore generate a position prediction to provide current instructions that can be executed and implemented to account for latency from position calculations and corrections for position implementation in time. Exemplary embodiments of the methods described herein may also include predicting future kinematic corrections for the object under observation. The predicted future kinematic corrections may be used to control the object under observation at future times such that by the time an instruction using the predicted future kinematic corrections is actually implemented by the controller, the object under observation will have been controlled at the future time, and thus the kinematic corrections may be synchronized with the implementation of the instruction providing the kinematic corrections.

[0129] In an exemplary embodiment, the position commands may be sent to a robot controller to be implemented in a closed loop to provide real-time position compensation.

[0130] 9 illustrates an exemplary system configuration according to an embodiment described herein. As shown, the system includes an external processor for high-speed reception and analysis for real-time positioning of controlled devices.

[0131] As illustrated in Figure 9, the object to be controlled is a KUKA robot. The robot transmits joint and look-ahead data as part of its control information. The robot may then receive corrections according to exemplary embodiments described herein. The KUKA's Robot Sensor Interface (RSI) may be used to implement the corrections received from the processor described herein. The KUKA is merely an exemplary embodiment and is representative of an object to be controlled according to embodiments described herein.

[0132] As shown, the system may include one or more sensors for acquiring position information about one of the target objects for precision position refinement. The system may include any combination of a laser tracker, photogrammetry, and an inertial measurement unit (IMU), among others. Exemplary sensors may include, for example, an AT960 laser tracker, OptiTrack high-speed photogrammetry, and an Emcore EN-300 IMU. The one or more sensors may provide position information, such as any or all of the six degrees of freedom position information, from the sensor to a real-time operating system.

[0133] In an exemplary embodiment, as shown, each sensor may be connected to a real-time operating system. The exemplary real-time operating system may be, for example, an industrial personal computer (IPC) such as a Beckhoff IPC. Other operating systems, such as, for example, an industrial personal computer or a programmable logic controller (PLC), may also be used. The exemplary embodiment of the real-time operating system may receive data from the sensors, fuse the received data, receive position information from the target object, determine corrections according to embodiments described herein, and transmit the correction information to the target object to correct the target object.

[0134] The sensor may communicate, for example, between the sensor and the operating system. The communication may be through any method available to the sensor, for example, wired and / or wireless communication. Exemplary embodiments may communicate between the sensor and the operating system via EtherCat, Ethernet, serial, or other communication. As shown, the Leica laser tracker may communicate through a real-time feature pack, which communicates via EtherCat.

[0135] Exemplary embodiments of the systems described herein may optionally include a data logger. The data logger may be configured to receive, collect, and store data from one or more sensors. For example, the logger may store information from raw sensor data, analyzed data from a processor (operating system), or other data, information, or analysis.

[0136] An exemplary embodiment of a processor may be configured to implement the algorithms described herein to fuse information from sensors and determine position corrections to the target object. For example, data may be filtered or augmented so that position information can be fused into a complete data set. Redundant data, such as position information from different sources, may be integrated through statistical analysis to obtain a desired level of accuracy. For example, information may be averaged, weighted averaged, substituted, or otherwise combined. Sensors with higher accuracy may be prioritized over other sensor data in obtaining a resulting position fix. The integration of data may be through Kalman filtering or may include other sensor fusion techniques. The algorithm executed by the processor may include comparing the integrated measurement data and position data from the target object to determine corrected position information. The algorithm may optionally include projecting the corrected position information into the future so that corrections to the target object's position will be consistent with the execution and / or implementation of instructions based on the latency or processing time of the system.

[0137] The exemplary embodiments described herein may include concepts and applications for customized development or implementation of real-time industrial-grade global metrology systems for industrial manufacturing and robotic cells. The exemplary embodiments described herein enable observation of all or a desired set of target objects within a given manufacturing cell with industrial-grade precision. The exemplary embodiments described herein may use the information to improve the accuracy and precision of assembly / manufacturing / inspection processes.

[0138] In an exemplary embodiment, to effectively saturate an area, the system herein may use multiple metrology sensors. This unique approach leverages advanced algorithms to fuse observations from all sensors within a cell. This can include the novel introduction of laser tracker data, high-speed photogrammetry data, and inertial dimensional metrology data provided by precision inertial measurement units. In addition, metrology data from mechanical and robotic devices within the cell can be fused into the solution.

[0139] In an exemplary optional embodiment, the resulting data may be used to initially calibrate all of the robots / machines in the cell and improve their accuracy. Calibration may begin with a series of poses to calculate parameters for an advanced kinematic model. Once calibrated, artificial intelligence algorithms with metrology feedback may be used to enable kinematic model adjustments during dynamic movement. In addition, metrology feedback may also be used for precision path corrections to nominal values.

[0140] In an exemplary embodiment, where a cell may have multiple robots, each robot may be calibrated and paths can be synchronized with a common global metrology frame. This allows for extremely precise coordination between parts and robots within the cell and greater precision throughout the entire working volume. Additionally, or alternatively, part positions can be constantly monitored and corrected. Dynamic adjustments to the assembly process can be applied in real time.

[0141] In an exemplary embodiment, each operation performed within the cell is observed by an industrial-grade metrology system, so that the inspection process is inherent to the manufacturing process, potentially eliminating downstream or offline inspection.

[0142] The computing devices described herein are unconventional systems at least due to the use of unconventional component parts and / or the use of unconventional algorithms, processes, and methods embodied, at least in part, in programming instructions stored and / or executed by the computing devices. For example, exemplary embodiments may use unique system configurations and associated processes for real-time augmented positioning systems as described herein, unique process and algorithm configurations and associated processes for object detection, location, feedback control looping, and positioning estimation for position accuracy and precision. Exemplary embodiments may be used for internal and / or external calibration of robots, which may be performed in real time. Exemplary embodiments of the systems and methods described herein may be used for robot or machine path correction for close-tolerance function. Exemplary embodiments of 7DoF real-time augmented positioning systems according to exemplary embodiments described herein may be used within a global metrology network to avoid collisions and reduce failure rates. Exemplary embodiments of the systems and methods described herein may be used for automation to improve quality. Exemplary embodiments of the systems and methods described herein may improve the accuracy and / or precision of less expensive instrumentation, with precision and accuracy capabilities for desired automation activities. Exemplary embodiments of the systems and methods described herein may be used to measure part (or object) position within a cell, including detecting, placing, and relative positioning of multiple objects within the environment. Parts and / or objects may be monitored simultaneously throughout a cellular environment, including monitoring the movement of multiple robots on a moving assembly line. Exemplary embodiments of system components may also offer unique attributes and / or technical benefits. For example, the use of a PLC processor may provide high-speed processing, reducing latency in calculations, enabling faster response and real-time processing.The algorithms described herein may include, for example, processing and position prediction to take machine operation into account so that corrections are consistent with machine implementation. Integration of sensor information and associated processing may improve accuracy, while filtering may be used to improve data processing time and position accuracy.

[0143] The exemplary embodiments described herein may be used in robotics, devices, drones, machinery for real-time positioning and correction. The exemplary embodiments may be used with object combinations for applications such as global recognition, precise and / or precise relative positioning, collision avoidance, etc. The exemplary embodiments may be used within an area for area positioning, mapping, tracking, etc. The exemplary embodiments may have applications in machining environments for manufacturing, tooling, agriculture, surveillance, etc.

[0144] The exemplary embodiments described herein may be used for a seven-degree-of-freedom real-time augmented positioning system. The exemplary embodiments may use up to six degrees of spatial positioning. Up to three degrees of spatial positioning may be translation along three perpendicular axes in three dimensions: forward / aft, i.e., surge; up / down, i.e., heave; and left / right, i.e., sway. Up to three degrees of spatial positioning may be in rotation about three perpendicular axes: rotation about a normal or up / down axis, i.e., yaw; rotation about a horizontal or left / right axis, i.e., pitch; and rotation about a longitudinal or forward / aft axis, i.e., roll. The final degree of freedom may be in time to enable real-time assessment and correction of the position of an augmented positioning system as described herein.

[0145] The exemplary embodiments described herein may use degrees of freedom in time. The exemplary embodiments described herein may use time to provide real-time processing and update the system position in real time. As used herein, real-time is understood to include receiving and processing information continuously, or at sufficiently short intervals, to make positioning determinations and / or corrections to positioning determinations in a time sufficient for the precision required for a given application, without interfering with the continued use of the system whose positioning is being determined. In other words, the machine, component, object, etc. being positioned does not need to be taken out of service for calibration, but can be positioned during use. Positioning may also occur in sufficient time such that the benefits of the positioning meet the requirements of the application. For example, if a machine has drift over a period of several seconds, the exemplary embodiments of the systems and methods for position refinement described herein may operate within one to several divisions before the drift exceeds a tolerance threshold.

[0146] Exemplary embodiments of the systems and methods described herein may improve costs and reduce line-of-sight limitations of current systems. Exemplary embodiments of the systems and methods described herein may also achieve higher angular and / or translational position accuracy. Exemplary embodiments of the systems and methods described herein may achieve higher position accuracy in faster times (such as within one second or less) compared to conventional robot position correction systems. Exemplary embodiments described herein may include a seven-degree-of-freedom (7DoF) global measurement system that can simultaneously monitor one or several components, objects, robots, etc. and correct their paths in real time.

[0147] Exemplary embodiments of the systems and methods provided herein may include real-time sensor data fusion for predictive machine path correction.

[0148] The exemplary embodiments described herein may use any combination of sensors to integrate multiple sensor data to create a global metrology system. Exemplary sensors may include any combination of inertial measurement units (IMUs), laser trackers, laser scanning devices, cameras, ranging systems, probing sensors, accelerometers, robotic encoders, etc.

[0149] Exemplary embodiments of the systems and methods described herein may include the incorporation of real-time device-independent metrology data for machine-independent predictive path correction. For example, the device-independent metrology data may be based on different sensors that may be used within the system. The device-independent metrology data may be because the systems and methods are configured to filter the data, extrapolate additional data points, transform the data, change the coordinate system of the data, change the data rate, synchronize the data, etc.

[0150] Exemplary embodiments of the systems and methods described herein may be used to provide a digital representation of a real-world environment, which may be used, for example, for remote viewing.

[0151] Exemplary embodiments of the systems and methods described herein may be used to determine actual object positions of one or more objects in an environment for collision avoidance. The exemplary embodiments described herein may use forward path prediction as described herein to provide control commands to one or more objects in the environment to slow down and / or start, and / or start and / or accelerate, and change direction to avoid a collision.

[0152] Exemplary embodiments of the systems and methods described herein may be used to determine actual object positions over time and / or to predict actual object positions into the future. Exemplary embodiments of such prediction may be used for obstacle avoidance, position compensation, vibration cancellation, etc.

[0153] Exemplary embodiments of the systems and methods described herein may include rolling calibration of a machine by post-processing metrology data and feeding it into a database for path corrections based on the machine's prior path of travel. For example, a robot or machine may travel various paths over time, and the system or method may include generating a database of corrections corresponding to a given path of travel and / or time. Exemplary embodiments may include lookup tables, databases, or corrections based on machine learning and predictions based on a database of path information and recorded offsets from actual positions. Exemplary embodiments may include providing predictive path corrections based on prior paths performed by the machine. Exemplary embodiments may include predictive updates based on a calibration model, such as when building a lookup table as the machine moves.

[0154] Exemplary embodiments of the systems and methods described herein may include a platform infrastructure for receiving joint values into the machine control layer, such as through custom controllers, to reduce or eliminate the use of original equipment manufacturer (OEM) kinematic modeling software, such as Kuka RSI, Fanuc DPM, and ABB EGM. Exemplary embodiments of the systems described herein may use encryptors and / or interpreters to provide this functionality in code. Exemplary embodiments described herein may therefore use OEM controllers, as this may be independent of the software used by such controllers. Alternatively, or in addition, a custom controller may be used to control the object under observation or may be directly integrated into the object to be controlled.

[0155] Exemplary embodiments of the systems and methods described herein include filtering the data stream using Kalman filtering in a PLC for real-time processing. Exemplary embodiments bypass the system processor to increase real-time processing of metrology data and provide real-time corrections while minimizing lag time between measurement and correction.

[0156] Exemplary embodiments comprise systems and methods that are communication protocol (industrial control language) independent, such as, for example, Profinet, Ethercat, Ethernet, Profinet RT (Real Time), etc.

[0157] Exemplary embodiments of the systems described herein can be based on software and / or hardware. While several specific embodiments of the invention are shown, the invention is not limited to these embodiments. For example, most functions performed by electronic hardware components may be replicated by software emulation. Thus, software programs written to perform those same functions may emulate the functionality of hardware components in input / output circuitry. The invention is not limited by the specific embodiments described herein, but is understood to be limited only by the scope of the appended claims.

[0158] As used herein, the terms "about," "substantially," or "approximately" with respect to any numerical value, range, shape, distance, relative relationship, etc., indicate suitable dimensional tolerances that enable a portion or collection of components to function for its intended purpose as described herein. Numerical ranges may also be provided herein. Unless otherwise indicated, each range is intended to include the endpoints and any quantity within the provided range. Thus, a range of 2 to 4 includes 2, 3, 4, and any subdivision between 2 and 4, such as 2.1, 2.01, and 2.001. Ranges also encompass any combination of ranges, such as 2 to 4 including 2 to 3 and 3 to 4.

[0159] Although the embodiments of the present invention have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the embodiments of the present invention as defined by the appended claims. Specifically, exemplary components are described herein. Any combination of these components may be used in any combination. For example, any component, feature, step, or portion may be integrated, separated, subdivided, removed, duplicated, added, or used in any combination while remaining within the scope of the present disclosure. The embodiments are exemplary only and provide illustrative combinations of features, but are not limited thereto.

[0160] As used in this specification and claims, the terms "comprises" and "comprising" and variations thereof mean that the specified features, steps, or integers are included. The terms are not to be interpreted as excluding the presence of other features, steps, or components.

[0161] Where appropriate, the features disclosed in the foregoing description, or the following claims, or the accompanying drawings, whether expressed in their specific form, or in terms of means for performing a disclosed function, or a method or process for achieving a disclosed result, may be utilized separately or in any combination of such features to realize the invention in various of its forms.

[0162] Also disclosed herein are the following numbered appendices: 1. A system comprising: one or more sensors each configured to generate a data stream and to produce one or more data streams; When executed by one or more processors, aggregating the data streams into a single integrated data stream to provide an actual position data stream for the object under observation; receiving an object position data stream from an object under observation; comparing the object position data streams to a single integrated data stream and determining a kinematic offset between the actual position of the object under observation and the programmed position of the object under observation; one or more processors in communication with one or more memories having machine-readable instructions stored thereon, configured to A system comprising: 2. The system of Appendix 1, wherein the one or more processors are in communication with one or more memories having machine-readable instructions stored thereon that, when executed by the one or more processors, are further configured to: filter the data streams to integrate the data streams; extrapolate information from one or more data streams before integrating the data streams; synchronize the data streams to integrate the data streams; determine an expected offset based on computational and machine-implemented latency; take the offset into account and provide control instructions to the object under observation to reposition the object under observation from its actual position to its intended position; or any combination thereof. 3. The system of claim 1 or 2, wherein the one or more processors are in communication with one or more memories having machine-readable instructions stored thereon that, when executed by the one or more processors, are further configured to provide a kinematic feedback loop to the object under observation, correct an actual position of the object under observation, and provide positional precision and accuracy of the object under observation in real time. 4. A method comprising: providing one or more sensors configured to observe an object under observation; generating a data stream from each of the one or more sensors to generate one or more data streams; aggregating the one or more data streams into a single aggregated data stream to provide an actual position data stream for the object under observation; receiving an object position data stream from an object under observation; comparing the object position data streams to a single integrated data stream to determine a kinematic offset between the actual position of the object under observation and the programmed position of the object under observation; A method comprising: 5. The method of claim 4, further comprising the step of providing a kinematic feedback loop to the object under observation to correct the actual position of the object under observation to be the programmed position of the object under observation, and providing positional precision and accuracy of the object under observation. 6. The method of claim 4 or 5, further comprising: synchronizing the object position data streams into a single integrated data stream; determining an expected offset by projecting the object position data streams and / or the single integrated data stream and / or comparison into the future; using the expected offset in a kinematic feedback loop to correct the actual position of the object under observation so that the expected offset aligns in time with the implementation of control commands to the object under observation using the expected offset; extrapolating data points in at least one of the one or more data streams; or filtering one or more data streams; or a combination thereof. 7. The method of claim 6, wherein the filtering step includes using Kalman filtering to produce a single integrated data stream. 8. The method of claim 6 or 7, wherein the generated one or more data streams relate to the position of an object under observation. 9. The method of any of appendixes 6-8, wherein another sensor is positioned above at least one of the one or more sensors configured to observe the object under observation. 10. The method of any of Appendices 6-9, wherein the receiving, processing, comparing, and feedback loop is provided in real time to provide updated kinematic corrections to the object under observation and update the object position during use. 11. A method according to any of the preceding appendices, wherein the single integrated data stream and / or object position data streams are extrapolated, filtered, synchronized, or some combination thereof, before being compared to provide additional data points for comparison. 12. A method according to any preceding clause, wherein the data stream is provided directly into one or more high speed PLC processors for processing and comparison with the object position data stream. 13. The method of any preceding clause, wherein the one or more sensors comprise any combination of an inertial measurement unit (IMU), a laser tracker, a laser scanning device, a camera, a ranging system, a probing sensor, an accelerometer, and a robotic encoder. 14. The method of any of the preceding clauses, further comprising using the forward path prediction to provide control commands to one or more objects in the environment for position correction based on the comparison. 15. The method of any of the preceding clauses, further comprising rolling calibration of the machine by post-processing the comparison and feeding it into a database for course correction based on the object's previous travel path.

Claims

1. 1. A system comprising: one or more sensors, each of the one or more sensors generating a data stream and configured to produce one or more data streams; One or more processors in communication with one or more memories, the one or more memories having machine-readable instructions stored thereon, the one or more processors, when executed by the one or more processors: aggregating said data streams into a single integrated data stream to provide an actual position data stream for the object under observation; receiving an object position data stream from an object under observation; comparing said object position data streams with said single integrated data stream to determine a kinematic offset between the actual position of said object under observation and the programmed position of said object under observation; one or more processors configured to perform A system comprising:

2. 10. The system of claim 1, wherein the one or more processors are in communication with one or more memories having machine-readable instructions stored thereon, the one or more processors being further configured, when executed by the one or more processors, to: filter the data streams to integrate them; extrapolate information from one or more data streams before integrating them; synchronize the data streams to integrate them; determine an expected offset based on computational and machine-implemented latency; provide control instructions to the object under observation to account for the offset and reposition the object under observation from its actual position to its intended position; or any combination thereof.

3. 10. The system of claim 1, wherein the one or more processors are in communication with one or more memories having machine-readable instructions stored thereon, and wherein the one or more processors are further configured to, when the machine-readable instructions are executed by the one or more processors, provide a kinematic feedback loop to the object under observation to correct an actual position of the object under observation and provide positional precision and accuracy of the object under observation in real time.

4. 1. A method comprising: providing one or more sensors configured to observe an object under observation; generating a data stream from each of the one or more sensors to generate one or more data streams; aggregating the one or more data streams into a single aggregated data stream to provide an actual position data stream for the object under observation; receiving an object position data stream from the object under observation; comparing said object position data streams with said single integrated data stream to determine a kinematic offset between the actual position of said object under observation and the programmed position of said object under observation; A method comprising:

5. 5. The method of claim 4, further comprising providing a kinematic feedback loop to the object under observation to correct the actual position of the object under observation to a programmed position of the object under observation, providing positional precision and accuracy of the object under observation.

6. 5. The method of claim 4, further comprising: synchronizing the object position data streams into the single integrated data stream; determining an expected offset by projecting the object position data streams and / or the single integrated data stream and / or the comparison into the future; using the expected offset in the kinematic feedback loop to correct the actual position of the object under observation so that the expected offset aligns in time with implementation of a control command to the object under observation using the expected offset; extrapolating data points in at least one of the one or more data streams; or filtering the one or more data streams; or a combination thereof.

7. The method of claim 6 , wherein the filtering includes using Kalman filtering to result in the single integrated data stream.

8. The method of claim 6 , wherein the generated one or more data streams relate to the position of the object under observation.

9. The method of claim 6 , wherein another sensor is positioned above at least one of the one or more sensors configured to observe the object under observation.

10. 7. The method of claim 6, wherein the receiving, processing, comparing, and feedback loop is provided in real time to provide updated kinematic corrections to the object under observation and update object position during use.

11. 5. The method of claim 4, wherein the single integrated data stream and / or the object position data streams are extrapolated, filtered, synchronized, or some combination thereof, before being compared to provide additional data points for comparison.

12. 5. The method of claim 4, wherein the data stream is provided directly into one or more high speed PLC processors for processing and comparison with the object position data stream.

13. 5. The method of claim 4, wherein the one or more sensors comprise any combination of an inertial measurement unit (IMU), a laser tracker, a laser scanning device, a camera, a ranging system, a probing sensor, an accelerometer, and a robotic encoder.

14. The method of claim 4 , further comprising using a forward path prediction based on the comparison to provide control commands to one or more objects in the environment for position correction.

15. 5. The method of claim 4, further comprising rolling calibration of the machine by post-processing the comparison and feeding it into a database for course correction based on a previous path of travel of the object.

Images