Method and system for improved automatic calibration of a robotic cell

Abstract

A robotic cell calibration method includes a robotic cell system having elements, the elements including one or more cameras, one or more sensors, a component, and a robotic arm. The method includes identifying the positions of the one or more cameras and the component relative to the position of the robotic arm using a common coordinate frame, moving the robotic arm in a movement pattern, and determining the position of the robotic arm at multiple points during the movement using the cameras and sensors. The method includes identifying discrepancies in the robotic arm position between predicted and determined positions in real time, and calculating compensations for the identified discrepancies by an auto-calibrator, which analyzes the elements in the robotic cell system as a system. The method includes modifying the behavior of the robotic arm in real time during the movement based on the compensations.

Description

Technical Field

[0001] (Related Application) This application claims priority to U.S. Provisional Patent Application No. 63 / 198,757, filed Nov. 10, 2020, the entire disclosure of which is incorporated herein by reference.

[0002] The present invention relates to calibration, and more particularly to the automatic calibration of robot cells.

Background Art

[0003] Robot assembly is becoming common in various types of devices that are automatically assembled. However, setting up and maintaining such robot assembly systems still requires calibration by experts.

[0004] Robot assembly units generally include a significant number of components, each of which must be calibrated. Each of these elements generally has a separate calibration routine that must be performed. Therefore, these calibrations must be adjusted in order to calibrate everything.

[0005] This is a slow and high-precision process, i.e., a process that takes time and requires expertise.

Brief Description of the Drawings

[0006] The present invention is shown by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals refer to like elements. [Figure 1A] A schematic block diagram of a system for a robot factory. [Figure 1B] A diagram showing one embodiment of a microfactory including a plurality of robot cells. [Figure 1C] A diagram showing one embodiment of a robot cell. [Figure 1D] A diagram showing one embodiment of a part of a work space including a plurality of fiducials. [Figure 1E] This diagram shows one embodiment of an element that defines the overall layout of a workspace. [Figure 2] This is a block diagram showing one embodiment of a robotic cell and an automatic calibration system. [Figure 3] This is a schematic flowchart illustrating one embodiment of automatic calibration. [Figure 4] This is an overview flowchart illustrating one embodiment of a holistic representational approach to calibration and use. [Figure 5] This is a schematic flowchart illustrating one embodiment of recalibration during use. [Figure 6] This is a flowchart showing one embodiment of the initial calibration process. [Figure 7] This is a flowchart showing one embodiment of reprocessing and calibration during use. [Figure 8] This is a flowchart illustrating one embodiment of setting up a robot cell using sensors and calibration. [Figure 9] This is a schematic flowchart illustrating one embodiment of setting up a workspace. [Figure 10] This is a flowchart showing one embodiment of error minimization. [Figure 11] This is a process flow illustrating one embodiment of the setup and use of a robotic cell. [Figure 12] This is a block diagram showing one embodiment of a computer system that may be used in conjunction with this application. [Modes for carrying out the invention]

[0007] This document describes the automatic calibration of robotic cells used in manufacturing or assembly. For the purposes of this description, a “robotic cell” refers to any computer-controlled assembly or manufacturing machine. For example, a robotic cell may be a modular robotic system for assembly, a computer numerical control (CNC) machine, a 3D printer, an automated optical inspection (AOI) system, a solder paste inspection (SPI) machine, an automated X-ray inspection (AXI) machine, or any other device that includes at least one camera and is computer-controlled. This system can be used to replace tasks previously performed manually by integrators, such as assembling parts into the robotic cell and monitoring the robotic cell during manufacturing.

[0008] By constructing a holistic or system view of the robot cell and its work area, an automated calibration system can provide high accuracy and precision, iterative feedback loops, refinement of constant postures, and refinement of cameras. Furthermore, in one embodiment, the automated calibration system can achieve this using off-the-shelf components rather than a customized assembly. In one embodiment, multiple low-cost cameras can be used in combination with geometric fiducial constraints to provide high accuracy with reduced cost and complexity. Automated calibration may also be used to encompass the entire robot cell environment, including all elements within the system, and is also called a holistic view, system view, or analysis as a system. For example, in a modular robot assembly system, this may include one or more cameras and / or other sensors, a robot arm, arm end tools, trays on which parts and additional tools are placed, a conveyor for moving workpieces in and out of the robot assembly system, and individual workpieces. In one embodiment, the overall view of the work area includes all elements, whether stationary or movable, and tracks and / or calibrates all elements.

[0009] In one embodiment, automatic calibration is enabled using a combination of camera lens calibration, frame alignment, arm end tool contact calibration, robot attitude error compensation, and calibration monitoring, with feedforward control and continuous correction to maintain calibration over time. Camera lens calibration calibrates the distortion and errors of the camera's sensors and lenses, and this calibration is used for the calibration of other elements. Frame alignment establishes the relative position and orientation of everything in the workspace, either to each other or to a shared coordinate system. Arm end tool contact calibration in one embodiment is referred to as tool center point (TCP) calibration, which calibrates the position of the tip of the tool in question, the line of the tool in question, or the tool contact plane that is the interface between the tool and the product or workpiece. Arm end tool contact calibration may have position and orientation in space. For a given component, the combination of all positions and orientations in question is referred to as attitude. The contact point may be a dynamic fluid or plasma, as in soldering or welding. Compensation for robot posture errors provides adjustments to compensate for the effects of accumulated manufacturing inaccuracies, wear, and other causes of inaccuracies in the robot arm components or other components. Robot posture errors can be caused by various factors, such as mechanical hysteresis, backlash, thermal expansion, and load. In one embodiment, calibration monitoring provides continuous monitoring of the calibration state of the robot cell system and constant correction. Thus, the system calibrates the dynamic and static components within the robot cell and models the dynamic and static components using a learning process.

[0010] In one embodiment, the robot cell structure includes a frame, a robot arm, and a work area defined by a space in which the robot arm and the part to be worked on reside. The frame is often one of the most rigid parts of the robot cell and can be used as the camera position and the origin or root. In one embodiment, the system is designed to create a holistic representation of the work area, including the elements of the robot cell itself and the product or workpiece within the structure.

[0011] In one embodiment, one or more fiducials observable from various viewpoints within the work area are attached to the robot cell structure. In one embodiment, the fiducials are distributed throughout the work area (or volume distribution), and each camera is configured to always see at least one fiducial attached to the work area, in addition to any fiducials on the robot arm and / or calibration board or other assembled elements held by the robot arm. However, since the robot cell may be physically moved, or someone may bump into the cell, or otherwise move the robot cell structure, such "rigid" parts may not be truly rigid. In addition, wear and tear can cause movement or displacement. In one embodiment, the system can identify such displacements, and if such deviations occur, the robot cell structure can be recalibrated.

[0012] In one embodiment, instead of, or in addition to, fiducials, the system may utilize innate features within the robot cell. These innate features may be any visually observable natural or artificial elements visible within the working area of ​​the robot cell, such as surface holes, product edges, corners, shapes found in marble, wood grain, paint spots, or printed patterns.

[0013] In addition to fiducials or identified innate features on the frame structure, the system may include several other fiducials or identified innate features, including those at the ends of the arm. In one embodiment, calibration may also be performed using one or more calibration boards or pieces moved by the robot arm. In one embodiment, the fiducial is a geometric fiducial constraint. In one embodiment, the robot arm includes several fiducials / patterns for calibration. The fiducials may be located on each link, on the tool, relative to the contact point of the arm end tool, or near each joint. In one embodiment, the fiducial is used by identifying its center point and using the center point for calibration and measurement. In another embodiment, the fiducial may be used by using another feature of the fiducial. In general, any consistently identified point can be used whether it is directly identified or identified in relation to another point (which may be on the fiducial or identified as an innate feature).

[0014] In one embodiment, the robot cell system also includes multiple cameras. The robot cell includes at least one camera. In one embodiment, the camera is rigidly mounted on the most rigid part of the frame, e.g., the top, side, and other locations. In another embodiment, the camera may be mounted in other locations. In one embodiment, the camera may be mounted on the end of an arm. In one embodiment, the field of view of each camera can observe the smallest set of fiducials within the spatial layout. In one embodiment, all positions and locations within the work area are observable by at least two cameras. In one embodiment, the cameras are calibrated to determine a distortion model. In one embodiment, the system utilizes a pair of cameras to provide an overall view of the workspace. In one embodiment, data from the cameras is integrated to provide a stereoscopic view or a multi-camera view to provide a 3D space view rather than individual 2D images. In another embodiment, a monocular camera may be used for the workspace.

[0015] In one embodiment, in addition to the camera, additional sensors, such as magnetic sensors, infrared (IR) sensors, and other sensors, may be mounted inside or outside the robot cell to monitor the work area. Other sensors that may be part of the robot cell include one or more of sound sensors, vibration sensors, motor torque sensors, force sensors (load cells), motion sensors for sensing velocity and / or acceleration, light sensors, and temperature sensors. In one embodiment, the system integrates all of this sensor data to accurately represent the environment as a whole.

[0016] In one embodiment, the system sets a reference system of the origin (0, 0, 0) of the cell, and all positions are based on this. In one embodiment, the origin is set with respect to the most rigid part of the cell. This origin is the reference point of the cell. In one embodiment, it is established using fiducials on the most rigid part of the frame and one or more cameras attached to the most rigid part of the frame. In one embodiment, additional cameras may be attached to other parts of the robot cell. In one embodiment, the system uses overview and / or internal measurement techniques to obtain the exact coordinate position of the center of the fiducial. In another embodiment, the system may utilize natural or artificial visual features or other points of the fiducial. The coordinate position of the fiducial is, in one embodiment, relative to the origin frame, also called the virtual origin frame (0, 0, 0).

[0017] The system uses the virtual origin frame to identify the position of the camera in the space relative to the robot cell. In one embodiment, the process then creates a map that includes a curve of accuracy versus precision. In one embodiment, the map is continuously updated.

[0018] In one embodiment, while the system observes the cell using the identified cameras and sensors at their positions, the robot arm moves at different speeds, along different paths, and under different loads to execute different tasks and create a model of the operation of the cell under different environmental conditions. In one embodiment, other factors such as the weight distribution, temperature, and other factors of the tool or substrate held by the robot arm may also be varied to more fully model the operation of the cell.

[0019] In one embodiment, the system provides a system-level or overall view of a robot cell, including the work area and robot arm. In one embodiment, this is done in part by using a virtual origin frame as a constant reference point on the cell's frame for all calculations and positioning. In one embodiment, the system constructs an accuracy map and uses accuracy metrics to pinpoint the positions of elements throughout the robot cell. By using metrics from calibration to map accuracy and continuously updating this mapping, the system can determine whether the accuracy map for an area of ​​the cell changes over time. These types of changes can then be taken into account when calculating the movement and position of elements within the cell.

[0020] In one embodiment, a calibration process is used, which automatically finds the fiducials within the camera's field of view and determines the orientation and identity of all elements within the robot cell's working area. "Orientation" includes both the position (x, y, z) and the orientation around that position (pitch, yaw, roll). Thus, the camera is positioned relative to a virtual origin frame.

[0021] In one embodiment, the process creates a camera-observable path for a robotic arm. The process can select the path using an error minimization framework to perform the best possible calculation of the location of visible objects within the cell's observable space. In other words, a path is selected that facilitates the movement of one or more of various movable components (robot, tools, trays, parts, and pallets, as well as / or robot-mounted cameras) and allows for the acquisition of data (camera images or data from other sensors).

[0022] The ecosystem has various potential sources of error, such as lens distortion in each camera, backlash in the robot's motion, hysteresis, and thermal expansion. The automatic calibration system takes into account the contributions from all potential sources of error and determines these errors at the system level, as a whole ecosystem. For example, if a tool is not in the expected location, the error could be due to lens distortion, backlash in the robot's motion, or both. To determine how much of the error is caused by the lens and backlash, the system can utilize data from other sensors in the system. However, those other sensors may have their own errors. Thus, analyzing the cause of one error depends on other sensors with their own errors. This is a system of simultaneous equations and constraints.

[0023] One constraint may be that the system can assume the fiducial lies on a flat surface, which allows for verification of whether the lens is distorted. If the lens is distorted, the fiducial will appear to be on a non-flat surface. Thus, the system can compensate for lens distortion. The algorithm can analyze and refine (iteratively and / or directly) a composite system with a series of multiple sensor observations and constraints as a global ecosystem. One way to describe this is "global error minimization and optimization by holistic mathematical analysis of observations and constraints across multiple objects and features of interest." This framework can be used for the calibration of camera, robot, and arm end tool contacts. In one embodiment, this framework is used for optimization by minimizing pose errors and reprojection errors.

[0024] The system utilizes models of each object, such as constraints, weight, and properties, to calculate probabilities and apply geometric and structural analysis. A set of correction parameters is applied to each sensor, and the system models the mechanical behavior of each tool / component / object / feature so that the corrected observations are rigorous and self-consistent with each other and with reality.

[0025] In one embodiment, this framework can use constraints on fiducials such as position (with precision maps), orientation, static / dynamic fiducials (fiducials on the frame versus fiducials on the robot arm), and planarity.

[0026] In one embodiment, the system performs small calibrations sequentially. In one embodiment, this is done during the processor's idle cycle, with data captured by cameras and / or sensors. In one embodiment, a watchdog process monitors the small calibrations to identify when a full recalibration is needed. In one embodiment, the data is used over time in a machine learning system to fine-tune when calibration is needed and to identify what changes or events necessitate recalibration. This can be used to improve the robotic cell itself or to address problems that cause the cell to fall out of calibration. In one embodiment, a user interface provides means for displaying calibration outputs and statistics. Over time, this can lead to improvements in cell design because problems that destabilize the cell or cause it to require recalibration are identified and eliminated.

[0027] In one embodiment, calibration data is stored and used in the steady-state operation of the robot.

[0028] In one embodiment, once a robot cell is calibrated, it operates using a state-dependent machine learning feedforward model.

[0029] The following detailed description of embodiments of the present invention refers to the accompanying drawings, where similar reference numerals indicate similar elements and illustrate specific embodiments for carrying out the invention. The descriptions of these embodiments are sufficiently detailed to enable those skilled in the art to carry out the invention. Those skilled in the art will understand that other embodiments may be utilized, and that logical, mechanical, electrical, functional, and other modifications may be made without departing from the scope of the invention. Accordingly, the following detailed description should not be construed as restrictive, and the scope of the invention is defined solely by the appended claims.

[0030] Figure 1A is a schematic block diagram of a system for a robot factory. Figure 1A is a simplified block diagram of one embodiment of a system in which robot cells may be implemented. In one embodiment, robot cell A10 comprises one or more individual robot cells, which together form a software-defined manufacturing line or micro-factory A12. In one embodiment, the individual robot cells A10 may be connected via conveyors and reverse conveyors so that a single item being manufactured or assembled through the micro-factory passes through one or more robot cells A10 (or passes through one or more cells A10 multiple times). Robot cell A10 can provide product manufacturing, assembly, inspection, and / or testing. For simplicity, the term “manufacturing” is used, but it should be understood that this term is used for any process that is part of making a product, including inspection, manufacturing, verification, and testing.

[0031] In one embodiment, the robot cell A10 is controlled by software. In one embodiment, configuration data and control data for the robot cell A10 are applied to the cell from memory A20. In one embodiment, memory A20 may be part of a remote system connected to the robot cell A10 via network A05. Configuration data A25 defines the software configuration for each robot cell A10 and each production line. Each robot cell is calibrated before use. In one embodiment, the robot cell A10 is also calibrated sequentially, as described in more detail below.

[0032] In one embodiment, the robot cell A10 collects operational data while being calibrated, tested, and used. This operational data A30 is stored in memory A20 and used by a machine learning system A35. In one embodiment, local storage A15 provides a backup of the robot cell's configuration data and the operational data generated by the robot cell during use. In one embodiment, local storage A15 functions as a buffer for memory A20. In one embodiment, even if the robot cell A10 is disconnected from network A05, the robot cell A10 can continue operating and collect real-time operational data using local storage A15. This also allows the robot cell A10 to operate offline.

[0033] In one embodiment, since the cell is configured by software, a single robotic cell A10 may perform multiple stages in the manufacturing process and may be reconfigured during the manufacturing process. In one embodiment, this also allows for the replacement of robotic cells A10 in a microfactory during manufacturing without large-scale reconfiguration. In one embodiment, this also allows for the addition of cells to a microfactory.

[0034] In one embodiment, the robot cell A10 includes a local user interface A55 that enables interaction with the robot cell A10 on the manufacturing floor. In one embodiment, the local user interface A55 may provide joystick-based interaction that enables direct control of elements of the robot cell.

[0035] In one embodiment, in addition to the local user interface (UI) A55, there may be a remote user interface A50 connected to the robot cell A10 via network A05. The remote user interface A50 may be a portable user interface such as a tablet. The remote user interface A50 may be linked to the robot cell A10 via a local area network (LAN), personal area network (PAN), or another type of network. In one embodiment, some remote user interfaces A50 may need to be located near the robot cell A10, while others may be operational from anywhere. In one embodiment, the functional and control elements presented on the user interface may vary based on one or more of the following: the robot cell A10, the configuration of the robot cell A10, the identification information / credentials of the individual logged into the user interface, and the physical proximity to the cell. In one embodiment, the local user interface A55 and the remote user interface A50 provide identical human-to-machine interface (HMI) elements, layout, and functionality, reducing complexity for the operator interacting with the robot cell A10. In one embodiment, the user interface provides a unified HMI across all robot cell types and configurations.

[0036] In one embodiment, the process for the production of the final product begins with development tool A40. In one embodiment, these tools can be made available remotely to the designer. In one embodiment, these tools may be provided online via a Software as a Service (SaaS) type interface. In one embodiment, development tool A40 enables the design of a manufacturing line including one or more robotic cells A10. In one embodiment, each robotic cell A10 has a specific capability. Development tool A10 enables the user to design a manufacturing line using one or more of the robotic cells A10 to create the final product.

[0037] In one embodiment, the CAD / generative design tool A60 can be used to create CAD drawings of the final product to be manufactured. In one embodiment, when using the CAD / generative design tool A60, the system can take into account the manufacturing / assembly constraints of the robot cell A10 when designing the final product. In one embodiment, the CAD / generative design tool A60 can receive data from the development tool A40 and iterate on the design of the final product based on the problems identified via the development tool A40. The output of the development tool A40 is a series of operations for each robot cell.

[0038] Once the design is generated, the translator A70 translates the sequence of movements into control commands for individual robot cells. In one embodiment, the output of the development tool A40 is described in a language that describes the configurations and actions performed by the robot cells. The translation is quite complex because each individual robot cell contains multiple elements that may utilize different control languages. Furthermore, different robot cells performing the same sequence of movements may have elements from different manufacturers or have different configurations. For example, a robot arm may have multiple motion joints, and the joints may have different limitations. Therefore, a single command in a series of movements may be translated differently for each individual robot cell.

[0039] The converted control commands can be applied to a virtualized robot cell A75, or a virtual representation of the robot cell system. The virtualized robot cell A75 is a software representation of an individual configured robot cell and, in one embodiment, is configured to simulate an actual robot cell, and is therefore also called a “digital twin,” and can be used for testing and verification. In one embodiment, the virtualized robot cell A75 may allow a user to remotely view the actions of the physical robot cell A10 using motion data A30 from the actual robot cell A10. In one embodiment, the user may use the virtualized robot cell A75 to preview the actions of the robot cell during a process, track the actual actions during a process, and / or review the actual actions after a process.

[0040] Once the output of translator A70 has been validated and verified, it is stored as configuration data A25. Configuration data A25 is applied to the physical robot cell, as described above.

[0041] In one embodiment, the machine learning system A35 is used to provide data for iterative learning and to provide improvements to the process.

[0042] In one embodiment, although the elements are shown here as individual elements, those skilled in the art will understand that the design tool A60, development tool A40, translator A70, virtualized robot cell A75, and machine learning system A35 are implemented on one or more computer systems. The computer systems may be standalone devices, servers, or cloud-based systems accessed via a network A05. In one embodiment, the elements described above may be implemented on a single server system. In one embodiment, the elements described above may be implemented on multiple unrelated computer / server systems. In one embodiment, only a single block is illustrated for an element such as the development tool A40, but the actual tools may be distributed across multiple devices.

[0043] Figure 1B shows an embodiment of a microfactory including multiple robotic cells. This figure shows three robotic cells, each containing a robotic arm to perform a specific action. As can be seen from the figure, a workpiece can flow through the three robotic cells, and each robotic cell performs some action in the process.

[0044] Figure 1C shows one embodiment of a robot cell. The robot cell includes an opening for a conveyor belt and an associated tray feeder through which workpieces can be supplied to the robot cell. A robot arm can be seen. In one embodiment, the robot arm is a 6-axis arm and has a range of motion throughout the entire workspace. The workspace in one embodiment is defined by an enclosed area where the robot arm is shown to move. The overall view of the workspace encompasses all components, workpieces, and other items within the workspace. In the case of a robot assembly system, the workspace may encompass the robot arm, tray feeder, conveyor belt, and associated workpieces that may be within the robot cell. The system is designed to automatically calibrate the elements in this 3D space to ensure that the processes performed by the robot cell are accurate, consistent, and repeatable.

[0045] Figure 1D is a diagram of one embodiment of a portion of a workspace containing multiple fiducials. In one embodiment, the system includes multiple fiducials within a robot cell. As shown herein, the fiducials in one embodiment may be AprilTag. Fiducials and / or other visual features can provide lens calibration, which can be used to locate and calibrate elements. In one embodiment, fiducials are used as part of initial automatic calibration, as well as for continuous feedback and calibration verification, as described below. Although AprilTag is illustrated, other types of fiducials may be used, including, for example, a QR code®, other patterns, or innate or artificial visual feature portions of a robot cell that can serve as reference points.

[0046] Figure 1E shows one embodiment of elements that define an overall view of the workspace of a robot cell. In one embodiment, the workspace is defined by a frame and enclosure within which a robot arm can move and perform actions on a workpiece. In one embodiment, the workspace includes two or more cameras to provide a stereoscopic or multi-camera three-dimensional view of the space. In one embodiment, there are additional cameras on the sides of the frame. In one embodiment, there are cameras at the ends of the arms. Additional cameras may also be used. Different cameras with different viewpoints offer different advantages.

[0047] In one embodiment, both upper cameras cover the entire workspace. Side cameras provide a better view of the arm end and offer additional checks regarding positioning. An on-arm camera provides a close-up image.

[0048] Camera data from multiple cameras, and optionally combinations of sensor data, provide a three-dimensional representation of the workspace. In one embodiment, this allows the system to observe the robotic arm in three dimensions and create a representation of the monitored 3D space. Having a holistic or system-level representation enables the system to construct predictions about where elements are located within the workspace and to fully understand the cellular configuration and the positions of objects within the workspace. This allows the system to react when the reality detected by the cameras deviates from the predictions constructed based on the holistic representation of the workspace. This provides a more accurate system and understanding of the robotic arm's position and movement. In contrast, prior art systems generally use monocular or multi-camera systems calibrated to only a limited number of two-dimensional work planes, flattening the data.

[0049] The use of a holistic representation of the 3D workspace is possible because automatic calibration and continuous feedback and recalibration ensure that the real-world positioning and movement of the robot arm and associated workpieces are accurately represented in the system. In one embodiment, this virtualized robot cell or virtual representation may be referred to as a "digital twin." In one embodiment, the digital twin representation may be shown as a visual reflection of the actual robot cell. In another embodiment, the digital twin is a holistic representation within the system that is not visually represented.

[0050] Figure 2 is a block diagram showing one embodiment of a robot cell and an automated calibration system. In one embodiment, the robot cell 210 includes a robot arm 215, a camera 220, and fiducials 225. In one embodiment, the robot arm 215 is a 6-axis robot arm available from any one of several manufacturers, such as FANUC®. In one embodiment, the camera 220 includes at least two cameras that monitor the workspace. In one embodiment, the fiducials 225 include a plurality of fiducials positioned within the workspace that are used to calibrate the system and perform continuous feedback regarding the calibration. In one embodiment, the robot cell 210 includes at least one camera 220 and fiducials or visual feature units (natural or artificial) to enable calibration. For example, in an AIO machine, the system includes a camera that autonomously scans the device under test to identify quality defects.

[0051] In one embodiment, the workspace may include, for example, a workpiece and other elements such as a conveyor belt 226 that moves the workpiece to and from the robot cell. Other elements may include a tray 227 on which the workpiece or other parts to be added to the workpiece are placed, and a path from a tray feeder. Other elements may include an arm end tool 228 and, optionally, a tool switch, if other tools are present. A “tool switch” refers to a special mounting interface on the robot arm that allows the arm end tool to be changed (manually or automatically via software control). This allows an automatic calibration routine to calibrate multiple tools. In one embodiment, this also allows the use of a special calibration tool for the robot, which the system calibrates and then remounts the arm end tool. Other elements in the workspace may include, for example, safety systems such as light curtains, electrical systems, lighting systems, sensors, cable management systems, and other elements located within the robot cell workspace and therefore considered part of the robot cell system.

[0052] In one embodiment, the robot cell 210 further includes one or more other sensors 230. These other sensors 230 may include a magnetic sensor 232, an infrared sensor 234, a sound sensor 236, a vibration sensor 238, a depth sensor, an ultraviolet sensor, a laser sensor, a pressure sensor, a current sensor, a temperature sensor, a polarization sensor, and other sensors. The other sensors 230 may also include a load cell sensor 240 that senses the force exerted by the robot arm. The other sensors 230 may also include a velocity and / or acceleration sensor 242 on the robot arm 215. The other sensors 230 may also include a light sensor 244, a temperature sensor 246, a torque sensor 248, and other sensors. In one embodiment, the temperature sensor 246 may be used to monitor temperature changes within the robot cell 210, because temperature changes may affect the dimensions of metal and plastic parts and may require modeling and adjustment of the calibration system.

[0053] The robot cell 210 includes additional components such as network connections, power and signal connections, a processor, user interface elements, a security system for shutting down the robot, and other components not shown for simplification.

[0054] The robot cell 210 transmits sensor data to the automatic calibrator 250 and receives control data from the automatic calibrator 250. In one embodiment, the automatic calibrator 250 may be a processor and subroutines implemented within the chassis of the robot cell. In another embodiment, the automatic calibrator 250 may be remote from the robot cell. In one embodiment, some of the systems described are remote and others are local.

[0055] In one embodiment, the automatic calibrator 250 is implemented by one or more processors 260 utilizing memory 255. The process described in one embodiment is designed for implementation on the hardware of the robot cell 210 and is software that constitutes the actual hardware. In one embodiment, a watchdog routine 265 provides continuous monitoring of the calibration after the initial calibration to ensure that the robot cell remains calibrated and triggers recalibration as needed.

[0056] In one embodiment, the calibration procedure design 270 includes a path design 272. The path design 272 designs movement patterns for the robot arm for various operations and calibrates any inaccuracies or discrepancies in movement between the predicted position in the movement pattern and the measured position of the robot arm. Robot movements are often highly repeatable but not highly precise. Since the robot arm has multiple joints and moving parts, it is useful to design an observable and accurately calibrated path. The calibration procedure design 270 further includes a positioning design 274 and a frame registry 276.

[0057] The positioning process 274 determines the position of one or more cameras within the robot cell relative to other features of the cell, such as robot arms and / or fiducials, fixtures, pallets, parts, feeders, trays, visual feature areas, points of interest, etc. In one embodiment, the system uses a virtual origin frame to determine the position of the cameras in space relative to the robot cell. In one embodiment, the process then creates a map including an accuracy-versus-precision curve. In one embodiment, the map is continuously updated. A frame registry provides the position data. In one embodiment, the system provides flexible storage and retrieval of images used to calculate the position data.

[0058] The feedback and refinement logic 280 provides lens distortion correction 282, posture refinement 284, and tool contact calibration 286.

[0059] The machine learning system 290 maintains the state logic 292 of the robot arm's current state and the holistic definition of the workspace. Task calibration 294 provides adjustments to the task (a set of actions to achieve a goal) based on the calibration. In one embodiment, a feedback loop 296 refines and maintains the holistic definition of the workspace while the robot cell is in use, utilizing continuous feedback from the robot cell 210 and the automatic calibrator 250.

[0060] The simplified elements shown are integrated into a robot cell having additional CPUs, GPUs, user interface elements, and network connectivity. Although the autocalibration package 250 is shown as separate logic, those skilled in the art will understand that the autocalibration package 250 can be integrated into a processor or a set of processors. These devices may be local to the robot cell 210 or remote. In one embodiment, some of the processes described may be utilized off-site or in the cloud, and the processed data may be returned to the robot cell 210.

[0061] Figure 3 is a schematic flowchart illustrating one embodiment of automated calibration. The process begins in block 310. In block 320, one or more robotic cells are assembled into the microfactory. As described above, the microfactory can be involved in the manufacturing process in ways such as manufacturing, assembly, inspection, verification, testing, or otherwise. The microfactory may be part of a larger system that does not utilize robotic cells, or it may provide a self-contained manufacturing system.

[0062] In block 330, the device is initialized with code. In one embodiment, the device initialization identifies each of the elements within the robot cell, including, for example, the robot arm, the tools being used, the camera, etc. The device initialization loads the basic sequence of steps that the robot cell will perform.

[0063] Block 340 identifies the calibration routine for the cell. In one embodiment, the calibration routine is based on the production line and cell configuration. For example, in the case of a pick-and-place robot cell, the calibration routine may differ from that of a screw-tightening robot cell, because the range of motion and path taken by the robot arm and workpiece are different. An automatic calibration process is performed.

[0064] In block 350, based on data from the calibration process, the system creates a holistic concept of the robot cell's workspace, including, for example, the robot arm, arm end tools, and workpieces within the workspace. This holistic concept is used to set expectations for movement and commands.

[0065] In block 355, a virtual representation of the robot cell is created based on a holistic concept. The virtual representation of the cell is designed to closely match the actual cell so that the mapped actions are performed similarly. This allows the system to build predictions about the position and movement patterns of the robot arm, as well as other elements in the workspace. The system uses these predictions to verify the cell's behavior.

[0066] In block 360, recipes are deployed and lines are executed. In one embodiment, a recipe is a sequence of commands sent to a robot cell to perform one or more actions. Recipes may be static or dynamic, simple or complex. A simple static recipe might be "move the conveyor by Y inches at time X." A complex recipe might include acquiring data from a camera and inserting a screw, provided the parts are correctly assembled. A recipe may include the entire assembly process or steps within the process. Once a recipe is executed, position and movement predictions based on a holistic representation are used to check the accuracy of the process. Therefore, if any discrepancies are identified, they can be addressed immediately.

[0067] In block 370, the system continues to monitor the line throughout the process. In one embodiment, the system utilizes virtual representations to reflect the expected positions of elements and compares these predictions to the observed reality from the camera. To enable this, the system must maintain accurate calibration. In one embodiment, the system uses idle-time recalibration to continuously recalibrate the robotic cells and update the holistic representations, and therefore the expected values, based on real-world data. The process ends in block 380.

[0068] Figure 4 is an overview flowchart illustrating one embodiment of a system-level or holistic approach to calibration and use. The process begins in block 410.

[0069] In block 420, the micro-factory line is calibrated, and the system has a holistic concept for each robot cell. The holistic concept encompasses the robot cell as a system, including all cameras, sensors, robot arms, and other components including workpieces within the robot cell's working area. This initial or offline calibration provides the initial parameters for the robot cell.

[0070] In block 430, the position and movement of the components within the cell are observed. This is done first as part of the automatic calibration, and then again when the robotic cell is used.

[0071] In block 440, the process determines whether the observed real-world situation differs from expectations based on the holistic concept of the robot cell, i.e., whether there is a discrepancy in the robot arm position between the predicted and observed positions. In one embodiment, the system compares the real-world orientation and movement patterns (e.g., approach vectors to each joint) of the robot arm, tool, and / or workpiece with expected values ​​based on the holistic 3D concept of the robot system. If the discrepancy is within an acceptable threshold, in block 450, the current step of the process is executed. In block 460, the process determines whether the routine is complete. If not, the process returns to block 430 and continues to observe the position / movement of the components. While this is shown as a flowchart in the real world, note that the monitoring is continuous, and in one embodiment, an interrupt is triggered when a difference between expectation and reality is detected.

[0072] Once the routine is complete, in one embodiment, a feedback loop is used in block 470 to provide feedback to the system. The feedback loop provides data to the machine learning system with expected values. The system may recalibrate if appropriate. In one embodiment, the system may utilize continuous micro-recalibration to keep the holistic concept fully aligned with the system. The process then terminates in block 475.

[0073] In block 460, if the system determines that the routine is not yet complete, the process returns to block 430 and continues observing the components within the cell.

[0074] In block 440, if the system determines that there is a difference between the observed situation and the prediction based on the holistic concept, i.e., if it determines that there is a discrepancy between the observed movement and the predicted position, the process proceeds to block 480. In block 480, the process determines whether the difference can be resolved automatically. The difference can be resolved automatically if the discrepancy can be resolved using additional data or by adjusting either the expected value or the real-world system. In one embodiment, if the system can resolve the problem automatically, in block 490 the system modifies the expected value or the real world to match them. The process then returns to block 430 and continues. In one embodiment, modifying the expected value includes performing a micro-automatic calibration or an online calibration. If the difference cannot be resolved, the system uses an alarm and stops the process in block 485. In one embodiment, the user may resolve the discrepancy, reset the system, or automatically calibrate the system to match the holistic concept and the expected value with the real world again. The process then terminates.

[0075] Figure 5 is a schematic flowchart of one embodiment of automatic calibration. The process begins in block 510. In block 520, camera and lens calibration is performed. In one embodiment, there is at least one camera mounted on the robot cell. In one embodiment, the camera is mounted on the reference section, which is the most rigid part of the robot cell. In one embodiment, this is the external metal frame that is least susceptible to deformation effects such as bending or changes in position relative to the rest of the robot cell. In another embodiment, the camera may be mounted on a different part of the robot cell.

[0076] Camera and lens calibration, in one embodiment, involves using an object with a known pattern that moves around a work area. In one embodiment, the pattern is known to have known visual features, and the system can therefore compare what is seen to known actual pattern characteristics. The difference is an error to “ground truth.” The pattern may be a naturally occurring pattern such as wood grain, an artificial pattern such as checkerboard, or a fiducial pattern. In another embodiment, instead of moving an object, the system can perform calibration using a stationary object and a moving camera, or using a pattern that covers most of the camera's field of view.

[0077] In one embodiment, lens calibration includes adjusting for lens distortion. In one embodiment, the camera and lens are each calibrated. In one embodiment, camera and lens calibration also includes calibrating other sensor technologies. In one embodiment, camera calibration utilizes fiducials within the spatial layout of the robot cell work area.

[0078] In block 530, the relative orientations of the camera and robot arm are estimated. In one embodiment, the relative orientations are relative to the origin. In one embodiment, establishing the relative positions of everything in the workspace relative to the origin is called frame registration. In one embodiment, everything is aligned to a virtual coordinate system. In one embodiment, all elements are registered to a single virtual coordinate system, i.e., the "origin" or "root". In one embodiment, the root is selected to be part of the most rigid part of the robot cell. In one embodiment, this is a metal frame around the workspace.

[0079] Block 535 models the robot arm error. This error represents the deviation or difference between the programmed posture of the robot arm and its actual posture.

[0080] In block 540, tool tip calibration is performed. Tool tip calibration determines the position of the tool tip (the end of the arm tool of the robot arm). Since the tool tip is the part of the robot arm that interacts with the workpiece, tool tip calibration is required to verify the current position of the robot arm and for all commands. In one embodiment, each part of the robot arm can also be calibrated.

[0081] In block 545, the other parts of the robot cell are calibrated against the root and / or against each other. The components of the robot cell may include workpieces, trays, conveyor belts, and any other elements in the workspace. Calibration in one embodiment takes into account all elements in the workspace.

[0082] In block 550, robot posture error correction is applied. Robot posture error is a shift in the robot's positioning caused by motion, load, deformation, temperature changes, or other changes in the robot arm, workspace, or environment, resulting in a discrepancy between the robot arm's actual position and its predicted position. When a command is sent to the robot arm to move to position X, Y, Z (or fully articulated orientation and posture), the actual position (or posture) the robot arm moves to will be different (e.g., X', Y', Z'), because the movement is not perfectly precise. The system compensates for this discrepancy. The kinematic engine that determines how the robot arm moves cannot be perfectly precise. However, by observing the robot arm in 3D workspace in real time, the system can calculate the deviation or inaccuracy of the movement compared to the intended destination. This can be used to adjust the commands to the arm to improve accuracy.

[0083] In block 560, the system is monitored during use. In one embodiment, a holistic representation is used for comparison with positional data observed for such monitoring. In one embodiment, the data is reprocessed during idle time to perform small-scale automatic calibration during use. In one embodiment, visual features (fiducial or innate or artificial features) are periodically extracted to provide continuous monitoring. The system takes into account "breathing," or other causes of inaccuracy, such as changes based on temperature, mechanical backlash, hysteresis, deflection, and load in the robotic arm.

[0084] In block 570, the process determines whether the calibration state is still sufficiently accurate. If so, a small correction is made in block 575, and the system returns to block 560 to continue monitoring.

[0085] If the calibration state is no longer sufficiently accurate, in block 580, the process determines whether a full recalibration is necessary. If not, the process proceeds to block 550 to perform robot attitude error compensation calculations. Conversely, if necessary, the process returns to block 520 to restart the fully automatic calibration process. In this way, the system continuously monitors the system, ensuring that it remains calibrated and continuously adjusts as needed. Note that although this and other figures are shown as flowcharts, in one embodiment these may be interrupt-driven processes. That is, the system can continuously monitor and determine when small adjustments are needed and perform those small adjustments automatically. When continuous monitoring of the system indicates a problem, the continuous monitoring can trigger a recalculation or a full recalibration.

[0086] Figure 6 is a flowchart showing one embodiment of the initial calibration process. The process begins in block 610. In block 620, the camera and robot elements within the cell are identified. In one embodiment, the system can operate with off-the-shelf components such as cameras, robot arms, trays, parts, feeders, and tools. The automated calibration process brings the components together into a holistic system, allowing various components to be integrated regardless of the manufacturer.

[0087] Block 625 identifies the calibration process for the cell. In one embodiment, the calibration process depends on the current configuration and / or case in which the robot cell is used, the positioning of the camera and other sensors, and the specific type of robot arm. For example, a robot cell such as an AOI machine may require a less complex calibration process than an assembly cell utilizing a 6-axis robot arm.

[0088] In block 630, an element having a known pattern of fiducial or visual feature parts is attached to the end of the arm. In one embodiment, the element is a checkerboard. In another embodiment, the element may be a dodecahedron, icosahedron, or other multiplanar shape having fiducials, one or more of which provide the camera with views of multiple fiducials on different planes to improve accuracy (and optionally provide an error minimization framework with additional planarity constraints). In yet another embodiment, the element attached to the end of the arm may have an undefined pattern of feature parts such as wood grain, scratches / stains / uneven finishes, which are viewed from multiple viewpoints from multiple cameras, and the visual features are matched and triangulated.

[0089] In block 635, the elements are moved around the cell, and image and sensor data are collected. In one embodiment, the movement pattern may depend on a recipe / process performed by the robotic cell. In one embodiment, the movement ensures that all cameras and sensors can reliably receive data from the elements.

[0090] In block 640, each fiducial is identified, and its orientation is identified. In one embodiment, frame / camera / timestamp association is used in block 645. In block 650, image data is processed, and in block 655, camera and lens calibration is performed. Lens calibration corrects lens distortion. By observing the movement of objects around the cell, distortion can be corrected for differences in various regions of the robot cell and differences between cameras.

[0091] In block 660, the relative orientations of the other components are calculated. These components include the robot arm and other elements within the workspace.

[0092] In block 665, the tool contact orientation is calculated for the robot arm. Registering the tool contact points is used to perform most recipes because these contact points are the parts of the robot system that generally interact with the workpiece.

[0093] In block 670, the system calculates an estimated pose error for the robot arm. In one embodiment, the error depends on the movement, orientation, load, and velocity of the robot arm. Since the arm consists of multiple joints, any final pose of the robot arm can be achieved through an infinite number of approaches. Therefore, in order for the system to reliably and accurately represent the real-world position of the robot arm in its holistic representation, it is necessary to calculate the error based on the orientation, load, and approach movement. For example, this approach can compensate for mechanical hysteresis and backlash from various approach vectors.

[0094] In block 675, the process determines whether the element is accurate enough to meet an accuracy threshold. In one embodiment, the system may refine the calibration multiple times. In one embodiment, the system may provide an initial calibration, then test whether the quality threshold is met, and refine it as necessary. The calibration, attitude estimates, and testing for whether the errors are within the parameters may be performed after any of the above steps, or may be performed here.

[0095] Once the calibration is sufficiently refined, in one embodiment, multi-parameter optimization is performed in block 680.

[0096] In block 685, in one embodiment, the system is validated with an external fiducial. In one embodiment, the validation involves validating that the calibration of each camera lens, camera pose, and tool contact point is all accurate using one or more separate fiducials or innate visual feature parts, and taking into account inaccuracies in the error model. The process then ends in block 690.

[0097] Figure 7 is a flowchart showing one embodiment of reprocessing and calibration in use. The process starts at block 710.

[0098] In block 715, the system receives sensor data and image data from the cell in use. In one embodiment, this sensor data and image data are received continuously.

[0099] In block 720, sensor data and image data are associated with each other based on frame / timestamp.

[0100] In block 725, sensor data and image data are processed when processing capacity is available. In one embodiment, this is done during the processor's idle cycle so that reprocessing does not affect the manufacturing process flow.

[0101] In block 730, camera calibration and lens calibration are performed.

[0102] In block 735, the orientation of each component within the workspace is calculated.

[0103] In block 740, the tool contact points with respect to the robot arm are calculated. As described above, the tool contact points may be the tool center point (TCP), one or more line segments or surfaces on which the tool contacts the workpiece, or any other criterion having a known relative orientation to the object's orientation.

[0104] In block 745, the error model of the robot arm is estimated.

[0105] The process in block 750 determines whether the holistic representation of the workspace, robot arm, and tool contact points is sufficiently accurate, based on real-world data compared to expectations from the holistic representation. If the representation is accurate, the process continues monitoring in block 715.

[0106] If the representation is inaccurate, the process determines in block 755 whether the calibration is out of range. If it is not out of range, the process proceeds to block 715. If it is out of range, in block 760 the data is reprocessed and new data is collected to verify the results.

[0107] If additional data indicates in block 765 that the system is out of range, the cell is stopped and a full recalibration procedure is initiated in block 770. If it is not out of range, the process returns to block 715. In one embodiment, the system may initialize a minor automatic recalibration when the calibration is within range but the accuracy is insufficient, but if it is determined that the system is out of range, a full automatic calibration process is initiated. The process then terminates in block 775.

[0108] Figure 8 is a flowchart illustrating one embodiment of setting up a robot cell using sensors and calibration. The process begins at block 810.

[0109] In block 815, fiducials are distributed within the cell. In one embodiment, fiducials are arranged on flat and inclined surfaces on the cell. In one embodiment, fiducials are also arranged on the robot arm. In one embodiment, each joint of the robot arm may have a fiducial.

[0110] Block 820 is where cameras and sensors are mounted. In one embodiment, the cameras are mounted on the most rigid part of the robot cell, such as a metal frame. However, even a metal frame can shift or deform over time. In one embodiment, at least two cameras are mounted on top of the cell to monitor the workspace. This allows data from multiple cameras to be integrated to provide a stereoscopic view of the workspace. In one embodiment, a side camera is used to ensure a view of the tool tip. A tool tip camera may be used for close-up images of the workpiece. In one embodiment, the cameras are arranged so that each camera views at least two fiducial or visual feature elements for calibration. In one embodiment, each camera views at least three fiducial or visual feature elements, with at least one of the fiducial or visual feature elements oriented on an element that is not parallel to the other elements.

[0111] In block 825, the robotic arm is programmed to perform a series of movements. In one embodiment, the series of movements can be performed with or without a load. In this context, the load is something that is grasped by the robotic arm or a tool attached to the robotic arm. In one embodiment, if the robotic arm has multiple types of tools in the recipe, the series of movements can be performed using each of the loads that will be attached to the robotic arm during the process.

[0112] In block 830, the origin (root or virtual 0,0,0 position) of the cell is established. In one embodiment, this root is selected with respect to the most rigid part of the cell. In one embodiment, the root is selected so that at least two cameras can see its position. In one embodiment, the root is defined based on fiducials. In one embodiment, the root is defined by calculating the average of the centers of four fiducials arranged in a square.

[0113] In block 835, the precise orientation of the camera and sensors relative to the origin is calculated.

[0114] In block 840, the motion of the arm is modeled and its accuracy estimated using the fiducial orientation on the arm during the motion sequence. In one embodiment, the system tracks fiducial points on the robot arm during the motion sequence to establish the accuracy of the arm and create a behavioral model of the arm. Since the system performs motion with and without load, the behavioral model takes the load into account. In one embodiment, the behavioral model also takes into account the differences in accuracy of the robot arm based on relative movement and position, as well as the load.

[0115] In block 845, the system calculates the position and orientation of elements visible within the cell's observable space (e.g., a work area observable by at least two cameras) and stores this data for calibration. In one embodiment, this includes all elements within the work space, such as conveyors and workpieces. The process then ends in block 850.

[0116] Figure 9 is a schematic flowchart illustrating one embodiment of setting up a workspace. The process begins in block 910.

[0117] In block 915, the positions of multiple cameras are identified. In one embodiment, the identification of positions determines the relative positions of the arm, the tool contact point, and other cameras with respect to a shared route.

[0118] In block 920, initial calibration is performed. In one embodiment, the process described above is used in this step.

[0119] In block 925, a single holistic representation of a calibrated system is created, including the robot, camera, sensors, tool tip, and other involved components. In one embodiment, the holistic representation includes a workpiece within a robot cell and a tray puller and conveyor belt used to load and unload the workpiece. The holistic representation provides the ability to predict the robot's movements and verify their accuracy by comparing the prediction or expected value to the real world.

[0120] In block 930, after initial calibration, the system performs a calibration refinement step during the idle cycle. This ensures that time-dependent robot attitude errors, which may result from temperature changes, motion drift, etc., are taken into account in the holistic representation, and therefore in the expected value.

[0121] In block 935, when the robot is commanded to move to positions X, Y, Z, but due to movement inaccuracies and the interactions of various joints and parts, the robot will end up moving to positions X', Y', Z', a target compensation is calculated to account for the error based on the holistic representation of the workspace described above. The compensation adjusts the command so that when the system needs to move the robot to positions X, Y, Z, it commands the robot to move to positions X'', Y'', Z'', and as a result the robot moves to positions X, Y, Z. The process ends in block 945.

[0122] Figure 10 is a flowchart of one embodiment of error minimization. The process starts at block 1010.

[0123] In block 1020, the camera's position is determined relative to the origin within space. In one embodiment, the origin is located on the frame of the robot cell and the most rigid part of the robot. In one embodiment, the camera's position is determined relative to a virtual root (0,0,0), and then its position is determined relative to the robot and frame.

[0124] In block 1030, the workspace is divided into multiple voxels. In one embodiment, the voxels may be the same size across the workspace. In another embodiment, the voxels may be different sizes based on the need for precision in different parts of the workspace, so that the system models the precision of the robot arm relative to each voxel.

[0125] In block 1040, the robotic arm moves around within the voxels in a predetermined pattern. In one embodiment, the robotic arm approaches the center of each voxel from various different approach angles and with various different weights. This allows for the modeling of robotic arm errors.

[0126] In block 1045, a model is constructed for each voxel based on the data. The model reflects the accuracy and precision of the robot arm within that voxel, based on the approach angle and weight. The accuracy of the robot arm depends on its position; for example, in one embodiment, the positioning of the robot arm may be more accurate near the center of the work area and more inaccurate towards the edge of the work area as the joint extends. This model is then stored and can be used when a robot command is received. Blocks 1050-1070 represent the use of the model.

[0127] In block 1050, desired destination coordinates for a given motion are received. The system obtains the starting coordinates, for example, the current position of the robot arm. In one embodiment, load data is also provided.

[0128] In block 1060, the system determines the correction values ​​to be applied to the motion based on the model described above. The model is based on the approach angle, weight, and end position of the robot arm for that motion.

[0129] In block 1070, the robotic arm is moved to a corrected destination and performs its motion.

[0130] The process terminates at block 1080.

[0131] Figure 11 is a simplified block diagram of a process utilizing a microfactory, which includes one or more robotic cells, to create an assembled final product. In some embodiments, the robotic cells may be inserted into a conventional manufacturing line to take over some of the lower parts of manufacturing. The process includes manufacturing layers, from recipe creation via the recipe creator C10 to fabrication / assembly via the microfactory C50. While the complete process from the initial conceptual / functional design to completed manufacturing is shown, those skilled in the art will understand that the system may implement a subset of these processes and include a subset of these functions.

[0132] Naturally, various processes are shown in flowchart form, but those skilled in the art will understand that the flowchart representation does not require each process element to be performed in a specific order. Furthermore, various aspects can be interrupt-driven rather than looping. Moreover, any process that does not depend on data from previous elements of the flowchart, or has no subsequent elements that depend on such data, can be reconfigured to be performed earlier, later, or in parallel with other processes.

[0133] Figure 12 is a block diagram showing one embodiment of a computer system that may be used in conjunction with the present invention. However, it will be apparent to those skilled in the art that other alternative systems with various system architectures may also be used.

[0134] The data processing system shown in Figure 12 includes a bus or other internal communication means 1240 for transmitting information and a processing unit 1210 connected to the bus 1240 for processing the information. The processing unit 1210 may be a central processing unit (CPU), a digital signal processor (DSP), or another type of processing unit 1210.

[0135] In one embodiment, the system further includes a random access memory (RAM) or other volatile storage device 1220 (referred to as memory) connected to the bus 1240 for storing information and instructions executed by the processor 1210. The main memory 1220 may be used to store temporary variables or other intermediate information during the execution of instructions by the processing unit 1210.

[0136] In one embodiment, the system also includes a read-only memory (ROM) 1250 and / or static storage device 1250 connected to the bus 1240 for storing static information and instructions for the processor 1210. In one embodiment, the system also includes a data storage device 1230, such as a magnetic disk or optical disk and its corresponding disk drive, flash memory, or other storage device that can store data when the system is not powered. In one embodiment, the data storage device 1230 is connected to the bus 1240 for storing information and instructions.

[0137] The system may further be connected to an output device 1270 for outputting information, such as a cathode ray tube (CRT) or liquid crystal display (LCD), which is connected to bus 1240 via bus 1260. The output device 1270 may be a visual output device, an audio output device, and / or a tactile output device (e.g., vibration).

[0138] An input device 1275 may be connected to bus 1260. Input device 1275 may be an alphanumeric input device, such as a keyboard with alphanumeric and other keys, to enable the user to transmit information and command selections to processing unit 1210. Additional user input devices 1280 may be included. One such user input device 1280 may be a cursor control device 1280, such as a mouse, trackball, stylus, cursor directional keys, or touchscreen, which is connected to bus 1240 via bus 1260 and transmits directional information and command selections to processing unit 1210 and controls movement on display device 1270.

[0139] Another device that may optionally be connected to the computer system 1200 is a network device 1285 for accessing other nodes of the distributed system via a network. The communication device 1285 may include any of the following: Ethernet, Token Ring, the Internet, or some commercially available networking peripherals used to connect to a wide area network, personal area network, wireless network, or other methods for accessing other devices. The communication device 1285 may further be a null modem connection or any other mechanism that provides connectivity between the computer system 1200 and the outside world.

[0140] It should be noted that any or all of the components and related hardware of this system shown in Figure 12 may be used in various embodiments of the present invention.

[0141] Those skilled in the art will understand that a particular machine implementing the present invention can be configured in various ways according to a particular embodiment. The control logic or software implementing the present invention can be stored in the main memory 1220, the mass storage device 1230, or other storage medium accessible locally or remotely by the processor 1210.

[0142] It will be apparent to those skilled in the art that the systems, methods, and processes described herein may be implemented as software stored in main memory 1220 or read-only memory 1250 and executed by processor 1210. This control logic or software may also reside on a product including a computer-readable medium, in which computer-readable program code is embodied and readable by mass storage device 1230, causing processor 1210 to operate according to the methods and teachings herein.

[0143] The present invention may also be embodied in a handheld or portable device that includes a subset of the computer hardware components described above. For example, the handheld device may be configured to include only a bus 1240, a processor 1210, and memory 1250 and / or 1220.

[0144] A handheld device may be configured to include a set of buttons or input signaling components that the user can select from a set of available options. These may be considered as a first input device 1275 or a second input device 1280. The handheld device may also be configured to include an output device 1270, such as a liquid crystal display (LCD) or a display element matrix, for displaying information to the user of the handheld device. Conventional methods may be used to implement such a handheld device. Implementation methods of the present invention for such a device will be obvious to those skilled in the art, given the disclosures of the present invention as provided herein.

[0145] The present invention may also be embodied in special-purpose equipment that includes a subset of the computer hardware components described above, such as a robotic cell. For example, such equipment may include a processing unit 1210, a data storage device 1230, a bus 1240, and a memory 1220, and may not include an input / output mechanism, or may include only a rudimentary communication mechanism, such as a small touchscreen, that allows a user to communicate with the device in a basic manner. Generally, the more specialized the purpose of the device, the fewer elements it needs to have to function. In some devices, communication with the user may occur through a touch-based screen or similar mechanism. In one embodiment, the device may not provide any direct input / output signals, but may be configured and accessed through a website or through other network-based connections via a network device 1285.

[0146] Those skilled in the art will understand that any configuration of a particular machine implemented as a computer system may be used according to its particular implementation. Control logic or software implementing the present invention may be stored on any machine-readable medium accessible locally or remotely by the processor 1210. The machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, the machine-readable medium includes read-only memory (ROM), random access memory (RAM), magnetic disk storage medium, optical storage medium, flash memory device, or other storage medium that may be used for temporary or permanent data storage. In one embodiment, the control logic may be implemented as transmittable data such as electrical, optical, acoustic, or other forms of propagating signals (e.g., carrier waves, infrared signals, digital signals, etc.).

[0147] In the aforementioned specification, the present invention is described with reference to certain exemplary embodiments. However, it will be apparent that various modifications and changes may be made without departing from the broader spirit and scope of the invention as described in the appended claims. Accordingly, this specification and the drawings should be considered illustrative rather than restrictive.

Claims

1. 1. A method for calibrating a robotic cell, comprising: providing a robotic cell system having elements including one or more cameras, one or more sensors, a component, and a robotic arm; determining a position of the one or more cameras and the component relative to a position of the robot arm within the robot cell using a common coordinate frame; moving the robotic arm through a movement pattern under a first load and determining a robotic arm position multiple times during the movement using the camera and the sensor; moving the robotic arm through the movement pattern under a second load and determining the robotic arm position multiple times during the movement using the camera and the sensor; identifying in real time a robot arm position discrepancy between a predicted position in the movement pattern and the determined robot arm position for the first load and the second load; calculating, by an auto-calibrator, compensations for the identified discrepancies, the auto-calibrator analyzing the elements in the robotic cell system as a system; modifying the motion of the robot arm in real-time during the movement based on the compensation to reduce the identified discrepancy in the robot arm position; The method includes:

2. 2. The robot cell calibration method of claim 1, further comprising: using the identified positions of the cameras, the sensors, and the robot arm to create a virtual representation of the robot cell system that tracks at least some movements of the robot cell, the virtual representation being used by the auto-calibrator.

3. determining whether the robot arm position is within a determined range of accuracy; initiating a recalibration by the auto-calibrator when the robot arm is outside the determined range of accuracy; The method of claim 1 further comprising:

4. The method of claim 1 further comprising providing a continuous calibration of the robot arm position as the robot arm moves through a motion pattern.

5. selecting a robot arm movement pattern to improve localization of a visible component within a viewable volume of the robot cell; performing the selected robot arm movement pattern while an object including one or more fiducials is attached to the robot arm; The method of claim 1 further comprising performing a pre-use calibration comprising:

6. The method of claim 1 , further comprising calibrating the one or more cameras, the calibrating taking into account camera lens distortion.

7. constructing a position accuracy map using the robot arm position metrics; adjusting the movement pattern based on the position accuracy map; and The method of claim 1 further comprising performing a pre-calibration including:

8. The method of claim 1 , wherein the calculating step includes modeling dynamic and static components within the robotic cell system.

9. 2. The robotic cell calibration method of claim 1, further comprising: performing an idle time calibration by performing a calibration refinement step on idle cycles of a processor while the robotic cell is in use, the idle time calibration determining whether the robotic cell remains calibrated.

10. The method of claim 1 , further comprising an initial calibration when the robotic cell is not in use.

11. 11. The method of claim 10, including an in-use calibration applied while the robotic cell is in use to update the calibration to account for changes in the robotic cell during use.

12. A robot cell system one or more cameras; one or more sensors; A robotic arm; 1. A processor comprising: constructing a position accuracy map using the position metrics of the robot arm; and adjusting a movement pattern based on the position accuracy map; and Perform a pre-calibration including Using a common coordinate frame to determine the position of the camera and component relative to the position of the robot arm within the robot cell; Controlling the movement of the robot arm in a movement pattern under a first load and determining a robot arm position at multiple times during the movement using the camera and the sensor; Controlling the movement of the robot arm through the movement pattern under a second load and determining a robot arm position multiple times during the movement using the camera and the sensor; identifying in real time a robot arm position discrepancy between a predicted position in the movement pattern and the determined robot arm position for the first load and the second load; calculating compensation for the identified discrepancies by an auto-calibrator by analyzing the elements in the robotic cell system as a system; a processor configured to modify a movement of the robot arm based on the compensation in real time during the movement to reduce the identified discrepancy in the robot arm position; A robot cell system comprising:

13. the processor further configured to create a virtual representation of the robotic cell system using the identified positions of the camera, the sensor, and the robot arm to track at least some movements of the robotic cell, the virtual representation being used by the auto-calibrator. The robot cell system according to claim 12.

14. 13. The robotic cell system of claim 12, further comprising the processor configured to determine if a robot arm position is within a predetermined range of accuracy and to initiate a recalibration by the auto-calibrator when the robot arm is outside of the predetermined range of accuracy.

15. The robotic cell system of claim 12 , wherein the system provides continuous calibration of the robot arm position as the robot arm moves through a motion pattern.

16. a robotic controller configured to move the robotic arm under different loads, at different speeds, and along different paths, and to identify the inconsistencies; The robotic cell system of claim 12 further comprising:

17. selecting a robot arm movement pattern to improve localization of visible components within a viewable volume of the robot cell; performing the selected robot arm movement pattern while an object including one or more fiducials is attached to the robot arm; and an auto-calibrator configured to perform a pre-use calibration including: The robot cell system according to claim 12.

18. 13. The robotic cell system of claim 12, wherein the processor is configured to perform an idle time calibration by executing a calibration refinement step on an idle cycle of a processor while the robotic cell is in use, the idle time calibration determining whether the robotic cell remains calibrated.

19. 1. A method for calibrating a robotic cell system, the robotic cell system including a robotic arm and one or more cameras, the method comprising: moving the robotic arm under a first load in a movement pattern and determining a robotic arm position multiple times during the movement pattern using a camera; moving the robotic arm through the movement pattern under a second load and determining the robotic arm position multiple times during the movement pattern using the camera; identifying in real time a robot arm position discrepancy between a predicted position in the movement pattern and the determined robot arm position for the first load and the second load; calculating, by an auto-calibrator, compensations for the identified discrepancies, the auto-calibrator analyzing the elements in the robotic cell system as a system; modifying a movement of the robotic arm during use based on the compensation to reduce the identified discrepancy in the robotic arm position; A method comprising:

20. creating one or more movement patterns such that a path traced by the movement patterns is visible to the one or more cameras; performing the movement pattern while an object including one or more fiducials is attached to the robotic arm; 20. The method of claim 19, further comprising:

Images