Automatic calibration for robot optical sensor

An automatic calibration system for robot optical sensors using a robotic arm and calibration chart enhances accuracy and reduces downtime by automating the calibration process, addressing the instability and skill-dependence of manual methods.

JP2024167264AActive Publication Date: 2024-12-03THE BOEING CO
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2024140038
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-03-05
Filing Date
2024-08-21
Publication Date
2024-12-03
Estimated Expiration
2040-02-18

AI Technical Summary

Technical Problem

Existing optical sensor calibration processes for robots are manual, unstable, and dependent on operator skill, requiring separate image acquisition in two stages, and are prone to recalibration needs due to external factors like vibration and contamination.

Method used

An automatic internal and external calibration system for robot-mounted optical sensors using a robotic arm, calibration chart, processors, and memory to determine poses, capture images, and calculate calibration parameters, reducing reliance on manual processes.

Benefits of technology

The system speeds up calibration, reduces robot downtime, and improves accuracy in computer vision applications by automating the calibration process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167264000001_ABST
    Figure 2024167264000001_ABST
Patent Text Reader

Abstract

To provide systems and methods for automatic intrinsic calibration and extrinsic calibration of a robot optical sensor.SOLUTION: An implementation includes: an optical sensor; a robot arm; a calibration chart; and one or more processors. Instructions include: determining a set of poses for calibrating the optical sensor; generating, based at least on the set of poses, pose data comprising three dimensional (3D) position data and orientation data; moving, based at least on the pose data, the robot arm into a plurality of poses; at each pose of the plurality of poses, capturing a set of images of the calibration chart with the optical sensor and recording the pose; calculating intrinsic calibration parameters, based at least on the set of captured images; and calculating extrinsic calibration parameters, based at least on the set of captured images.SELECTED DRAWING: Figure 1A
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Optical sensors such as cameras are used in several robotic applications, e.g., computer vision (CV) applications, for navigation, quality control (QC), and other functions to detect objects, such as those commonly found during material handling in manufacturing operations or warehouses. To properly interpret images collected by CV applications, optical sensors must be calibrated. This is necessary to achieve accurate position determination of objects relative to the end effector. [Background technology]

[0002] There are two main types of calibration: intrinsic calibration In general, internal calibration is used. The application takes into account, for example, the optical aberrations and inconsistencies inherent in the system. In order to achieve this, the parameters of the optical sensor lens are estimated in a laboratory environment by manually acquiring a first data set of images under ideal conditions. The external calibration parameter values ​​of the optical sensor mounting position are also typically calculated in a laboratory environment by manually acquiring a second data set of images under ideal conditions, after the internal calibration process is completed. During the use of the robot, the need to recalibrate the optical sensor often arises due to various external factors such as vibration, dirty lenses, collisions of the sensor, etc. The calibration process described above is manual and requires separate image collection in two stages. Also, such calibration, being a manual process, may not be stable and is dependent on the operator's skill. Summary of the Invention

[0003] Disclosed embodiments are described in detail below with reference to the accompanying drawings, in which: A summary is presented below to illustrate embodiments disclosed herein, but is not intended to limit all embodiments to a particular configuration or sequence of steps.

[0004] Some aspects and embodiments disclosed herein relate to automatic internal and external calibration of a robot-mounted optical sensor. One embodiment includes an optical sensor, a robot arm, a calibration chart, one or more processors, and a memory storing instructions for causing the one or more processors to perform steps including: determining a set of poses for calibrating the optical sensor, generating pose data including three-dimensional (3D) position and orientation data based on at least the set of poses, moving the robot arm to a plurality of poses based on at least the pose data, capturing a set of images of the calibration chart with the optical sensor at each of the plurality of poses, calculating internal calibration parameters based on at least the set of captured images, and calculating external calibration parameters based on at least the set of captured images.

[0005] The described features, functions, and advantages can be achieved individually in various embodiments or may be combined in yet other embodiments, further details of which will become apparent by reference to the following description and drawings. [Brief description of the drawings]

[0006] The disclosed embodiments are described in detail below with reference to the accompanying drawings, in which: Common reference numbers refer to common parts throughout.

[0007] [Figure 1A] FIG. 1 is a front view of a robot 100 according to an embodiment. [Figure 1B]FIG. 1 is a side view of a robot 100 according to one embodiment. [Diagram 2] FIG. 5 illustrates a view from a robot-based mounted optical sensor 502a according to one embodiment. [Diagram 3] FIG. 3 illustrates a calibration chart 300 according to one embodiment. [Figure 4] FIG. 2 illustrates the relationship between various coordinates according to one embodiment. [Diagram 5] FIG. 5 illustrates a process 500 for converting the output of a generic light sensor 502 into a calibrated image 508 (image correction), according to one embodiment. [Figure 6] FIG. 6 is a block diagram illustrating a process 600 for automatic internal and external calibration of a robot optical sensor 502a, according to one embodiment. [Figure 7] FIG. 7 is a block diagram illustrating a controller 700 for automatic internal and external calibration of a robot optical sensor 502a, according to one embodiment. [Figure 8] 10 is a flow chart illustrating an implementation of automatic internal and external calibration of a robot optical sensor 502a, according to one embodiment. [Figure 9] FIG. 9 is a block diagram illustrating a process 900 for automatic internal and external calibration of a robot optical sensor 502a, according to one embodiment. [Figure 10] 10 is a block diagram of a computing device 1000 suitable for implementing various aspects of the present disclosure, according to one embodiment. [Figure 11] FIG. 11 is a block diagram of a device manufacturing and maintenance method 1100 that advantageously employs automatic internal and external calibration of a robot optical sensor 502a, according to one embodiment. [Figure 12] FIG. 12 is a block diagram of an apparatus 1200 that advantageously employs automatic internal and external calibration of a robot optical sensor 502a, according to one embodiment. [Figure 13]12 is a schematic perspective view of a particular flight module 1201 according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Various embodiments will now be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts. References to specific embodiments throughout this disclosure are made for illustrative purposes only and are not intended to be limiting of all embodiments unless otherwise indicated.

[0009] The above summary, as well as the following detailed description of some embodiments, will be more clearly understood with reference to the accompanying drawings. In this specification, an element or step described in the singular and preceded by the word "a" does not necessarily exclude a plurality of elements or steps. Furthermore, reference to "one embodiment" does not exclude the existence of other embodiments that also include the described features. Furthermore, unless otherwise specified, an embodiment "comprising" or "having" an element or elements having a particular characteristic may additionally include other elements that do not have that characteristic.

[0010] Optical sensors, such as cameras, are used in several robotic applications, e.g., computer vision (CV) applications, for navigation, quality control (QC), and other functions on objects they sense, such as those commonly performed during manufacturing operations or material handling in warehouses. However, to properly interpret images collected by CV applications, the optical sensors must be calibrated so that the actual location of the object in the image is as close as possible to the estimated location. This is necessary to achieve accurate positioning of the object relative to the end effector. The embodiments described herein take these and other factors into consideration. One technical effect of the embodiments described herein is As a result, the time required for calibration of the robotic application is reduced. Additionally, one technical effect of the embodiments described herein is to improve the accuracy of the CV application. Additionally, one technical effect of the embodiments described herein is to reduce the reliance on manual calibration.

[0011] There are two main types of calibration: internal calibration and external calibration. External calibration parameters are also referred to as "external calibration values" and / or "external calibration parameter values". Internal calibration parameters are also referred to as "internal calibration values" and / or "internal calibration parameter values". Internal calibration, as used herein, generally relates to compensating for artifacts inherent to a camera system, such as inherent irregularities in a lens, e.g. optical aberrations. Internal calibration parameters are also referred to as "internal calibration values" and / or "internal calibration parameter values". The calibration parameters are intended to eliminate or take into account the effects of such irregularities. Typically, the internal calibration parameters of the optical sensor lens are calculated in a laboratory environment by manually acquiring a first set of images under ideal conditions. Each image contains a detectable fiducial of known size, such as a chessboard or a ChArUco board. Each image is then processed separately to determine the location of the fiducial keypoints. This set of measured fiducial keypoints is then matched to a set of nominal fiducial Bode points. An optimization method such as Levenberg-Marquardt optimization is then used to determine the optical parameters of the lens.

[0012] As used herein, external calibration generally refers to determining a set of values ​​that define the transformation from sensor coordinates to the robot coordinate system. Typically, this involves mapping the kinetic movement of the robot's end effector relative to the sensor (e.g., including hand / eye coordination calibration). For example, this mapping is modeled as AX=XB, where A and B are sensor measurements from different positions, and X is unknown transformation data that correlates the sensor measurements to different positions in space. Any technique can be used to derive this transformation data, as known in the art. The external calibration parameters of the optical sensor mounting positions are also typically calculated in a lab environment by manually acquiring images of the second data set under ideal conditions, after the internal calibration process is completed. Since the internal calibration has already been completed, the images have been de-distorted, and the position of each fiducial is found, for example, using Perspective-n-Point (PnP) optimization. PnP involves estimating the pose of an internally calibrated optical sensor from a set of n 3D points in a world coordinate system and their corresponding 2D projected points in an image. The pose of the optical sensor, also known as external calibration, consists of six degrees of freedom (DOF), which are the rotation (roll, pitch, and yaw) and 3D translation (e.g., X, Y, and Z) of the optical sensor relative to the world coordinate system.

[0013] During the use of the robot, various external factors (e.g. vibrations), dirty lenses, sensor inaccuracies, drift, asynchronicity, occlusions, and collisions of the optical sensors can occur. Collisions often result in the need for recalibration. Unfortunately, these processes are manual and require separate image collection in two stages.

[0014] Aspects and embodiments disclosed herein relate to automatic internal and external calibration of a robotic optical sensor. One implementation includes a system including an optical sensor, a robotic arm, a calibration chart, one or more processors, and a memory storing instructions, the instructions including: determining a set of poses for calibrating the optical sensor; generating pose data including three-dimensional (3D) position and orientation data based on at least the set of poses; and calibrating the robotic arm based on at least the pose data. The method has the one or more processors perform steps including: moving the camera to a plurality of postures; capturing a set of images of the calibration chart with an optical sensor in each of the plurality of postures; calculating internal calibration parameters based on at least the set of captured images; and calculating external calibration parameters based on at least the set of captured images.

[0015] One embodiment involves mounting a calibration end effector (an end effector with a known pattern on the calibration chart) on a robot arm. A set of suitable poses is determined that provide the camera with a perspective of the calibration board with a uniform extent. The robot arm is then automatically moved to each pose and images are taken and recorded along with the position and orientation of the calibration chart. The resulting set of images is used to calculate an internal lens calibration. The lens calibration is then used to remove distortions in each image and determine the accurate pose of the calibration chart for each image. Two sets of transformation data are constructed using the measured pose of the calibration chart and the recorded pose of the robot arm for each image. These transformation data are then used to solve the optimization equation AX=XB, which results in an external calibration of the optical sensor using the set of images. This speeds up the calibration process, resulting in reduced robot downtime and improved utilization, and is applicable to a variety of scenarios including mobile manipulator robots, their manufacturing and maintenance, as well as warehouse operations.

[0016] Referring more specifically to the figures, FIG. 1A is a front view of the robot 100, and FIG. 1B is a side view of the robot 100. The robot 100 has a robot arm 102 having a number of sections 104a-104e and a number of joints 106b-106e between the sections 104a-104e. The joint 106a connects the section 104a of the robot arm 102 to a robot body 108. As shown, the robot body 108 has an upper segment 108a that rotates on a base 108b. By manipulating the position (e.g., by pivoting the joints 106a-106e), the robot arm 102 can be moved to a variety of positions in a variety of orientations. Some embodiments of the robot arm 102 have six degrees of freedom (6DoF) with control of 3D position (e.g., X, Y, and Z translation along Cartesian axes) and control of 3D orientation (e.g., roll, pitch, and yaw). A controller 700 controls the movement of the robot 100. The controller is described below with reference to FIG. 7.

[0017] Although the light sensor housing 110 is shown attached to the front of the upper segment 108a of the robot body 108, in other embodiments the light sensor housing 110 is attached to the robot arm 102. Although the light sensor housing 110 is shown as having three openings 112a-c, in alternative embodiments other numbers of openings are used. Behind the central opening 112b is the first robot light sensor 502a. Behind the opening 112a is the second robot light sensor 502b. Behind the opening 112c is another light sensor or light emitter (e.g., LED). In one embodiment, the robot light sensors 502a and 502b are cameras that operate at visible wavelengths. In some embodiments, the robot light sensors 502a and 502b operate at wavelengths outside the visible light, such as infrared (IR). In some embodiments, the robot light sensors 502a and 502b operate at different wavelengths. The use of dual sensors allows for stereo imaging as well as comparison, for example, the output of robot optical sensor 502a can be compared to the output of robot optical sensor 502b to determine the need for recalibration. The end effector 114 is in position to hold a calibration chart 300, which is described in more detail in connection with FIG.

[0018] FIG. 2 shows a view 200 from the robot optical sensor 502a when the robot arm 102 is in a position similar to that shown in FIG. 1. The view 200 includes the robot arm 102, the robot arm's shadow 202, and the end effector 114. The end effector 114 shown in FIGS. 1B and 2 has a checkerboard pattern, but in some embodiments, a 7×8 ChArUco calibration board pattern is used, shown as calibration chart 300 in FIG. 3. The ChArUco board is a 7×8 ChArUco calibration board pattern, shown as a checkerboard pattern in FIG. 3. The ChArUco board is a combination of a checkerboard (aka chessboard) and improves the accuracy of alignment by providing a larger number of edges and shapes. For example, the ChArUco board in Figure 3 is not rotationally symmetric.

[0019] 4 shows a set of relationships between various coordinates. World coordinates 402 represent the position of an object in a global reference frame, and camera coordinates 404 represent the position of the object relative to the focal point of a camera or other sensor (e.g., robot light sensor 502a). World coordinates 402 and camera coordinates 404 are related by a 3D-to-3D transformation. Camera coordinates 404 identify 2D pixel coordinates 406, which indicate the object's location in the image, by a 3D-to-2D mapping. Extrinsic calibration parameters 408 relate world coordinates 402 to camera coordinates 404, and intrinsic calibration parameters 410 relate camera coordinates 404 to pixel coordinates 406.

[0020] FIG. 5 shows a process 500 for converting the output of a generic light sensor 502 into a calibrated image 508. Various specific embodiments of the light sensor 502 include a robot light sensor 502a and a robot light sensor 502b. The light sensor 502 outputs a raw image 504. A calibration process 506 is performed on the raw image 504 to generate a calibrated image 508 suitable for data extraction (e.g., in a CV application). In general, internal calibration is related to removing lens distortions to generate a correct looking image. Internal calibration parameters are also used to project from 2D coordinates to 3D coordinates (e.g., to determine where an object is in a room based on where it is in the image). External calibration is related to locating the sensor optical surface relative to an external frame (e.g., a robot frame or a mounting point).

[0021] 6 is a block diagram illustrating a process 600 for automatic internal and external calibration of a robotic optical sensor 502a. In step 602, desired coverage parameters are determined. In one embodiment, this is done by The calibration chart 300 is then aligned to the desired calibration accuracy. In step 604, a set of poses is determined. In one embodiment, a uniform spread of the calibration board perspective across the field of view (e.g., the image range acquired thereby) of the robotic optical sensor 502a is determined. In one embodiment, an algorithm is used to automatically determine the poses to effectively distribute the perspective of the calibration chart 300 across the data set of collected images. This eliminates the need to manually acquire the data set and results in a more evenly distributed data set that contributes to optimization.

[0022] Step 604 includes a number of steps 606-630. In step 606, a 2D item of interest within the field of view of the robot optical sensor 502a is determined. In step 608, a magnification adjustment is applied, in step 610, a translation is applied, in step 612, a skew adjustment is applied, e.g., to correct for skew in the image data, and in step 614, a rotation is applied. Step 616 then refines the pose in steps 618-628. In step 618, the 2D item of interest is converted to a 3D pose, and step 620 includes a PnP optimization. In step 622, the image is transformed to a 3D pose, and step 624 includes a PnP optimization. Reachability is determined using inverse kinematics. Inverse kinematics refers to using the kinematic equations of the robot arm 102 to determine the joint parameters that will achieve the desired position of the end effector 114. Specifying the movement of the robot arm 102 so that the end effector 114 can accomplish a desired task is known as motion planning. Inverse kinematics converts the motion plan into actuator trajectories for the joints of the robot arm 102. If a pose is determined to be reachable in a determination step 624, the pose is accepted and saved in the pose set in a step 626. Otherwise, the pose is rejected in a step 628. In one embodiment, determining reachability includes both determining whether the robot arm 102 can move to the specified position and determining whether the calibration chart 300 is within the field of view of the robot optical sensor 502a. In step 630, the sequence of poses is optimized, for example, to the sequence that minimizes the movement required by the robot arm to go through all the poses.

[0023] In step 632, calibration data is acquired. This step includes steps 634-642. Step 634 is repeated for each pose in the set of poses. Step 636 includes moving the robot arm 102 to each of a plurality of poses based on at least the pose data. Step 638 includes acquiring position data for the calibration chart 300 based on where the robot arm moved the end effector 114. In step 640, an image is acquired and step 642 includes recording the image and transformation data pair. The image data set is then available for calibration. Step 644 includes calculating the intrinsic calibration parameters 410 and step 646 includes calculating the extrinsic calibration parameters 408. In summary, the process involves calculating internal calibration parameters 410 from a set of images and known characteristics of the calibration chart 300, applying internal calibration to remove distortion from the captured images, applying marker detection to the removed images, and using an optimization program to compare the marker detection results with previously recorded positions (per step 638), thereby generating external calibration parameters 408.

[0024] More specifically, the acquired data is used to derive internal lens distortion calibration values ​​(e.g., using Levenberg-Marquardt optimization, which compares the measured calibration board reference points to a nominal set). The images are then de-distorted. The position of the end effector 114 (holding the calibration chart 300) in each image is calculated using PnP optimization. Each pose is paired with a corresponding calculated position for the robot optical sensor 502a. In one embodiment, the mounting position of the robot optical sensor 502a is determined using, for example, a dual-quaternion technique. It is obtained by solving the equation AX=XB using the attitude generation algorithm.

[0025] FIG. 7 is a block diagram illustrating a controller 700 for automatic internal and external calibration of the robot optical sensor 502a. In one embodiment, the controller 700 is in the robot body 108 of the robot 100 (see FIG. 1A). In one embodiment, the controller 700 embodies a computing device 1000, which is described in more detail with reference to FIG. 10. The controller 700 includes one or more processors 1004 coupled to a memory 1002. The controller 700 is coupled to the optical sensor 502, which represents the robot optical sensor 502a and / or the robot optical sensor 502b. The controller 700 is also coupled to a position sensor 732, which determines the position and / or position of the robot arm 102 (specifically, the sections 104a-104e, the joints 106a-106e, and the end effector 114). The controller 700 senses the position and orientation of the robot arm 102. Position data 716 is obtained in step 638 (of FIG. 6) and is acquired by a position sensor 732. The controller 700 is also connected to a robot arm servo motor 730, which moves the joints 106a-106e, thereby moving the robot arm 102.

[0026] The memory 1002 includes instructions executable by the processor 1004 and stored data. The memory 1002 includes a light sensor controller 702, which controls the robot light sensors 502a and 502b. The memory also includes a robot arm controller 704, which controls the robot arm servo motors 730 using feedback from a position sensor 732 to place the end effector 114 in a specified pose (e.g., a specified 3D position and orientation). The memory also includes previous calibration data 706 to be used for a reference or default calibration for images collected during the calibration process described herein. In one embodiment, the previous calibration data 706 is replaced with default calibration data. A calibration monitor component 708 determines a calibration trigger event according to trigger criteria 710, for example, by comparing object positions calculated from different images or by monitoring a timer. A pose generation and optimization component 712 generates pose data 714 for moving the robot arm 102 during the calibration process.

[0027] Raw images 504 are output from the optical sensor 502 at various poses and are used for calibration. The calibration also uses the image and transformation data pairs 718 recorded in step 642. A calibration process control 720 performs the internal and external calibration using internal calibration support data 722 and external calibration support data 724, which include the logic and data necessary to perform the calculations described herein. The internal calibration support data 722 also includes the data for the calibration chart 300. Calculated calibration parameters 726, which include both the external calibration parameters 408 and the internal calibration parameters 410, are also in the memory 1002. The memory 1002 also holds the calibrated images 508 and other logic and data 728 necessary to perform the steps described herein.

[0028] Referring to FIG. 8, a flow chart 800 illustrates an embodiment of a process for auto-calibration of the robot optical sensor 502a. In one embodiment, the steps illustrated in FIG. 8 are performed at least in part by executing instructions by one or more processors 1004. Step 802 includes determining a trigger event for calibrating the robot optical sensor 502a. In one embodiment, step 802 includes comparing an image captured by the robot optical sensor 502a with an image captured by the robot arm optical sensor 502b. For example, a position of an object is calculated using a CV algorithm performed on an image from the robot optical sensor 502a, and a position of the same object is calculated using a CV algorithm performed on an image from the robot optical sensor 502b. If the calculated positions differ by a difference greater than a given event threshold, this is determined as a trigger event. Also, in one embodiment, the trigger event is the expiration of a timer setting. Also, in one embodiment, the trigger event is a user input. Furthermore, in one embodiment, the trigger event is the detection of damage to the workpiece or unexpected physical resistance to a CV-controlled operation.

[0029] Step 804 selects a set of poses for calibrating the robot optical sensor 502b. 8. In one embodiment, this includes at least some of the operations described for step 604 (of FIG. 6). Step 806 includes generating pose data 714 including 3D position and orientation data based on at least the set of poses. Step 808 includes using the position and orientation data to determine whether a pose in the set of poses is reachable. In one embodiment, determining reachability includes both determining whether the robot arm 102 can move to a specified position and determining whether the calibration chart 300 is within the field of view of the robot optical sensor 502a. Step 812 includes accepting the pose based on at least the pose being reachable in determining step 810. Step 814 includes rejecting the pose based on at least the pose being not reachable in determining step 810. Step 816 then includes determining an order of poses based on at least the set of poses. In one embodiment, these poses are sequenced to minimize the total movement of the robot arm 102 so that the robot arm 102 does not move back and forth unnecessarily.

[0030] Step 818 includes moving the robot arm to a number of poses based on at least the pose data. For example, a set of poses is selected to present images of the calibration chart 300 in a preferred range of distances (from close to maximum distance with the fewest change steps), a range of orientations (from a first angle to a second angle with the fewest change steps), and a range of positions in both horizontal and vertical translations, etc., to comprehensively cover (without discernible gaps) the 6 DoF of the end effector 114. For example, at a range and translational position, the end effector 114 holding the calibration chart 300 steps through a range of orientation angles with a sufficient number of change angles within each orientation term to ensure a gap in each of roll, pitch, and yaw. The end effector 114 is then moved to another translational position and the orientation change is again stepped through. This is repeated until sufficient images cover the range of the 6 DoF that can be reached.

[0031] Step 820 includes taking a set of images of the calibration chart 300 with the robot optical sensor 502b in each of these multiple poses. In one embodiment, the calibration chart 300 is attached to the end effector 114 of the robot arm 102, and the robot optical sensor 502b is mounted to the robot body 108. Also, in one embodiment, the robot optical sensor 502b is mounted to the robot arm 102 (e.g., the end effector 114), such that the robot optical sensor 502b moves with the robot arm 102. In such an embodiment, the calibration chart 300 is placed in another location, such as, for example, a wall in a known position or the robot body 108.

[0032] Step 822 includes calculating internal calibration parameters 410 based at least on the set of captured images. Step 824 applies the internal calibration parameters 410 to the set of captured images, and step 826 includes calculating external calibration parameters 408 based at least on the set of captured images.

[0033] 9 is a block diagram illustrating a process 900 for automatic internal and external calibration of the robotic optical sensor 502a. Specifically, process 900 identifies elements within the controller 700 that, in one embodiment, perform the various steps of flowchart 800. The calibration monitor component 708 performs step 802. The pose generation and optimization component 712 performs steps 804-806. The robot arm controller 704 performs step 818, and the optical sensor controller 702 performs step 820. Finally, the calibration process controller 720 performs steps 822-826.

[0034] Referring to FIG. 10, there is shown a block diagram of a computing device 1000 suitable for implementing various aspects of the present disclosure. In some embodiments, the computing device 1000 includes one or more processors 1004, one or more presentation components 1006, and memory 1002. The disclosed embodiments relating to the computing device 1000 may be implemented by a variety of computing devices, including personal computers, laptops, smartphones, mobile tablets, handheld devices, consumer electronics, specialized computing devices, and the like. No distinction is made between categories such as "workstations," "servers," "laptops," "handheld devices," and the like, all of which are intended to be within the scope of "computing device" in FIG. 10 and herein. The disclosed embodiments may also be implemented in distributed computing environments, where tasks are performed by remote processing devices connected by a communications network. Additionally, while the computing device 1000 is depicted at first glance as a single device, in one embodiment, multiple computing devices cooperate and share resources of the illustrated device. For example, in one embodiment, memory 1002 is distributed across multiple devices and the processors 1004 depicted are housed in different devices.

[0035] In one embodiment, the memory 1002 includes any of the computer-readable media described herein. In one embodiment, the memory 1002 is used to store and access instructions configured to perform various operations described herein. In some embodiments, the memory 1002 includes computer storage media in the form of volatile and / or non-volatile memory, removable or non-removable memory, a data disk of a virtual environment, or a combination thereof. In one embodiment, the processor 1004 includes any number of processing units that read data from various entities, such as the memory 1002 or input / output (I / O) components 1010. In particular, the processor 1004 is programmed to execute computer-executable instructions to implement aspects of the present disclosure. In one embodiment, the instructions are executed by the processor, multiple processors within the computing device 1000, or a processor external to the computing device 1000. In some embodiments, the processor 1004 is programmed to execute instructions as shown in the flow charts described below and illustrated in the accompanying drawings.

[0036] The presentation component 1006 presents a display of the data to an operator or another device. In one embodiment, the presentation component 1006 includes a display device, a speaker, a printing component, a vibrating component, and the like. As one skilled in the art will appreciate, computer data can be presented in many ways, such as visually in a graphical user interface (GUI), audibly through a speaker, wirelessly connected to the computing device 1000, wired connection, or otherwise. In one embodiment, the presentation component 1006 is not used where processes and steps are sufficiently automated to reduce or eliminate the need for human intervention. The I / O ports 1008 allow the computing device 1000 to be logically connected to other devices, including input / output (I / O) components 1010, some of which are built-in. Examples of I / O components 1010 include, but are not limited to, microphones, keyboards, mice, joysticks, game pads, satellite dishes, scanners, printers, wireless devices, and the like.

[0037] Computing device 1000 includes a bus 1016 that directly or indirectly connects the following components: memory 1002, one or more processors 1004, one or more presentation components 1006, input / output (I / O) ports 1008, input / output components 1010, a power source 1012, and a network component 1014. Computing device 1000 may include any one or any combination of components illustrated herein. Neither should it be interpreted as having any dependency or requirement regarding timing. Bus 1016 represents one or more buses (e.g., an address bus, a data bus, or a combination thereof). Although the various blocks in Figure 10 are illustrated with lines for clarity, in some implementations, there is no clear distinction in functionality between the various components illustrated therein.

[0038] In some embodiments, the computing device 1000 is communicatively coupled to the network 1018 using a network component 1014. In some embodiments, the network component 1014 includes a network interface card and / or computer-executable instructions (e.g., drivers) for operating the network interface card. In one embodiment, the computing device 1000 communicates with other devices via a wired or wireless connection 1020 using any protocol or mechanism. In some embodiments, the network component 1014 communicates with a short-range communication technology (e.g., Near Field Communication (NFC), Bluetooth® communication, etc.), The network may be operable to communicate data wirelessly between devices using public, private, or hybrid (public and private) transport protocols, or a combination thereof.

[0039] Although embodiments of the present disclosure have been described in conjunction with a computing device 1000, they may be implemented in numerous other general purpose or specialized computing system environments, configurations, or devices. Examples of well-known computing systems, environments, and / or configurations suitable for use with aspects of the present disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing devices and / or communication devices in wearable or accessory form factors (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, VR devices, holographic devices, and the like. Such systems or devices receive input from a user from an input device such as a keyboard or pointing device, or in any manner, including gesture input, proximity input (such as by hovering), and / or voice input, and the like.

[0040] An embodiment of the present disclosure is described in the context of computer-executable instructions, such as program modules, executed by one or more computers or other devices in software, firmware, hardware, or a combination thereof. In one embodiment, computer-executable instructions are organized into one or more computer-executable components or modules. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. In one embodiment, aspects of the present disclosure can be implemented with any number and configuration of such components or modules. For example, aspects of the present disclosure are not limited to the specific computer-executable instructions or the specific components or modules illustrated and described herein. Other embodiments of the present disclosure include different computer-executable instructions or components having more or less functionality than illustrated and described herein. In an embodiment involving a general-purpose computer, aspects of the present disclosure transform the general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

[0041] By way of example, and not limitation, computer-readable media include computer storage media and communication media. Computer storage media includes any method or technology implemented for storage of information, such as computer-readable instructions, data structures, program modules, and the like, including volatile memory and communication media. Includes non-volatile memory, removable memory and non-removable memory. Computer storage media is tangible and mutually exclusive with communication media. Computer storage media is implemented in hardware and excludes carrier waves and propagating signals. Computer storage media for the purposes of this disclosure is not the signal itself. In one embodiment, computer storage media includes hard disks, flash drives, solid state memory, phase change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium used to store information accessed by a computing device. In contrast, communication media typically embodies computer readable instructions, data structures, program modules, etc. in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0042] Some examples of the robot 100 (of FIGS. 1A and 1B) are used in manufacturing and maintenance applications as shown and described with respect to FIGS. 11-13. Accordingly, embodiments of the present disclosure will be described with respect to the apparatus manufacturing and maintenance method 1100 shown in FIG. 11 and the apparatus 1200 shown in FIG. 12. FIG. 11 shows a diagram of an apparatus manufacturing and maintenance method according to one embodiment. In one embodiment, prior to the start of production, the apparatus manufacturing and maintenance method 1100 includes specification and design 1102 and material procurement 1104 of the apparatus 1200 of FIG. 12. During production, component and subassembly manufacturing 1106 and system integration 1108 of the apparatus 1200 of FIG. 12 occurs. The apparatus 1200 of FIG. 12 then goes through certification and delivery 1110 and enters service 1112. During its service with the customer, the device 1200 of FIG. 12 is subject to a schedule of routine maintenance and upkeep 1114, which in one embodiment includes retrofits, reconfigurations, modifications, and other maintenance and upkeep as described herein.

[0043] In one embodiment, each process of equipment manufacturing and maintenance method 1100 is performed or implemented by a system integrator, a third party, and / or an operator. In such an embodiment, the operator is a customer. Note that the system integrator may include any number of equipment manufacturers and major system subcontractors, and the third parties may include any number of vendors, subcontractors, and suppliers. In one embodiment, the operator is an owner of the equipment or equipment, a manager responsible for the equipment or equipment, a user who operates the equipment, a leasing company, a military entity, a service organization, etc.

[0044] Referring to FIG. 12, a device 1200 is shown. As shown in FIG. 12, one example of the device 1200 is a flying device 1201, such as a spacecraft, an aircraft, an air cargo, a flying car, etc. As further shown in FIG. 12, another example of the device 1200 is a ground transportation device 1202, such as a car, a truck, a heavy equipment, a construction machine, a boat, a ship, a submarine, etc. A further example of the device 1200 shown in FIG. 12 is a modular device 1203 that includes at least one of the following modules: an air module, a payload module, and a ground module. The air module provides an air lift or flight function. The payload module provides a function of transporting objects, such as cargo or living objects (people, animals, etc.). The ground module provides a ground movement function. The approaches disclosed herein are applied to each of these modules individually or by group, such as the air module and the payload module, the payload module and the ground module, or all the modules.

[0045] 13, there is shown a more detailed view of the flight module 1201 in which embodiments of the present disclosure may be advantageously employed. In this example, the flight module 1201 is the same as that shown in FIG. 13. An aircraft manufactured by the method 1100 for manufacturing and maintaining an aircraft in an aerospace industry, the aircraft including an airframe 1303 having a number of systems 1304 and an interior 1306. Embodiments of the number of systems 1304 include one or more of a propulsion system 1308, an electrical system 1310, a hydraulic system 1312, and an environmental system 1314, although other systems are possible. Additionally, while an aerospace application has been described as an example, various advantageous embodiments have application in other industries, such as the automotive industry.

[0046] The disclosed embodiments are described in the context of computer code or machine usable instructions. Such instructions include computer executable instructions, such as program components, executed by a computer or other machine, such as a personal data assistant or other handled device. Generally, program components, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The disclosed embodiments may be practiced in a variety of system configurations, including personal computers, laptops, smart phones, mobile tablets, handheld devices, consumer electronics, specialized computing devices, etc. The disclosed embodiments may also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked through a communications network.

[0047] The following notes describe further aspects of the present disclosure.

[0048] A system for auto-calibration of a robotic optical sensor, comprising: An optical sensor; A robot arm; A calibration chart; one or more processors; and a memory for storing instructions that, when executed by the one or more processors, determining a set of poses for calibrating the optical sensor; generating pose data including three-dimensional (3D) position and orientation data based on at least the set of poses; moving the robot arm to a plurality of poses based at least on the pose data; capturing a set of images of the calibration chart with the optical sensor in each of the plurality of positions; calculating internal calibration parameters based on at least the set of captured images; and calculating external calibration parameters based on at least the set of captured images.

[0049] Addendum 2. The system of Addendum 1, wherein the process further includes applying the internal calibration parameters to the set of captured images before calculating the external calibration parameters.

[0050] Addendum 3. The steps include determining whether a posture of the set of postures is reachable; 3. The system of claim 1 or 2, further comprising: rejecting the pose based at least on the pose being not reachable.

[0051] Appendix 4. The system according to any one of appendices 1 to 3, wherein the calibration chart is attached to an end effector of the robot arm, and the optical sensor is mounted on a body of the robot.

[0052] Addendum 5. The system according to any one of Addendums 1 to 3, wherein the optical sensor is attached to the robot arm.

[0053] Addendum 6. The process further includes determining a trigger event for calibrating the optical sensor; 6. The system of claim 1, wherein determining the set of postures for calibrating the optical sensor includes determining the set of postures for calibrating the optical sensor based on at least the trigger event.

[0054] Addendum 7. The optical sensor is a first optical sensor; The system further includes a second optical sensor. 7. The system of claim 6, wherein determining the trigger event includes comparing an image captured by the first optical sensor with an image captured by the second optical sensor.

[0055] A method for auto-calibration of a robot optical sensor, comprising: determining a set of poses for calibrating the optical sensor; generating pose data including three-dimensional (3D) position and orientation data based on at least the set of poses; moving the robot arm to a plurality of poses based at least on the pose data; capturing a set of images of a calibration chart with the optical sensor in each of the plurality of positions; calculating internal calibration parameters based on at least the set of captured images; and calculating extrinsic calibration parameters based on at least the set of captured images.

[0056] Addendum 9. The method of Addendum 8, further comprising applying the internal calibration parameters to the set of captured images prior to calculating the external calibration parameters.

[0057] Addendum 10. Determining whether a pose of the set of poses is reachable; and 10. The method of claim 8 or 9, further comprising rejecting the pose based at least on the pose being not reachable.

[0058] Addendum 11. The method according to any one of Addendums 8 to 10, wherein the calibration chart is attached to an end effector of the robot arm, and the optical sensor is mounted on a body of the robot.

[0059] Addendum 12. The method according to any one of Addendums 8 to 12, wherein the optical sensor is mounted on the robot arm.

[0060] Clause 13. The method according to claim 1, further comprising determining a trigger event for calibrating the optical sensor; Determining the set of poses for calibrating the optical sensor includes at least 13. The method of any one of claims 8 to 12, comprising determining the set of poses for calibrating the optical sensor based on at least the trigger event.

[0061] Addendum 14. The optical sensor is a first optical sensor; 14. The method of claim 13, wherein determining the trigger event includes comparing an image captured by the first optical sensor with an image captured by a second optical sensor.

[0062] Clause 15. A computer program product including a computer usable medium having computer readable program code embodied therein, the computer readable program code configured to be executed to implement a method for automatically calibrating a robot optical sensor, the method comprising: determining a set of poses for calibrating the optical sensor; generating pose data including three-dimensional (3D) position and orientation data based on at least the set of poses; moving the robot arm to a plurality of poses based at least on the pose data; capturing a set of images of a calibration chart with the optical sensor in each of the plurality of positions; calculating internal calibration parameters based on at least the set of captured images; and calculating extrinsic calibration parameters based on at least the set of captured images.

[0063] Addendum 16. The computer program product of Addendum 15, further comprising applying the internal calibration parameters to the set of captured images prior to calculating the external calibration parameters.

[0064] Addendum 17. Determining whether a pose of the set of poses is reachable; and 17. The computer program product of claim 15 or 16, further comprising: rejecting the pose based at least on the pose being not reachable.

[0065] Addendum 18. A computer program product according to any one of Addendums 15 to 17, wherein the calibration chart is attached to an end effector of the robot arm, and the optical sensor is mounted on a body of the robot.

[0066] Addendum 19. The computer program product according to any one of Addendums 15 to 18, wherein the optical sensor is attached to the robot arm.

[0067] Addendum 20. The optical sensor is a first optical sensor; determining a trigger event for calibrating the first light sensor; determining the trigger event includes comparing an image captured by the first optical sensor to an image captured by a second optical sensor; 20. The computer program product of any of Addendum 15-19, wherein determining the set of poses for calibrating the first optical sensor includes determining the set of poses for calibrating the first optical sensor based on at least the trigger event.

[0068] When describing aspects of the disclosure or elements of an embodiment thereof, articles such as "a," "an," "the," and "said" mean that there is one or more of the element. The terms "may," "including," "having," and the like are inclusive and mean that there may be additional elements other than the listed elements. The term "embodiment" is intended to mean "one example." Also, "one or more of the following A, B, and C" means "at least one of A and / or at least one of B and / or at least one of C."

[0069] While aspects of the present disclosure have been described in detail, it will be apparent that modifications and variations can be made without departing from the scope of the aspects of the present disclosure as defined in the appended claims. Because various modifications can be made to the structures, products, and methods described above without departing from the scope of the aspects of the present invention, all matter contained in the above description or shown in the accompanying drawings should be construed in an illustrative and not limiting sense.

Claims

1. 1. A system for auto-calibration of a robotic optical sensor, comprising: An optical sensor; A robot arm; A calibration chart; one or more processors; and a memory for storing instructions that, when executed by the one or more processors, determining a set of poses for calibrating the optical sensor; generating pose data including three-dimensional (3D) position and orientation data based on at least the set of poses; moving the robot arm to a plurality of poses based at least on the pose data; capturing a set of images of the calibration chart with the optical sensor in each of the plurality of positions; calculating internal calibration parameters based on at least the set of captured images; and calculating external calibration parameters based on at least the set of captured images.

2. The system of claim 1 , wherein the process further comprises applying the internal calibration parameters to the set of captured images prior to calculating the external calibration parameters.

3. The steps include determining whether a pose of the set is reachable; The system of claim 1 or 2, further comprising: rejecting the pose based at least on the pose being not reachable.

4. The system according to any one of claims 1 to 3, wherein the calibration chart is attached to an end effector of the robot arm, and the optical sensor is mounted on a body of the robot.

5. The system according to any one of claims 1 to 3, wherein the optical sensor is mounted on the robot arm.

6. The process further includes determining a trigger event for calibrating the light sensor; 6. The system of claim 1, wherein determining the set of poses for calibrating the optical sensor comprises determining the set of poses for calibrating the optical sensor based on at least the trigger event.

7. the optical sensor is a first optical sensor, The system further includes a second optical sensor. The system of claim 6 , wherein determining the trigger event includes comparing an image captured by the first optical sensor to an image captured by the second optical sensor.

8. 1. A method for auto-calibration of a robotic optical sensor, comprising: determining a set of poses for calibrating the optical sensor; generating pose data including three-dimensional (3D) position and orientation data based on at least the set of poses; moving the robot arm to a plurality of poses based at least on the pose data; capturing a set of images of a calibration chart with the optical sensor in each of the plurality of positions; calculating internal calibration parameters based on at least the set of captured images; and calculating extrinsic calibration parameters based on at least the set of captured images.

9. The method of claim 8 , further comprising applying the internal calibration parameters to the set of captured images prior to calculating the external calibration parameters.

10. determining whether a pose of the set is reachable; and The method of claim 8 or 9, further comprising: rejecting the pose based at least on the pose being not reachable.

11. The method according to any one of claims 8 to 10, wherein the calibration chart is attached to an end effector of the robot arm, and the optical sensor is mounted on a body of the robot.

12. The method according to any one of claims 8 to 10, wherein the optical sensor is mounted on the robot arm.

13. determining a trigger event for calibrating the light sensor; 13. The method of claim 8, wherein determining the set of poses for calibrating the light sensor comprises determining the set of poses for calibrating the light sensor based on at least the trigger event.

14. the optical sensor is a first optical sensor, The method of claim 13 , wherein determining the trigger event includes comparing an image captured by the first optical sensor to an image captured by a second optical sensor.

Citation Information

Patent Citations

  • High-precision mechanical arm hand-eye camera calibration method and system

    CN107256568A

  • Controller for robot

    JP1992326404A

  • System and method for determining camera miscalibration at runtime

    JP2013515959A

  • Apparatus and method for robust calibration between machine vision systems and robots

    JP2013526423A

  • Robot, control device, robot system and control method

    JP2015089575A