Automatic calibration of optical sensors for robots

Abstract

The present invention relates to automatic calibration of a robot optical sensor. A system and method for automatic internal and external calibration of a robot optical sensor are provided. One implementation includes: an optical sensor; a robot arm; a calibration map; one or more processors; and a memory storing instructions that cause one or more processors to perform operations including the following steps: determining a set of poses for calibrating a first optical sensor; generating pose data containing three-dimensional (3D) position and orientation data based at least on the set of poses; moving the robot arm into a plurality of poses based at least on the pose data; acquiring a set of images of the calibration map using the first optical sensor in each of the plurality of poses and recording the pose; calculating intrinsic calibration parameters based at least on the set of acquired images; and calculating extrinsic calibration parameters based at least on the set of acquired images.

Description

Technical Field

[0001] The present invention relates to automatic calibration of optical sensors for robots. Background Art

[0002] Optical sensors such as cameras are used in certain robotics applications, for example, in conjunction with computer vision (CV) applications, to perform navigation, quality control (QC), and other functions on detected objects, such as those common in manufacturing and warehouse material handling. Correct interpretation of the images collected by CV applications requires calibration of the optical sensor. This is required to achieve accurate positioning of the object relative to the end effector.

[0003] There are two main types of calibration: intrinsic and extrinsic. Typically, intrinsic calibration estimates the parameters of the optical sensor lens to, for example, account for inherent optical aberrations or inconsistencies in the system and is calculated in a laboratory environment by manually acquiring a first data set of images under ideal conditions. The extrinsic calibration parameter values ​​for the optical sensor mounting position are also typically calculated in a laboratory environment by manually acquiring a second data set of images under ideal conditions after the intrinsic calibration process has been completed. During the use of the robot, various environmental factors (such as vibrations), lens contamination, and sensor collisions often trigger the need to recalibrate the optical sensor. The above-mentioned calibration procedure is manual and requires separate image collection in two stages. As a manual process, calibration can be inconsistent and relies on the skills of the operator. Summary of the invention

[0004] The disclosed examples are described in detail below with reference to the accompanying drawings listed below. The following overview is provided to illustrate the implementation disclosed herein. However, this is not meant to limit all examples to any specific configuration or sequence of operations.

[0005] Aspects and implementations disclosed herein are directed to automatic intrinsic and extrinsic calibration of an optical sensor for robot mounting. One implementation includes: an optical sensor; a robot arm; a calibration map; one or more processors; and a memory storing instructions that cause the one or more processors to perform operations including the following steps: determining a set of poses for calibrating a first optical sensor; generating pose data including three-dimensional (3D) position and orientation data based at least on the set of poses; moving the robot arm into a plurality of poses based at least on the pose data; acquiring a set of images of the calibration map with the first optical sensor at each of the plurality of poses; calculating intrinsic calibration parameters based at least on the set of acquired images; and calculating extrinsic calibration parameters based at least on the set of acquired images.

[0006] The features, functions, and advantages that have been discussed can be achieved independently in various implementations or may be combined in yet other implementations, further details of which will be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The disclosed examples are described in detail below with reference to the following figures:

[0008] Figure 1A is a front view of the robot 100 according to an example.

[0009] Figure 1B is a side view of the robot 100 according to an example.

[0010] Figure 2 A view from a robot base mounted optical sensor 502a is illustrated according to an example.

[0011] Figure 3 A calibration graph 300 according to an example is illustrated.

[0012] Figure 4 Relationships between various coordinates according to examples are illustrated.

[0013] Figure 5 A process 500 for converting the output of a general optical sensor 502 into a calibration image 508 (image correction) is illustrated according to an example.

[0014] Figure 6 is a block diagram illustrating a process 600 for automatic intrinsic and extrinsic calibration of a robotic optical sensor 502a, according to an example.

[0015] Figure 7 is a block diagram illustrating a controller 700 for automatic intrinsic and extrinsic calibration of a robotic optical sensor 502a according to an example.

[0016] Figure 8 is a flow chart illustrating an implementation of automatic intrinsic and extrinsic calibration for a robotic optical sensor 502a according to an example.

[0017] Fig. 9 is a block diagram illustrating a process 900 for automatic intrinsic and extrinsic calibration of a robotic optical sensor 502a, according to an example.

[0018] Fig.10 is a block diagram of a computing device 1000 suitable for implementing aspects of the present disclosure, according to an example.

[0019] Fig.11is a block diagram of an apparatus that advantageously employs a manufacturing and repair method 1100 for automated internal and external calibration of a robotic optical sensor 502a, according to an example.

[0020] Fig.12 is a block diagram of an apparatus 1200 advantageously employed for automatic intrinsic and extrinsic calibration of a robotic optical sensor 502a, according to an example.

[0021] Fig.13 is a schematic perspective view of a particular flight module 1201 according to an example.

[0022] Corresponding reference characters indicate corresponding parts throughout the several views. DETAILED DESCRIPTION

[0023] Various implementations will be described in detail with reference to the accompanying drawings. Throughout the drawings, the same reference numerals will be used to refer to the same or similar components as much as possible. References made throughout this disclosure to specific embodiments and implementations are provided for illustrative purposes only and are not intended to limit all implementations unless otherwise indicated.

[0024] When read in conjunction with the accompanying drawings, the following detailed description of the aforementioned overview and certain implementations will be better understood. As used herein, elements or steps stated in singular form and beginning with the word "one" or "an" should be understood to not necessarily exclude multiple elements or steps. In addition, reference to "an implementation" is not intended to be interpreted as excluding the existence of other embodiments that also include the features. Moreover, unless explicitly stated to the contrary, an implementation that "includes" or "has" an element or multiple elements with a particular characteristic may include other elements that do not have the characteristic.

[0025] Optical sensors such as cameras are used in certain robotic applications, for example, in conjunction with computer vision (CV) applications, to perform navigation, quality control (QC), and other functions on detected objects, such as those common in manufacturing and warehouse material handling. However, correct interpretation of the collected images by the CV application requires that the optical sensor be calibrated so that the actual position of the object in the image is as close to the expected position as possible. This is required to achieve accurate positioning of the object relative to the end effector. The embodiments described herein take these and other factors into account. The technical effects of the embodiments herein include reducing the amount of time to calibrate a robotic application. The technical effects of the embodiments herein include increasing the accuracy of CV applications. The technical effects of the embodiments herein include reducing reliance on manual calibration.

[0026] There are two main types of calibration: intrinsic and external. Extrinsic calibration parameters are also referred to as "extrinsic calibration values" and / or "extrinsic calibration parameter values". Intrinsic calibration parameters are also referred to as "intrinsic calibration parameters" and / or "intrinsic calibration parameter values". As used herein, intrinsic calibration generally involves compensating for artifacts inherent in a camera system, for example, inconsistencies such as inherent optical aberrations in a lens. The intrinsic calibration parameters attempt to counteract or account for the effects of inconsistencies. Typically, the intrinsic calibration parameters of an optical sensor lens are calculated in a laboratory environment by manually acquiring a first data set of images under ideal conditions. Each image contains a benchmark of known size that can be detected, such as a chessboard or charuco board. Once the data is acquired, each image is processed separately to find the location of the benchmark key points. The set of measured benchmark key points is then correlated to the set of nominal benchmark board points. An optimization method such as Levenberg-Marquardt optimization is used to solve for the optical parameters of the lens.

[0027] The external calibration used herein generally involves determining a set of values ​​to define the transformation from the sensor coordinates to the robot coordinate system. Typically, this involves mapping the kinematic movement of the robot's end effector relative to the sensor (e.g., hand / eye coordinate calibration). For example, the mapping can be modeled as AX=XB, where A and B are sensor measurements from different positions, and X is an unknown transformation that relates the sensor measurements to different positions in space. As is known in the art, any method can be used to achieve the transformation. The external calibration parameters of the optical sensor mounting position are also typically calculated in a laboratory environment by manually acquiring a second data set of images under ideal conditions after the internal calibration process has been completed. Because the internal calibration has been completed, each image is undistorted, and the position of each benchmark can be determined using, for example, Perspective-n-Point (PnP) optimization. PnP involves estimating the pose of an internally calibrated optical sensor given a set of n 3D points in the world and their corresponding 2D projections in the image. The optical sensor pose (also called extrinsic calibration) consists of 6 degrees of freedom (DOF) consisting of the rotation (roll, pitch and yaw) and 3D translation (e.g., X, Y and Z) of the optical sensor relative to the world.

[0028] During the use of the robot, various environmental factors (e.g., vibrations), lens contamination / sensor inaccuracies, drift, asynchrony, blockages, and optical sensor collisions often trigger the need for recalibration. Unfortunately, these procedures are manual and require separate image collection in two stages.

[0029] Aspects and implementations disclosed herein are directed to automatic intrinsic and extrinsic calibration for a robotic optical sensor. One implementation includes: an optical sensor; a robotic arm; a calibration map; one or more processors; and a memory storing instructions that cause the one or more processors to perform operations including the following steps: determining a set of poses for calibrating a first optical sensor; generating pose data containing three-dimensional (3D) position and orientation data based at least on the set of poses; moving the robotic arm into a plurality of poses based at least on the pose data; acquiring a set of images of the calibration map using the first optical sensor in each of the plurality of poses; calculating intrinsic calibration parameters based at least on the set of acquired images; and calculating extrinsic calibration parameters based at least on the set of acquired images.

[0030] One implementation involves attaching a calibration end effector (an end effector with a known pattern on a calibration map) to a robot arm. A set of suitable poses is determined that provides a uniform distribution of the camera's view angle of the calibration plate. The robot arm is then automatically moved to each pose, and images are acquired and recorded along with the position and orientation of the calibration map. The resulting set of images is used to calculate the internal lens calibration. Thereafter, the lens calibration is used to undistort each image and determine the exact pose of the calibration map for each image. Using the measured pose of the calibration map and the recorded robot arm pose of each image, two sets of transformations can be constructed. These lists of transformations are then used to solve the AX=XB optimization problem, thereby solving the external calibration of the optical sensor - using a single set of images. This speeds up the calibration process, thereby reducing robot downtime and improving utilization, and is applicable to a variety of situations involving mobile manipulator robots, such as manufacturing, maintenance, and warehouse operations.

[0031] More particularly, with reference to the accompanying drawings, Figure 1A is a front view of the robot 100, and Figure 1B is a side view of the robot 100. The robot 100 has a robotic arm 102 having a plurality of segments 104a to 104e and a plurality of joints 106b to 106e between the segments 104a to 104e. The joint 106a connects the segments 104a of the robotic arm 102 to a robot body 108. As shown, the robot body 108 has a top segment 108a that rotates on a base 108b. By manipulating the position (e.g., by rotating the joints 106a to 106e), the robotic arm 102 can be moved into a plurality of positions with a plurality of orientations. Certain implementations of the robotic arm 102 have six degrees of freedom (6DoF) with 3D position (e.g., X, Y, and Z translations along Cartesian axes) control and 3D orientation control (e.g., roll, pitch, and yaw). Reference Figure 7The controller 700 , which controls the operation of the robot 100 , is described in more detail.

[0032] The optical sensor housing 110 is illustrated as being mounted to the front of the top section 108a of the robot body 108, however in another implementation, the optical sensor housing 110 is mounted on the robot arm 102. The optical sensor housing 110 is illustrated as having three holes 112a to 112c, however it should be understood that a different number of holes are used in alternative implementations. The first robot optical sensor 502a is behind the middle hole 112b. The second robot optical sensor 502b is behind the hole 112a. Another optical sensor or illuminator (e.g., LED) is behind the hole 112c. In one implementation, the robot optical sensors 502a and 502b are cameras operating at visible light wavelengths. In some implementations, the robot optical sensors 502a and 502b operate outside of visible light wavelengths (e.g., infrared (IR)). In some implementations, the robot optical sensors 502a and 502b each operate at a different wavelength. Using dual sensors not only permits stereoscopic imaging, but also permits comparison, for example, by comparing the outputs of the robot optical sensors 502a and 502b to each other to determine whether recalibration is needed. The end effector 114 is in a suitable position to hold the calibration map 300, relative to Figure 3 This calibration graph is described in more detail.

[0033] Figure 2 The example shows that when the robot arm 102 is in Figure 1B The view 200 from the robot optical sensor 502a is similar to the position shown. The view 200 includes the robot arm 102, the shadow 202 of the robot arm, and the end effector 114. Figure 1B and Figure 2 The end effector 114 shown in FIG. 1 has a checkerboard pattern, although in some implementations a checkerboard pattern is used. Figure 3 A 7×8 charuco calibration plate pattern is shown as calibration diagram 300 in FIG. The charuco plate combines the ArUco pattern with a checkerboard (also called a chessboard) and provides an additional number of edges and shapes to improve alignment accuracy. For example, Figure 3 The charuco plate is not rotationally symmetric.

[0034] Figure 4A set 400 of relationships between various coordinates is illustrated. World coordinates 402 describe the location of an object in a global reference frame, while camera coordinates 404 describe the location of an object relative to the focus of a camera or other sensor (e.g., a robotic optical sensor 502a). World coordinates 402 and camera coordinates 404 are related by a 3D to 3D transformation. Using a 3D to two-dimensional (2D) mapping, camera coordinates 404 specify 2D pixel coordinates 406 that represent the location of an object within an image. Extrinsic calibration parameters 408 relate world coordinates 402 to camera coordinates 404, while intrinsic calibration parameters 410 relate camera coordinates 404 to pixel coordinates 406.

[0035] Figure 5 A process 500 for converting the output of a general optical sensor 502 into a calibration image 508 is illustrated. Various specific implementations of the optical sensor 502 include a robotic optical sensor 502a and a robotic optical sensor 502b. The optical sensor 502 outputs a raw image 504. A calibration operation 506 operating on the raw image 504 produces a calibration image 508 suitable for data extraction (e.g., in a CV application). In general, intrinsic calibration is associated with removing lens distortion to produce a correct appearance image. Intrinsic calibration parameters are also used to project from 2D coordinates to 3D coordinates (e.g., determine the location of an object in a room based on the location of the object in the image). Extrinsic calibration is associated with positioning the sensor optical plane with reference to an external frame (e.g., a robot frame or mounting point).

[0036] Figure 6 is a block diagram illustrating a process 600 for automatic internal and external calibration of a robotic optical sensor 502a. In operation 602, desired coverage parameters are determined. In one implementation, this includes a desired calibration accuracy. In operation 604, a set of poses is determined. In one implementation, a uniform distribution of calibration plate viewing angles over the field of view (e.g., the resulting image range) of the robotic optical sensor 502a is determined. One implementation uses an algorithm to automatically determine the poses to effectively distribute the viewing angles of the calibration map 300 across a dataset of collected images. This eliminates the need to manually acquire a dataset and provides a more evenly distributed dataset, which facilitates optimization.

[0037] Operation 604 includes a plurality of operations 606 to 630. In operation 606, a 2D coverage within the field of view of the robot optical sensor 502a is determined. Scaling is applied in operation 608, translation is applied in operation 610, skewing is applied in operation 612, for example, to skew the image data, and rotation is applied in operation 614. Then, in operations 618 to 628, operation 616 improves the pose. In operation 618, the 2D coverage is converted to a 3D pose, and operation 620 includes a PnP optimization. In operation 622, reachability is determined using inverse kinematics. Inverse kinematics refers to using the kinematic equations of the robot arm 102 to determine the joint parameters that provide the desired position of the end effector 114. The movement of the robot arm 102 is specified so that the end effector 114 achieves the desired task, which is called a motion plan. Inverse kinematics transforms the motion plan into joint actuator trajectories of the robot arm 102. If the pose is reachable, as determined in decision operation 624, it is accepted into (retained in) the set of poses in operation 626; otherwise, it is rejected in operation 628. In one implementation, determining reachability includes both determining whether the robot arm 102 can be moved into a defined position and determining that the calibration map 300 is within the field of view of the robot optical sensor 502a. In operation 630, the pose sequence is optimized, for example, by minimizing the motion required for the robot arm to go through all poses.

[0038] In operation 632, calibration data is acquired, which operation 632 includes operations 634 to 642. Operation 634 iterates through each pose in the pose set. Operation 636 includes moving the robotic arm 102 into each of the plurality of poses based at least on the pose data. Operation 638 includes acquiring position data for the calibration map 300 based on the positions to which the robotic arm has moved the end effector 114. In operation 640, images are acquired, and operation 642 includes recording image and transformation pairs. The image data set is now available for calibration. Operation 644 includes calculating intrinsic calibration parameters 410, and operation 646 includes calculating extrinsic calibration parameters 408. An overview of the process is that the intrinsic calibration parameters 410 are calculated based on the known properties of the image set and the calibration map 300; an internal calibration is applied to undistort the acquired image; a marker detection is applied to the undistorted image, and the marker detection results are compared to the earlier recorded positions (from operation 638) using an optimizer, thereby producing the extrinsic calibration parameters 408.

[0039] More specifically, the internal lens distortion calibration values ​​are solved using the acquired data (e.g., comparing the measured calibration plate keypoints to the nominal set using Levenberg-Marquardt optimization). The images will then be undistorted. The position of the end effector 114 (holding the calibration map 300) in each image is calculated using PnP optimization. Each pose is paired with a corresponding calculated position relative to the robot optical sensor 502a. In one implementation, the AX=XB problem is solved to solve for the mounted position of the robot optical sensor 502a using, for example, a dual quaternion method pose generation algorithm.

[0040] Figure 7 is a block diagram illustrating a controller 700 for automatic internal and external calibration of the robot optical sensor 502a. In one implementation, the controller 700 is positioned on the robot body 108 of the robot 100 (see Figure 1A In one implementation, controller 700 is a representation of computing device 1000, relative to Fig.10 1004 is described in more detail. The controller 700 includes one or more processors 1004 coupled to a memory 1002. The controller 700 is coupled to an optical sensor 502, which represents a robot optical sensor 502a and / or a robot optical sensor 502b. The controller 700 is also coupled to a position sensor 732, which senses the position and orientation of the robot arm 102 (particularly the segments 104a to 104e, the joints 106a to 106e, and the end effector 114). In ( Figure 6 The position data 716 acquired in operation 638 of the controller 700 is acquired by the position sensor 732. The controller 700 is also coupled to the robot arm servo motor 730, which moves the robot arm 102 by moving the joints 106a to 106e.

[0041] The memory 1002 includes instructions that can be executed by the processor 1004 and stored data. The memory 1002 includes an optical sensor control unit 702 that controls the robot optical sensors 502a and 502b. The memory also includes a robot arm control unit 704 that uses feedback from a position sensor 732 to control a robot arm servo motor 730 to position the end effector 114 in a specified posture (e.g., a specified 3D position and orientation). The memory also includes previous calibration data 706, which is used as a baseline or default calibration for images collected during the calibration procedure described herein. In one implementation, the default calibration data is used instead of the previous calibration data 706. The calibration monitoring component 708 determines a calibration trigger event based on a trigger criterion 710, such as by comparing the object positions calculated from different images or monitoring a timer. The posture generator and optimizer component 712 generates posture data 714 to move the robot arm 102 during the calibration procedure.

[0042] Raw images 504 are output from the optical sensor 502 at various poses and used for calibration, which also uses the image and transformation pairs 718 already recorded in operation 642. The calibration process control 720 performs internal and external calibration using internal calibration support data 722 and external calibration support data 724, which contain the logic and data required to perform the calculations described herein. The internal calibration support data 722 also includes data for the calibration map 300. The calculated calibration parameters 726, including both the extrinsic calibration parameters 408 and the intrinsic calibration parameters 410, are also in the memory 1002. The memory 1002 also stores the generated calibration images 508 and other logic and data 728 required to perform the operations described herein.

[0043] Now refer to Figure 8 , flowchart 800 illustrates an implementation of operations for automatically calibrating the robotic optical sensor 502a. In one implementation, at least in part, the operations are performed by executing instructions by one or more processors 1004. Figure 8. Operation 802 includes determining a trigger event for calibrating the robot optical sensor 502a. In one implementation, operation 802 includes comparing an image acquired using the robot optical sensor 502a with an image acquired using the robot arm optical sensor 502b. For example, the position of an object is calculated using a CV algorithm that operates on an image from the robot optical sensor 502a, and the position of the same object is calculated using a CV algorithm that operates on an image from the robot optical sensor 502b. If the calculated positions differ by more than a certain threshold event, this is a determined trigger event. In one implementation, the trigger event is the expiration of a timer. In one implementation, the trigger event is a user input. In one implementation, the trigger event is the detection of damage to a workpiece or an unexpected physical resistance to an operation controlled by CV.

[0044] Operation 804 includes determining a set of poses for calibrating the robot optical sensor 502b. In one implementation, this includes determining a set of poses for calibrating the robot optical sensor 502b. Figure 6 At least some of the operations described in operation 604 of (hereinafter referred to as "operation 806"). Operation 806 includes generating pose data 714 including 3D position and orientation data based at least on the set of poses. Using the position and orientation data, operation 808 includes determining whether a pose in the set of poses is reachable. In one implementation, determining reachability includes both determining whether the robot arm 102 can be moved into a defined position and determining that the calibration map 300 is within the field of view of the robot optical sensor 502a. Based on the decision operation 810, at least based on the pose being reachable, operation 812 includes receiving the pose. Based on the decision operation 810, at least based on the pose not being reachable, operation 814 includes rejecting the pose. Then operation 816 includes determining a pose order based at least on the set of poses. In one implementation, the poses are sorted in a manner that minimizes the total motion of the robot arm 102 so that the robot arm 102 does not move back and forth unnecessarily.

[0045] Operation 818 includes moving the robot arm into a plurality of poses based at least on the pose data. For example, a set of poses is selected to provide images of the calibration map 300 at a preferred range of distances (from closest to maximum distance with a minimum number of transition steps), a range of orientations (from a first angle to a second angle with a minimum number of transition steps); and a range of positions of horizontal and vertical translations, etc., to provide full coverage of the 6DoF of the end effector 114 (without significant gaps). For example, at a given range and transition position, the end effector 114 holding the calibration map 300 steps through the range or orientation angles in each of the roll, pitch, and yaw using a sufficient number of transition angles in each orientation dimension to avoid gaps. The end effector 114 is then moved to a different transition position and again stepped through the orientation changes. This step is repeated until sufficient images cover the achievable 6DoF range.

[0046] At each of the plurality of poses, operation 820 includes acquiring, with the robot optical sensor 502b, a set of images of the calibration chart 300. In one implementation, the calibration chart 300 is attached to the end effector 114 of the robot arm 102, and the robot optical sensor 502b is mounted on the robot body 108. In one implementation, the robot optical sensor 502b is mounted on the robot arm 102 (e.g., on the end effector 114), and thus the robot optical sensor 502b moves with the robot arm 102. In such an implementation, the calibration chart 300 is placed elsewhere, such as on a wall at a known location or on the robot body 108.

[0047] Operation 822 includes calculating intrinsic calibration parameters 410 based on at least the set of acquired images. Operation 824 applies intrinsic calibration parameters 410 to the set of acquired images, and operation 826 includes calculating extrinsic calibration parameters 408 based on at least the set of acquired images.

[0048] Fig. 9 800. The process 900 of FIG. 800 is a block diagram illustrating a process 900 for automatically calibrating the robot optical sensor 502a. Specifically, in one implementation, the process 900 identifies elements within the controller 700 that perform the various operations of the flowchart 800. The calibration monitoring component 708 performs operation 802. The posture generator and optimizer component 712 performs operations 804 to 806. The robot arm control 704 performs operation 818, and the optical sensor control 702 performs operation 820. Finally, the calibration process control 720 performs operations 822 to 826.

[0049] Now refer to Fig.10, a block diagram of a computing device 1000 suitable for implementing aspects of the present disclosure is provided. In some implementations, the computing device 1000 includes one or more processors 1004, one or more presentation components 1006, and memory 1002. The disclosed implementations associated with the computing device 1000 are practiced by a variety of computing devices including personal computers, laptops, smart phones, mobile tablets, handheld devices, consumer electronics, special-purpose computing devices, etc. No distinction is made between these categories such as "workstations," "servers," "laptops," "handheld devices," etc., all of which are considered in the Fig.10 1000 is within the scope of and is a reference to a "computing device" herein. The disclosed implementations are also practiced in distributed computing environments, where tasks are performed by remote processing devices that are linked through a communications network. In addition, although the computing device 1000 is depicted as appearing to be a single device, in one implementation, multiple computing devices work together and share the depicted device resources. For example, in one implementation, memory 1002 is distributed across multiple devices, processors 1004 are provided and housed on different devices, and so on.

[0050] In one implementation, the memory 1002 includes any of the computer-readable media discussed herein. In one implementation, the memory 1002 is used to store and access instructions configured to perform various operations disclosed herein. In some implementations, the memory 1002 includes a computer storage medium in the form of a volatile and / or non-volatile memory, a removable or non-removable memory, a data disk in a virtual environment, or a combination thereof. In one implementation, the processor 1004 includes any number of processing units that read data from various entities such as the memory 1002 or the input / output (I / O) component 1010. Specifically, the processor 1004 is programmed to execute computer executable instructions for implementing aspects of the present disclosure. In one implementation, the instructions are executed by a processor, by multiple processors within the computing device 1000, or by a processor outside the computing device 1000. In some implementations, the processor 1004 is programmed to execute instructions such as those illustrated in the flowchart discussed below and depicted in the accompanying drawings.

[0051] The presentation component 1006 presents data representation to an operator or another device. In one implementation, the presentation component 1006 includes a display device, a speaker, a printing component, a vibration component, etc. Those skilled in the art will understand and recognize that computer data is presented in a variety of ways (e.g., visually in a graphical user interface (GUI), audibly through a speaker, wirelessly between computing devices 1000, through a wired connection, or in other ways). In one implementation, when processing and operation are fully automated so that the need for human interaction is reduced or not required, the presentation component 1006 is not used. The I / O port 1008 allows the computing device 1000 to be logically connected to other devices including the I / O component 1010, some of which are built-in. The implementation of the I / O component 1010 includes, but is not limited to, for example, a microphone, a keyboard, a mouse, a joystick, a game controller, a satellite dish, a scanner, a printer, a wireless device, etc.

[0052] The computing device 1000 includes a bus 1016 that directly or indirectly couples the following devices: memory 1002, one or more processors 1004, one or more display components 1006, input / output (I / O) ports 1008, I / O components 1010, power supply 1012, and network components 1014. The computing device 1000 should not be interpreted as having any dependency or requirement relating to any single component or combination of components illustrated therein. The bus 1016 represents one or more buses (e.g., an address bus, a data bus, or a combination thereof). Although shown with lines for clarity, Fig.10 , but some implementations blur the functionality of the various different components described herein.

[0053] In some implementations, the computing device 1000 is communicatively coupled to the network 1018 using a network component 1014. In some implementations, the network component 1014 includes a network interface card and / or computer executable instructions (e.g., a driver) for operating the network interface card. In one implementation, communications between the computing device 1000 and other devices are conducted using any protocol or mechanism over a wired or wireless connection 1020. In some implementations, the network component 1014 is operable to communicate wirelessly using short-range communication technologies (e.g., near field communication (NFC), Data is communicated between devices using public, private or mixed (public and private) transmission protocols (such as brand communications, etc.) or a combination thereof.

[0054] Although described in conjunction with computing device 1000, implementations of the present disclosure can be implemented using many other general-purpose or special-purpose computing system environments, configurations, or devices. Implementations of well-known computing systems, environments, and / or configurations suitable for use with aspects of the present disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, wearable or accessory-based mobile computing and / or communication devices (e.g., watches, glasses, headsets, or headphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, VR devices, holographic devices, and the like. Such systems or devices accept input from a user in any manner, including from an input device such as a keyboard or pointing device, via gesture input, proximity input (such as by hovering), and / or via voice input.

[0055] Embodiments of the present disclosure are described in the general context of computer executable instructions (e.g., program modules), which are executed by one or more computers or other devices in software, firmware, hardware, or a combination thereof. In one implementation, computer executable instructions are organized into one or more computer executable components or modules. Typically, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. In one implementation, various aspects of the present disclosure are implemented using any number and organization of such components or modules. For example, aspects of the present disclosure are not limited to specific computer executable instructions or specific components or modules illustrated in the accompanying drawings and described herein. Other implementations of the present disclosure include different computer executable instructions or components having more or less functions than those illustrated and described herein. In an implementation involving a general-purpose computer, aspects of the present disclosure transform a general-purpose computer into a special-purpose computing device when configured to execute instructions described herein.

[0056] By way of example and not limitation, computer-readable media include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable memory implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media are tangible and mutually exclusive with communication media. Computer storage media are implemented in hardware and do not include carrier waves and propagated signals. For the purposes of this disclosure, computer storage media themselves are not signals. In one implementation, computer storage media include hard disks, flash drives, solid-state memory, phase change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage devices, cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other non-transmission media for storing information for access by a computing device. In contrast, communication media typically embodies computer readable instructions, data structures, program modules, etc. in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0057] As compared to Figures 11 to 13 As shown, used in manufacturing and service applications ( Figure 1A and Figure 1B Thus, in Fig.11 The manufacturing and servicing method 1100 of the apparatus and the Fig.12 Implementations of the present disclosure are described in the context of the device 1200 shown in FIG. Fig.11 In FIG. 1 , a diagram illustrating an exemplary device manufacturing and service method is depicted according to an implementation. In one implementation, during pre-production, the device manufacturing and service method 1100 includes: Fig.12 The equipment 1200 is specified and designed 1102 and the materials are purchased 1104. During production, Fig.12 Component and subassembly manufacturing 1106 and system integration 1108 of the device 1200 in FIG. Fig.12 The device 1200 in undergoes certification and delivery 1110 to be put into service 1112. When serviced by a customer, Fig.12 The equipment 1200 in is scheduled for routine maintenance and servicing 1114, which in one implementation includes modification, reconfiguration, refurbishment, and other maintenance or servicing as described herein.

[0058] In one embodiment, each of the processes of the equipment manufacturing and service method 1100 is performed or carried out by a system integrator, a third party, and / or an operator. In these implementations, the operator is a customer. For the purpose of this description, the system integrator includes any number of equipment manufacturers and main system subcontractors; the third party includes any number of contractors, subcontractors, and suppliers; and in one implementation, the operator is the owner of the equipment or equipment fleet / the administrator responsible for the equipment or equipment fleet, the user who operates the equipment, a leasing company, a military entity, a service organization, etc.

[0059] Now refer to Fig.12 , a device 1200 is provided. Fig.12 As shown, an example of the device 1200 is a flying device 1201, such as an aerospace vehicle, an aircraft, air cargo, a flying car, etc. Fig.12 As shown, another example of equipment 1200 is ground transportation equipment 1202, such as a car, truck, heavy equipment, construction equipment, motorboat, boat, submarine, etc. Fig.12 Another example of the device 1200 shown in is a modular device 1203, which includes at least one or more of the following modules: an air module, a payload module, and a ground module. The air module provides aerial lifting or flight capabilities. The payload module provides the ability to transport objects, such as cargo or living objects (people, animals, etc.). The ground module provides ground movement capabilities. The solutions disclosed herein are applied individually or in groups to each of the modules, such as air and payload modules, or payload and ground, etc., or all modules.

[0060] Now refer to Fig.13 , depicts a more detailed diagram of a flight module 1201, in which implementations of the present disclosure are advantageously employed. In this example, the flight module 1201 is implemented by Fig.11 An aircraft is produced by the equipment manufacturing and service method 1100 in the embodiment of the present invention. And includes a fuselage 1303 with multiple systems 1304 and an interior 1306. The implementation of multiple systems 1304 includes one or more of a propulsion system 1308, an electrical system 1310, a hydraulic system 1312, and an environmental system 1314. However, other systems may also be included. Although an aviation example is shown, different advantageous implementations are applied to other industries, such as the automotive industry, etc.

[0061] The implementation disclosed herein is described in the general context of computer code or machine usable instructions, which include computer executable instructions (e.g., program components) executed by a computer or other machine (e.g., a personal data assistant or other handheld device). In general, program components including routines, programs, objects, components, data structures, etc. refer to codes that perform specific tasks or implement specific abstract data types. The disclosed embodiments are practiced in a variety of system configurations (including personal computers, laptop computers, smart phones, mobile tablet computers, handheld devices, consumer electronics, special computing devices, etc.). The disclosed implementation is also practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network.

[0062] The following paragraphs describe other aspects of the disclosure:

[0063] Clause 1. A system for automatic calibration of a robotic optical sensor, the system comprising:

[0064] Optical sensors;

[0065] Robotic arm;

[0066] Calibration chart;

[0067] one or more processors; and

[0068] A memory storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:

[0069] determining a set of poses for calibrating the optical sensor;

[0070] generating pose data comprising (3D) position and orientation data based at least on the set of poses;

[0071] moving the robotic arm into a plurality of poses based at least on the pose data;

[0072] acquiring, using the optical sensor, a set of images of the calibration chart in each of the plurality of postures;

[0073] calculating intrinsic calibration parameters based at least on the set of acquired images; and

[0074] Extrinsic calibration parameters are calculated based at least on the acquired set of images.

[0075] Clause 2. The system of clause 1, wherein the operations further comprise:

[0076] The intrinsic calibration parameters are applied to the set of acquired images before calculating the extrinsic calibration parameters.

[0077] Clause 3. The system of any one of clauses 1 to 2, wherein the operations further comprise:

[0078] determining whether a pose in the set of poses is reachable; and

[0079] The gesture is rejected based at least on the fact that the gesture is not attainable.

[0080] Clause 4. The system of any one of clauses 1 to 3, wherein the calibration map is attached to the end effector of the robot arm, and wherein the optical sensor is mounted to a body of the robot.

[0081] Clause 5. The system of any one of clauses 1 to 3, wherein the optical sensor is mounted to the robotic arm.

[0082] Clause 6. The system of any one of clauses 1 to 5, wherein the operations further comprise:

[0083] determining a trigger event for calibrating the optical sensor; and

[0084] Wherein, determining the set of postures for calibrating the optical sensor comprises: determining the set of postures for calibrating the optical sensor based at least on the trigger event.

[0085] Clause 7. The system of clause 6, wherein the optical sensor is a first optical sensor, and wherein the system further comprises:

[0086] a second optical sensor; and

[0087] Wherein, determining the trigger event includes: comparing the image acquired by using the first optical sensor with the image acquired by using the second optical sensor.

[0088] Clause 8. A method for automatic calibration of a robotic optical sensor, the method comprising:

[0089] determining a set of poses for calibrating the optical sensor;

[0090] generating pose data comprising (3D) position and orientation data based at least on the set of poses;

[0091] moving the robotic arm into a plurality of poses based at least on the pose data;

[0092] acquiring, using the optical sensor, a set of images of a calibration chart in each of the plurality of postures;

[0093] calculating intrinsic calibration parameters based at least on the set of acquired images; and

[0094] Extrinsic calibration parameters are calculated based at least on the acquired set of images.

[0095] Clause 9. The method according to clause 8, further comprising:

[0096] The intrinsic calibration parameters are applied to the set of acquired images before calculating the extrinsic calibration parameters.

[0097] Clause 10. The method according to any one of clauses 8 to 9, further comprising:

[0098] determining whether a pose in the set of poses is reachable; and

[0099] The gesture is rejected based at least on the fact that the gesture is not attainable.

[0100] Clause 11. The method of any one of clauses 8 to 10, wherein the calibration map is attached to the end effector of the robot arm, and wherein the optical sensor is mounted to a body of the robot.

[0101] Clause 12. The method of any one of clauses 8 to 12, wherein the optical sensor is mounted to the robotic arm.

[0102] Clause 13. The method according to any one of clauses 8 to 12, further comprising:

[0103] determining a trigger event for calibrating the optical sensor; and

[0104] Wherein, determining the set of postures for calibrating the optical sensor comprises: determining the set of postures for calibrating the optical sensor based at least on the trigger event.

[0105] Clause 14. The method of clause 13, wherein the optical sensor is a first optical sensor, and wherein determining the triggering event comprises:

[0106] An image acquired using the first optical sensor is compared with an image acquired using the second optical sensor.

[0107] Clause 15. A computer program product comprising a computer usable medium having computer readable program code embodied therein, the computer readable program code being adapted to be executed to implement a method for automatically calibrating a robotic optical sensor, the method comprising:

[0108] determining a set of poses for calibrating the optical sensor;

[0109] generating pose data comprising (3D) position and orientation data based at least on the set of poses;

[0110] moving the robotic arm into a plurality of poses based at least on the pose data;

[0111] acquiring, using the optical sensor, a set of images of a calibration chart in each of the plurality of postures;

[0112] calculating intrinsic calibration parameters based at least on the set of acquired images; and

[0113] Extrinsic calibration parameters are calculated based at least on the acquired set of images.

[0114] Clause 16. The computer program product of clause 15, wherein the operations further comprise:

[0115] The intrinsic calibration parameters are applied to the set of acquired images before calculating the extrinsic calibration parameters.

[0116] Clause 17. The computer program product of any one of clauses 15 to 16, wherein the operations further comprise:

[0117] determining whether a pose in the set of poses is reachable; and

[0118] The gesture is rejected based at least on the fact that the gesture is not attainable.

[0119] Clause 18. The computer program product of any one of clauses 15 to 17, wherein the calibration map is attached to the end effector of the robot arm, and wherein the optical sensor is mounted to a body of the robot.

[0120] Clause 19. The computer program product of any one of clauses 15 to 18, wherein the optical sensor is mounted to the robotic arm.

[0121] Clause 20. The computer program product of any one of clauses 15 to 19, wherein the optical sensor is a first optical sensor, and wherein the operations further comprise:

[0122] Determining a trigger event for calibrating the first optical sensor, wherein determining the trigger event comprises:

[0123] comparing an image acquired using the first optical sensor with an image acquired using a second optical sensor; and

[0124] Wherein, determining the set of postures for calibrating the first optical sensor comprises:

[0125] Based at least on the triggering event, a set of gestures for calibrating the first optical sensor is determined.

[0126] When introducing elements of aspects of the disclosure or implementations thereof, the articles "a," "an," "the," and "said" are intended to indicate that there are one or more elements. The terms "comprising," "including," and "having" are intended to be inclusive and indicate that there may be additional elements in addition to the listed elements. The term "implementation" is intended to mean "an example thereof." The phrase "one or more of: A, B, and C" means "at least one of A and / or at least one of B and / or at least one of C."

[0127] Having described aspects of the disclosure in detail, it will be apparent that modifications and variations may be made without departing from the scope of aspects of the disclosure as defined in the appended claims. As various changes may be made to the above constructions, products and methods without departing from the scope of aspects of the disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

Claims

1. A system for automatic calibration of a robot optical sensor, the system comprising: Optical sensor (502a); Robot (100) arm (102); CalibrationMap(300); one or more processors (1004); and A memory (1002) storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising: determining a set of poses for calibrating the optical sensor (804); generating gesture data (714) (806) comprising three-dimensional (3D) position (716) and orientation data based at least on the set of gestures; Based at least on the pose data, moving the robotic arm into a plurality of poses (818); acquiring, using the optical sensor, a set of images (504) of the calibration chart in each of the plurality of postures (820); calculating intrinsic calibration parameters (410) (822) based at least on a set of acquired images of the calibration map at each of the plurality of poses and known properties of the calibration map (300); applying the calculated intrinsic calibration parameters to the set of acquired images of the calibration map to obtain an updated set of acquired images of the calibration map (824); and Extrinsic calibration parameters (408) are calculated based at least on the updated set of acquired images of the calibration map (826).

2. The system according to claim 1, wherein: The operations also include: determining whether a gesture in the set of gestures is reachable (808); and The gesture is rejected based at least on the fact that the gesture is not reachable (814).

3. The system according to claim 1 or 2, wherein: The calibration map is attached to an end effector (114) of the robot arm, and wherein the optical sensor is mounted to a body (108) of the robot.

4. The system according to claim 1 or 2, wherein: The optical sensor is mounted to the robotic arm.

5. The system according to claim 1 or 2, wherein: The operations also include: determining a trigger event for calibrating the optical sensor (802); and Wherein, determining the set of postures for calibrating the optical sensor comprises: Based at least on the triggering event, a set of gestures for calibrating the optical sensor is determined.

6. The system according to claim 5, wherein: The optical sensor is a first optical sensor, and wherein the system further comprises: a second optical sensor (502b); and Wherein, determining the trigger event comprises: comparing the image acquired by using the first optical sensor with the image acquired by using the second optical sensor.

7. A method for automatic calibration of a robot optical sensor, the method comprising: determining a set (804) of poses for calibrating the optical sensor (502a); generating gesture data (714) (806) comprising three-dimensional (3D) position (716) and orientation data based at least on the set of gestures; Based at least on the posture data, moving the robot (100) arm (102) into a plurality of postures (818); Acquiring a set (820) of images (504) of the calibration chart (300) using the optical sensor in each of the plurality of postures; calculating intrinsic calibration parameters (410) (822) based at least on a set of acquired images of the calibration map at each of the plurality of poses and known properties of the calibration map (300); applying the calculated intrinsic calibration parameters to the set of acquired images of the calibration map to obtain an updated set of acquired images of the calibration map (824); as well as Extrinsic calibration parameters (408) are calculated based at least on the updated set of acquired images of the calibration map (826).

8. The method according to claim 7, further comprising: determining whether a gesture in the set of gestures is reachable (808); as well as The gesture is rejected based at least on the fact that the gesture is not reachable (814).

9. The method according to claim 7 or 8, wherein: The calibration map is attached to an end effector (114) of the robot arm, and wherein the optical sensor is mounted to a body (108) of the robot.

10. The method according to claim 7 or 8, wherein: The optical sensor is mounted to the robotic arm.

11. The method according to claim 7 or 8, further comprising: Determining a trigger event for calibrating the optical sensor (802); and Wherein, determining the set of postures for calibrating the optical sensor comprises: Based at least on the triggering event, a set of gestures for calibrating the optical sensor is determined.

12. The method according to claim 11, wherein: The optical sensor is a first optical sensor, and wherein determining the triggering event comprises: An image acquired using the first optical sensor is compared with an image acquired using a second optical sensor (502b).

Images