Method and control system for verifying and updating calibration information for robotic control - Patents.com

Abstract

To provide a computing system and a method for calibration verification.SOLUTION: The computing system is configured to perform a first calibration operation, and to control a robot arm to move a verification symbol to a reference location. The robot control system further receives, from a camera, a reference image of the verification symbol, and determines a reference image coordinate for the verification symbol. The robot control system further controls the robot arm to move the verification symbol to the reference location again during an idle period, receives an additional image of the verification symbol, and determines a verification image coordinate. The robot control system determines a deviation parameter value based on the reference image coordinate and the verification image coordinate, determines whether the deviation parameter value exceeds a defined threshold, and performs a second calibration operation if the threshold is exceeded.SELECTED DRAWING: Figure 12A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. application Ser. No. 16 / 732,832, filed Jan. 2, 2020, entitled "Method and Control System for Verifying and Updating Camera Calibration for Robot Control," which is a continuation of U.S. application Ser. No. 16 / 525,004, filed July 29, 2019, entitled "Method and Control System for Verifying and Updating Camera Calibration for Robot Control," which is a continuation of U.S. application Ser. No. 16 / 369,630, filed March 29, 2019, entitled "Method and Control System for Verifying and Updating Camera Calibration for Robot Control," the entire contents of which are incorporated herein by reference. This application further claims the benefit of U.S. Patent Application No. 62 / 916,798, filed October 18, 2019, entitled "Method and Control System for Verifying Calibration for Robot Control," the entire contents of which are also incorporated by reference herein.

[0002] The present invention relates to a method and control system for verifying and updating calibration information for robotic control. [Background technology]

[0003] As automation becomes more common, robots are being used in more environments, such as warehousing and manufacturing environments. For example, robots may be used in warehouses to palletize or unpalletize items, or in factories to lift objects from conveyor belts. The movement of the robot may be fixed or may be based on inputs, such as images taken by a camera in the warehouse or factory. In the latter situation, a calibration operation may be performed to determine the characteristics of the camera and to determine the relationship between the camera and the environment in which the robot is located. The calibration operation may generate calibration information that is used to control the robot. In some implementations, the calibration operation may require manual operation by a human, who may manually control the movement of the robot or manually control a camera to capture images of the robot. Summary of the Invention

[0004] An aspect of the embodiments herein relates to performing calibration verification for robotic control, such as verifying a camera calibration or other system calibration. Calibration verification can be performed by a robot control system including a communication interface and control circuitry. The communication interface can be configured to communicate with a robot having a base and a robot arm having a verification symbol disposed thereon, and to communicate with a camera having a camera field of view. The control circuitry of the robot control system is configured to: a) perform a first calibration operation (e.g., a first camera calibration) to determine calibration information (e.g., camera calibration information); b) output a first motion command to the communication interface, the communication interface configured to communicate the first motion command to the robot during or after the first calibration operation such that the robot arm moves the verification symbol to a location within the camera field of view, the location being a reference location of one or more reference locations for verifying the first calibration operation; and c) receive an image of the verification symbol from the camera via the communication interface. The calibration verification may be configured to perform by: (a) receiving an image of the verification symbol, where the camera is configured to capture an image of the verification symbol at a reference location, the image being a reference image for verification; (b) determining reference image coordinates for the verification symbol, where the reference image coordinates are coordinates at which the verification symbol appears in the reference image; and (c) outputting to the communications interface a second operation command based on the calibration information, the communications interface being configured to communicate the second operation command to the robot to move the robot arm and perform a robotic operation.

[0005] In one embodiment, the control circuitry includes: f) detecting a quiescent period during robot operation; g) outputting a third motion command to the communication interface, the communication interface configured to communicate the third motion command to the robot such that during the quiescent period, the robot arm moves the verification symbol to at least the reference location; h) receiving an additional image of the verification symbol from the camera via the communication interface, the camera configured to capture the additional image of the verification symbol at least at the reference location during the quiescent period, the additional image being a verification image for verification; i) determining verification image coordinates used for verification, the verification image coordinates being coordinates at which the verification symbol appears in the verification image; and j) determining a verification image coordinate based on an amount of deviation between the reference image coordinates and the verification image coordinates. the reference image coordinates and the verification image coordinates are both associated with the reference location, the deviation parameter value being indicative of a change since the first calibration operation of a camera with which the communications interface is configured to communicate, or a change since the first calibration operation of a relationship between the camera with which the communications interface is configured to communicate and the robot; k) determining whether the deviation parameter value exceeds a defined threshold; and l) in response to a determination that the deviation parameter value exceeds the defined threshold, performing a second calibration operation (e.g., a second camera calibration operation) and determining updated calibration information (e.g., updated camera calibration information). [Brief description of the drawings]

[0006] The above and other features, objects and advantages of the present invention will be apparent from the following description of embodiments of the invention as illustrated in the accompanying drawings, which are incorporated in and form a part of this specification, and further serve to explain the principles of the invention and to enable those skilled in the art to make and use the invention. The drawings are not to scale.

[0007] [Figure 1A] 1 illustrates a block diagram of a system in which validation of calibration information is performed, according to embodiments herein; [Figure 1B] 1 illustrates a block diagram of a system in which validation of calibration information is performed, according to embodiments herein; [Figure 1C] FIG. 1 illustrates a block diagram of a robotic control system configured to perform calibration validation, according to an embodiment of the present disclosure. [Figure 1D] FIG. 2 illustrates a block diagram of a camera for which camera calibration is performed, according to an embodiment of the present disclosure. [Diagram 2] 1 illustrates a system illustrating a robot being controlled based on calibration information obtained from a calibration operation, according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates a system for performing a calibration operation, according to one embodiment of the present disclosure. [Figure 4A] 1 provides a flow diagram illustrating a method for performing validation of calibration information, according to an embodiment herein. [Figure 4B] 1 provides a flow diagram illustrating a method for performing validation of calibration information, according to an embodiment herein. [Figure 5A] 1 illustrates a system in which a verification symbol is placed on a robot and used to perform verification of calibration information, according to one embodiment herein. [Figure 5B]1 illustrates a system in which a verification symbol is placed on a robot and used to perform verification of calibration information, according to one embodiment herein. [Figure 5C] 1 illustrates an exemplary verification symbol according to one embodiment of the present disclosure. [Figure 6A] 1 illustrates an example of a reference location where a respective image of a verification symbol is captured, according to one embodiment of the present disclosure. [Figure 6B] 1 illustrates an example of a reference location where a respective image of a verification symbol is captured, according to one embodiment of the present disclosure. [Figure 6C] 1 illustrates an example of a reference location where a respective image of a verification symbol is captured, according to one embodiment of the present disclosure. [Figure 6D] 1 illustrates an example of a reference location where a respective image of a verification symbol is captured, according to one embodiment of the present disclosure. [Figure 7A] 1 illustrates an example of determining reference image coordinates, according to an embodiment of the present disclosure. [Figure 7B] 1 illustrates an example of determining verification image coordinates according to an embodiment herein. [Figure 8] 1 illustrates an exemplary timeline for validation of calibration information, according to an embodiment of the present disclosure. [Figure 9] A flow diagram illustrating an example method for performing validation of calibration information according to an embodiment of the present specification is provided. [Figure 10A] 1 illustrates a system in which a group of verification symbols are placed on a robot and used to perform verification of calibration information, according to one embodiment herein. [Figure 10B] 1 illustrates a system in which a group of verification symbols are placed on a robot and used to perform verification of calibration information, according to one embodiment herein. [Figure 10C]1 illustrates a system in which a group of verification symbols are placed on a robot and used to perform verification of calibration information, according to one embodiment herein. [Figure 11A] 1 illustrates a system in which a group of verification symbols are placed on a robot and used to perform verification of calibration information, according to one embodiment herein. [Figure 11B] 1 illustrates a system in which a group of verification symbols are placed on a robot and used to perform verification of calibration information, according to one embodiment herein. [Figure 11C] 1 illustrates a group of verification symbols having different respective sizes, according to one embodiment herein. [Figure 12A] 1 provides a flow diagram illustrating a method for performing validation of calibration information, according to an embodiment herein. [Figure 12B] 1 provides a flow diagram illustrating a method for performing validation of calibration information, according to an embodiment herein. [Figure 13A] 13A-13C illustrate a reference image and a validation image, respectively, associated with a first pose of the robot arm. [Figure 13B] 13A-13C illustrate a reference image and a validation image, respectively, associated with a first pose of the robot arm. [Figure 14A] 13A-13C illustrate a reference image and a validation image, respectively, associated with a second pose of the robot arm. [Figure 14B] 13A-13C illustrate a reference image and a validation image, respectively, associated with a second pose of the robot arm. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The following detailed description is merely exemplary in nature and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be limited by any expressed or implied theory presented in the preceding technical field, background, brief summary or the following detailed description.

[0009] The embodiments described herein relate to verifying and / or updating calibration information used to control a robot, such as a robot used in a warehouse, manufacturing plant, or any other environment. More specifically, the calibration information may be determined by performing a calibration operation, which may be performed to facilitate control of a robotic manipulation system and may be referred to as a system calibration. The system calibration may include a calibration of a camera (which may be referred to as a camera calibration or a camera calibration operation), a calibration of a robot (which may be referred to as a robot calibration), a calibration of another element of the robotic manipulation system, or any combination thereof. The system calibration may be performed, for example, by a robot control system (also referred to as a robot controller) to generate calibration information (e.g., camera calibration information) that facilitates the ability of the robotic control system to control the robot based on an image captured (e.g., photographed) by a camera. For example, a robot may be used to lift packages in a warehouse, where the positioning of a robotic arm or other component of the robot may be based on an image of the package captured by a camera. In that case, when the calibration information includes camera calibration information, the camera calibration information may be used along with the image of the package to determine, for example, a location and orientation of the package relative to the robotic arm of the robot. When the system calibration includes a camera calibration, the camera calibration may include determining an estimate of each of the intrinsic parameters (sometimes called intrinsic parameters) of the camera and determining an estimate of a relationship between the camera and its external environment. The intrinsic parameters of the camera may have one or more parameter values, such as a matrix, vector, or scalar value. Further examples of intrinsic parameters include a projection matrix and a distortion parameter.In one example, camera calibration can include determining a position of the camera relative to a fixed position in the external environment, which can be expressed as a transformation function that represents a relationship between the camera and the fixed position in the external environment. In some cases, camera calibration can be performed with the aid of a calibration pattern, which can have pattern elements disposed at defined positions on the calibration pattern. The camera can capture images of the pattern elements of the calibration pattern (also referred to as calibration images), and camera calibration can be performed based on comparing the images of the pattern elements to the defined positions of the pattern elements. Camera calibration is discussed in more detail in U.S. Patent Application Serial No. 16 / 295,940, filed March 7, 2019 (Attorney Docket No. MJ0021US1), entitled "METHOD AND DEVICE FOR PERFORMING AUTOMATIC CAMERA CALIBRATION FOR ROBOT CONTROL," the entire contents of which are incorporated herein by reference.

[0010] As mentioned above, one aspect of the disclosure relates to verifying that a camera calibration or other calibration operation performed at an earlier time point is still accurate at a later time point. A camera calibration performed at an earlier time point can generate camera calibration information that reflects the characteristics of the camera at that time point, such as the camera's intrinsic parameters or the relationship between the camera and its external environment at that time point. In some cases, the characteristics of the camera may change over time, such that the earlier camera calibration may not maintain accuracy over time. In a first example, the camera's intrinsic parameters may change over time. Such changes may be caused, for example, by temperature changes, which change the shape of the camera's housing and / or lens. In a second example, the relationship between the camera and its external environment may change over time. For example, the camera may change position or orientation, for example, relative to the base of the robot, or a location in a warehouse. Such changes may be caused, for example, by temperature changes that cause any components used to mount the camera to expand or contract, a person or other object colliding with the camera, vibrations in the camera's external environment (e.g., a warehouse), forces due to the camera's own weight (i.e., due to gravity), or some other factor. These changes may cause the camera calibration information or other calibration information to become stale, which may result in errors when the camera calibration information or other calibration information is used to position the robot arm or other components of the robot at a later time. In other words, if the characteristics associated with the camera change over time but the camera calibration information is not updated to reflect such changes, the robot may operate based on stale or otherwise inaccurate camera calibration information, which may cause undesirable errors in the robot's operation.To address the possibility that changes may occur in one or more characteristics of the camera, the robotic control system may automatically perform a validation that detects when the camera calibration information from the camera calibration (or, more generally, when the calibration information from a calibration operation is no longer sufficiently accurate). Detecting such a condition may provide an indication regarding the change in the characteristics of the camera, or some other element of the robotic manipulation system. If the validation detects that the calibration information is no longer sufficiently accurate, the robotic control system may perform a calibration operation to determine updated calibration information that may reflect a more current characteristic or characteristics of the camera or another element of the robotic manipulation system. The updated calibration information may be used to control the positioning of the robot arm, or some other aspect of the robot's operation. Thus, automatic validation of the calibration information and / or updating of the calibration information is performed to ensure that the robot operates based on accurate information about one or more characteristics associated with the camera or any other element of the robotic manipulation system.

[0011] One aspect of the embodiments herein relates to verifying calibration information for a camera by comparing a reference image captured by the camera with a verification image captured by the camera. In some cases, the reference image can be an image of an object captured when the object is in a particular position at an earlier time, and the verification image can be an image of the object captured at a later time in the same position. The verification can determine whether the deviation between the reference image and the verification image is too large, such as whether the deviation exceeds a certain threshold. In some implementations, the object can be a verification symbol. More specifically, the robot arm or other components of the robot can have a verification symbol that is used to verify the calibration information. Both the reference image and the verification image can capture or otherwise include a verification symbol, and the robot control system can compare the two images by comparing the appearance of the verification symbol in the reference image with the appearance of the verification symbol in the verification image. For example, after the robot control system executes a calibration operation that generates calibration information at a particular time, the robot control system can control the robot arm (e.g., via motion commands) to move the verification symbol to a set of predefined locations within the field of view of the camera (also referred to as the camera field of view of the camera), which can be used as a set of reference locations for verification. The camera can capture a reference image of each of the verification symbols at the set of reference locations. In some cases, the reference images can be captured immediately after the calibration operation is executed. The operation of the robot arm, or more specifically, the motion commands used to move the robot arm, can be based on the calibration information from the just executed calibration operation, or can be independent of the calibration information. In some cases, the reference images can be captured before the robot starts a robot operation.After the reference image has been captured, the robot can be considered ready to begin robotic operations to perform a task, and the robot control system can control the positioning of the robot arm, for example, based on the next image captured by the camera.

[0012] As mentioned above, the reference image can be compared to a subsequently captured verification image. In one embodiment, the verification image can be captured during one or more idle periods detected by the robot control system. More specifically, when a robotic operation begins, the robot can begin performing a robot task (e.g., by interacting with a load or other object). When the robot is performing a robotic operation, the robot control system can detect one or more idle periods for the robot. In some cases, an idle period can be a period during which the robot is not performing a robotic task during a robotic operation. In some cases, the robot control system can schedule a robotic operation based on detecting or otherwise predicting an object with which the robot needs to interact, and can detect an idle period based on detecting or otherwise predicting the absence of an object with which the robot needs to interact.

[0013] During the idle period(s), the robot control system may control the robot arm or other components of the robot to move (e.g., via motion commands) to the reference locations and capture a respective verification image (e.g., via camera commands) at each reference location. If the robot has a verification symbol disposed thereon, the robot control system may more specifically control the robot arm to move the verification symbol to the reference locations and capture a verification image. The robot control system may then determine how much each verification image deviates from the corresponding reference image at each reference location. In some cases, the deviation between the verification image and each reference image may be represented as a deviation parameter. If the value of the deviation parameter (also referred to as the deviation parameter value) exceeds a defined threshold for the deviation parameter (also referred to as a defined deviation threshold), the robot control system may further perform a further calibration operation (e.g., further camera calibration) to determine updated calibration information for the camera (e.g., updated camera calibration information). When the value of the deviation parameter exceeds the defined deviation threshold, this condition may indicate that using the previously generated calibration information may result in an undesirable amount of error in the robot operation. Thus, in some cases, robotic operation may be interrupted or halted while further calibration operations are performed (the interruption may be considered another pause period). After the further calibration operations are completed, a new set of reference images may be captured and robotic operation may continue with the updated calibration information. During the subsequent pause period(s), a new set of verification images may be captured and the robotic control system may perform verification of the further calibration operations by comparing the new set of reference images to the new set of verification images.

[0014] As mentioned above, if the value of the deviation parameter exceeds a defined deviation threshold, the robot control system may perform a further calibration operation. If the value of the deviation parameter does not exceed the deviation threshold, the robot control system may continue the robot operation after the rest period without performing a further calibration operation. In this scenario, the camera may capture a new set of verification images at the respective reference locations during the subsequent rest period(s). Once the new set of verification images is captured, the robot control system may again perform verification of the calibration information by determining how much the new set of verification images deviate from the respective reference images at the respective reference locations.

[0015] As mentioned above, the robot arm may have a verification symbol, such as a ring pattern, disposed thereon, which may be captured by or otherwise included in the reference image and the verification image. In one embodiment, the robot control system may identify deviations between the reference image and each verification image based on respective positions at which the verification symbol appears in the reference image and based on respective positions at which the verification symbol appears in the verification image. For example, the robot control system may identify reference image coordinates for each reference location. The reference image coordinates for a particular location may be the coordinates at which the verification symbol appears in a reference image captured when the verification symbol is placed at that reference location. More specifically, the reference image coordinates may be associated with a particular reference location and may refer to image coordinates at which the verification symbol appears in a reference image captured by a camera when the verification symbol is placed at that reference location. In the above example, the image coordinates may refer to coordinates within an image, such as pixel coordinates. When the robot control system subsequently places a verification symbol at the particular reference location at a later time and acquires a corresponding verification image, the robot control system may determine the verification image coordinates. The verification image coordinates may also be associated with the reference location and may refer to the image coordinates (e.g., pixel coordinates) at which the verification symbol appears in the verification image captured by the camera when the verification symbol is placed at the reference location. The robotic control system may compare the reference image coordinates associated with a particular reference location with the verification image coordinates associated with the same reference location. This comparison may be performed for each reference location for which a verification image and a reference image were captured.

[0016] In one example, the reference image coordinates at which the verification symbol appears in the reference image may be the coordinates of the center of the verification symbol in the reference image (also referred to as the center coordinates of the verification symbol in the reference image). Similarly, the verification image coordinates at which the verification symbol appears in the verification image may be the coordinates of the center of the verification symbol in the verification image (also referred to as the center coordinates of the verification symbol in the verification image). For each reference location at which the robot arm and / or the verification symbol was located when the corresponding verification image was captured, the robot control system may determine a deviation between the reference image coordinates associated with the reference location and the verification image coordinates associated with the same reference location. In the case where the robot arm and / or the verification symbol has been positioned at multiple reference locations, the robot control system may determine a respective deviation amount between the respective reference image coordinates and the respective verification image coordinates for the multiple reference locations. The robot control system may further determine a value of the deviation parameter based on the respective deviation amount between the reference image coordinates and the respective verification image coordinates for each reference location.

[0017] In one example, the verification symbol may include multiple shapes that are concentric with each other such that the centers of each of the multiple shapes in the verification symbol are at the same or substantially the same location. For example, the verification symbol may be a ring pattern including two or more concentric circles. In some cases, when the reference image coordinates of the verification symbol are the center coordinates of the verification symbol in the reference image, the robot control system may identify the center coordinates of the verification symbol based on the center coordinates of each of the multiple shapes in the reference image, and the center coordinates of a particular shape are the coordinates of the center of that shape. In some cases, when the verification symbol is a ring pattern, the center coordinates of the ring pattern in the reference image may be identified as the average of the center coordinates of a first circle forming the ring pattern in the reference image and the center coordinates of a second circle forming the ring pattern. Similarly, the center coordinates of the verification symbol in the verification image may be identified based on the center coordinates of each of the multiple shapes forming the verification symbol in the verification image. In some cases, the accuracy of the verification may be improved by using multiple shapes to form the verification symbol. For example, the robustness of the verification to image noise may be improved by identifying the center coordinates of the verification symbol using the center coordinates of each of the multiple shapes in an image. More specifically, if the image of the verification symbol includes image noise, the image noise may reduce the accuracy with which the robot control system detects the center coordinate of a particular feature of the verification symbol. However, if the center coordinate of the feature is averaged with the center coordinate of another feature to determine the center coordinate of the verification symbol, the averaged center coordinate may reduce the effect of the image noise. As a result, the accuracy in determining the center coordinate of the verification symbol may be improved.

[0018] In one example, the verification symbol can have multiple regions, each of different colors, and the areas of each of the multiple regions can have a distinguishable, defined ratio. For example, the verification symbol can have a first region having a first color (e.g., black) and a second region having a second color (e.g., white), and the ratio of the area of ​​the first region to the area of ​​the second region is defined or otherwise known. The distinguishable ratio can facilitate identifying the verification symbol in the image, especially when the image captures or otherwise includes other features, such as dots of a calibration pattern. For example, a robotic arm moving the verification symbol may have a calibration pattern disposed thereon. The robotic control system can use the ratio to distinguish between the verification symbol and the dots of the calibration pattern. More specifically, since the ratio of the areas of the multiple regions of the verification symbol is defined as a distinguishable ratio, the robotic control system can identify the verification symbol in the image based on the defined ratio. During identification of the verification symbol appearing in the image, the robotic control system can distinguish between the verification symbol and the calibration pattern or other features based on the defined ratio. In some cases, the verification symbol may be identified within the image as a portion of the image having multiple regions, each of a different color, with a prescribed ratio between the respective areas of the multiple regions. If the robotic control system or other system or device determines that a particular portion of the image does not have multiple regions, each of a different color, or the respective areas of the multiple regions have a ratio different than the prescribed ratio, then the robotic control system may determine that the portion of the image is not a verification symbol.

[0019] In one example, the robot control system can perform the validation based on the temperature of the robot's surroundings. For example, the robot control system can adjust the defined deviation threshold (i.e., define a new value for the deviation threshold) based on the temperature. For example, the temperature can affect various components in the camera and / or in the robot, since certain materials may be temperature sensitive and / or may expand / contract depending on the temperature. A change in temperature may change the intrinsic parameter(s) of the camera and / or the relationship between the camera and its external environment. In one embodiment, the deviation threshold can be set to have a first value when the temperature is outside of a defined range, while the deviation threshold can be set to have a second value lower than the first value when the temperature is within the defined range. For example, the deviation threshold can be a first value when the temperature is within a defined normal operating temperature range (e.g., within 10 degrees of room temperature). When the temperature is outside of the normal operating temperature range, the deviation threshold can have a second value lower than the first value. The second value may be lower than the first value to more easily trigger further calibration operations when temperatures are outside the normal operating range, because temperatures outside the normal operating range may be more likely to cause changes in the camera or in the camera's relationship to the external environment, and therefore more likely to cause errors in operating the robot with previously generated camera calibration information or any other calibration information.

[0020] In one embodiment, the verification of the calibration information may rely only on a single reference point. Alternatively, the verification of the calibration information may rely on multiple reference points. The reference point may be any position within the camera field of view, or may be a specific defined position. For example, the reference point may be defined as a position on the surface of at least one virtual sphere that is concave with respect to the camera. At each reference point in this scenario, the robot arm may be controlled to position the verification symbol such that the verification symbol is positioned tangentially to the surface of the at least one virtual sphere when facing the camera. This positioning may allow the verification symbol to be better photographed or otherwise captured by the camera head-on (with the verification symbol directly facing the camera), so that the image of the verification symbol is closer to a plan view rather than a perspective view of the verification symbol. For example, if the verification symbol is a ring pattern, the ring pattern may be positioned tangentially to the surface of the virtual sphere so that the resulting ring pattern image still appears circular rather than appearing elliptical. The resulting image may exhibit little perspective distortion (for scenarios where the ring pattern appears elliptical in the image) or no perspective distortion at all. The absence of perspective distortion may aid in accurate identification of the center coordinates of the ring pattern. In some cases, the reference point may be divided into multiple virtual spheres that are all concave with respect to the camera. The multiple virtual spheres may share a common center and may be of different sizes, such that each virtual sphere has a spherical surface with a different distance from the camera. In some cases, the camera may be the common center for all virtual spheres.

[0021] FIG. 1A illustrates a block diagram of a robotic operation system 100 (also referred to as system 100) for performing automatic camera calibration and automatic verification of the camera calibration. Although some of the following examples discuss performing automatic camera calibration and verifying camera calibration information determined from the automatic camera calibration, these examples may apply more generally to any type of automatic calibration operation and verifying any type of calibration information determined from the automatic calibration operation. The robotic operation system 100 includes a robot 150, a robotic control system 110 (also referred to as a robot controller), and a camera 170. In one embodiment, the system 100 may be located in a warehouse, a manufacturing plant, or other facility. The robotic control system 110 may be configured to perform camera calibration, which will be discussed in more detail below, and to identify camera calibration information, which is later used to control the robot 150 to perform a robotic operation, such as lifting a load, in a warehouse. The robotic control system 110 may be further configured to perform camera calibration verification, which will also be discussed in more detail below, to verify whether the camera calibration information is still sufficiently accurate. In some cases, the robotic control system 110 is configured to perform camera calibration and control the robot 150 to perform robotic operations based on the camera calibration information. In some cases, the robotic control system 110 can form a single device (e.g., a single console or a single computer) that communicates with the robot 150 and the camera 170. In some cases, the robotic control system 110 can include multiple devices.

[0022] In some cases, the robot control system 110 may be a dedicated system for performing camera calibration and / or verification of camera calibration, and may communicate the latest camera calibration information to another control system (also referred to as another controller, not shown), which then controls the robot 150 to perform a robotic operation based on the latest camera calibration information. The robot 150 may be positioned based on images captured by the camera 170 and based on the camera calibration information. More specifically, the robot control system 110, in one embodiment, may be configured to generate motion commands based on the images and based on the camera calibration information, and to communicate the motion commands to the robot 150 to control the motion of its robotic arm. In some cases, the robot control system 110 is configured to perform verification of camera calibration during idle periods during robotic operations. In some cases, the robot control system 110 is configured to perform verification while performing robotic operations with the robot 150.

[0023] In one embodiment, the robot control system 110 can be configured to communicate with the robot 150 and the camera 170 via wired or wireless communication. For example, the robot control system 110 can be configured to communicate with the robot 150 and / or the camera 170 via an RS-232 interface, a Universal Serial Bus (USB) interface, an Ethernet interface, a Bluetooth interface, an IEEE 802.11 interface, or any combination thereof. In one embodiment, the robot control system 110 can be configured to communicate with the robot 150 and / or the camera 170 via a local computer bus, such as a Peripheral Component Interconnect (PCI) bus.

[0024] In one embodiment, the robot control system 110 can be separate from the robot 150 and can communicate with the robot via a wireless or wired connection as discussed above. For example, the robot control system 110 can be a stand-alone computer configured to communicate with the robot 150 and the camera 170 via a wired or wireless connection. In one embodiment, the robot control system 110 can be an integral component with the robot 150 and can communicate with other components of the robot 150 via a local computer bus as discussed above. In some cases, the robot control system 110 can be a dedicated control system (also referred to as a dedicated controller) that controls only the robot 150. In other cases, the robot control system 110 can be configured to control multiple robots, including the robot 150. In one embodiment, the robot control system 110, the robot 150, and the camera 170 are located within the same facility (e.g., a warehouse). In one embodiment, the robot control system 110 may be remote from the robot 150 and the camera 170 and may be configured to communicate with the robot 150 and the camera 170 via network communications (e.g., a local area network (LAN) connection).

[0025] In one embodiment, the robot control system 110 can be configured to retrieve or otherwise receive from the camera 170 images of the calibration pattern 160 and / or the verification symbol 165 located on the robot 150 (e.g., on the robot arm of the robot). In some cases, the robot control system 110 can be configured to control the camera 170 to capture such images. For example, the robot control system 110 can be configured to generate a camera command that causes the camera 170 to capture an image of the field of view of the camera 170 (also referred to as the camera field of view) and communicate the camera command to the camera 170 via a wired or wireless connection. The same command can also cause the camera 170 to communicate the image to the robot control system 110, or more generally, to a storage device accessible by the robot control system 110. Alternatively, upon receiving the camera command, the robot control system 110 can generate another camera command that communicates the image(s) captured by the camera 170 to the robot control system 110. In one embodiment, camera 170 can automatically capture images within its camera field of view periodically or in response to defined trigger conditions without requiring a camera command from robotic control system 110. In such an embodiment, camera 170 can also be configured to automatically communicate images to robotic control system 110, or more generally to a storage device accessible by robotic control system 110, without the use of a camera command from robotic control system 110.

[0026] In one embodiment, the robot control system 110 can be configured to control the movement of the robot 150 via movement commands generated by the robot control system 110 and communicated to the robot 150 via a wired or wireless connection. The robot 150 can be configured to have one or both of a calibration pattern 160 and a verification symbol 165 on the robot 150. For example, FIG. 1B illustrates a robot manipulation system 100A in which a verification symbol 165 is placed on the robot 150 in the absence of the calibration pattern 160 of FIG. 1A. In one example, the verification symbol 165 can be part of the robot 150 and can be permanently placed on the robot 150. For example, the verification symbol 165 can be permanently applied to the robot 150 or can be part of a sticker or board that is permanently attached to the robot 150. In another example, the verification symbol 165 can be a separate component that is removable from the robot 150. The verification symbol 165 can be permanently placed on the robot 150 or can be a separate component that is removable from the robot 150.

[0027] In one embodiment, the only images used in the system 100 to control the robot 150 may be those captured by the camera 170. In another embodiment, the system 100 may include multiple cameras and the robot 150 may be controlled by images from the multiple cameras.

[0028] 1B further illustrates an embodiment in which the robot control system 110 communicates with a user interface device 180. The user interface device 180 can provide an interface with an operator of the robot 150, such as an employee at a warehouse in which the robot 150 is located. The user interface device 180 can include, for example, a tablet computer or a desktop computer that provides a user interface that displays information related to the operation of the robot 150. As mentioned above, the robot control system 110 can be configured to detect when a deviation parameter value exceeds a defined deviation threshold. In one embodiment, the user interface device 180 can provide an alarm or other warning to notify an operator that a deviation parameter value exceeds a defined deviation threshold.

[0029] 1C illustrates a block diagram of a robotic control system 110. As shown in the block diagram, the robotic control system 110 includes control circuitry 111, a communication interface 113, and a non-transitory computer-readable medium 115 (e.g., memory). In one embodiment, the control circuitry 111 can include one or more processors, programmable logic circuits (PLCs) or programmable logic arrays (PLAs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or any other control circuitry.

[0030] In one embodiment, the communication interface 113 can include one or more components configured to communicate with the camera 170 of FIG. 1A or FIG. 1B and the robot 150 of FIG. 1A or FIG. 1B. For example, the communication interface 113 can include communication circuitry configured to perform communication via a wired protocol or a wireless protocol. By way of example, the communication circuitry can include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth controller, a PCI bus controller, any other communication circuitry, or a combination thereof. In one embodiment, the control circuitry 111 can be configured to generate motion commands (e.g., motor motion commands) and output the motion commands to the communication interface 113. In this embodiment, the communication interface 113 can be configured to communicate the motion commands to the robot 150 to control the motion of a robot arm or other components of the robot 150. In one embodiment, the control circuitry 111 can be configured to generate camera commands and output the camera commands (e.g., image capture commands) to the communication interface 113. In this embodiment, communications interface 113 may be configured to communicate camera commands to camera 170 to control camera 170 to photograph or otherwise capture images of objects within the camera's field of view. In one embodiment, communications interface 113 may be configured to receive images or other data from camera 170, and control circuitry 111 may be configured to receive images from communications interface 113.

[0031] In one embodiment, the non-transitory computer-readable medium 115 can include computer memory. The computer memory can include, for example, dynamic random access memory (DRAM), solid state integrated memory, and / or a hard disk drive (HDD). In some cases, the camera calibration can be performed through computer-executable instructions (e.g., computer code) stored on the non-transitory computer-readable medium 115. In such a case, the control circuitry 111 can include one or more processors configured to execute the computer-executable instructions and perform the camera calibration verification (e.g., the steps illustrated in FIGS. 4A, 4B, and 9).

[0032] FIG. 1D illustrates a block diagram of a camera 170 including one or more lenses 171, an image sensor 173, and a communication interface 175. The communication interface 175 can be configured to communicate with the robotic control system 110 of FIG. 1A, FIG. 1B, or FIG. 1C and can be similar to the communication interface 113 of FIG. 1C of the robotic control system 110. In one embodiment, the one or more lenses 171 can focus light coming from outside the camera 170 onto the image sensor 173. In one embodiment, the image sensor 173 can include an array of pixels configured to represent an image via respective pixel intensity values. The image sensor 173 can include a charge-coupled device (CCD) sensor, a complementary metal-oxide semiconductor (CMOS) sensor, a quantum image sensor (QIS), or any other image sensor.

[0033] As mentioned above, camera calibration can be performed to facilitate control of the robot based on images captured by the camera. For example, FIG. 2 illustrates a robot manipulation system 200 (also referred to as system 200) that uses images to control a robot 250 to perform a robotic operation, such as lifting an object 292 in a warehouse. More specifically, the system 200 can be an embodiment of the system 100 of FIG. 1A and includes a camera 270, a robot 250, and a robot control system 110. The camera 270 can be an embodiment of the camera 170 of FIG. 1A, FIG. 1B, or FIG. 1D, and the robot 250 can be an embodiment of the robot 150 of FIG. 1A or FIG. 1B. The camera 270 can be configured to capture images of an object 292 (e.g., a package for shipping) disposed on a conveyor belt 293 in a warehouse, and the robot control system 110 can be configured to control the robot 250 to lift the object 292. When one or more objects are present on the conveyor belt 293, the robot control system 110 can be configured to schedule operation of the robot 250 to lift the objects. The robot control system 110 can be configured to detect idle periods for robot operations, in some cases, by detecting when there are no objects present on the conveyor belt 293 or when there are no objects present on the conveyor belt 293 within reach of the robot 250.

[0034] In the embodiment of FIG. 2, the robot 250 can have a base 252 and a robot arm operable relative to the base 252. More specifically, the robot arm can include a number of links 254A-254E and a robot hand 255 attached to the link 254E. The links 254A-254E can be pivotable relative to one another and / or can be prismatic links that can move linearly relative to one another. Since FIG. 2 pertains to a robot 250 used to lift an object, the robot hand 255 can include grippers 255A and 255B that are used to grasp the object 292. In one embodiment, the robot control system 110 can be configured to communicate motion commands to rotate one or more of the links 254A-254E. The motion commands can be low-level commands, such as motor motion commands, or high-level commands. If the motion commands from the robot control system 110 are high-level commands, the robot 150 can be configured to translate the high-level commands into low-level commands.

[0035] In one embodiment, the camera calibration information determined from the camera calibration describes a relationship between the camera 270 and the robot 250, or more specifically, a relationship between the camera 270 and a world point 294 that is stationary relative to the base 252 of the robot 250. The world point 294 may represent the world or other environment in which the robot 250 is located and may be any imaginary point that is stationary relative to the base 252. In other words, the camera calibration information may include information that describes a relationship between the camera 270 and the world point 294. In one embodiment, the relationship may refer to a position of the camera 270 relative to the world point 294 and an orientation of the camera 270 relative to a reference direction for the robot 250. The above relationship between the camera 270 and the world point 294 may be referred to as a camera-world relationship and may be used to represent a relationship between the camera 270 and the robot 250. In some cases, the camera-world relationship can be used to determine the relationship between the camera 270 and the object 292 (also referred to as the camera-object relationship) and the relationship between the object 292 and world points 294 (also referred to as the object-world relationship). The camera-object relationship and the object-world relationship can be used to control the robot 250 to lift the object 292.

[0036] In one embodiment, the camera calibration information may describe intrinsic parameters of the camera 270, which may be any parameters whose values ​​are independent of the position and orientation of the camera 270. The intrinsic parameters may characterize properties of the camera 270, such as the focal length of the camera, the size of the camera's image sensor, or the effects of lens distortion introduced by the camera 270.

[0037] An embodiment showing a detailed structure of an embodiment of a robot 350 is illustrated in FIG. 3, which illustrates a robot manipulation system 300 including a camera 370 and a robot control system 110 in communication with the robot 350. The camera 370 can be an embodiment of the camera 170 / 270 of FIG. 1A, FIG. 1B, FIG. 1D, or FIG. 2, respectively, and the robot 350 can be an embodiment of the robot 150 / 250 of FIG. 1A, FIG. 1B, or FIG. 2, respectively. The camera 370 may be capable of capturing an image within a camera field of view 330. The robot 350 can include a base 352 and a robot arm that is movable relative to the base 352. The robot arm includes one or more links, such as links 354A-354E, and a robot hand 355. In one embodiment, the links 354A-354E can be pivotally attached to one another. For example, the link 354A can be pivotally attached to the robot base 352 via a joint 356A. The remaining links 354B-354E can be pivotally attached to each other via joints 356B-356E. In one embodiment, the base 352 can be used to mount the robot 350, for example, to a mounting frame or a mounting surface (e.g., a warehouse floor). In one embodiment, the robot 350 can include a plurality of motors configured to move the robot arm by rotating the links 354A-354E. For example, one of the motors can be configured to rotate the first link 354A relative to the joint 356A and the base 302, as shown by the dashed arrow in FIG. 3. Similarly, the other motors of the plurality of motors can be configured to rotate the links 354B-354E. The plurality of motors can be controlled by the robot control system 110. FIG. 3 further illustrates a robot hand 355 positioned to be fixed on the fifth link 354E.The robotic hand 355 can have a calibration pattern 320 thereon such that the robotic control system 110 can capture an image of the calibration pattern 320 via the camera 370 and perform camera calibration based on the captured image of the calibration pattern 320. For example, the robotic control system 110 can move the robot arm such that the calibration pattern 320 can be within the camera field of view 330 and is visible to the camera 370 when the camera 370 is used to capture an image of the calibration pattern 320 (also referred to as a calibration image). After the camera calibration is performed, the robotic hand 355 can be removed and replaced with another robotic hand, such as a robotic hand having a verification symbol disposed thereon, as discussed in more detail below.

[0038] As mentioned above, according to one embodiment, calibration verification can include comparing a reference image coordinate where a verification symbol appears in the reference image with a verification image coordinate where a verification symbol appears in the verification image. The comparison can determine a deviation between the verification image coordinate and the reference image coordinate, and the deviation can be used to determine whether a further calibration operation should be performed. The verification image can be captured during a quiescent period of robotic manipulation. FIGS. 4A and 4B illustrate a flow diagram illustrating a method 400 for verification of camera calibration, according to one embodiment. Although some of the following embodiments discuss verifying camera calibration information determined from a camera calibration, the method 400 can be used more generally to verify calibration information determined from any calibration operation performed for a robotic manipulation system. In one embodiment, the method 400 can be performed by the control circuitry 111 of the robotic control system 110. As mentioned above, the robot control system 110 can include a communication interface 113 of Figure 1C configured to communicate with the robot 150 of Figure 1A or 1B and the camera 170 of Figure 1A, 1B, or 1D. In one embodiment, the robot can have a base (e.g., base 252 of Figure 2 or base 352 of Figure 3) and a robot arm (e.g., the robot arm of Figure 2 or 3) on which the verification symbol is disposed, the robot arm being operable relative to the base.

[0039] An exemplary environment in which method 400 may be implemented is illustrated in Figures 5A and 5B, which illustrate a robot manipulation system 500 / 500A, each including a robot control system 110 in communication with a camera 570 and a robot 550. The camera 570 may be an embodiment of the camera 170 / 270 / 370 of Figures 1, 2, or 3, respectively, and the robot 550 may be an embodiment of the robot 150 / 250 / 350 of Figures 1A, 1B, 2, or 3, respectively. The robot 550 may include a base 552 and a robot arm (labeled 553 in Figures 5A and 5B) operable relative to the base 552. The robot arm includes one or more links, such as links 554A-554E. Links 554A-554E may also be examples of arm portions of a robot arm 553 that are operably attached to one another. In one embodiment, the links 554A-554E can be pivotally attached to one another. For example, the link 554A can be pivotally attached to the robot base 552. The remaining links 554B-554E can be pivotally attached to one another via multiple joints. In one embodiment, the base 552 can be used to mount the robot 550 to, for example, a mounting frame or a mounting surface (e.g., a warehouse floor). The robot 550 can be operated in a similar manner to the robot 350 of FIG. 3. For example, the robot 550 can include multiple motors configured to move the robot arm by rotating the links 554A-554E relative to one another. The robot arm can further include a robot hand attached to the link 554E. For example, FIG. 5A illustrates a first robot hand 555, a second robot hand 557, and a third robot hand 559, each of which can be detachable from the fifth link 554E. The robotic hand 555 / 557 / 559 can include, for example, a gripper or a suction device configured to lift an object (e.g., 582A, 582B, 582C) from the conveyor belt 573. When the robotic hand 555 / 557 / 559 is attached to the fifth link 554E, the attachment can be made in a mixed manner.The attachment and detachment can be performed manually or automatically. In one example, the fifth link 554E can be attached to a first robotic hand 555 as shown in Figures 5A and 5B, and the robot control system 110 can control the robot 550 to cause the fifth link 554E to release the first robotic hand 555 and attach the fifth link 554E to a second robotic hand 557. In another embodiment, the fifth link 554E can be permanently attached to a robotic hand (e.g., robotic hand 559).

[0040] In one embodiment, the robot 550 can have a verification symbol 530 disposed thereon. In some cases, the verification symbol 530 can be permanently disposed on the robot 550. In some cases, the verification symbol 530 can be disposed on a robot arm or on a robot hand of the robot 550, such as on one of the links 554A-554E. For example, FIG. 5A illustrates the verification symbol 530 disposed on the first robot hand 555 and the third robot hand 559, while FIG. 5B illustrates the verification symbol 530 disposed on the link 554E. The verification symbol 530 can be applied directly on the robot 550 or can be attached to the robot 550 via a sticker, flat plate, or the like. In the embodiment illustrated in FIG. 5A, the second robotic hand 557 or the third robotic hand 559 may be used to perform camera calibration as each has a respective calibration pattern 520 / 527 disposed thereon, while the first robotic hand 555 or the third robotic hand 559 may be used to perform verification of the camera calibration as each has a respective verification symbol 530 disposed thereon.

[0041] Returning to FIG. 4A, in one embodiment, method 400 may begin at step 401, where control circuitry 111 performs a first camera calibration to determine camera calibration information associated with a camera (e.g., camera 170 / 270 / 370 / 570 of FIG. 1, FIG. 2, FIG. 3, or FIG. 5, respectively). More specifically, the camera calibration information may include camera calibration values ​​for the camera. In this embodiment, control circuitry 111 may perform the first camera calibration based on an image of a calibration pattern (also referred to as a calibration image).

[0042] For example, to perform the first calibration, the robot 550 of FIG. 5A can be attached to a second robot hand 557 having a calibration pattern 520 or to a third robot hand 559 having a calibration pattern 527. FIG. 3 illustrates a similar environment in which the first camera calibration can be performed. During this step, the first robot hand 555, whose calibration pattern 320 is used to perform the camera calibration, can be detached from the fifth link 554E. The first camera calibration can be performed before starting a robot operation. For example, the robot operation can start with a robot task such as the first robot hand 555 interacting with a first object 582A on a conveyor belt. During the first camera calibration, the robot 550 can be equipped with the second robot hand 557. The robot control system 110 can control the robot arm of the robot 550 via movement commands to move the calibration pattern 520 to various positions within the camera field of view 510 of the camera 570 and capture respective images of the calibration pattern 520 at such positions. The robotic control system 110 can perform a first camera calibration based on the captured image of the calibration pattern 520 to determine camera calibration information for the camera 570. In one example, the camera calibration information can include information describing a relationship between the camera 570 and the robot 550. In one example, the camera calibration information can describe intrinsic parameters of the camera 570. Camera calibration is discussed in more detail in U.S. Patent Application Serial No. 16 / 295,940, filed March 7, 2019 (Attorney Docket No. MJ0021US1), entitled “METHOD AND DEVICE FOR PERFORMING AUTOMATIC CAMERA CALIBRATION FOR ROBOT CONTROL,” the entire contents of which are incorporated herein by reference.

[0043] Returning to FIG. 4A, the method 400 may further include step 403, in which the control circuit 111 controls the robot arm to move the verification symbol (e.g., 530 in FIG. 5) to a location within the camera field of view (e.g., 510) of the camera (e.g., 570) during or after the first camera calibration by outputting a first motion command to the communication interface 113 of the robot control system 110. The communication interface 113 may be configured to communicate a motion command to the robot arm for the robot arm to move the verification symbol (e.g., 530) to that location within the camera field of view (e.g., 510) during or after the first camera calibration. The motion command may also cause the robot arm to orient the verification symbol to face the camera (e.g., 570) or, more generally, to be visible to the camera. The location may be used as one of one or more reference locations for verification of the first camera calibration. For example, as the verification process acquires images of the verification symbol over time, the control circuitry 111 can control the robot arm to continually position the verification symbol (e.g., 530) at the same one or more locations so that the one or more locations can be used as one or more reference locations. Additionally, as described below with respect to steps 405-459, the verification process can compare later images of the verification symbol to a set of earlier images of the verification symbol (e.g., 530), such as images obtained immediately after a first camera calibration is performed. The later images can be used as verification images, while the images to which the later images are compared can be used as reference images.

[0044] In step 405, the control circuitry 111 may receive (e.g., retrieve) an image of the verification symbol (e.g., 530) from the camera (e.g., 170 / 270 / 370 / 570) via the communication interface 113, which is a reference image for verification. The image may have been captured by the camera while the verification symbol was at the reference location or was at the reference location. In one embodiment, the communication interface 113 may first receive the reference image from the camera, and then the control circuitry 111 may receive the reference image from the communication interface 113. In one embodiment, step 405 is performed without the control circuitry 111 generating a camera command for the camera. In one embodiment, step 405 may include the control circuitry 111 generating a camera command and communicating the camera command to the camera via the communication interface 113. The camera command may control the camera to capture an image of the verification symbol at the reference location.

[0045] 5A-6B illustrate aspects of steps 403 and 405. In the embodiment of FIG. 5A, after a first camera calibration is performed, for example, with a second robotic hand 557, the second robotic hand 557 can be replaced with a third robotic hand 559 having a verification symbol 530 disposed thereon. In this example, the robot control system 110 controls the robot arm of the robot 550 (e.g., via one or more motion commands) to move the verification symbol 530 to one or more reference locations within the camera field of view 510 of the camera 570. The one or more reference locations can include any location within the camera field of view 510, or can be a set of one or more specific locations, such as locations disposed on a surface of a virtual sphere, as discussed in more detail below. In another example, in the embodiment of FIG. 5B, during or after the first camera calibration, the robot control system 110 can control the robot arm to move the verification symbol 530 to one or more reference locations within the camera field of view 510. In this example, the one or more reference locations can include any location where the verification symbol 530 was photographed (with the calibration pattern 520) during the first camera calibration, or can be a set of one or more specific locations to which the verification symbol 530 was moved after the first camera calibration was performed. The robot control system 110 can control the movement of the robot arm of the robot 550 in this step with instructions from the camera 570 based on the camera calibration information obtained from the first camera calibration, or can control the movement without such instructions. In one embodiment, the reference locations can be defined locations that can be stored and retrieved in a local or remote storage device. The reference locations can be stored in the form of coordinates (e.g., Cartesian coordinates), or as motor commands for rotating the links 554A-554E, or in some other manner.

[0046] In one embodiment, the one or more reference locations to which the robot arm moves the verification symbol (e.g., 530) can include multiple reference locations, each of the multiple reference locations being a location located on a surface of a virtual sphere that is concave with respect to the camera. In such an embodiment, the control circuitry 111 can be further configured to control the robot arm to move the verification symbol at each of the multiple reference locations to be tangential to the surface of the virtual sphere. For example, as illustrated in FIGS. 6A, 6B, 6C, and 6D, the robot control system 110 can control the robot arm of the robot 550 to move the verification symbol 530 to the reference locations 610A-610I and can control the camera 570 to capture a respective reference image at each of the reference locations 610A-610I. The reference locations 610A-610I in FIGS. 6A and 6B can be divided among multiple virtual spheres within the camera field of view 510. The reference points 610A and 610B may be located on a first spherical surface 621 of a first virtual sphere 620, where the first spherical surface 621 is within the camera field of view 510. The reference points 610C, 610D, and 610E may be located on a second spherical surface 631 of a second virtual sphere 630, where the second spherical surface 631 is within the camera field of view 510. The reference points 610F, 610G, 610H, and 610I may be located on a third spherical surface 641 of a third virtual sphere 640, where the third spherical surface 641 is within the camera field of view 510. As illustrated in Figures 6A and 6B, the first spherical surface 621, the second spherical surface 631, and the third spherical surface 641 are each concave with respect to the camera 570. 6A and 6B show three spheres based on three spheres, the number of different spheres on which the reference points can be placed can be more or less than three. In one embodiment, a camera 570 can be at the center of each of the virtual spheres 620, 630, 640.

[0047] In one embodiment, as shown in Figures 6A-6D, when the verification symbol 530 is moved to a reference location, the robot control system 110 can control (e.g., via motion commands) the robot arm 553 of the robot 550 to position the verification symbol 530 tangentially to the spherical surface on which the reference location is located. For example, Figures 6A and 6B illustrate the verification symbol 530 oriented tangentially to the second spherical surface 631 at the reference location 610D, and Figures 6C and 6D illustrate the verification symbol 530 oriented tangentially to the second spherical surface 631 at the reference location 610C. More specifically, the verification symbol 530 may be disposed on a plane (e.g., a sticker) and the plane of the verification symbol 530 may be tangential to the second spherical surface 631 at reference 610D in FIG. 6A and FIG. 6B and tangential to the second spherical surface 631 at reference 610C in FIG. 6C and FIG. 6D. In one embodiment, the robot arm 553 may be in a first pose in FIG. 6A and FIG. 6B and in a second pose in FIG. 6C and FIG. 6D. The pose of the robot arm 553 may refer, for example, to a shape formed by the links or other arm portions of the robot arm 553, or more generally to a geometric shape. For example, the pose of the robot arm 553 may refer to a particular permutation of the angle or distance that the links of the robot arm 553 have rotated or translated (e.g., extended or retracted) relative to the preceding links of the robot arm 553. As an example, the first pose illustrated in Figure 6A may correspond to a first permutation of angles formed between successive links of the series of links of the robot arm 553, while the second pose illustrated in Figure 6C may correspond to a second permutation of angles between successive links of the series of links of the robot arm 553. In such an example, the reference 610D of the verification symbol 630 may be associated with the first pose for the robot arm 553 illustrated in Figure 6A, while the reference 610C of the verification symbol 630 may be associated with the second pose for the robot arm 553 as illustrated in Figure 6C.

[0048] In one embodiment, the control circuitry 111 is configured to control the robot arm to move the verification symbol (e.g., 530) to directly face the camera when the verification symbol is moved to the reference location. For example, as illustrated in FIG. 6A, the robot control system 110 can control the robot arm 553 of the robot 550 to move the verification symbol 530 to directly face the camera 570 when the verification symbol 530 is moved to the reference location 610D. In this example, the robot control system 110 can control the robot hand 555 to rotate the verification symbol 530 to directly face the camera 570. In some cases, the verification symbol can directly face the camera 570 by facing tangentially to the sphere in the camera field of view 510. When the verification symbol 530 directly faces the camera 570, the camera 570 may be able to capture the verification symbol 530 from the front, which results in no or less squint effect on the image of the verification symbol 530.

[0049] In one embodiment, the verification symbol (e.g., 530) includes a first region having a first color and a second region having a second color, and a ratio between the area of ​​the first region and the area of ​​the second region is specified and stored on a non-transitory computer-readable medium (e.g., a storage device) of the robot control system 110. In such an embodiment, the control circuitry 111 can be configured to identify the verification symbol in the reference image or the verification image based on the specified ratio. For example, as illustrated in FIG. 5C, the verification symbol 530 can be ring-shaped and include a first region 531 having a first color (e.g., a black region), surrounded by the first region 531 and including a second region 533 having a second color (e.g., a white region). The ratio between the area of ​​the black first region 531 and the area of ​​the white second region 533 in the verification symbol 530 can be a specified value that can be identified. By analyzing the colors in the captured image, the robot control system 110 may be able to identify the portion of the image that corresponds to the verification symbol 530 by determining whether the portion is a ring-like region surrounding a circular region and determining whether the ratio between the area of ​​the ring-like region and the area of ​​the circular region matches a prescribed ratio. This may allow the robot control system 110 to distinguish the verification symbol 530 from other features captured in the image. For example, as illustrated in FIG. 5A, the robot 550 may be configured to utilize a third robot hand 559 having a combination of a calibration pattern 527 and a verification symbol 530. In this example, the reference image may show both the verification symbol 530 and the calibration pattern 527. In this example, the calibration pattern 527 may not have any ring patterns or may have ring patterns with a different ratio than the prescribed ratio discussed above. The control circuit 111 can determine whether a portion of the reference image is a verification symbol 530 or a calibration pattern 527 by determining whether the portion of the reference image has a first image region having a first color and a second image region having a second color, and determining whether the ratio of the area of ​​the first image region to the area of ​​the second image region is equal to a specified ratio.

[0050] In some cases, the robot control system 110 can determine whether a particular portion of the captured image has a first region having a first color and a second region having a second color, and determine whether the ratio of the area of ​​the first region to the area of ​​the second region is within a specified range. In one example, if the specified ratio is 1.5, the robot control system 110 can determine that the particular region corresponds to the verification symbol 530 if the ratio in the particular region falls within a range of 1.4 to 1.6. The two colors of the first region and the second region are not limited to black and white, and can be any two different colors that are distinguishable by the robot control system 110.

[0051] In one embodiment, the verification symbol (e.g., 530) can include a first shape and a second shape that are concentric with one another, with the respective centers of the first and second shapes being at substantially the same location. For example, the verification symbol can be formed as a circular ring, with the circular ring including a first circle and a second circle that are concentric with one another. More specifically, as shown in FIG. 5C, the verification symbol 530 can include a first shape 535 (e.g., an outer circle) and a second shape 537 (e.g., an inner circle). The first shape 535 and the second shape 537 can be concentric with one another, such that the center of the first shape 535 and the center of the second shape 537 are at substantially the same location. For example, the center of the first shape 535 is located at the coordinate

number

number

number

number

[0052] Returning to Figure 4A, method 400 may further include step 407, in which control circuitry 111 determines reference image coordinates for a verification symbol, the reference image coordinates being coordinates at which the verification symbol (e.g., 530) appears in the reference image. For example, as illustrated in Figure 6A, an image of verification symbol 530 may be captured at reference location 610D and used as the reference image. Verification symbol 530 may appear in the reference image at particular coordinates, which may be referred to as reference image coordinates.

[0053] In one embodiment, as discussed above, a verification symbol (e.g., 530) may include a first shape and a second shape that are concentric with one another, with the centers of the first and second shapes being at substantially the same location. In such an embodiment, control circuitry 111 may be configured to determine reference image coordinates for such a verification symbol in step 407 by determining a first coordinate of a center of the first shape in the reference image, determining a second coordinate of a center of the second shape in the reference image, and determining the reference image coordinates as an average of the first and second coordinates in the reference image.

[0054] For example, FIG. 7A illustrates a reference image 710 captured at reference location N of the reference locations, where N is an integer. The reference image 710 includes a verification portion 730, which is an image portion in the reference image 710 that shows the verification symbol 530 of FIG. 5A, 5B, or 5C. The robot control system 110 of FIG. 1A or 1B may be configured to identify a first shape 735 (e.g., an outer circle) from the verification portion 730 that is the same as or substantially the same as the first shape 535 of the verification symbol 530 of FIG. 5C. The robot control system 110 may be further configured to identify a second shape 737 (e.g., an inner circle) from the verification portion 730 that is the same as or substantially the same as the second shape 537 of the verification symbol 530 of FIG. 5C. Then, for reference location N, the robot control system 110 may determine a first coordinate of a center of the first shape 735 shown in the reference image 710.

number

number

number

number

number

number

[0055] In one embodiment, the reference image coordinate for the verification symbol may be its center coordinate, and determining the center coordinate of the verification symbol 530 based on the center coordinates of the first shape 735 and the second shape 735 may improve the robustness of the verification process against image noise. For example, image noise may introduce errors in determining the center coordinate for the first shape 735 but may not introduce errors in determining the center coordinate for the second shape 737. In some cases, the second shape 737 may actually share the same center location as the first shape 735, but image noise may cause the center coordinate of the second shape 737 to appear different in the image than the center coordinate of the first shape 735. In this scenario, simply using the center coordinate of the second shape 737 as the center coordinate of the verification symbol 530 may result in an undesirable amount of error. The amount of error may be reduced by using the average of the center coordinate for the first shape 735 and the center coordinate for the second shape 737 as the center coordinate of the verification symbol 530.

[0056] In one embodiment, the one or more reference locations discussed above may be multiple reference locations corresponding to multiple reference image coordinates, respectively. In this embodiment, the reference image coordinate may be one of the multiple reference image coordinates. For example, as illustrated in FIG. 6A and FIG. 6B, there may be multiple reference locations, such as reference locations 610A-610I, at which the verification symbol 530 is moved or otherwise positioned. For each reference location 610A-610I of the verification symbol 530, the robot control system 110 may retrieve or otherwise receive a respective reference image captured by the camera 570 of the verification symbol 530 at that location, and may identify respective reference image coordinates indicating where the verification symbol 530 appears within the respective reference image.

[0057] Returning to FIG. 4A, the method 400 may further include step 409, in which the control circuitry 111 controls the movement of the robot arm to perform a robotic operation based on the calibration information. In one embodiment, this step may include the control circuitry 111 generating a second movement command based on the camera calibration information and outputting the second movement command to the communication interface 113. The communication interface 113 may then communicate the second movement command to the robot to control the movement of the robot arm. For example, as illustrated in FIG. 5A, after the first camera calibration, the robot control system 110 controls the robot 550 to perform a robotic operation including a robotic task, such as lifting objects 582A, 582B, and 582C. The movement of the robot 550 may be based on the camera calibration information obtained from the first camera calibration and based on images of the objects 582A, 582B, 582C captured by the camera 570.

[0058] In step 411, the control circuitry 111 detects a pause period during the robotic operation. In one aspect, a pause period of the robot may be a period during the robotic operation during which the robot is not performing a robotic task. In some cases, if the robotic operation is based on picking up an object from the conveyor belt 573, the pause period may be based on the absence of an object on the conveyor belt 573. More specifically, the conveyor belt 573 may be reachable by the robotic arm 553, and the control circuitry 111 is configured to detect the pause period by detecting the absence of an object on the conveyor belt 573 or the distance between the robot 550 and the nearest object on the conveyor belt 573 exceeding a defined distance threshold. In some cases, the control circuitry 111 may receive a signal indicating that a pause period is about to occur, which may be received from another device or component that monitors the robotic operation. For example, as illustrated in FIG. 5A, the robot control system 110 can detect a pause during a robot operation between a robot task including lifting a second object 582B and a robot task including lifting a third object 582C due to a long distance between the second object 582B and the third object 582C. During this pause period, the robot can have a pause period during which it is not performing a robot task because after the robot 550 lifts the second object 582B, the object 582C is not yet reachable by the robot 550. In one example, the robot control system 110 can detect a pause period when an object on the conveyor belt 573 is not reachable by the robot 550 and / or when the robot control system 110 determines that the distance between the robot 550 and the closest object upstream on the conveyor belt 573 (e.g., the third object 582C) exceeds a certain threshold.

[0059] Returning to FIG. 4A and FIG. 4B, the method 400 may further include step 451, in which the control circuitry 111 controls the robot arm 553 to move the verification symbol 530 to at least the reference location used in step 403 (used to capture the reference image) during the idle period. In one embodiment, step 451 may include the control circuitry 111 generating a third motion command and outputting the third motion command to the communication interface 113. The communication interface 113 may be configured to communicate the third motion to the robot such that the robot arm 553 moves based on the motion command. In some cases, the third motion command may include a set of stored motor commands corresponding to the reference location. In some cases, the third motion command may be generated based on the camera calibration information from step 401. In some cases, the third motion command of step 451 does not rely on the camera calibration information from step 401.

[0060] In step 453, the control circuitry 111 reads or otherwise receives a further image of the verification symbol (e.g., 530) from the camera (e.g., 570) during the dwell period, the further image being a verification image for verification, an image of the verification symbol at least at the reference location during the dwell period. That is, the verification image for the reference location is captured while the verification symbol (e.g., 530) is at or was at the reference location. In one embodiment, step 453 includes the control circuitry 111 generating a camera command to control the camera (e.g., 570) to capture the verification image. The control circuitry 111 can output the camera command to the communication interface 113, which can communicate the camera command to the camera (e.g., 570). In one embodiment, step 451 can include controlling the robot arm to move the verification symbol to a plurality of reference locations and receiving a plurality of respective verification images captured by the camera. For example, as illustrated in Figures 6A and 6B, during the pause period, the robot control system 110 can control the robot arm 553 of the robot 550 to move the verification symbol 530 to one of the reference locations 610A-610I and capture an image of the verification symbol 530 at that location as a verification image. If the pause period has not yet ended, more specifically, if a sufficient amount of time remains within the pause period, the robot control system 110 can control the robot arm 553 of the robot 550 to move the verification symbol 530 to another location of the reference locations 610A-610I and capture an image of the verification symbol 530 at the other location as another verification image. If the pause period ends, the robot control system 110 can stop capturing the verification image. In that case, during each pause period, the robot control system 110 can control the robot arm 553 of the robot 550 to move the verification symbol 530 to one or more of the reference locations 610A-610I and capture a verification image at each of the one or more of the reference locations 610A-610I.

[0061] Returning to FIG. 4B, method 400 may further include step 455, in which control circuitry 111 determines a verification image coordinate used for verification, the verification image coordinate being the coordinate at which the verification symbol appears in the verification image. If the verification symbol (e.g., 530) is moved to multiple reference locations (e.g., 610A-610I), the camera (e.g., 570) may capture multiple verification images respectively corresponding to the multiple reference locations, and control circuitry 111 may identify multiple verification image coordinates respectively corresponding to the multiple verification images and respectively corresponding to the multiple reference locations. The multiple verification images may all be captured by the camera (e.g., 570) in a single dwell period (e.g., if the single dwell period is long enough for the robotic arm to move the verification symbol (e.g., 530) to all of the reference locations 610A-610I), or may be captured in multiple different dwell periods (e.g., if each dwell period is not long enough for the robotic arm to move the verification symbol 530 to all of the reference locations 610A-610I).

[0062] In one embodiment, the verification image coordinates can be determined in the same manner as the reference image coordinates. For example, the verification image coordinates can be the center coordinates of the verification symbol (e.g., 530) and can be determined as the average of the center coordinates of the first shape of the verification symbol (e.g., 530) in the verification image (e.g., 760) and the center coordinates of the second shape of the verification symbol. For example, FIG. 7B shows a verification image 760 captured at a reference location N of the reference locations. The verification image 760 shows a verification portion 780, which is an image portion in the verification image 760 showing the verification symbol 530. The robot control system 110 can identify a first shape 785 from the verification portion 780 that is the same as or substantially the same as the first shape 585 of the verification symbol 530 in FIG. 5C. The robot control system 110 can further identify a second shape 787 from the verification portion 780 that is the same as or substantially the same as the second shape 587 of the verification symbol 530. Additionally, the robot control system 110 determines the center coordinates of the first shape 785 shown in the verification portion 780 of the verification image 760.

number

number

number

number

[0063] Returning to FIG. 4B , method 400 may further include step 457, in which control circuit 111 determines a deviation parameter value based on an amount of deviation between the reference image coordinates of step 403 and the verification image coordinates of step 455, the reference image coordinates and the verification image coordinates both being associated with a reference location N. In one example, the deviation between the reference image coordinates and the verification image coordinates may be a distance between the reference image coordinates and the verification image coordinates. For example, the reference image coordinates at reference location N are

number

number

number

[0064] In an embodiment where the one or more reference locations are multiple reference locations, as discussed above, the control circuitry 111 can be configured to identify multiple verification image coordinates respectively corresponding to the multiple reference locations, where the verification image coordinate discussed above is one of the multiple verification image coordinates. In such an embodiment, the deviation parameter value is based on a respective deviation amount between the multiple reference image coordinates and the multiple verification image coordinates for the multiple reference locations, where each deviation amount among the respective deviation amounts is between (a) a reference image coordinate corresponding to a respective reference location of the multiple reference locations and (b) a verification image coordinate corresponding to the same reference location. The multiple verification image coordinates can be respective coordinates where the verification symbol appears in the multiple verification images, where the verification image discussed above is one of the multiple verification images. The control circuitry 111 can be configured to control the camera to capture all of the multiple verification images within one dwell period and / or can be configured to control the camera to capture the multiple verification images within different dwell periods.

[0065] 6A and 6B, when there are multiple reference locations, the robot control system 110 can determine a plurality of respective reference image coordinates corresponding to the plurality of reference locations, can determine a plurality of respective verification image coordinates corresponding to the plurality of reference locations, and can determine respective deviation amounts between the plurality of reference image coordinates and the plurality of verification image coordinates. The deviation parameter value can be based on the respective deviation amounts between the plurality of reference image coordinates and the plurality of verification image coordinates. For example, the deviation parameter can be an average of the respective deviation amounts, as follows:

number

[0066] In the above formula, N may refer to the Nth reference location, while M may refer to the total number of reference locations.

[0067] Returning to FIG. 4B, the method 400 may further include step 459, in which the control circuitry 111 determines whether the deviation parameter value exceeds a defined threshold (sometimes referred to as a defined deviation threshold). Moreover, in step 461, in response to the control circuitry 111 determining that the deviation parameter value exceeds the defined threshold, a second camera calibration may be performed to identify updated camera calibration information for the camera. For example, a deviation parameter value that exceeds a defined threshold may indicate that the camera calibration information for the camera is out of date and / or is likely to cause an undesirable amount of error in the robotic operation. Therefore, if the deviation parameter value exceeds the defined threshold, a second camera calibration may be performed for the camera to update the camera calibration information for the camera (e.g., 570). The second camera calibration may use the same technique as the first camera calibration, but may be based on images captured by the camera subsequent to the first camera calibration. In one example, if step 459 indicates that the deviation parameter value exceeds a defined threshold, the robot operation can be stopped or paused, and then a second camera calibration can begin to be performed, which can begin by capturing an image for the second camera calibration. After the second camera calibration is completed and the camera calibration information for the camera is updated, the robot control system 110 can resume the robot operation with the updated camera calibration information.

[0068] In one embodiment, the control circuitry 111 may be configured to control the robot to continue robot operation after the pause period without performing further camera calibration (e.g., by outputting a fourth movement command to the robot via the communications interface) in response to determining that the deviation parameter value does not exceed a defined threshold. Such a condition may indicate that the camera calibration information from step 401 is still sufficiently accurate and that robot operation can continue without being subject to undesirable amounts of error.

[0069] In one embodiment, the control circuitry 111 can be configured to determine the temperature of the environment in which the robot is located and adjust at least one of the defined deviation threshold (also referred to as redefining the deviation threshold) or the camera calibration information for the camera based on the measured temperature. For example, the control circuitry 111 can determine the temperature of the environment by measuring the temperature or by receiving temperature data from another device or component. In such an embodiment, the control circuitry 111 can be configured to adjust the defined threshold based on the measured temperature by setting the defined threshold to have a first value when the measured temperature is outside of a defined range and setting the threshold to have a second value lower than the first value when the measured temperature is within the defined range. For example, an excessively high or low temperature may cause changes in the camera. More specifically, the temperature change may affect the intrinsic parameters of the camera. For example, components within the camera may expand when the temperature increases and contract when the temperature decreases, which may affect the intrinsic parameters of the camera. Therefore, it may be advantageous to adjust the defined deviation threshold based on the temperature or the amount of temperature change. For example, when the temperature is within a range of normal operating temperatures (e.g., a range defined based on the ambient room temperature), the defined deviation threshold may be lowered since the temperature will not adversely affect the camera. On the other hand, when the temperature is outside of the range of normal operating temperatures, the deviation threshold may be raised since lower or higher temperatures will adversely affect the camera. In an alternative example, the deviation threshold may be defined to be lower when the temperature is outside of the normal operating temperatures in order to trigger further camera calibrations more frequently. In this example, the deviation threshold may be defined to be higher when the temperature is within the normal operating temperatures in order to trigger further camera calibrations less frequently.

[0070] FIG. 8 illustrates an exemplary timeline 800 in which camera calibration and verification of camera calibration are performed. Although the following examples discuss verifying camera calibration, they may be more generally applied to verifying any type of calibration operation performed for a robotic manipulation system. Before robotic manipulation begins, the robotic control system 110 of FIG. 1A or FIG. 1B performs a first camera calibration during a calibration period 811 to determine camera calibration information for a camera (e.g., camera 570 of FIG. 5A or FIG. 5B). After the first camera calibration is completed, the robotic control system 110 captures reference images of a verification symbol (e.g., verification symbol 530) at various reference locations during a reference acquisition period 813 and determines reference image coordinates at which the verification symbol appears in each reference image (e.g., reference image 710 of FIG. 7A). Once the reference image coordinates are identified, the robotic manipulation can begin after the reference acquisition period 813.

[0071] After the robot operation starts, during a working period 815, the robot control system 110 controls the robot (e.g., robot 550 in FIG. 5A or FIG. 5B) to perform one or more robotic tasks, and therefore, in one embodiment, may not be able to collect a verification image (e.g., verification image 760 in FIG. 7B). The robot control system 110 detects a pause period 817 after the working period 815, during which the robot is not performing a robotic task. Thus, during the pause period 817, the robot control system 110 captures one or more verification images of the verification symbol at one or more reference locations (e.g., 610A and 610B) of the first set of reference locations, respectively. After the pause period 817 ends, during a working period 819, the robot control system 110 resumes controlling the robot to perform one or more robotic tasks, and therefore, may not collect a verification image. The robot control system 110 detects a pause period 821 after the working period 817, during which the robot is not performing a robotic task. During the idle period 821, the robot control system 110 captures one or more verification images of the verification symbol at one or more reference locations of the second set of reference locations (e.g., 610C-610E), respectively. After the idle period 821, during a robot work period 823, the robot control system 110 resumes controlling the robot to perform one or more robotic tasks and therefore may not collect verification images. The robot control system 110 detects an idle period 825 after the work period 823 during which the robot is not performing a robotic task. During the idle period 825, the robot control system 110 captures one or more verification images of the verification symbol at one or more locations of the third set of reference locations (e.g., 610F-610I), respectively.

[0072] The verification images (e.g., 760) captured during the idle periods 817, 821, and 825 may be captured at different respective locations of the reference locations. For example, one or more locations of the first set, the second set, and the third set may be different from one another and may not overlap. Additionally, during the idle period 825, the robot control system 110 may determine that the verification image capture is complete, which may indicate that a sufficient number of verification images have been captured for verifying the camera calibration. In one embodiment, the robot control system 110 may determine that the verification image capture is complete when a verification image has been captured at all of the reference locations (e.g., 610A-610I). In one embodiment, the robot control system 110 may determine that the verification image capture is complete when the number of verification images reaches a defined target total number.

[0073] Upon determining that the verification image capture is complete, the robot control system 110 identifies verification image coordinates at which the verification symbol appears in each verification image. The robot control system 110 then identifies a deviation parameter value based on the amount of deviation of each of the verification image coordinates from the reference image coordinates. If the deviation parameter exceeds a defined threshold, the robot control system 110 performs another camera calibration. However, in this example, the deviation parameter does not exceed the defined threshold, and therefore, the robot control system 110 continues to perform the robot task during the task period 827 after the idle period 825 without performing a further camera calibration.

[0074] FIG. 9 illustrates an example flow diagram 900 showing the verification process for the timeline of FIG. 8. In step 901, the robot control system 110 of FIG. 1A, FIG. 1B, or FIG. 1C performs a first camera calibration of a camera (e.g., camera 570 of FIG. 5A or FIG. 5B) to determine camera calibration information of the camera. In step 903, the robot control system 110 controls the robot (e.g., robot 550 of FIG. 5A or FIG. 5B) to move the verification symbol (e.g., verification symbol 530 of FIG. 5A or FIG. 5B) to the reference locations, and captures each instance of a reference image (e.g., reference image 710 of FIG. 7A) of the verification symbol at each reference location via the camera. In step 905, the robot control system 110 starts the robot operation of the robot based on the camera calibration information obtained from the first camera calibration.

[0075] In step 907, the robot control system 110 detects a pause period during the robot operation. In step 909, the robot control system 110 controls the robot (e.g., the robot 550 in FIG. 5A or 5B) to move the verification symbol (e.g., the verification symbol 530 in FIG. 5A or 5B) to one or more of the reference locations during the pause period, and captures one or more verification images (e.g., the verification image 760 in FIG. 7B) at the one or more of the reference locations, respectively, via the camera. In some cases, the robot control system 110 can control the robot to move the verification symbol to as many reference locations as the duration of the pause period allows. In step 911, the robot control system 110 determines whether the total number of captured verification images has reached a defined target total number. If the total number of captured verification images has not reached the target total number, the robot control system 110 attempts to detect another subsequent pause period during the robot operation by returning to step 907 to capture more verification images.

[0076] If the total number of captured verification images has reached the target total number, then in step 913, the robot control system 110 performs a camera calibration verification based on the reference image (e.g., 710) and the verification image (e.g., 760). The camera calibration verification generates a deviation parameter. In step 915, the robot control system 110 determines whether the deviation parameter exceeds a defined threshold. If the deviation parameter does not exceed the threshold, then in step 919, the robot control system 110 can reset the total number of captured verification images to 0 and continue the robot operation after the pause period while attempting to detect another pause period to capture a new set of verification images by returning to step 907.

[0077] If the deviation parameter exceeds a threshold, the robot control system 110 may stop the robot operation and perform a second camera calibration in step 917. After the second camera calibration in 917, the robot control system 110 may reset the total number of captured verification images to zero in 921. After step 921, the flow diagram may return to step 903, where the robot control system 110 controls the robot (e.g., 550) to move the verification symbol (e.g., 530) to the reference locations and captures a new set of reference images (e.g., 710) of the verification symbol at each reference location via the camera (e.g., 570), so that the new set of reference images is available for subsequent verification.

[0078] As mentioned above, one aspect of the disclosure includes receiving both a reference image (e.g., reference image 710 of FIG. 7A ) and a verification image (e.g., verification image 760 of FIG. 7B ) that capture or otherwise represent a common verification symbol (e.g., verification symbol 530 of FIG. 6A-6D ) located at a physical location on a robot arm, such as robot arm 553 of robot 550 of FIG. 6A-6D . In one embodiment, the reference image (e.g., 710 ) can be generated at a first time point, such as an earlier time point, while the verification image (e.g., 760 ) can be a new image generated at a second time point, such as a later time point. In some cases, the reference image may be generated based on one or more commands (also referred to as one or more instructions) generated by a computing system, such as robot control system 110 of FIG. 1C . In some cases, the one or more commands can include a motion command and / or a camera command. The motion command can be for the robot arm to move the verification symbol to the reference location. For example, the motion command can cause the robot arm to adopt a particular pose, thereby moving the verification symbol to a reference location associated with the particular pose of the robot arm. The camera command can be to cause a camera, such as camera 570 of FIGS. 6A-6D, to generate a reference image while the robot arm is in a particular pose and / or while the verification symbol is in a reference location associated with the particular pose. Thus, the reference image can correspond to the reference location. Similarly, the verification image can be generated based on one or more commands, such as motion commands and / or camera commands, which can be the same or similar to the motion commands and / or camera commands discussed above with respect to the reference image. The one or more commands for the verification image can be generated by the same computing system that generated the reference image, such as the robot control system 110, or by a different computing system. In this example, the one or more commands for generating the verification image can move the verification symbol back to the reference location. For example, the one or more commands can cause the robot arm to adopt a particular pose associated with the reference location.

[0079] In the above examples, the computing system receiving the verification image can compare it against the reference image to determine whether the calibration information is still sufficiently accurate. The calibration information can be determined by a calibration operation performed on the robotic manipulation system. The calibration operation, which may also be referred to as a system calibration, can include a camera calibration, a robot calibration, or any other calibration for controlling one or more components of the robotic manipulation system. In one embodiment, the calibration information can include a camera calibration information, a robot calibration information, or any other calibration information. The comparison discussed above can include determining a deviation parameter value that is based, for example, on the difference between where the verification symbol appears in the reference image and where the verification symbol appears in the verification image. If the deviation parameter value is too large, such as exceeding a defined deviation threshold (e.g., a predetermined deviation threshold), the computing system can determine that the calibration information is no longer sufficiently accurate. In such circumstances, the calibration information may be referred to as reflecting or including a miscalibration or misalignment. If the camera calibration information contains miscalibration or misalignment, it may not be able to accurately describe the intrinsic properties of the camera (e.g., projection properties or lens distortion properties) and / or may not be able to accurately describe the relationship between the camera and its external environment (e.g., the spatial relationship between the camera and the robot's base). When the robot calibration information contains miscalibration or misalignment, it may not be relied upon to accurately move a robot arm or other component to a desired location and / or orientation.

[0080] In one embodiment, the method of comparing the reference image and the verification image may include a plurality of verification symbols. For example, FIGS. 10A-10C illustrate a group of a plurality of verification symbols 530A-530C. More specifically, the group of verification symbols 530A-530C may be part of a robotic manipulation system 500A, which may be an embodiment of the robotic manipulation system 500. The robotic manipulation system 500A may include a robot 550A (which may be an embodiment of the robot 550), a camera 570 having a camera field of view 510, and a computing system, such as a robot controller 110. Similar to the robot 550, the robot 550A has a robotic arm 553 including a plurality of arm portions operatively attached to each other. For example, the plurality of arm portions may include links 554A-554E, including a robotic end effector, such as a robotic hand 555, attached to the link 554E. In some cases, as illustrated in FIGS. 10A-10C, multiple arm segments may be connected or otherwise arranged in series from the base 552 of the robot 550A to the robot end effector (e.g., the robot hand 555). In such cases, the series of arm segments may form a kinematic chain, and movement of a particular arm segment in the series may cause movement of some or all of the arm segments downstream of the particular arm segment. An arm segment downstream of a particular arm segment may refer to an arm segment that follows the particular arm segment in the series of arm segments. For example, links 554B-554E and the robot hand 555 may be downstream of link 554A. In other words, link 554A may be upstream of links 554B-554E and upstream of the robot hand 555. In this example, each arm segment in the series of arm segments or a subset thereof may be pivotable, extendable, retractable, or otherwise movable relative to the respective arm segment immediately preceding it in the series of arm segments. For example, link 554C can be pivotable relative to link 554B, which can be the arm portion immediately preceding link 554C in the series of arm portions illustrated in FIG. 10A.In the embodiment of FIGS. 10A-10C, links 554A-554E and robotic hand 555 may be operatively attached to one another via joints 556A-556D.

[0081] As discussed above, the robot arm (e.g., 553) may be moved to different poses, where a pose may refer to a shape formed by the arm portions (e.g., links) of the robot arm, or more generally, a geometric shape. For example, FIGS. 10A-10C illustrate three different respective poses for the robot arm 553. In one embodiment, the robot 550A in FIGS. 10A-10C may include one or more actuators (e.g., motors) configured to rotate, translate (e.g., extend or retract), or otherwise move the links 554A-554E and the robot hand 555 relative to one another. In such an embodiment, each of the poses in FIGS. 10A-10C may be associated with a particular permutation of the movements output by the one or more actuators. For example, the permutation may describe the angle values ​​by which the various arm portions of the robot arm 553 are rotated relative to the respective arm portions immediately preceding the various arm portions, and / or the distances by which the various arm portions are translated relative to the respective arm portions immediately preceding the various arm portions. As an example, the poses in Figures 10A-10C can each be associated with different permutations of five angle values ​​that describe the direction and amount of rotation of link 554B relative to link 554A, the direction and amount of rotation of link 554C relative to link 554B, the direction and amount of rotation of link 554D relative to link 554C, the direction and amount of rotation of link 554E relative to link 554D, and the direction and rotation of robot hand 555 relative to link 554E.

[0082] As mentioned above, FIGS. 10A-10C illustrate a group of multiple verification symbols 530A-530C disposed on one or more arm portions of a robot arm 553. More specifically, verification symbol 530A may be disposed on link 554B, while verification symbol 530B may be disposed on link 554C, and verification symbol 530C may be disposed on a robot hand 555. In some cases, a robot (e.g., robot 550) may have any number of verification symbols, which may be disposed on the robot in any manner. For example, verification symbols 530A-530C may be randomly disposed at respective locations on robot 550A. In some cases, the number and relative placement of verification symbols (e.g., 530A-530C) may be subject to one or more prescribed constraints. For example, their relative placement may be subject to a specified constraint that requires adjacent verification symbols (e.g., 530B and 530C) to be separated by a specified minimum distance (also called symbol spacing), such as a specified minimum distance of 5 cm. Such a constraint may reduce the likelihood that a computing system (e.g., robot controller 110) will confuse a particular verification symbol (e.g., 530B) for an adjacent verification symbol (e.g., 530C).

[0083] 11A-11C illustrate another example including a group of multiple verification symbols 1130A-1130C. More specifically, the verification symbols 1130A-1130C may be part of a robot manipulation system 1100 including a robot 1150, a camera 1170 having a camera field of view 1110, and a computing system such as a robot controller 110. The robot manipulation system 1100 and the camera 1170 may each be, for example, an embodiment of the robot manipulation system 500 and the camera 570, respectively. The robot 1150 may have a robot arm 1153 including multiple arm portions, such as links 1154A-1154E, and a robot hand 1155 (or other robot end effector). The multiple arm portions may be operatively attached to one another, for example, via joints 1156A-1156D. FIG. 11A shows the robot arm 1153 in a first pose, and FIG. 11B shows the robot arm 1153 in a second pose. Similar to FIGS. 10A-10C, multiple arm segments may be connected or otherwise arranged in series from the base 1152 of the robot 1150 to the robot hand 1155. A series of arm segments may form a kinematic chain in which the motion of one arm segment in a series may propagate to downstream arm segments in the chain. As illustrated in FIG. 11A, the verification symbol 1130A may be located on link 1154C, while the verification symbol 1130B may be located on link 1154D, and the verification symbol 1130C may be located on the robot hand 1155.

[0084] In one embodiment, one or more of the verification symbols of the group of verification symbols (e.g., 530A-530C, or 1130A-1130C) can have a circular shape. For example, each of the verification symbols 530A-530C of FIG. 10A-10C or the verification symbols 1130A-1130C of FIG. 11A-11C can be a ring pattern, or more specifically, a circular ring, as discussed above with respect to FIG. 5C. In the example of FIG. 5C, the ring pattern can have concentric circular regions (e.g., 531 and 533) or concentric circles (e.g., 535 and 537). The concentric circular regions or concentric circles can include, for example, inner and outer circular regions, or an inner circle and an outer circle. In the example illustrated in FIG. 11C, the verification symbol 1130A can have, in one example, a radius r 1,1130A and radius r 2,1130A In this example, the verification symbol 1130B can be formed as a circular ring having an outer circle with a radius r 1,1130B and radius r 2,1130B The verification symbol 1130C may also be another circular ring having an outer circle with a radius r 1,1130C and radius r 2,1130C It may be shaped as a circle having an outer circle having

[0085] In one embodiment, a group of verification symbols (e.g., 530A-530C, or 1130A-1130C) may be shaped as respective circular rings having different respective sizes. For example, as illustrated in FIG. 11C, verification symbols 1130A-1130C may have different radii for their respective outer circle regions or circles. That is, they may all have different r 2,1130A , r 2,1130B , and r 2,1130C In one embodiment, the circular rings for the verification symbols 1130A-1130C may have different ratios between the radius of their respective inner circular regions or circles and the radius of their respective outer circular regions or circles. That is, the ratio r 2,1130A / r 1,1130A , ratio r 2,1130B / r 1,1130B , and the ratio r 2,1130C / r1,1130C may all be different from one another. As discussed in more detail below, a computing system (e.g., robotic control system 110) may be configured to identify verification symbols 1130A / 1130B / 1130C based on the size of the respective circular rings that form the verification symbols 1130A / 1130B / 1130C and / or based on the respective ratios between the radii of the inner circles of the circular rings and the radii of the outer circles of the circular rings.

[0086] In one embodiment, some or all of the verification symbols (e.g., 530a-530C in FIGS. 10A-10C or 1130A-1130C in FIGS. 11A-11B) may be permanently attached or otherwise disposed on the robot arm (e.g., 533 or 1130). Such an embodiment may include a calibration pattern (e.g., 520 in FIGS. 5A and 5B) used to perform camera calibration, which has a size that is large enough to accommodate a pattern that is highly complex and / or large enough to accommodate a sufficient number of pattern elements to obtain sufficiently accurate results for the camera calibration. However, such a large size of the calibration pattern (e.g., 520) may be too large to be a permanent or otherwise persistent part of the robot arm, as the large size of the calibration pattern (e.g., 520) may interfere with normal robotic operations. In such an example, the calibration pattern 520 may be removed from the robot arm (e.g., 553), for example, before resuming normal robotic operations. In this embodiment, some or all of the verification symbols (e.g., 530A-530C or 1130A-1130C) may be less complex and / or smaller than the calibration pattern (e.g., 520). The smaller size of the verification symbols may allow them to remain on the robot arm (e.g., 553 or 1153) during normal robot operation while presenting no or only minimal interference to normal robot operation. Thus, in such an example, some or all of the verification symbols (e.g., 530A-530C or 1130A-1130C) may be permanently or otherwise persistently placed on the robot arm (e.g., 533 or 1133), as the case may be. Such placement provides the advantage that the robot controller or other computing system may more frequently and / or more quickly perform calibration verification and evaluate whether an updated camera calibration needs to be determined.

[0087] 12A and 12B illustrate a method 1200 relating to using a plurality of verification symbols to verify calibration information, such as camera calibration information. In one embodiment, the method 1200 may be performed by a computing system, such as the robot control system 110 of FIGS. 10A-10C or 11A-11B, or more specifically, by a control circuit of the computing system, such as the control circuit 111 of the robot control system 110 of FIG. 1C. As shown in FIG. 1C, the computing system may include a communication interface 113 configured to communicate with a camera having a camera field of view, such as the camera 570 of FIG. 10A or the camera 1170 of FIG. 11A, having a camera field of view 510 or 1110, respectively. The communication interface 113 may be further configured to communicate with a robot, such as the robot 550A of FIGS. 10A-10C or the robot 1150 of FIGS. 11A-11C. As mentioned above, the robot 550A / 1150 may include a robot arm 553 / 1153 having multiple arm portions operatively attached to one another and may include a group of verification symbols disposed on each of the multiple arm portions. In the example of FIGS. 10A-10C, the group of verification symbols may include verification symbols 530A-530C, which may be disposed on link 554B, link 554C, and robot hand 555, respectively. In the example of FIGS. 11A-11C, the group of verification symbols may include verification symbols 1130A-1130C, which may be disposed on links 1154C, 1154D, and robot hand 1155, respectively.

[0088] In one embodiment, method 1200 may include step 1201, in which control circuitry 111 of a robot control system or other computing system outputs a motion command to control a motion of a robot arm (e.g., robot arm 533 / 1153) to perform a robotic operation. The motion command may be based on, for example, calibration information. Calibration information, such as camera calibration information, may be determined from a first calibration operation, such as a first camera calibration. In some cases, step 1201 may be the same as or similar to step 409 of method 400, in which control circuitry 111 controls a motion of a robot arm to perform a robotic operation. For example, the robotic operation may include retrieving a box or other object in a warehouse. In this example, control circuitry 111 may be configured to determine based on an image of the box generated by a camera (e.g., camera 570 of FIGS. 10A-10C, or camera 1170 of FIGS. 11A-11B), and based on the calibration information, a spatial relationship between the camera and the box, and / or a spatial relationship between a robot (e.g., robot 550A-1150) and the box. In one embodiment, method 1200 may include control circuitry 111 performing a first calibration operation to determine calibration information. Such a step of performing a first calibration operation may be similar to step 401 of method 400 of FIG. 4A and may be performed before step 1201. As an example, the first calibration operation may be a camera calibration that includes determining estimates of camera calibration parameters based on a calibration image generated by a camera.

[0089] In one embodiment, method 1200 may include step 1203, in which control circuitry 111 or other components of the computing system determine a group of reference image coordinates. The group of reference image coordinates may be, for example, respective coordinates at which a group of verification symbols (e.g., 530A-530C / 1130A-1130C) appear in the reference image, and the reference image may be an image for representing the group of verification symbols (e.g., 530A-530C / 1130A-1130C). In one embodiment, the group of reference image coordinates may be used to verify the calibration information.

[0090] For example, FIG. 13A illustrates a reference image 1120 representing the group of verification symbols 1130A-1130C of FIG. 11A. In the example of FIG. 13A, the group of reference image coordinates may include a first reference image coordinate, a second reference image coordinate, and a third reference image coordinate. In such an example, the first reference image coordinate may identify where the verification symbol 1130A appears in the reference image 1120, the second reference image coordinate may identify where the verification symbol 1130B appears in the reference image 1120, and the third reference image coordinate may identify where the verification symbol 1130C appears in the reference image 1120. In a more specific example, each of the first reference image coordinate, the second reference image coordinate, and the third reference image coordinate may be a pixel coordinate [uv]. T More specifically, FIG. 13A shows three reference image coordinates of the verification symbols 1130A, 1130B, and 1130C, respectively, as [u ref_1 v ref_1 ] T 1130A , [u ref_1 v ref_1 ] T 1130B , [u ref_1 v ref_1 ] T 1130C13A is illustrated as a reference image generated when the robot arm 1153 is in an Nth pose, such as a first pose (when N=1). As discussed in more detail below, the reference image 1120 in FIG. 13A may correspond to or be more generally associated with a first pose, which may be an example pose of the robot arm 1153 shown in FIG. 11A. Different poses may position the verification symbols 1130A-1130C in different groups of respective locations within the camera field of view 1110. For example, the pose in FIG. 11A positions the verification symbols 1130A-1130C at 3D locations, [x ref_1 y ref_1 z ref_1 ] T 1130A , [x ref_1 y ref_1 z ref_1 ] T 1130B , [x ref_1 y ref_1 z ref_1 ] T 1130C In this example, these 3D locations [x ref_1 y ref_1 z ref_1 ] T 1130A , [x ref_1 y ref_1 z ref_1 ] T 1130B , [x ref_1 y ref_1 z ref_1 ] T 1130C is the pixel coordinate [u ref_1 v ref_1 ] T 1130A , [u ref_1 v ref_1 ] T 1130B , [u ref_1 v ref_1 ] T 1130C In one embodiment, the pixel coordinates [uref_1 v ref_1 ] T 1130A , [u ref_1 v ref_1 ] T 1130B , [u ref_1 v ref_1 ] T 1130C Each of the verification symbols 1130A / 1130B / 1130C may identify the center at which its respective verification symbol 1130A / 1130B / 1130C appears within the reference image 1120.

[0091] In one embodiment, step 1203 may include the control circuitry 111 receiving a reference image (e.g., 1120) from the communication interface 113 and / or the non-transitory computer readable medium 115 of the robot control system 110 or other computing system. For example, the reference image 1120 of FIG. 13A may be generated by the camera 1170 when the robot arm 1153 is in the pose of FIG. 11A. The computing system 110 may receive the reference image 1120 from the camera 1170 via the communication interface 113 and may store the reference image 1120 in the non-transitory computer readable medium 115. In step 1203, the control circuitry 111 may obtain or otherwise receive the reference image 1120 from the non-transitory computer readable medium 115 in one example. The non-transitory computer readable medium 115 may also store a verification image (discussed below), and the control circuitry 111 may receive the verification image from the non-transitory computer readable medium 115.

[0092] In one embodiment, the control circuitry 111 can be configured to identify at least one verification symbol (e.g., 1130A) of a group of verification symbols (e.g., 1130A-1130C) in a reference image (e.g., 1120) based on a prescribed model that describes the geometry of a robot arm (e.g., 1153). For example, the prescribed model can describe which links, robot end effectors, or other arm portions form a robot arm, their respective sizes (e.g., lengths), and how they are connected, and / or which arm portions have at least one verification symbol (e.g., 1130A) disposed thereon. In such an embodiment, the control circuitry can be configured to determine an area in the reference image (e.g., 1120) where at least one verification symbol is expected to appear based on the model, and search for at least one verification symbol (e.g., 1130A) in the area of ​​the reference image. In one embodiment, the model can store, or more generally describe, the location of a verification symbol (e.g., 1130A) on a robotic arm (e.g., 1153). The location of the verification symbol (also referred to as the symbol location) can be the approximate location of the verification symbol on the robotic arm (e.g., 1153).

[0093] For example, if a reference image (e.g., 1120) is stored with one or more parameter values ​​of a movement command used to generate a pose associated with the reference image, or more specifically, a pose of the robot arm that appears in the reference image, the one or more parameter values ​​and the model may be used to estimate a pose of the robot arm (e.g., 1153) when the robot arm is moved according to the movement command. As discussed in more detail below, the one or more parameter values, in one embodiment, may belong to one or more actuator parameters used to control one or more actuators (e.g., one or more motors) used to move the robot arm (e.g., 1153). The estimated pose may be used to estimate a location of a verification symbol (e.g., 1130A) on the robot arm (e.g., 1153), and the estimated location may be used to estimate where the verification symbol is likely to appear in the reference image (e.g., 1120).

[0094] In the above embodiments, the control circuitry 111 may be configured to focus on a region(s) of the reference image (e.g., 1120) in which the verification symbol or symbols are expected to appear. Such a technique may allow the control circuitry 111 to avoid searching the entire reference image (e.g., 1120) for the verification symbol (e.g., 1130A-1130C) and therefore more quickly identify the verification symbol (e.g., 1130A-1130C) in the reference image (e.g., 1120). The models in the above embodiments may also be used to search for the verification symbol in the verification image (discussed below).

[0095] As mentioned above, the group of verification symbols (e.g., 1130A-1130C) may, in one embodiment, be shaped as respective circular rings. In such an embodiment, the control circuitry 111 may be configured to recognize or otherwise identify the verification symbol (e.g., 1130A) in the reference image (e.g., 1120) by identifying the circular rings that form the verification symbol. When the group of verification symbols are shaped as respective circular rings having different respective sizes, as shown above with respect to FIG. 11C, the control circuitry 111 may determine the size (e.g., radius r) of the respective circular rings that form the verification symbol. 2,1130A ) to identify the verification symbol (e.g., 1130A). In some cases, where the verification symbol (e.g., 1130A) is shaped as a circular ring having at least a first circular region or first circle and a second circular region or second circle, the control circuitry 111 can be configured to identify the verification symbol based on a ratio between a radius of the first circular region or first circle and a radius of the second circular region or second circle. For example, the control circuitry 111 can identify a circular ring and determine whether the ring is shaped such that the ratio r 2,1130A / r 1,1130A The verification symbol 1130A may be configured to identify the verification symbol 1130A based on verifying that the verification symbol 1130A has a

[0096] As further mentioned above, the reference image of step 1203 (e.g., 1120 in FIG. 13A ) may be generated by a camera (e.g., 1170 in FIG. 11A ) when the robot arm (e.g., 1153 ) is in a first pose, such as the pose illustrated in FIG. 11A . The reference image may be generated by the camera during a first time period, or more generally, at a first time point. The first time period may refer to a time period (e.g., a period of milliseconds, seconds, or minutes) used or otherwise associated with generating the reference image. For example, the first time period may include a time for a camera operation to capture the reference image, and in some examples may further include a time for a robot operation to position the verification symbol within the camera field of view. In some cases, the first time period during which the reference image is generated may be, for example, hours, days, or weeks prior to step 1201 and / or step 1203. In one embodiment, the computing system, or more specifically, the control circuitry 111 performing steps 1201 and 1203, may be uninvolved in generating the reference image (e.g., 1120). In one embodiment, the computing system, or more specifically, the control circuitry 111 performing steps 1201 and 1203, may be involved in generating the reference image (e.g., 1120). For example, the computing system may output a movement command during a first time period to move the robot arm (e.g., 1153) to a first pose in a step preceding steps 1201 and / or 1203. In such an example, this preceding step may be part of the method 1200. The first pose may be a step in which the group of verification symbols (e.g., 1130A-1130C) are moved to a position [x] as discussed above. ref_1 y ref_1 z ref_1 ] T 1130A , [x ref_1 y ref_1 z ref_1 ] T 1130B , [x ref_1 y ref_1 z ref_1 ] T 1130C1C or another non-transitory computer-readable medium. In some cases, such a step may be similar to step 403 of FIG. 4A. In the above example, the computing system, or more specifically, the control circuitry 111, may further output a camera command in this step, which causes the camera (e.g., 1170) to generate a reference image (e.g., 1120) while the robot arm (e.g., 1153) is in the first pose. In one embodiment, the computing system may receive the reference image (e.g., 1120) from the camera and store the reference image in the non-transitory computer-readable medium 115 of FIG. 1C or another non-transitory computer-readable medium. In some cases, the computing system may further store information that enables the robot arm (e.g., 1153) to return to the first pose and / or return the verification symbols (e.g., 1130A-1130C) to the first group of respective locations. For example, the computing system may store the first group of respective locations, or, more specifically, their 3D coordinates, and / or parameter values ​​of a motion command for moving the robotic arm (e.g., 1153) to a first pose. In one embodiment, the coordinates and / or motion commands may be stored in the non-transitory computer-readable medium 115 in a manner that associates the stored information with the reference image (e.g., 1120) discussed above.

[0097] In one embodiment, the movement command discussed above for moving the robot arm to a first pose can include one or more parameter values ​​that describe the movement of the robot arm (e.g., 1153). As mentioned above, in some cases, the one or more parameter values ​​may belong to one or more actuator parameters that control one or more actuators that create the movement for the robot arm (e.g., 1153). In such cases, the one or more parameter values ​​can be referred to as one or more actuator parameter values ​​(also referred to as robot joint values). For example, the one or more actuator parameter values ​​can describe, for example, the amount of rotation of each of the arm portions of the robot arm relative to each other, the respective locations and / or orientations of the arm portions relative to each other, and / or the respective locations of the joints connecting the arm portions (e.g., 1156A-1156D). For example, the one or more actuator parameter values ​​of the movement command can describe respective angle values ​​that cause motors in the robot 1150 to rotate various arm portions (e.g., links 1154A-1154E and robot hand 1155) relative to their immediately preceding arm portion.

[0098] In one embodiment, the motion commands discussed above may have any parameter value(s) (e.g., random parameter value(s)), and the first pose caused by the motion command may be any pose. In one embodiment, the motion commands discussed above may be motion commands that allow some or all of the verification symbols (e.g., 1130A-1130C) to have a desired appearance in the reference image (e.g., 1120). For example, as discussed above, some or all of the group of verification symbols (e.g., 1130A-1130C) may have a circular shape, such as concentric circular regions or a ring shape formed by concentric circles. In one such example, when at least one of the verification symbols (e.g., 1130A) is in a certain orientation with respect to the camera (e.g., 1170), the at least one verification symbol (e.g., 1130A) may appear elliptical in the resulting reference image, rather than appearing perfectly circular. The elliptical appearance may result in inaccurate calibration verification. For example, if the reference image coordinate for the verification symbol 1130A is where the center of the symbol appears in the reference image, it may be more difficult to accurately determine when the verification symbol 1130A appears as an oval in the reference image. Furthermore, identifying the verification symbol 1130A in the reference image (e.g., distinguishing the verification symbol 1130A from other features in the reference image) may involve identifying a ring pattern in the reference image and whether the ring pattern corresponds to a particular ratio between concentric circles associated with the symbol (e.g., the ratio r associated with the verification symbol 1130A). 2,1130A / r 1,1130A ), this identification may be more difficult to perform accurately when the ring pattern appears elliptical in the reference image. Thus, if control circuitry 111 includes outputting motion commands associated with generating a reference image, control circuitry 111 may attempt to generate motion commands that cause a group of verification symbols (e.g., 1130A-1130C) to be positioned in a manner that appears perfectly or substantially circular in the reference image.

[0099] As an example, the control circuitry 111 can generate movement commands that move the robot arm (e.g., 1153) to a pose in which a group of verification symbols (e.g., 1130A-1130C) directly face a camera (e.g., 1170). For example, the pose may cause at least one verification symbol of the group of verification symbols (e.g., 1130A-1130C) to be oriented tangentially to a surface of a virtual sphere that is concave with respect to the camera (e.g., 1170). In FIG. 11A, the illustrated pose for the robot arm 1153 may cause the verification symbol 1130A and the verification symbol 1130B to have respective orientations that are tangential to a surface of a virtual sphere 1121 that is concave with respect to the camera 170, and may allow the verification symbol 1130C to have an orientation that is tangential to a surface of a virtual sphere 1123 that is also concave with respect to the camera 1170. In such a pose, which may be the first pose discussed above, the group of verification symbols (e.g., 1130A-1130C) may appear as respective circular shapes in the reference image (e.g., 1120). More specifically, the group of verification symbols (e.g., 1130A-1130C) in such an example may be positioned in such a manner that they appear in the reference image without eccentricity or with a respective amount of eccentricity that is less than a defined eccentricity threshold. In the above example, the imagined circles 1121 and 1123 may be centered on the camera (e.g., 1170). In one embodiment, the control circuitry 111 may be configured to generate random motion commands and search through the random motion commands to find one that can generate the orientation discussed above for the group of verification symbols. The found motion commands may be output by the control circuitry 111 to generate the reference image (e.g., 1120).

[0100] Returning to FIGS. 12A-12B, the method 1200 in one embodiment includes step 1205, in which the control circuit 111 outputs a motion command to control the robot arm to move to a first pose, such as the first pose illustrated in FIG. 10A or FIG. 11A, where the first pose is the pose at which the reference image (e.g., 1120) is generated during a first time period, as discussed above. In some cases, the motion command can be output to the robot (e.g., 550A / 1150) via the communication interface 113. In one embodiment, the motion command in step 1205 can be referred to as an additional motion command because it is in addition to the motion command in step 1201. The additional motion command can be output during a second time period following the first time period, or more generally, a second time point. The second time period can refer to a time period (e.g., a period of milliseconds, seconds, or minutes) used or otherwise associated with generating a verification image. For example, the second time period may include time for robotic movement to position the verification symbol within the camera field of view and / or time for camera manipulation to capture a verification image. In some cases, the second time period (or more generally, the second time point) may follow the first time period (or more generally, the first time point) by hours, days, or weeks. As discussed in more detail below, additional motion commands may be used to generate verification images during the second time period. In one embodiment, the additional motion commands may move the robot arm (e.g., 1153) to a pose (e.g., the first pose in FIG. 11A) in which the group of verification symbols (e.g., 1130A-1130C) directly faces the camera (e.g., 1170). For example, the group of verification symbols may have respective orientations tangent to one or more imaginary spheres that are concave with respect to the camera. In such a pose, the group of verification symbols (e.g., 1130A-1130C) may each appear as a circular shape in the verification image (e.g., 1160 in FIG. 13B).

[0101] In one embodiment, when the method 1200 includes outputting a motion command for generating a reference image during a first time period as discussed above, the motion command for generating a reference image during a first time period may be a first additional motion command, while the motion command in step 1205 for generating a verification image during a second time period may be a second additional motion command. In some cases, the first additional motion command may be a faster motion command, while the second additional motion command may be a slower motion command. In some cases, the first additional motion command may have one or more actuator parameter values ​​(or robot joint values) for controlling the robot arm to move to the first pose, and the second additional motion command may also have one or more actuator parameter values. More specifically, the first additional motion command and the second additional motion command may have the same actuator parameter values. As an example, if a robot (e.g., 1150) includes multiple motors that rotate various arm portions (e.g., links 1154A-1154E and robot hand 1155) relative to one another, the one or more actuator parameter values ​​may include multiple respective angle values ​​that control how much rotation is output by the multiple motors. In this example, the first additional movement command and the second additional movement command may both include the same multiple respective angle values ​​that control how much rotation is output by the multiple motors.

[0102] In one embodiment, the first additional motion command and / or the second additional motion command may be output during a respective idle period, such as the idle period discussed above with respect to step 411 of FIG. 4A. In some cases, if the calibration information is determined by performing a calibration operation, the first additional motion command may be output immediately after the calibration operation is performed or during the earliest idle period following the calibration operation. In some circumstances, the second additional motion command may be responsive to a defined trigger condition, such as the passage of a defined period of time since the calibration operation was performed, a collision event involving the robot, or any other event that may result in a possible displacement or misalignment between the camera and the robot or a part thereof (e.g., a natural disaster such as an earthquake), or some other trigger condition. If generating the verification image includes the control circuitry 111 outputting a camera command, in some examples, the camera command may also be output in response to a defined trigger condition.

[0103] In one embodiment, the first pose of step 1205 for the robot arm (e.g., 1153) may be associated with a particular group of respective reference points for a group of verification symbols (e.g., 1130A-1130C) located on the robot arm, as discussed above. For example, each group of reference points may be associated with a 3D point [x ref_1 y ref_1 z ref_1 ] T 1130A , [x ref_1 y ref_1 z ref_1 ] T 1130B , [x ref_1 y ref_1 z ref_1 ] T 1130CWhen the reference image (e.g., 1120) is generated, the group of verification symbols (e.g., 1130A-1130C) may be located at the respective group of reference locations associated with the first pose. When an additional movement command returns the robot arm to the first pose in step 1205, the group of verification symbols (e.g., 1130A-1130C) may be located at the respective group of reference locations (e.g., [x ref_1 y ref_1 z ref_1 ] T 1130A , [x ref_1 y ref_1 z ref_1 ] T 1130B , [x ref_1 y ref_1 z ref_1 ] T 1130C In such an embodiment, the groups of verification symbols (e.g., 1130A-1130C) may be placed in their respective reference groups when the reference image (e.g., 1120) is generated, and again placed in their respective reference groups when the verification image (e.g., 1160) is generated.

[0104] 12A-12B, the method 1200 in one embodiment can include step 1207, in which the control circuitry 111 receives a verification image, such as verification image 1160 in FIG. 13B. As mentioned above, the reference image (e.g., 1160) can be an image for representing a group of verification symbols (e.g., 1130A-1130C). In this example, the verification image (e.g., 1160) can be a further image for also representing a group of verification symbols (e.g., 1130A-1130C), and can be generated when the robot arm (e.g., 1153) is moved to a first pose as a result of the additional movement command of step 1205, such as the first pose illustrated in FIG. 11A.

[0105] In one embodiment, the method 1200 can include step 1209, in which the control circuitry 111 determines a group of verification image coordinates. In this embodiment, the group of verification image coordinates can be respective coordinates at which a group of verification symbols (e.g., 1130A-1130C) appear in the verification image (e.g., 1160 in FIG. 13B). In one embodiment, the group of verification image coordinates can be for verifying calibration information, such as camera calibration information. Similar to the above discussion of the reference image coordinates in step 1203, the group of verification image coordinates in one example can include a first verification image coordinate, a second verification image coordinate, and a third verification image coordinate. In a more specific example, each of the first verification image coordinate, the second verification image coordinate, and the third verification image coordinate can be, for example, a pixel coordinate that identifies a center of its respective verification symbol. For example, FIG. 13B illustrates three verification image coordinates, or more specifically, pixel coordinates [u verify_1 v verify_1 ] T 1130A , [u verify_1 v verify_1 ] T 1130B , [u verify_1 v verify_1 ] T 1130C , with the centers of each of the verification symbols 1130A-1130C appearing in the verification image 1160. Similar to the discussion of reference image coordinates, the label verify_N (e.g., verify_1) may refer to coordinates associated with a verification image generated when the robot arm is in an Nth pose, such as the first pose (N=1).

[0106] In one embodiment, the control circuitry 111 can perform steps 1203-1209 multiple times for multiple reference images and multiple verification images (e.g., five reference images and five verification images). The multiple reference images and multiple verification images can correspond to multiple respective poses. For example, FIGS. 10A-10C illustrate a series of three poses for the robot arm 553. In this example, the control circuitry 111 of the robot controller 110 or any other computing system can receive a first reference image and a first verification image both associated with a first pose as shown in FIG. 10A, a second reference image and a second verification image both associated with a second pose as shown in FIG. 10B, and a third reference image and a third verification image both associated with a third pose as shown in FIG. 10C. In another example, the reference image 1120 of FIG. 13A may be a first reference image, and the verification image 1160 of FIG. 13B may be a first verification image, both of which may be associated with the first pose illustrated in FIG. 11A for the robot arm 1153. In this example, the control circuitry 111 may further receive a second reference image 1122 of FIG. 14A and a second verification image 1162 of FIG. 14B, both of which may be associated with the second pose illustrated in FIG. 11B. In this example, the reference image coordinates [u ref_1 v ref_1 ] T 1130A , [u ref_1 v ref_1 ] T 1130B , [u ref_1 v ref_1 ] T 1130C may be the first group of reference image coordinates, while the verification image coordinates [u verify_1 v verify_1 ] T 1130A , [u verify_1 v verify_1 ] T 1130B , [u verify_1 v verify_1 ] T 1130Cmay be a group of first verification image coordinates. The control circuit 111 of this embodiment further controls the group of second reference image coordinates [u ref_2 v ref_2 ] T 1130A , [u ref_2 v ref_2 ] T 1130C , and the group of second verification image coordinates [u verify_2 v verify_2 ] T 1130A , [u verify_2 v verify_2 ] T 1130B , [u verify_2 v verify_2 ] T 1130C In the above examples, the control circuitry 111 can in some instances be configured to output different respective movement commands for moving the robot arm (e.g., 1153) to a plurality of respective poses, where a plurality of reference images and / or a plurality of verification images are associated with the plurality of respective poses.

[0107] 12A-12B, in one embodiment, the method 1200 may include step 1211, where the control circuit 111 selects a group of reference image coordinates (e.g., [u ref_1 v ref_1 ] T 1130A , [u ref_1 v ref_1 ] T 1130B , [u ref_1 v ref_1 ] T 1130C ) and a group of verification image coordinates (e.g., [u verify_1 v verify_1 ] T 1130A , [u verify_1 v verify_1 ] T 1130B , [u verify_1 vverify_1 ] T 1130C 13A and 13B , the respective deviation parameter value groups may include a first deviation parameter value, a second deviation parameter value, and a third deviation parameter value. The first deviation parameter value is determined based on a respective deviation amount between the first reference image coordinate [u ref_1 v ref_1 ] T 1130A and the first verification image coordinates [u verify_1 v verify_1 ] T 1130A For example, the first deviation parameter value may be equal to, or more generally based on, the distance between the first reference image coordinate and the first verification image coordinate, as discussed above with respect to step 457 of FIG. 4B. Similarly, the second deviation parameter value may be equal to, or more generally based on, the distance between the second reference image coordinate [u ref_1 v ref_1 ] T 1130B and the second verification image coordinates [u verify_1 v verify_1 ] T 1130B Further, the third deviation parameter value may be based on (e.g., equal to) the amount of deviation between the third reference image coordinates [u ref_1 v ref_1 ] T 1130C and the third verification image coordinates [u verify_1 v verify_1 ] T 1130CIn the above embodiment, each group of deviation parameter values ​​is associated with a group of verification symbols 1130A-C, i.e., a first deviation parameter value is associated with verification symbol 1130A, while a second deviation parameter value is associated with verification symbol 1130B, and a third deviation parameter value is associated with verification symbol 1130C.

[0108] In one embodiment, some or all of the groups of each deviation parameter value may be based on a single pair of a reference image and a verification image, both of which may be associated with a common pose for the robot arm. For example, the first deviation parameter value discussed above may be associated with the verification symbol 1130A and may be based on a single pair of a reference image 1120 and a verification image 1160, both of which are associated with a common pose, such as the pose of FIG. 11A.

[0109] In one embodiment, some or all of the groups of respective deviation parameter values ​​may be based on multiple pairs of a respective reference image and a respective verification image, each pair associated with a respective pose of a plurality of poses for the robot arm. As an example, the first deviation parameter value discussed above associated with the verification symbol 1130A may be based on a first pair of reference image 1120 and verification image 1160 (of FIGS. 13A and 13B) and a second pair of reference image 1122 and verification image 1162 (of FIGS. 14A and 14B). The first pair may be associated with the first pose of FIG. 11A, while the second pair may be associated with the second pose of FIG. 11B. More specifically, the first deviation parameter value in this example may be based on [u ref_1 v ref_1 ] T 1130A and [u verify_1 v verify_1 ] T 1130A (associated with the first pose), ref_2 v ref_2 ] T1130A and [u verify_2 v verify_2 ] T 1130A In one example, the first deviation parameter value may be equal to or based on an average of the two deviation measures. More generally, each group of deviation parameter values ​​in this example may be based on a respective deviation amount between the first group of reference image coordinates (associated with the first pose) and the first group of verification image coordinates, and may further be based on a respective deviation amount between the second group of reference image coordinates (associated with the second pose) and the second group of verification image coordinates.

[0110] In some cases, the reference image coordinates in the first reference image and the reference image coordinates in the second reference image may be part of a set of reference image coordinates indicating where a common verification symbol (e.g., 1130A) appears in a set of reference images, such as the two reference images 1120, 1122 discussed above. The set of reference images and the set of reference image coordinates in this example may correspond to respective sets of poses for the robot arm (e.g., 1153), such as the two poses in FIG. 11A and FIG. 11B. Similarly, the verification image coordinates in the first verification image and the verification image coordinates in the second verification image may be part of a set of verification image coordinates indicating where a verification symbol appears in a set of verification images, such as the verification images 1160, 1162 discussed above. The set of verification images and the set of verification image coordinates may also correspond to a set of poses. In such a case, the deviation parameter value associated with the verification symbol may be based on the respective amount of deviation between the set of reference image coordinates and the set of verification image coordinates.

[0111] Returning to FIGS. 12A-12B, method 1200 may include step 1213, in which control circuitry 111 determines whether at least one deviation parameter value of each group of deviation parameter values ​​exceeds a defined deviation threshold. In some cases, step 1213 may be similar to step 459 of FIG. 4B. In one embodiment, step 1213 may include determining whether each deviation parameter value of each group of deviation parameter values ​​exceeds a respective deviation threshold. For example, control circuitry 111 may determine whether a deviation parameter value associated with verification symbol 1130A exceeds a defined deviation threshold, whether a deviation parameter value associated with verification symbol 1130B exceeds a defined deviation threshold, and / or whether a deviation parameter value associated with verification symbol 1130C exceeds a defined deviation threshold. In some cases, the respective deviation thresholds may have the same value, thus forming a common deviation threshold for the verification symbols, or may have different values.

[0112] In one embodiment, the method may include step 1215, in which the control circuitry 111, in response to determining that at least one deviation parameter value of the respective group of deviation parameter values ​​exceeds a defined deviation threshold, may perform at least one of: (a) outputting a notification that at least one of the respective group of deviation parameter values ​​exceeds a defined deviation threshold; and (b) performing a calibration operation to determine updated calibration information (e.g., updated calibration information). For example, step 1215 may include outputting a notification to a user interface device, such as an electronic display in communication with the robot controller 110. The electronic display may display, for example, the at least one deviation parameter value or an indication that the at least one deviation parameter value exceeds a defined deviation threshold. In one embodiment, performing the calibration operation in step 1215 may be similar or identical to step 461 in FIG. 4B. In one embodiment, if the calibration information of step 1201 is determined by performing a first calibration operation, the calibration operation of step 1215 may be a second calibration operation subsequent to the first calibration operation.

[0113] In one embodiment, at least one deviation parameter value exceeding a defined deviation threshold can indicate that there is a change to the camera (e.g., 1170 in FIGS. 11A-11B) and / or a change to the camera's environment or robotic handling system (e.g., 1100). In some cases, the change to the camera may be an internal change, such as when the camera's lens or camera's image sensor changes shape or size due to temperature change or physical damage. In some cases, the change to the camera may include a change to where the camera (e.g., 170) is mounted, such as due to vibrations in the structure (e.g., ceiling) to which the camera is mounted. In some cases, a change in the camera's environment or to the robotic handling system may include a change in the position or orientation of the base (e.g., 1152) of the robot (e.g., 1150), such as due to vibrations in the structure (e.g., floor) to which the robot is mounted. In some cases, the change to the camera's environment or robotic manipulation system may be a change in the relationship between arm portions of the robot arm (e.g., between links 1154A-1154E of the robotic arm 1153) or the arm portions themselves. For example, one of the arm portions (e.g., link 1154D or the robotic hand 1155) may bend or otherwise deformed or damaged due to an event that was not planned for by the computing system 110. The unplanned event may be a collision event or some other unexpected event that may lead to a possible change in the robot (e.g., 1150) or other elements of the robotic manipulation system. The calibration validation discussed above may provide a fast and efficient technique for detecting changes to the camera (e.g., 1170) and / or changes to the camera's environment or robotic manipulation system 1100. For example, changes in the camera (e.g., 1170) and / or robot (e.g., 1150) may be detected by comparing the reference image coordinate(s) and the corresponding validation image coordinate(s) to determine the difference between them. In many cases, such a comparison can be performed without placing significant demands on the computational resources of computing system 110.For example, the comparison can be performed in a computation that acquires only a limited amount of processor execution time and / or a limited amount of memory. Thus, the comparison can facilitate accurate monitoring of the accuracy of the calibration information in a computationally efficient manner.

[0114] In one embodiment, at least one deviation parameter value exceeding a defined deviation threshold may indicate that there is a calibration error, such as a camera calibration error where the camera calibration information from the first camera calibration is no longer sufficiently accurate. In one embodiment, the method 1200 may include the control circuit 111 determining a type of calibration error (also referred to as a type of misalignment) that causes the at least one deviation parameter value to exceed a defined deviation threshold. The type of calibration error may indicate, for example, whether a loss of accuracy for the calibration information (e.g., camera calibration information) is caused by a change to the robot or otherwise indicative of a change to the robot, or whether a loss of accuracy is caused by a change to the camera or otherwise indicative of a change to the camera. For example, a change to the camera (e.g., 1170) may include an internal change to the camera and / or a change to where the camera is mounted, as discussed above. Changes to the robot may include changes to the location or orientation of the base (e.g., 1152) of the robot (e.g., 1150), changes to the relationship between the arm portions of the robot arm (e.g., 1153), and / or changes to the arm portions themselves, as discussed above.

[0115] In one embodiment, the determination of the type of calibration error can be based on a comparison between the groups of respective deviation parameter values, and more specifically, on whether the groups of respective deviation parameter values ​​exceed a defined deviation threshold in a substantially uniform manner. For example, if the groups of deviation parameter values ​​associated with different verification symbols (e.g., 1130A-1130C) all exceed a defined deviation threshold and do so in a substantially uniform manner, the control circuitry 111 can determine that the loss of precision is caused by the camera (e.g., 1170), or more specifically, by changes to the camera. This is because the appearance of each of the verification symbols (e.g., 1130A-1130C) in the reference image or verification image depends on the internal characteristics and / or the positioning of the camera (e.g., 1170). More specifically, the reference image coordinates and the verification image coordinates of the verification symbols (e.g., 1130A-1130C) in the reference image or verification image may all depend on the internal characteristics or the positioning of the camera. Therefore, a change to the internal characteristics or positioning of the camera (e.g., 1170) may often affect the respective deviation parameter values ​​for all verification symbols (e.g., 1130A-1130C) and, more specifically, may often increase all deviation parameter values ​​in a substantially uniform manner. In comparison, if a part of the robot (e.g., 1150), such as an arm portion, is damaged, malfunctions, or otherwise changed, it is unlikely that other parts of the robot will all change in exactly the same way (e.g., be damaged or all malfunction). Therefore, if there is a change in a part of the robot that would cause at least one deviation parameter value to exceed a defined deviation threshold, other deviation parameter values ​​of the group of deviation parameter values ​​may remain below the defined deviation threshold, or the group of deviation parameter values ​​may all exceed the defined deviation threshold, but do so in a non-uniform manner.Therefore, in one embodiment, if at least one of the groups of respective deviation parameter values ​​associated with different verification symbols (e.g., 1130A-1130C) exceeds a defined deviation threshold, but the groups of respective deviation parameter values ​​do not all exceed the defined deviation threshold in a substantially uniform manner, the control circuit 111 can determine that the loss of accuracy is caused by a change to the robot (e.g., 1150), or more specifically, the robot.

[0116] In one embodiment, the control circuitry 111 can use a defined uniformity threshold to evaluate whether a group of deviation parameter values ​​all exceed a defined deviation threshold in a substantially uniform manner. For example, the control circuitry 111 can determine whether at least one of the deviation parameter values ​​exceeds a defined deviation threshold, and can further determine whether the difference between the deviation parameter values ​​(or between the respective amounts by which they exceed the defined deviation threshold) is within a defined uniformity threshold. The uniformity threshold can be dynamically defined (e.g., based on current operating conditions of the robotic manipulation system) or can be predefined. As an example of using a defined uniformity threshold, if the control circuit 111 determines that the respective deviation parameter values ​​for the verification symbols 1130A-1130C all exceed the defined deviation threshold, but the deviation parameter value associated with the verification symbol 1130C differs from the deviation parameter value associated with the verification symbol 1130A by more than the defined uniformity threshold, and / or differs from the deviation parameter value associated with the verification symbol 1130B by more than the defined uniformity threshold, the control circuit 111 can determine that the loss of precision is caused by a change in the robot 1150, such as a change in at least the robot hand 1155 or other arm portion on which the verification symbol 1130C is located. In the above example, the control circuit 111 directly compares the deviation parameter values. In other embodiments, the control circuit 111 can compare the respective amounts by which the deviation parameter values ​​exceed the defined deviation threshold, and whether their respective amounts differ by more than the defined uniformity threshold. In another example, if the control circuit 111 determines that a first deviation parameter value of the group of deviation parameter values ​​exceeds a defined deviation threshold, but one or more of the group of deviation parameter values ​​does not exceed the defined deviation threshold, the control circuit 111 may also determine that a calibration error (also called a misalignment) is caused by a change to the robot (e.g., 1150), such as an arm portion on which the verification symbol associated with the first deviation parameter value is located.In another embodiment, if the control circuitry 111 determines that a group of deviation parameter values ​​all exceed a defined deviation threshold and that they do not differ from each other by more than a defined uniformity threshold, then the control circuitry 111 can determine that the calibration error is caused by a change in the camera (e.g., 1170).

[0117] In some cases, the control circuitry 111 of the robot controller 110 or other computing system may be in communication with a conveyor belt, such as the conveyor belt 1173 of Figures 11A and 11B. In such a case, the control circuitry 111 may be configured to stop the conveyor 1173 belt in response to determining that at least one deviation parameter value exceeds a defined deviation threshold. Stopping the conveyor belt 1173 may prevent the robot arm 1153 from having an undesired interaction with an object on the conveyor belt 1173 based on inaccurate calibration information.

[0118] In one embodiment, the control circuitry 111 can be configured to determine that if the calibration information is sufficiently accurate for a particular arm portion (e.g., link 1154C on which verification symbol 1130A is located, or link 1154D on which verification symbol 1130B is located, or robot hand 1155 on which verification symbol 1130C is located), then the calibration information is also sufficiently accurate for one or more arm portions upstream of that particular arm portion. As mentioned above, multiple arm portions can be arranged in a series of arm portions from the base of the robot to the robot end effector. An arm portion can be upstream of another arm portion if the former arm portion precedes the latter arm portion in the series of arm portions. For example, link 1154D in FIGS. 11A-11B can be upstream of link 1154E and robot hand 1155. In one example, if the control circuitry 111 determines that the calibration information (e.g., camera calibration information) is sufficiently accurate for, for example, the robotic hand 1155, the control circuitry 111 can determine that the calibration is sufficiently accurate for the upstream arm portions, such as links 1154E, 1154D, 1154C, 1154B, and 1154A. In this example, the control circuitry 111 can determine that the calibration information is sufficiently accurate for the arm portions if a deviation parameter value associated with a verification symbol located on the arm portion is below a defined deviation threshold. In one embodiment, if the control circuitry 111 determines that there is a calibration error for a particular arm portion such that the calibration information is not sufficiently accurate for that arm portion, the control circuitry 111 can determine that there is a calibration error for some or all downstream arm portions. For example, if the control circuitry 111 determines that the calibration information is not sufficiently accurate for a particular robot portion, such as link 1154D, the control circuitry 111 can determine that the calibration information is not sufficiently accurate for the downstream arm portions, such as link 1154E and the robotic hand 1155.

[0119] In one embodiment, one or more steps of method 1200, such as steps 1203-1215, can be executed by the robot controller 110 or other computing system in response to a user command. For example, a user (e.g., a system operator) can manually trigger a calibration verification operation, including steps 1203-1215. In one embodiment, steps 1203-1215 can be executed during an idle period. An idle period can be a period during which no robotic operations are being performed, such as, for example, lifting an object from a conveyor belt or a pallet. In one embodiment, one or more steps of method 1200, such as steps 1203-1215, can be executed by the robot controller 110 or other computing system in response to a defined trigger condition. As discussed above, the trigger condition can include an unplanned event, such as, for example, a collision involving the robot (e.g., 1150), an earthquake, or other natural disaster, which can result in changes in the robot (e.g., 1150) and / or the camera (e.g., 1170). In some cases, the trigger condition may include a particular period of time that elapses after an earlier calibration operation, such as the calibration operation used to determine the calibration information of step 1201. In such an example, the earlier calibration operation may be the first calibration operation, while the calibration operation of step 1215 may be the second calibration operation.

[0120] Further discussion of various embodiments Embodiment A1 relates to a robot control system comprising a communication interface configured to communicate with a robot having a base and a robot arm having a verification symbol disposed thereon and to communicate with a camera having a camera field of view. The robot control system further comprises a control circuit configured to perform a first camera calibration (or, more generally, a calibration operation) to determine camera calibration information associated with the camera (or, more generally, a calibration information associated with the robot control system). The control circuitry is configured to: a) control the robot arm to move the verification symbol to a location within the camera field of view during or after the first camera calibration by outputting, via the communication interface, a first movement command to the robot, the location being a reference location among one or more reference locations for verification of the first camera calibration; b) receive, via the communication interface, an image of the verification symbol from the camera, the camera configured to capture an image of the verification symbol at the reference location, the image being a reference image for verification; c) determine reference image coordinates for verification, the reference image coordinates being coordinates at which the verification symbol appears in the reference image; and d) control the robot arm to perform a first camera calibration (or, more generally, a calibration operation) to determine camera calibration information associated with the camera (or, more generally, a calibration information associated with the robot control system) via the communication interface. controlling movement of the robot arm to perform a robotic operation based on the camera calibration information by outputting a second movement command based on the camera calibration information to the robot arm; e) detecting a quiescent period during the robotic operation; f) controlling the robot arm to move the verification symbol at least to a reference location during the quiescent period by outputting a third movement command to the robot via the communications interface; g) receiving a further image of the verification symbol from the camera via the communications interface during the quiescent period, the camera being configured to capture a further image of the verification symbol at least at the reference location, the further image being a verification image for verification; and h) determining verification image coordinates used for verification.The system is further configured to: i) determine a verification image coordinate, the verification image coordinate being a coordinate at which a verification symbol appears in the verification image; and i) determine a deviation parameter value based on an amount of deviation between the reference image coordinate and the verification image coordinate, where both the reference image coordinate and the verification image coordinate are associated with the reference location, the deviation parameter value being indicative of a change in the camera since the first camera calibration or a change in a relationship between the camera and the robot since the first camera calibration; i) determine whether the deviation parameter value exceeds a defined threshold; and j) in response to determining that the deviation parameter value exceeds the defined threshold, perform a second camera calibration to determine updated camera calibration information (or, more generally, perform a second calibration operation to determine updated calibration information).

[0121] Embodiment A2 includes the robot control system of embodiment A1, wherein the control circuit is configured to control the robot to continue robot operation after the idle period without performing further camera calibration by outputting a fourth movement command to the robot via the communications interface in response to determining that the deviation parameter value does not exceed a defined threshold.

[0122] Embodiment A3 includes the robot control system of embodiment A1 or A2, wherein the one or more reference locations are a plurality of reference locations respectively corresponding to a plurality of reference image coordinates, and the reference image coordinate is one of the plurality of reference image coordinates. In this embodiment, the control circuitry is further configured to determine a plurality of verification image coordinates respectively corresponding to the plurality of reference locations, the verification image coordinate is one of the plurality of verification image coordinates, and the deviation parameter value is based on respective deviation amounts between the plurality of reference image coordinates and the plurality of verification image coordinates for the plurality of reference locations, each deviation amount among the respective deviation amounts being between (a) a reference image coordinate corresponding to a respective reference location of the plurality of reference locations and (b) a verification image coordinate corresponding to the same reference location.

[0123] Embodiment A4 includes the robot control system of embodiment A3, wherein the multiple verification image coordinates are respective coordinates at which a verification symbol appears in the multiple verification images, the verification image is one of the multiple verification images, and the control circuit is configured to control the camera to capture all of the multiple verification images within the idle period.

[0124] Embodiment A5 includes the robot control system of embodiment A3, wherein the multiple verification image coordinates are respective coordinates at which a verification symbol appears in the multiple verification images, the verification image is one of the multiple verification images, and the control circuit is configured to control the camera to capture the multiple verification images within different idle periods, the idle period being one of the different idle periods.

[0125] Embodiment A6 includes a robot control system of any one of embodiments A1 to A5, wherein the verification symbol includes a first region having a first color and a second region having a second color, and a ratio between the area of ​​the first region and the area of ​​the second region is specified and stored on a storage device of the robot control system as the specified ratio.

[0126] Embodiment A7 includes the robotic control system of embodiment A6, where the control circuitry is configured to identify a verification symbol in the reference image or the verification image based on a defined ratio.

[0127] Embodiment A8 includes the robot control system of embodiment A7, wherein the robot arm has a calibration pattern disposed thereon, the reference image includes a verification symbol and the calibration pattern, and the control circuit is configured to determine whether the portion of the reference image has a first image area having a first color and a second image area having a second color, and to determine whether the portion of the reference image is the verification symbol or the calibration pattern by determining whether the ratio of the area of ​​the first image area to the area of ​​the second image area is equal to a specified ratio.

[0128] Embodiment A9 includes any one of the robot control systems of embodiments A1 to A8, and the verification symbol includes a first shape and a second shape that are concentric with each other, and the centers of the first shape and the second shape are substantially at the same location.

[0129] Embodiment A10 includes the robotic control system of embodiment A9, wherein the control circuitry is configured to determine the reference image coordinates by: a) determining a first coordinate of a center of a first shape in the reference image, b) determining a second coordinate of a center of a second shape in the reference image, and c) determining the reference image coordinates as an average of the first and second coordinates in the reference image. In this embodiment, the control circuitry is configured to determine the verification image coordinates by: d) determining a first coordinate of a center of the first shape in the verification image, e) determining a second coordinate of a center of the second shape in the verification image, and f) determining the verification image coordinates as an average of the first and second coordinates in the verification image.

[0130] Embodiment A11 includes the robot control system of any one of embodiments A1 to A10, wherein the control circuit is configured to identify a verification symbol in the reference image or the verification image by identifying a circular ring, and the verification symbol is formed as a circular ring.

[0131] Embodiment A12 includes the robot control system of any one of embodiments A1 to A11, wherein the control circuit is further configured to determine a temperature of an environment in which the robot is located and adjust at least one of the defined thresholds or the calibration information based on the measured temperature.

[0132] Embodiment A13 includes the robotic control system of embodiment A12, wherein the control circuit is configured to adjust the defined threshold value based on temperature by setting the defined threshold value to have a first value when the temperature is outside the defined range and setting the threshold value to have a second value lower than the first value when the temperature is within the defined range.

[0133] Embodiment A14 includes the robot control system of any one of embodiments A1 to A13, wherein the one or more reference points at which the control circuit is configured to move the verification symbol via the robot arm include a plurality of reference points disposed on a surface of a sphere that is concave relative to the camera.

[0134] Embodiment A15 includes the robot control system of embodiment A14, wherein the control circuitry is further configured to control the robot arm to move the verification symbol at each of the multiple reference locations so that it is oriented tangentially relative to the surface of the sphere.

[0135] Embodiment A16 includes the robot control system of any one of embodiments A1 to A15, wherein the control circuit is configured to control the robot arm to move the verification symbol directly opposite the camera when the verification symbol is moved to the reference location.

[0136] Embodiment A17 includes the robot control system of any one of embodiments A1 to A16, and the control circuit is configured to detect idle periods of robot operation by detecting periods during robot operation when the robot is not performing a robot task.

[0137] Embodiment A18 includes the robot control system of embodiment A17, wherein the control circuit is configured to control the robot arm to interact with objects on a conveyor belt that are reachable by the robot arm, and the control circuit is configured to detect idle periods by detecting the absence of objects on the conveyor belt or by detecting that the distance between the robot and the nearest object on the conveyor belt exceeds a defined distance threshold.

[0138] Embodiment B1 relates to a computing system comprising a communications interface and control circuitry configured to communicate with a robot having (i) a camera having a camera field of view and (ii) a robotic arm comprising a plurality of arm segments operatively attached to one another and comprising a group of verification symbols disposed on each of the plurality of arm segments.The control circuitry is further configured to: output motion commands for controlling movement of the robot arm to perform a robotic operation when the robot arm is in the camera field of view, the motion commands being based on the calibration information; determine a group of reference image coordinates, the group of reference image coordinates being respective coordinates at which a group of verification symbols appear in the reference image, the reference image being an image for representing the group of verification symbols and generated by the camera during a first time period when the robot arm is in a first pose; output an additional motion command for controlling the robot arm to move to the first pose during a second time period following the first time period; receive a verification image, the additional image for representing the group of verification symbols and generated by the camera when the robot arm is moved to the first pose as a result of the additional motion command; and determine a group of verification image coordinates, the group of verification image coordinates being based on the calibration information. The loop is configured to perform at least one of: determining respective coordinates at which a group of verification symbols appear in the verification image, determining respective groups of deviation parameter values ​​based on respective amounts of deviation between the group of reference image coordinates and the group of verification image coordinates, where each group of deviation parameter values ​​is associated with a group of verification symbols, determining whether at least one deviation parameter value of the respective groups of deviation parameter values ​​exceeds a defined deviation threshold, and in response to determining that at least one deviation parameter value of the respective groups of deviation parameter values ​​exceeds the defined deviation threshold, outputting a notification that at least one of the respective groups of deviation parameter values ​​exceeds the defined deviation threshold, and performing a calibration operation to determine updated calibration information. The control circuitry can perform the method, for example, by executing instructions on a non-transitory computer readable medium.

[0139] Embodiment B2 includes the computing system of embodiment B1, wherein the first pose is associated with a first additional movement command output during a first time period, the first additional movement command having one or more actuation parameter values ​​for controlling the robot arm to move to the first pose, and the reference image is generated by the camera when the robot arm is in the first pose as a result of the first additional movement command. Further in this embodiment, the additional movement command output during a second time period is a second additional movement command and also includes one or more actuation parameter values.

[0140] Embodiment B3 includes the computing system of embodiment B2, wherein each verification symbol of the group of verification symbols has a circular shape, and the one or more operating parameter values ​​of the first additional motion command and the second additional motion command position the group of verification symbols in the reference image and in the verification image in a manner such that the group of verification symbols appears without eccentricity or with a respective amount of eccentricity that is less than a specified eccentricity threshold.

[0141] Embodiment B4 includes the computing system of any one of embodiments B1 to B3, wherein the control circuitry is configured to determine one or more actuation parameter values ​​for an additional motion command that moves each verification symbol of the group of verification symbols directly facing the camera.

[0142] Embodiment B5 includes the computing system of embodiment B4, in which the one or more actuation parameter values ​​cause the group of verification symbols to be oriented tangentially relative to one or more virtual spheres that are concave relative to the camera.

[0143] Embodiment B6 includes the computing system of any one of embodiments B1 to B5, and when at least one verification symbol of the group of verification symbols is formed as a circular ring, the control circuit is configured to identify at least one verification symbol in the reference image and in the verification image by identifying the circular ring.

[0144] Embodiment B7 includes the computing system of embodiment B6, wherein when the group of verification symbols are formed as respective circular rings having different respective sizes, the control circuitry is configured to identify the at least one verification symbol based on a size of the respective circular rings that form the at least one verification symbol.

[0145] Embodiment B8 includes any one of the computing systems of embodiments B1 to B7, wherein the control circuit is configured to identify at least one verification symbol of a group of verification symbols in the reference image based on a defined model describing a geometric shape of the robot arm.

[0146] Embodiment B9 includes the computing system of embodiment B8, wherein the control circuitry is configured to determine, based on the model, an area in the reference image in which at least one verification symbol is expected to appear, and to search for the at least one verification symbol in the area of ​​the reference image.

[0147] Embodiment B10 includes any one of the computing systems of embodiments B1 to B9, wherein the control circuit is configured to determine a type of calibration error that causes at least one deviation parameter value to exceed a defined deviation threshold based on a comparison between the groups of respective deviation parameter values, and the type of calibration error indicates whether a loss of accuracy to the calibration information represents a change in the robot or whether a loss of accuracy represents a change in the camera.

[0148] Embodiment B11 includes the computing system of embodiment B10, where the control circuitry is configured to determine whether the respective groups of deviation parameter values ​​all exceed a defined deviation threshold and whether the respective groups of deviation parameters differ from one another by more than a defined uniformity threshold. In response to determining that the respective groups of deviation parameter values ​​all exceed the defined deviation threshold and do not differ from one another by more than a defined uniformity threshold, the control circuitry in this embodiment is configured to determine that the type of calibration error is a calibration error representative of a change in the camera.

[0149] Embodiment B12 includes the computing system of embodiment B11, wherein the control circuitry is further configured to determine that the type of calibration error is a calibration error representative of a change in the robot in response to a determination that one or more of the groups of respective deviation parameter values ​​do not exceed a defined deviation threshold or a determination that the groups of respective deviation parameter values ​​differ from one another by more than a defined uniformity threshold.

[0150] Embodiment B13 includes the computing system of any one of embodiments B1 to B12. In this embodiment, the calibration information is associated with a first calibration operation, the calibration operation for generating the updated calibration information is a second calibration operation subsequent to the first calibration operation, and the control circuitry is configured to output additional motion commands and to output camera commands for receiving a verification image in response to a defined trigger condition. The defined trigger condition includes at least one of a defined period of time that has elapsed since the first calibration operation or an event that is not planned by the computing system and that results in a change to the robot or the camera.

[0151] Embodiment B14 includes the computing system of any one of embodiments B1 to B13, and when a plurality of robot arm portions are arranged in a series of arm portions from a robot base to a robot end effector, the control circuit is configured to determine whether a deviation parameter value for a first verification symbol of a group of verification symbols exceeds a defined deviation threshold, identify from among the plurality of arm portions a first arm portion in which the first verification symbol is arranged, and in response to a determination that the deviation parameter value for the first verification symbol does not exceed a defined deviation threshold, determine that the calibration information is accurate for the first arm portion and for at least one additional arm portion preceding the first arm portion in the series of arm portions.

[0152] Embodiment B15 includes the computing system of any one of embodiments B1-B14, wherein when the computing system is in communication with a conveyor belt used in robotic operations, the control circuitry is configured to stop the conveyor belt in response to determining that at least one deviation parameter value exceeds a defined deviation threshold.

[0153] Embodiment B16 includes a computing system of any one of embodiments B1 to B15, wherein the reference image is a first reference image associated with a first pose of the robot arm, the verification image is a first verification image associated with the first pose, the group of reference image coordinates is a group of first reference image coordinates associated with the first pose, and the group of verification image coordinates is a group of first verification image coordinates associated with the first pose. In this embodiment, the control circuitry is configured to: determine a group of second reference image coordinates, the group of second reference image coordinates being respective coordinates at which the group of verification symbols appears in the second reference image, the second reference image being generated by the camera when the robot arm is in a second pose; outputting a further movement command for controlling the robot arm to move the robot arm to the second pose subsequent to the second reference image being generated; receive a second verification image, the second verification image also representing the group of verification symbols and being generated by the camera when the robot arm is moved to the second pose as a result of the further movement command; and determine a group of second verification image coordinates, the group of second verification image coordinates being respective coordinates at which the group of verification symbols appears in the second verification image. In this embodiment, each group of deviation parameter values ​​is further based on a respective deviation amount between the group of second reference image coordinates and the group of second verification image coordinates.

[0154] Although various embodiments have been described above, it should be understood that these embodiments are presented merely as illustrations and examples of the present invention, and not as limitations. It will be apparent to those skilled in the art that various changes in form and detail can be made therein without departing from the spirit and scope of the invention. Thus, the breadth and scope of the present invention should not be limited by any of the exemplary embodiments described above, but should be defined only by the appended claims and their equivalents. It will also be understood that each feature of each embodiment discussed herein, and each reference cited herein, can be used in combination with the features of any other embodiment. All patents and publications discussed herein are incorporated by reference in their entirety.

Claims

1. 1. A computing system comprising: a communications interface configured to communicate with a robot having a robotic arm comprising a plurality of arm portions operatively attached to one another and comprising a group of verification symbols disposed on one or more of each of the plurality of arm portions; A control circuit, The control circuit includes: determining a group of reference image coordinates, the group of reference image coordinates being respective coordinates at which the group of verification symbols appear in a reference image captured by a camera; receiving a verification image, the verification image being an image representative of the group of verification symbols, the verification image being an image captured by the camera during one or more pauses in robotic operation; determining a group of verification image coordinates, the group of verification image coordinates being respective coordinates at which the group of verification symbols appear within the verification image; determining a respective group of deviation parameter values ​​based on a respective amount of deviation between the group of reference image coordinates and the group of verification image coordinates, the respective group of deviation parameter values ​​being associated with the group of verification symbols; determining whether at least one deviation parameter value of the respective group of deviation parameter values ​​exceeds a defined deviation threshold; 1. A computing system configured to:

2. The computing system of claim 1 , wherein the control circuitry is configured to detect the one or more inactive periods.

3. The control circuit includes: outputting movement commands for controlling movement of the robot arm to perform a robotic operation, the movement commands being generated based on calibration information; in response to determining that at least one deviation parameter value of the respective groups of deviation parameter values ​​exceeds the defined deviation threshold, performing at least one of: outputting a notification that at least one of the respective groups of deviation parameter values ​​exceeds the defined deviation threshold; and performing a calibration operation to determine updated calibration information.

3. A computing system according to claim 1 or 2, configured to:

4. the reference image is associated with the robot arm in a first pose, the first pose being associated with a first movement command output during a first time period, the first movement command having one or more actuation parameter values ​​for controlling the robot arm to move to the first pose, the reference image being generated when the robot arm is in the first pose as a result of the first movement command; 4. The computing system of claim 1, wherein a second pause is associated with a second movement command output during a second time period and also includes the one or more operating parameter values.

5. 5. The computing system of claim 4, wherein each verification symbol of the group of verification symbols has a circular shape, and the one or more operating parameter values ​​of the first motion command and the second motion command cause the group of verification symbols to be positioned in the reference image and in the verification image in a manner such that the group of verification symbols appears with no eccentricity or with a respective amount of eccentricity that is less than a defined eccentricity threshold.

6. The computing system of claim 4 , wherein the control circuitry is configured to determine the one or more operating parameter values ​​for the first motion command that moves each verification symbol of the group of verification symbols.

7. The computing system of claim 6 , wherein the one or more operating parameter values ​​cause the group of verification symbols to be oriented tangentially relative to one or more imaginary spheres.

8. 5. The computing system of claim 1, wherein when at least one verification symbol of the group of verification symbols is formed as a circular ring, the control circuitry is configured to identify the at least one verification symbol in the reference image and in the verification image by identifying the circular ring.

9. 9. The computing system of claim 8, wherein when the group of verification symbols are formed as respective circular rings having different respective sizes, the control circuitry is configured to identify the at least one verification symbol based on a size of the respective circular rings forming the at least one verification symbol.

10. 5. The computing system of claim 1, wherein the control circuitry is configured to identify at least one verification symbol of the group of verification symbols in the reference image based on a defined model describing a geometry of the robot arm.

11. The control circuit includes: determining an area in the reference image in which the at least one verification symbol is expected to appear based on the model; and searching for the at least one verification symbol within the region of the reference image.

12. 12. The computing system of claim 1, wherein the control circuitry is configured to determine a type of calibration error that causes the at least one deviation parameter value to exceed the defined deviation threshold based on a comparison between the groups of the respective deviation parameter values.

13. The computing system of claim 12 , wherein the type of calibration error indicates whether a loss of precision to calibration information represents a change in the robot.

14. The control circuit includes: determining whether the respective groups of deviation parameter values ​​all exceed the defined deviation threshold and whether the respective groups of deviation parameter values ​​differ from one another by more than a defined uniformity threshold; determining a type of the calibration error in response to determining that the respective groups of deviation parameter values ​​all exceed the defined deviation threshold and do not differ from one another by more than the defined uniformity threshold; The computing system of claim 13 configured to:

15. 14. The computing system of claim 13, wherein the control circuitry is further configured to determine that the type of calibration error is a calibration error representative of a change in the robot in response to determining that one or more of the respective groups of deviation parameter values ​​do not exceed the defined deviation threshold or determining that the respective groups of deviation parameter values ​​differ from one another by more than a defined uniformity threshold.

16. the control circuitry is configured to output movement commands for controlling movement of the robot arm to perform a robotic operation, the movement commands being generated based on calibration information; When the plurality of arm segments are arranged as a series of arm segments from a robot base to a robot end effector, the control circuitry determining whether a deviation parameter value for a first verification symbol of the group of verification symbols exceeds the defined deviation threshold; identifying a first arm portion from among the plurality of arm portions, the first arm portion having the first verification symbol disposed thereon; 3. The computing system of claim 1, further configured to: in response to determining that the deviation parameter value for a first verification symbol does not exceed the defined deviation threshold, determine that the calibration information is accurate for the first arm portion and for at least one additional arm portion that precedes the first arm portion in the series of arm portions.

17. 17. The computing system of claim 1, wherein when the computing system is in communication with a conveyor belt used in robotic operations, the control circuitry is configured to stop the conveyor belt in response to determining that the at least one deviation parameter value exceeds the defined deviation threshold.

18. A non-transitory computer-readable medium having instructions stored therein, comprising: The instructions, when executed by control circuitry of a computing system, cause the control circuitry to: determining a group of reference image coordinates based on a reference image captured by a camera and received by the computing system, the computing system being configured to communicate with a robot having a robot arm including a plurality of arm sections operatively attached to one another and including a group of verification symbols disposed on one or more of each of the plurality of arm sections, the group of reference image coordinates being respective coordinates at which the group of verification symbols appear in the reference image; receiving a verification image, the verification image being an image representative of the group of verification symbols, the verification image being an image captured by the camera during one or more pauses in robotic operation; determining a group of verification image coordinates, the group of verification image coordinates being respective coordinates at which the group of verification symbols appear within the verification image; determining a respective group of deviation parameter values ​​based on a respective amount of deviation between the group of reference image coordinates and the group of verification image coordinates, the respective group of deviation parameter values ​​being associated with the group of verification symbols; determining whether at least one deviation parameter value of the respective group of deviation parameter values ​​exceeds a defined deviation threshold; A non-transitory computer-readable medium for causing

19. The control circuit includes: outputting movement commands for controlling movement of the robot arm to perform a robotic operation, the movement commands being generated based on calibration information; in response to determining that at least one deviation parameter value of the respective groups of deviation parameter values ​​exceeds the defined deviation threshold, performing at least one of: outputting a notification that at least one of the respective groups of deviation parameter values ​​exceeds the defined deviation threshold; and performing a calibration operation to determine updated calibration information.

20. The non-transitory computer-readable medium of claim 18 configured to:

20. 1. A method for verifying a reference image, comprising: communicating, via a communications interface, with a robot having a robot arm with a plurality of arm sections operatively attached to one another, wherein a group of verification symbols is disposed on one or more of each of the plurality of arm sections; determining, by a control circuit, a group of reference image coordinates, the group of reference image coordinates being respective coordinates at which the group of verification symbols appear in a reference image captured by a camera; receiving, by the control circuitry, a verification image, the verification image being an image intended to represent the group of verification symbols and captured by the camera during one or more pauses in robotic operation; determining, by the control circuitry, a group of verification image coordinates, the group of verification image coordinates being respective coordinates at which the group of verification symbols appear within the verification image; determining, by the control circuitry, respective groups of deviation parameter values ​​based on respective amounts of deviation between the group of reference image coordinates and the group of verification image coordinates, each of the groups of deviation parameter values ​​being associated with the group of verification symbols; determining, by the control circuitry, that at least one deviation parameter value of the respective group of deviation parameter values ​​exceeds a defined deviation threshold; A method comprising:

21. The control circuit includes: outputting movement commands for controlling movement of the robot arm to perform a robotic operation, the movement commands being generated based on calibration information; in response to determining that at least one deviation parameter value of the respective groups of deviation parameter values ​​exceeds the defined deviation threshold, performing at least one of: outputting a notification that at least one of the respective groups of deviation parameter values ​​exceeds the defined deviation threshold; and performing a calibration operation to determine updated calibration information.

21. The method of claim 20, configured to:

Images