Effective methods for calculating the minimum distance to a dynamic object

Abstract

To provide a method and system for calculating a minimum distance from a robot to dynamic objects in a robot workspace.SOLUTION: The method uses images from one or more three-dimensional cameras, where edges of objects are detected in each image, and the robot and the background are subtracted from the resultant image, leaving only object edge pixels. Depth values are then overlaid on the object edge pixels, and distance calculations are performed only between the edge pixels and control points on the robot arms. Two or more cameras may be used to resolve object occlusion, where each camera's minimum distance is computed independently and the maximum of the cameras' minimum distances is used as the actual result. The use of multiple cameras does not significantly increase the computational load, and does not require calibration of the cameras with respect to each other.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to the field of object detection in the motion control of industrial robots. More specifically, it relates to a method for calculating the minimum distance between a robot and a dynamic object present in the robot working space using images from one or more 3D cameras. In this method, the edges of the object are detected in each image, and the robot and the background are removed from the resulting image. The depth values are overlaid on the remaining object edge pixels, and distance calculations are performed only between the object edge pixels and the control points on the robot arm.

Background Art

[0002] It is well known that industrial robots are used to perform various manufacturing, assembly, and material movement operations. Many robot working space environments contain obstacles, which may also be present within the path of the robot's movement. Obstacles can be permanent structures such as machines and furniture, and due to their static nature, they can be easily avoided by the robot. Additionally, obstacles can be dynamic objects that move randomly into or through the robot working space. Dynamic objects need to be calculated in real time by the robot control device, and the robot needs to avoid the objects while performing its operations. Collisions between the robot and the obstacles must be absolutely avoided.

[0003] Prior art techniques for detecting dynamic objects based on camera images are known in the art, but such known techniques are subject to significant limitations. In one known technique, a large number of depth sensors are used to create a point cloud defining the outer surface of any object within the robot working space, and the minimum distance between the points in the point cloud and the robot arm is calculated at each motion time step of the robot. However, due to the number of points in the point cloud, such calculations are extremely computationally intensive and time-consuming.

[0004] Due to the inherent disadvantages of point cloud-based object distance methods, other methods have been developed. The single-camera depth space approach is known, where the distance between an obstacle point and a robot control point is calculated from depth images. However, for obstacles closer to the camera than the robot, the extent of the obstacle's depth is unknown, and the occluded space behind the obstacle is considered occupied. This can lead to a significant underestimation of the minimum distance, potentially causing the robot arm to move unnecessarily large around object space that is not even occupied.

[0005] To overcome the limitations of single-camera depth spatial methods, a multi-sensor method designed to address occlusion issues has been developed. In this method, the primary sensor is used to construct a spatial depth grid, while other sensors are used to detect occluded cells behind obstacles within the depth grid. Although this method is less computationally intensive than the point cloud approach, it can still require long computation times for large robots with many control points and large workspaces with many cells in the depth grid. Furthermore, this method is prone to errors when constructing the 3D depth grid, requiring precise camera / sensor alignment to minimize these errors. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] Given the circumstances described above, an improved method is needed to calculate the minimum distance between a robot and dynamic objects within the robot's workspace. [Means for solving the problem]

[0007] A method and system for calculating the minimum distance from a robot to a dynamic object in the robotic workspace is described and shown in accordance with the teachings of this disclosure. This method uses images from one or more 3D cameras, where the edges of the object are detected in each image, and the robot and background are removed from the resulting image, leaving only the edge pixels of the object. The depth value is then superimposed on the edge pixels of the object, and the distance calculation is performed only between the edge pixels of the object and the control points on the robotic arm. Two or more cameras may be used to resolve object occlusion, and the minimum distance from each camera is calculated independently, with the largest of the minimum distances from the cameras being used as the actual result. Using multiple cameras does not significantly increase the computational load, and there is no need to calibrate the cameras against each other.

[0008] Further features of the apparatus and methods of this disclosure will become apparent when the following description and the accompanying claims are used in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This is an example of a human occupying the workspace of an industrial robot, illustrating the calculation of the minimum distance that must be performed to ensure that collisions between the robot and the object are not occurred. [Figure 2] This paper presents a single-camera depth-space method, known in this field, for calculating the minimum distance between a robot and an object. [Figure 3] This paper presents the multi-sensor depth space technique, known in this field for calculating the minimum distance between a robot and an object. [Figure 4A] The images shown are a series of images according to the embodiment of this disclosure, used to calculate the minimum distance between the robot and the obstacle, ultimately leading to the detection of the obstacle's edge using depth information. [Figure 4B] The images shown are a series of images according to the embodiment of this disclosure, used to calculate the minimum distance between the robot and the obstacle, ultimately leading to the detection of the obstacle's edge using depth information. [Figure 4C] The images shown are a series of images according to the embodiment of this disclosure, used to calculate the minimum distance between the robot and the obstacle, ultimately leading to the detection of the obstacle's edge using depth information. [Figure 4D] The images shown are a series of images according to the embodiment of this disclosure, used to calculate the minimum distance between the robot and the obstacle, ultimately leading to the detection of the obstacle's edge using depth information. [Figure 5] This is a flowchart illustrating an embodiment of the present disclosure, showing a method and data flow for calculating the minimum distance between a robot and an obstacle. [Figure 6] This is a top view showing a multi-camera system for calculating the minimum distance between a robot and an obstacle, where each camera independently applies the calculation method shown in Figure 5 according to an embodiment of this disclosure. [Figure 7] Figure 6 is a flowchart illustrating an embodiment of the present disclosure showing a method for calculating the minimum distance between a robot and an obstacle using the multi-camera system. [Figure 8A] To determine the minimum robot-obstacle distance that closely correlates with actual measurements, Figure 6 and Figure 7 illustrate how the system and method according to the embodiments of this disclosure are applied, showing scenes captured simultaneously by two cameras with different viewpoints. [Figure 8B] To determine the minimum robot-obstacle distance that closely correlates with actual measurements, Figure 6 and Figure 7 illustrate how the system and method according to the embodiments of this disclosure are applied, showing scenes captured simultaneously by two cameras with different viewpoints. [Figure 9A] This document describes three different obstacle scenarios and the calculation of the minimum distance using the three-camera system according to the embodiment of this disclosure, which results from each scenario. [Figure 9B] This document describes three different obstacle scenarios and the calculation of the minimum distance using the three-camera system according to the embodiment of this disclosure, which results from each scenario. [Figure 9C]This document describes three different obstacle scenarios and the calculation of the minimum distance using the three-camera system according to the embodiment of this disclosure, which results from each scenario. [Modes for carrying out the invention]

[0010] The following description of embodiments of this disclosure relating to the calculation of the minimum distance to a dynamic object in a robotic workspace is purely illustrative and is not intended to limit the disclosed equipment and techniques, or their application or use.

[0011] Industrial robots are well known to be used for various manufacturing, assembly, and material handling operations. Many robot workspaces contain obstacles, sometimes even within the robot's path of movement. In other words, without proactive motion planning, a part of the robot may collide with or come close to an obstacle as it moves from its current position to its destination. Obstacles can be fixed structures such as machinery, fixtures, and tables, or they can be dynamic (moving) objects such as people, forklifts, and other machinery. Methods have been developed in this field to calculate robot motion so that the tool follows a path to its destination while avoiding collisions with obstacles. However, in the case of dynamic objects, it has always been difficult to determine the location of the object and calculate the minimum distance from the object to the robot in real time, which is necessary for collision avoidance motion planning.

[0012] One method for calculating the distance between a robot and an object involves defining control points on the robot and calculating the 3D coordinates of these control points based on the robot's posture at every time step to determine the distance from the obstacle point to the robot control point. Obstacles can be defined by point clouds derived from sensor data or by other methods discussed further below.

[0013] Figure 1 shows an example where a human occupies the working space of an industrial robot, and illustrates the calculation of the minimum distance that needs to be carried out to reliably prevent collisions between the robot and the object. In Figure 1, the robot 100 is operating within the working space. A plurality of control points (such as 102 - 106) are defined on the robot 100. The control points 102 - 106 can be defined at important points of the robot's skeleton (for example, the center of joints or points along the center line of the arm), or the control points can be defined on the outer surface of the actual arm shape. The human 130 occupies a part of the working space in which the robot 100 is operating. The human 130 represents a dynamic object that can enter the working space, move around the periphery of the working space, or move out of the working space. Determining the minimum distance between the robot 100 and the human 130 at all time steps is essential for planning a collision - free robot path.

[0014] One method for obtaining the minimum distance between a robot and an obstacle is to place a number of sensors around the working space and construct a point cloud that defines the obstacle from the sensor data. Then, the 3D distance from the points in the point cloud to the robot control points is calculated. Some of these distances are shown by the lines indicated as 110 in Figure 1, and the minimum distance is specified from these lines. The point cloud method can result in accurate minimum distance results between the robot and the obstacle. However, the point cloud of the obstacle typically contains thousands of 3D points, and since each point in the point cloud needs to be checked against each robot control point, the calculation of the point cloud minimum distance is very computationally intensive. For this reason, other minimum distance calculation methods have been developed. One problem inherent in these other methods is that, as shown as distance 140 in Figure 1, the apparent minimum distance between the obstacle and the robot control point obtained from any single viewpoint can be significantly smaller than the actual 3D distance. This effect will be further discussed below.

[0015] Figure 2 shows the depth space method using a single camera for calculating the minimum distance between a robot and an object, which is known in the art. One 3D camera 200 is represented by the camera center point 202. The camera 200 has an imaging plane 210, and each pixel in the imaging plane has coordinates (px , p y ) has. Since camera 200 is a 3D camera, each pixel also has a depth or distance value d. Using the depth space method depicted in Figure 2, camera 200 provides a series of consecutive images, each image is analyzed to detect obstacles that may require a collision avoidance strategy by the robot. In the following explanation, assume that camera 200 in Figure 2 is providing images of the scene shown in Figure 1.

[0016] Camera 200 in Figure 2 detects obstacle points 220 and 230. Obstacle point 220 could be a point on the arm or hand of human 130. Obstacle point 230 could be a point on some other object in the background of Figure 1. A control point 240 is also shown, which could be one of the control points 102-106 on robot 100 in Figure 1. The location of control point 240, including the distance value d4, can be determined from the known robot kinematics and the spatial calibration of the camera to the workspace coordinate system. From the distance value d2, it can be seen that obstacle point 220 is much closer to camera 200 than control point 240. However, since camera 200 cannot "see through" obstacles, it is not possible to know how much space is occupied behind obstacle point 220. This effect is known as occlusion. Therefore, for the pixel representing obstacle point 220, all depth values ​​greater than the depth d2 must be considered occupied. The occupied space is indicated by line 222.

[0017] The 3D coordinates of the obstacle point 220 can be obtained based on the location of the corresponding pixel in the imaging plane 210, the camera calibration parameters, and the distance d2. The 3D coordinates of the control point 240 are known as described above. Therefore, the distance can be calculated between the obstacle point 220 and the control point 240. However, there is no guarantee that the calculated distance is the minimum distance between the object detected at the obstacle point 220 and the robot. On the contrary, due to the possibility that any or all points along the occlusion and line 222 are occupied, the worst-case minimum distance between the object and the control point 240 needs to be calculated as the minimum distance between the control point 240 and line 222, and this minimum distance exists along a line perpendicular to line 222. This minimum distance is the distance projected onto a plane parallel to the camera imaging plane 210, indicated by arrow 224.

[0018] The obstacle point 230 has a distance d3 measured by the camera, and this distance is greater than the distance d4 to the control point 240. Therefore, in the case of the obstacle point 230, the occluded space indicated by line 232 is in front of the control point 240 and thus does not affect the calculation of the minimum distance. Therefore, the distance from the obstacle point 230 to the control point 240 can be calculated based on the actual 3D coordinates of each point (using the sum of the squares of the differences in x, y, and z coordinates). The resulting distance indicated by arrow 234 is the exact minimum distance between the object detected at the obstacle point 230 and the control point 240 on the robot.

[0019] Figure 2 shows the drawbacks of the depth space method using a single camera for calculating the minimum distance between a robot and an obstacle. That is, the minimum distance of an object closer to the camera 200 than the distance from the robot to the camera 200 needs to be calculated using a worst-case assumption due to occlusion. This is evident from the fact that the distance indicated by arrow 224 in Figure 2 (the worst-case minimum distance) is much smaller than the actual distance between the obstacle point 220 and the control point 240. Using the worst-case assumed minimum distance between the robot and the obstacle, the robot control device will calculate a path that deviates significantly from its own path unnecessarily in order for the robot to avoid the obstacle point 220.

[0020] Figure 3 shows a multi-sensor depth-space method for calculating the minimum distance between a robot and an object, which is known in this field. The multi-sensor depth-space method shown in Figure 3 was developed to overcome the shortcomings of the aforementioned single-camera depth-space method.

[0021] On the left side of Figure 3, the depth-space arrangement with a single camera as in Figure 2 is again depicted. This includes the aforementioned imaging plane 210, obstacle points 220, 230, and control point 240. On the right side of Figure 3, the multiple sensor expansion is shown. Camera 200 (from Figure 2) is shown on the left. Camera 200 is now the primary camera, and at least one additional camera or sensor 300 is provided. For the purposes of this explanation, this additional camera will be referred to as camera 300 and is assumed to have both imaging plane and distance-measuring capabilities as described above, relative to camera 200.

[0022] In the multi-sensor depth space method, a depth grid map 310 is constructed to overcome the occupancy problem by determining the actual spatial occupation by obstacles in the robot workspace. The depth grid map 310 is a tapered hexahedron 3D volume grid radiating from the camera center point 202. Data from camera 200 (main camera) is used to construct the overall cell structure of the depth grid map 310. The cell structure of the depth grid map 310 is kept constant for all images from camera 200, and the depth grid map 310 preferably covers substantially the entire range of motion of the robot workspace.

[0023] For each camera image from camera 200, an obstacle point is identified as described above. Each obstacle point is then assigned to a corresponding occupied cell in the depth grid map 310. For example, obstacle point 220 corresponds to occupied cell 320 in the depth grid map 310. Cell 320 is determined to be occupied based on the distance d2 from camera 200 to obstacle point 220. In the multi-sensor depth space method, data from camera 300 (second camera / sensor) is used to determine whether cells "behind" the occupied cell (which is occupying from camera 200) are occupying by an obstacle. Cells 330 and 340 are "behind" the occupied cell 320 as seen from camera 200. Therefore, data from camera 300 is used to determine whether cells 330 and 340 are actually occupied by an obstacle. For example, if obstacle point 220 is on a thin object with almost no depth, then cells 330 and 340 are empty. Every cell in the depth grid map 310 is determined to be occupied or empty in this way.

[0024] Once the occupancy status of cells in the depth grid map 310 is determined, the minimum distance between the robot and the obstacle can be calculated by determining the distance from each control point on the robot to each occupied cell in the depth grid map 310. This multi-sensor depth space method largely overcomes the occupancy problem by using data from additional sensors. However, for large workspaces, the depth grid map 310 (which is actually based on an image with thousands of pixels, much finer than shown in Figure 3) can easily contain hundreds of thousands of grid cells. Therefore, the calculation of the minimum distance to the robot control point can become computationally intensive.

[0025] This disclosure describes a method developed to overcome the occlusion problem of single-camera depth-space methods and the computational complexity problem of multi-sensor depth-space methods. This disclosure describes a method for calculating the minimum distance between a robot and an object, which dramatically reduces the number of points that need to be evaluated and can be applied to either a single-camera configuration or a multi-camera configuration. This method is illustrated with reference to the following figures.

[0026] Figures 4A to 4D show a series of images according to the embodiment of this disclosure, used to calculate the minimum distance between a robot and an obstacle, ultimately leading to obstacle edge detection using depth information. The image processing steps in Figures 4A to 4D provide the basis for the efficient minimum distance calculation method disclosed.

[0027] Figure 4A shows a camera image 400 of the robot workspace 402 prior to any of the disclosed processing steps. Image 400 is a digital image from a 3D camera of the type known in the art and described above. Image 400 includes the robot 410 along with background objects such as a control device 420, a pipe 422, a door 424, and a floor 426. Image 400 also includes a human 430 who has entered the workspace 402 holding an object 432. The human 430 and object 432 represent obstacles that the robot 410 needs to detect and avoid.

[0028] Figure 4B shows image 440 with the robot 410 "removed" (digitally deleted). The human 430 and object 432 have simply been omitted from image 440 for clarity. The robot 410 has a base fixed to the floor 426 and several robot arms, all of which move around the workspace 402. Using the known robot arm shapes (from CAD models or approximated using geometric primitives) and the robot's posture data (joint positions) known from the control device 420, the position of the robot arms in image 440 can be calculated at each image time step. The calculated pixels, including the robot 410, are removed from image 440 as indicated by the white space 442. The position of the robot arms in image 440 can be calculated as long as the camera position and orientation are calibrated in advance to the workspace coordinate system, and the robot kinematics and arm shapes are given as described above. Removing the robot 410 simplifies image 440 to which further processing is performed to remove pixels that are not related to dynamic objects (obstacles).

[0029] In another embodiment, image 400 is analyzed to detect recognizable components of the robot 410 (arms, joints, arm-end tools), and only those parts visible in image 400 of the robot 410 are captured to create image 440. This technique prevents unintentionally removing parts of dynamic objects between the camera and the robot 410.

[0030] Figure 4C shows image 450 containing only the pixels of the edges of background objects 420-426. The background objects 420-426 do not move. This fact can be used to remove the background objects 420-426 in a separate image processing step. By processing images such as image 440 (without obstacles) or multiple images where the robot 400 is in different positions without obstacles, pixels containing the edges of background objects 420-426 can be detected. The result is stored as image 450 and can be used for further image processing in the disclosed method.

[0031] In one embodiment of the disclosed method, edge filtering is performed on image 400 (Figure 4A) to create an intermediate image containing only edge pixels, and the robot and background objects are removed from the edge-only intermediate image. In other embodiments, the edge filtering step may be performed at a different point in the process, such as after the robot has been removed.

[0032] Figure 4D shows image 460 containing only the edge pixels of the human 430 and object 432. Image 460 is created by starting with the original image 400, performing edge filtering to leave only the edge pixels in the first processed image, and removing the robot's edge pixels (Figure 4B) and the background object's edge pixels (Figure 4C) as described above. Image 460 contains only the edge pixels of dynamic objects in the workspace 402 (i.e., things other than the robot and background, i.e., potential obstacles). Each pixel in image 460 has a depth value given by the 3D camera that captured image 400. "Gaps" in the pixel depth data of image 460 (in 3D camera images, some pixels may have missing depth data because the depth signal around objects in front is often blocked) can be filled using the minimum depth (from the camera) of adjacent pixels, thereby ensuring that the depth value is taken from the dynamic object rather than the background.

[0033] Based on the aforementioned image processing and analysis, each pixel in image 460 has x and y coordinates on the imaging plane and a depth value from the camera; therefore, each pixel can be represented by its 3D coordinates in the working space coordinate system. The 3D coordinates of the robot control points are known as described above with respect to Figure 2. Thus, the 3D distance between each pixel in image 460 and the robot control point can be calculated. However, if the pixels in image 460 (representing dynamic objects, i.e., humans 430 and objects 432) are closer to the camera than the robot, there is no guarantee that the 3D distance will be the minimum distance between objects 430, 432 and the robot. Rather, due to occlusion and the fact that any or all points behind objects 430, 432 are occupied, the minimum (shortest) distance between objects 430-432 and the robot control point in the worst case must be calculated as the distance projected onto a plane parallel to the camera imaging plane, as described with respect to Figure 2. In this worst-case scenario, occlusion-based distance calculation is applied only if the distance from the camera to pixels 430 and 432 of objects is less than the distance from the robot control point to the camera. Otherwise, the actual 3D distance from each pixel to each robot control point is used.

[0034] Figure 5 is a flowchart 500 according to an embodiment of the present disclosure, illustrating a method and data flow for calculating the minimum distance between a robot and an obstacle. A 3D camera 502 is shown on the left, and line 504 provides a new image at every camera time interval. The camera time interval or time step may be based on any suitable frame rate, such as in the range of 10 to 30 frames per second. Higher or lower frame rates may also be used. The image on line 504 is processed on two parallel tracks, namely, a pixel processing (upper) track using a color image 510 and a depth processing (lower) track using a depth image 520.

[0035] Edge filtering is performed on the color image 510 in figure 512 to create an intermediate image. As described above, the edge filtering creates a version of the color image 510 that contains only the edge pixels of the image content. In figure 514, the robot and background are removed from the intermediate image to create the final processed pixel image. The robot and background are removed as shown in Figures 4B and 4C, respectively, as described above. The final processed pixel image contains only the edge pixels of any dynamic objects that were in the color image 510 (i.e., what remains after the robot and background have been removed). An example of the final processed image is shown in Figure 4D.

[0036] Using the depth image 520, a depth hole-filling filter is performed in figure 522. The depth hole-filling filter step in figure 522 provides depth values ​​to pixels where depth data is missing. This can occur when a foreground object blocks the depth signal from a background object in a neighboring pixel. The gaps in depth data are filled using the minimum depth value (from the camera) in the adjacent pixel, ensuring that the depth value is taken from a dynamic object in the foreground rather than a background object such as a door or wall.

[0037] The obstacle edge pixel depth image 530 is created from the final pixel image (figure 514) after processing to remove the robot and background, and the pixel depth data (figure 522) after depth hole filling filter processing. The obstacle edge pixel depth image 530 contains only the pixels representing the edges of the obstacle (dynamic object) from the color image 510, and has depth data for each pixel given from the depth image 520. Figure 4D above represents the obstacle edge pixel depth image 530.

[0038] At decision symbol 540, it is determined whether the obstacle edge pixel depth image 530 contains a number of pixels exceeding a predetermined threshold. Even if no dynamic objects are present in the camera image, the image processing step in Figure 5 is likely to produce a small number of "noise" pixels in the obstacle edge pixel depth image 530. The threshold check at decision symbol 540 is intended to determine whether a dynamic object actually exists in the image (and in the robot workspace). In other words, if only a dozen or twenty pixels remain in the obstacle edge pixel depth image 530, it is highly likely that no dynamic object is present in the image. On the other hand, if hundreds or thousands of pixels remain in the obstacle edge pixel depth image 530 and define a characteristic shape as shown in Figure 4D, this clearly represents a real object in the image. The threshold check at decision symbol 540 can be designed in any appropriate way to distinguish between mere "noise" pixels and real objects in the image. If the threshold is not exceeded at judgment symbol 540, this indicates that there are no moving objects in the image, and the process returns to receiving the next image from camera 502 at line 504.

[0039] If the threshold is exceeded in judgment symbol 540, this indicates the presence of a dynamic object in the image, and in figure 550, the distance from each pixel in the obstacle edge pixel depth image 530 to each robot control point is calculated. If the distance from the camera to the pixel in the obstacle edge pixel depth image 530 is less than the distance from the camera to the robot control point, the calculation in figure 550 is based on the aforementioned occlusion assumption. If the distance from the camera to the pixel in the obstacle edge pixel depth image 530 is greater than the distance from the camera to the robot control point, the calculation in figure 550 is based on the actual 3D distance from the pixel coordinates to the control point coordinates.

[0040] In unobstructed situations (where the robot is closer to the camera than the distance from the object to the camera), the 3D distance in figure 550 is calculated by first replacing each pixel in the obstacle edge pixel depth image 530 from its image coordinate system (x / y on the imaging plane plus depth) to the previously described and illustrated robot workspace coordinate system. The coordinates of the robot control points in the workspace coordinate system are already known from the robot's posture, as previously mentioned. Then, the actual 3D distance from the pixel in the obstacle edge pixel depth image 530 to the control point is calculated using the sum of squares of the differences in coordinates in three dimensions.

[0041] In occluded situations (where the robot is further from the camera than the object is from the camera), the distance calculation in figure 550 is performed assuming that all space behind the pixels in the obstacle edge pixel depth image 530 is occupied by the object. Therefore, the distance is calculated as the apparent distance from the pixel to the control point in a plane parallel to the camera imaging plane, as described above.

[0042] In figure 560, the minimum distance from the dynamic object to the robot is provided as the minimum value for distance calculations (for all pixels and control points) in figure 550. If the minimum distance in figure 560 is less than a threshold, the robot control unit includes the presence of the dynamic object in the calculations to ensure collision prevention in the robot motion program. The robot control unit may consider more data than just one minimum distance. For example, if the minimum distance is less than a threshold, the robot control unit may consider all obstacle pixels within a certain distance from any point on the robot when performing collision prevention calculations. The minimum distance from the dynamic object to the robot (and, if applicable, distance data from other pixels to the robot) is provided in figure 560, and the process returns to receiving the next image from camera 502 on line 504.

[0043] The method in Figure 5 can be performed on the robot control device itself, such as the control device 420 in Figure 4, or on a separate computer provided for image processing and minimum distance calculation. The control device 420 calculates robot joint motion commands, as is known in the art. If a separate computer is used to calculate the minimum distance between the robot and obstacles, a minimum distance vector (and, optionally, distance data between other pixels and the robot) can be provided to the robot control device at each image step, so that the control device can perform joint motion calculations, including collision avoidance considerations, based on the given minimum distance between the robot and obstacles.

[0044] The sequential image processing in Figures 4A to 4D and the flowchart in Figure 500 are all based on a single 3D camera providing images of the robot's workspace. The same method can be applied to a multi-camera system as described below.

[0045] Figure 6 is a top view showing a multi-camera system for calculating the minimum distance between a robot and an obstacle, with each camera independently applying the calculation method of Figure 5 according to an embodiment of this disclosure. The method of Figure 5 provides an effective and efficient calculation of the minimum robot-obstacle distance by using only edge pixels in the calculation when an unknown or dynamic object enters the robot workspace. However, occlusion can still be a problem with a single camera, causing the camera to calculate an apparent (conservative, worst-case) robot-obstacle distance that is much smaller than the actual 3D robot-obstacle distance.

[0046] Figure 6 shows a method that can improve minimum distance accuracy in occluded situations by very simply scaling up the methods of Figures 4 and 5 to multiple cameras. The robot 600 operates within the workspace as described above. 3D cameras 610, 610, and 630 are positioned near the periphery of the workspace to provide images from different viewpoints. Since dynamic objects entering the workspace (people, forklifts, etc.) are likely to move along the floor, cameras 610-630 are typically oriented nearly horizontally, that is, the cameras are oriented within a vector of ±20° from the horizontal, and the camera imaging planes are oriented within a vector of ±20° from the vertical.

[0047] Cameras 610, 620, and 630 communicate with computer 640 via hardware connection or wirelessly. Computer 640 performs image processing and minimum distance calculations and provides the minimum distance result to control device 650, which controls robot 600 by communicating with robot 600 in a manner known in the art. Alternatively, computer 640 may be removed, and control device 650 receives images from cameras 610, 620, and 630 and performs image processing and minimum distance calculations itself.

[0048] Cameras 610, 620, and 630 each have a field of view indicated by a triangle originating from that particular camera. Camera 610 has a field of view 612, camera 620 has a field of view 622, and camera 630 has a field of view 632. Fields of view 612, 622, and 632 are each labeled twice in Figure 6 for clarity. A dynamic object 640 (person) enters the robot workspace and is inside all of the fields of view 612, 622, and 632. The object 640 creates occlusion areas within the field of view of each camera, aligned with the field of view direction. Occluding area 614 is created in the field of view 612 (camera 610) in the field of view direction of object 640, occlusion area 624 is created in the field of view 622 (camera 620) in the field of view direction of object 640, and occlusion area 634 (very small) is created in the field of view 632 (camera 630) in the field of view direction of object 640.

[0049] Using the edge pixel distance calculation method described above for Figures 4 and 5, the minimum distance between the robot and the obstacle can be calculated for cameras 610, 620, and 630, respectively. First, we consider camera 610 and its field of view 612. Object 640 is closer to camera 610 than the distance from robot 600 to camera 610. Therefore, it is necessary to assume that an occluded situation exists and that object 640 occupies the occluded area 614. After the image processing steps described above for Figure 5, camera 610 calculates the minimum distance between the robot and the obstacle based on occluding, labeled as line 616.

[0050] Next, we consider camera 620 and its field of view 622. Object 640 is closer to camera 620 than the distance from robot 600 to camera 620. Therefore, we must assume that an occluded situation exists and that object 640 occupies the occluded area 624. After the image processing steps described above for Figure 5, camera 620 calculates the minimum robot-obstacle distance based on occluded areas, labeled as line 626.

[0051] Next, we consider camera 630 and its field of view 632. Object 640 is farther from camera 630 than robot 600 is from camera 630. Therefore, there is no occluded situation, and the occluded area 634 is not used in the minimum distance calculation. Instead, for camera 630, the actual 3D distance between the edge pixels and the robot control point is calculated and used in the minimum distance calculation. After the image processing steps described above for Figure 5, camera 630 calculates the actual minimum robot-obstacle distance, which is labeled as line 636.

[0052] After calculations for all three cameras, the minimum robot-obstacle distance 616 exists for camera 610, the minimum robot-obstacle distance 626 exists for camera 620, and the minimum robot-obstacle distance 636 exists for camera 630. At this point, the largest of the three minimum distances can be selected as the actual value without issue, because the actual 3D distance viewed from different observation points can never be smaller than the largest of those observed values. This fact becomes clear when considering occlusion assumptions that artificially reduce the minimum distances 616 and 626. Therefore, the minimum distance 636 is selected as the most accurate value among the three cameras, and this distance is the value used by the control unit 650 in calculating the robot collision avoidance motion plan.

[0053] Figure 7 is a flowchart 700 according to an embodiment of the present disclosure, illustrating a method for calculating the minimum distance between a robot and an obstacle using the multi-camera system of Figure 6. In step 710, an image frame is obtained from a first 3D camera identified as camera 1. As previously stated, each of the cameras in Figure 7 (and in Figure 6) provides a continuous stream of images at a certain frame rate, typically many frames per second. In figure 712, edge filtering is applied to the image to obtain an intermediate processed image containing only edge pixels. In figure 714, the robot and static background are removed from the intermediate processed image of figure 712 to obtain an image containing only the edges of the dynamic object, as previously stated. In figure 716, depth data (from the cameras) is applied to the object pixels remaining in the dynamic object edge image. After the depth data has been applied to the dynamic object edge image, in figure 718, the minimum distance of the object edge pixels to the robot control point is calculated as a true 3D distance or based on the aforementioned occlusion assumption. The calculation of the minimum distance in Figure 718 provides the minimum distance value from camera 1 to the current image frame, which matches the distance 616 depicted in Figure 6.

[0054] In step 720, an image frame is obtained from a second 3D camera identified as camera 2. The image from camera 2 is processed in figures 722-728, and a distance calculation is performed in figure 728 to provide the minimum distance value from camera 2 to the current image frame, which matches the distance 626 shown in figure 6.

[0055] In step 730, an image frame is obtained from the next 3D camera identified as camera N. The image from camera N is processed in figures 732-738, and a distance calculation is performed in figure 738 to provide the minimum distance value from camera N to the current image frame. When the value of N is 3, the distance calculated in figure 738 matches the distance 636 in Figure 6.

[0056] In Figure 740, the largest of the minimum distances calculated in Figures 718, 728, and 738 is selected as the most accurate representation of the minimum distance between the robot and the obstacle at the current camera image time step. As mentioned earlier regarding Figure 6, the minimum distance calculated from some of the multiple cameras may be artificially reduced due to occlusion assumptions, and it has been found that the largest minimum distance calculated from a large number of cameras is the most accurate value.

[0057] The multi-camera method shown in Figure 7 can be applied to any number of cameras, two or more. The more cameras there are (in Figure 740), the more accurate the final minimum distance value becomes. Increasing the number of cameras in the method of Figure 7 does not increase the computational load significantly because the minimum distance for each camera is calculated independently and in parallel, and ultimately only the maximum value needs to be selected (Figure 740). This is in contrast to the prior art method shown in Figure 3, which involves creating a 3D depth grid with numerous cameras and complex calculations related to the interrelationships of data from different cameras.

[0058] Furthermore, the methods of this disclosure shown in Figures 6 and 7 only require that the cameras be calibrated with respect to the robot workspace coordinate system. While the methods of this disclosure do not require the cameras to be calibrated relative to each other, this is not a trivial movement and, if not performed precisely, will result in errors in the depth grid shown in Figure 3.

[0059] Figures 8A and 8B show scenes captured simultaneously by two cameras with different viewpoints, illustrating how the system of Figure 6 and the method of Figure 7 are applied according to embodiments of this disclosure to determine the minimum robot-obstacle distance that closely correlates with actual measurements. Figure 8A represents an image of the robot workspace captured by a first camera (not shown). The robot 800 is operating within the workspace, controlled by the control device 810. Dynamic objects, including a person 820 holding an object 830, are also present in the workspace. Figure 8B represents the robot workspace in an image captured by a second camera (not shown). The robot 800, control device 810, person 820, and object 830 are also visible in the image of Figure 8B. Figures 8A and 8B represent two images captured using different cameras so that they can be processed using the method of Figure 7 substantially simultaneously, i.e., in the most recent camera time step with both cameras operating at the same frame rate.

[0060] Figures 8A and 8B show examples of the two-camera case for the system in Figure 6 and the method in Figure 7. In Figure 8A, it is clear that from the positions of person 820 and robot 800 on the floor, person 820 and object 830 are closer to the first camera than robot 800 is closer to the first camera. Therefore, it must be assumed that an occluded situation exists and that the space "behind" person 820 and object 830 from the perspective of the first camera is occupied. By applying the image processing technique in Figure 4 and the method in Figure 5 to the first camera, the minimum robot-obstacle distance based on occlusion is identified as shown by line 840. The minimum distance line 840 is measured in a plane parallel to the camera imaging plane.

[0061] In Figure 8B, the person 820 and the object 830 are at approximately the same distance from the second camera as they are from the second camera to the robot 800. Therefore, any occlusion is minimal. By applying the image processing technique in Figure 4 and the method in Figure 5 to the second camera, the minimum distance between the robot and the obstacle is identified as indicated by line 850. Regardless of whether the minimum distance line 850 is measured in a plane parallel to the camera imaging plane or in actual 3D workspace coordinates, the length of line 850 is considerably longer than the length of line 840.

[0062] Using the method shown in Figure 7, the minimum distance between the robot and the obstacle for the scenes depicted in Figures 8A and 8B is selected as line 850 (Figure 8B), which is the larger of the two minimum distances identified by the first and second cameras.

[0063] Figures 9A–9C illustrate three different obstacle scenarios and the calculation of the minimum distance using the three-camera system according to the embodiment of this disclosure, resulting in each scenario. Unlike Figures 8A / 8B, which show images of the same scene captured simultaneously by different cameras, Figures 9A–9C show three different images from the same camera captured at different times, with the moving object moving to a new location in each scenario. Each of Figures 9A–9C is discussed separately to show how the relationship between the camera's location and the obstacle's location around the workspace affects the calculated minimum distance for each camera. In each scenario, the distance values ​​from actual system tests are listed for each camera.

[0064] Camera 910 is positioned in the "front center" of the workspace, as depicted in Figures 9A to 9C. That is, Figures 9A to 9C are shown from the viewpoint of camera 910. As depicted in Figures 9A to 9C, camera 920 is positioned in the "rear left" of the workspace, and camera 930 is positioned in the right center of the workspace. In each of Figures 9A to 9C, the robot 900, accompanied by the control device 902, operates within the workspace. The robot 900 is depicted in the same posture in each of Figures 9A to 9C. Person 940 holds object 950, and both person 940 and object 950 represent potential obstacles to the robot 900. Both person 940 and object 950 are dynamic in that they can move into, out of, and around the workspace.

[0065] In Figure 9A, person 940 is located "front left" of the workspace, holding object 950 with their arm extended roughly horizontally towards the center of the workspace. In this situation, camera 910 is much closer to object 950 than robot 900 is, so camera 910 is definitely in an occluded situation. However, there is considerable lateral and vertical distance between object 950 and robot 900, so camera 910 perceives a minimum robot-obstacle distance based on an occlusion of approximately 69 centimeters (cm). Similarly, camera 920 is also closer to object 950 than robot 900 is, so camera 920 is in an occluded situation. Due to the occlusion assumption and the relative positions of object 950 and robot 900, camera 920 perceives a minimum robot-obstacle distance based on an occlusion of approximately 39 cm. Camera 930 is closer to robot 900 than object 950 is, so camera 930 is not in an occluded situation. Therefore, camera 930 calculates the minimum robot-obstacle distance based on the actual 3D coordinates of the edge pixels of object 950 and the control points on robot 900. Using the actual 3D coordinates, camera 930 calculates a minimum robot-obstacle distance of approximately 82 cm. According to the method in Figure 7, the largest of the three minimum distances is selected as the most accurate. Thus, a minimum distance of 82 cm from camera 930 is selected. This minimum distance value is preferably equivalent to a minimum distance of 81 cm measured by a motion capture system placed in the actual workspace for the situation in Figure 9A.

[0066] In Figure 9B, person 940 is positioned to the left of the center of the workspace, with their arms extended roughly horizontally towards the center of the workspace, still holding object 950. In this situation, camera 910 is much closer to object 950 than robot 900 is, so camera 910 is still occluded. The lateral and vertical distances between object 950 and robot 900 are smaller here than in Figure 9A, and camera 910 recognizes a minimum robot-obstacle distance of approximately 35 centimeters (cm) based on occlusion. Camera 920 is also closer to object 950 than robot 900 is, so camera 920 is also occluded. Due to the occlusion assumption and the relative positions of object 950 and robot 900, camera 920 also recognizes a minimum robot-obstacle distance of approximately 35 cm. Camera 930 is closer to robot 900 than object 950 is, so camera 930 is still not occluded. Therefore, camera 930 calculates the minimum distance between the robot and the obstacle based on the edge pixels of object 950 and the actual 3D coordinates of the control points on robot 900, and this distance is approximately 83 cm. According to the method in Figure 7, the largest of the three minimum distances is selected as the most accurate, resulting in a minimum distance of 83 cm from camera 930. This minimum distance value is preferably equivalent to the minimum distance of 89 cm measured by a motion capture system in the actual workspace for the situation in Figure 9B.

[0067] In Figure 9C, person 940 is facing backward along the left side of the workspace, still holding object 950 with their arm extended roughly horizontally towards the center of the workspace. In this situation, camera 910 is at approximately the same distance from object 950 and robot 900, so camera 910 may or may not be occluded. Using the image processing technique shown in Figure 4 and the method in Figure 5, occlusion is assumed if the distance from the edge pixels of object 950 to the camera is less than the distance from the control point on the robot arm to the camera. Using appropriate calculations (either projection based on occlusion or true 3D distance), camera 910 calculates a minimum robot-to-obstacle distance of approximately 78 cm. In Figure 9C, camera 920 is definitely occluded. Due to the assumption of occlusion and the relative positions of object 950 and robot 900, camera 920 perceives a minimum robot-to-obstacle distance based on occlusion of approximately 20 cm. In Figure 9C, camera 930 is closer to robot 900 than to object 950, so camera 930 is still not in an occluded situation. Therefore, camera 930 calculates the minimum distance between robot and obstacle based on the edge pixels of object 950 and the actual 3D coordinates of the control points on robot 900, and this distance is approximately 77 cm. According to the method in Figure 7, the largest of the three minimum distances is selected as the most accurate, resulting in a minimum distance of 78 cm from camera 910. This minimum distance value is preferably equivalent to the minimum distance of 78 cm measured by a motion capture system in the actual workspace for the situation in Figure 9C.

[0068] The preceding explanation in Figures 9A to 9C is provided to show how the system in Figure 6 and the method in Figure 7 can be applied to real-world robot-obstacle situations to quickly, reliably, and accurately determine the minimum distance using three cameras. For the purposes of discussion, if the person 940 and the object 950 are located to the right and in front of the workspace, camera 920 is more likely to yield the largest (and most accurate) minimum distance.

[0069] As mentioned above, in Figures 9A to 9C, the control device 902 receives images from cameras 910, 920, and 930, and can perform all image processing and the distance calculation steps described in Figures 4 to 9. Alternatively, an independent computer may be provided to perform all image processing and the distance calculation steps, which will determine whether a dynamic object is present in the workspace and, if so, what the minimum distance between the robot and the obstacle is. In either case, if a dynamic object is present, the minimum distance between the robot and the obstacle is used by the control device 902 to determine whether a collision avoidance strategy is required for the robot 900.

[0070] Throughout the above description, various computers and control devices have been described and implied. It should be understood that the software applications and modules for these computers and control devices run on one or more computer devices having processors and memory modules. In particular, this includes the processors of the robot control device 650 and computer 640 (if used) in Figure 6, as previously mentioned. Specifically, the processors of the control device 650 and / or computer 640 (if used) are configured to perform image processing steps on images from one or more cameras in the manner described throughout the above disclosure to calculate the minimum distance between the robot and obstacles. The same applies to the robot control devices depicted in Figures 4, 8, and 9, which can be used to perform the disclosed calculations with or without an independent computer.

[0071] As outlined above, the disclosed method for calculating the minimum distance to a dynamic object in a robotic workspace offers significant advantages over prior art methods. The disclosed image processing step for detecting edge pixels of an object provides efficient identification of the minimum distance vector relative to the camera, with or without occlusion. By combining multiple cameras in parallel computing, accurate minimum distances can be detected quickly and accurately without the time-consuming and error-prone camera calibration and 3D depth grid calculations required in prior art methods.

[0072] Numerous preferred embodiments and models of methods and systems for calculating the minimum distance to a dynamic object in a robotic workspace have been described above, but those skilled in the art will recognize modifications, substitutions, additions, and sub-combinations thereto. Therefore, the following appended claims and any claims introduced hereafter are intended to be construed as including all such modifications, substitutions, additions, and sub-combinations, as all such modifications, substitutions, additions, and sub-combinations are in the true spirit and scope of the claims.

Claims

1. A method for calculating the minimum distance from a robot to an obstacle in the robot's workspace, To provide an image of the workspace from a 3D camera, Using a computer having a processor and memory, edge filtering is performed on the image to obtain an edge image having only pixels that define the edges of objects in the workspace. The robot and background objects are removed from the aforementioned edge image to obtain an edge pixel image of the dynamic object. The process involves assigning a depth value to each pixel in the edge pixel image of the dynamic object to create a final image, wherein the depth value is taken from the depth data in the image provided by the camera. The distance from each pixel in the final image to each of the multiple control points on the robot is calculated, The minimum distance between the robot and the obstacle is determined from the calculated distance, Includes, Calculating the distance from each pixel in the final image to each of the plurality of control points on the robot includes performing an occlusion check, the occlusion check is positive if the distance from the camera to the pixel is less than the distance from the camera to the control point, If the occlusion confirmation is positive, the distance from the pixel to the control point is calculated as the distance projected onto a plane parallel to the camera imaging plane. Providing simultaneous images of the workspace from one or more additional three-dimensional cameras, each of which is located at a different location near the periphery of the workspace; determining the minimum robot-obstacle distance from the images from each of the additional cameras; and selecting the largest of the determined minimum robot-obstacle distances for use in robot motion programming. method.

2. The method according to claim 1, wherein removing the robot from the edge image includes determining the robot posture using robot joint position data based on forward kinematics calculations, replacing the robot posture with the respective positions and orientations of the robot arms in the image coordinate system, and removing each of the robot arms from the edge image.

3. The method according to claim 1, wherein removing the background object from the edge image comprises preparing a previously captured background reference image in which it is known that no dynamic objects exist in the workspace, performing edge filtering on the background reference image to create a background reference edge image, and removing the background reference edge image from the edge image.

4. A method for calculating the minimum distance from a robot to an obstacle in the robot's workspace, To provide an image of the workspace from a 3D camera, Using a computer having a processor and memory, edge filtering is performed on the image to obtain an edge image having only pixels that define the edges of objects in the workspace. The robot and background objects are removed from the aforementioned edge image to obtain an edge pixel image of the dynamic object. The process involves assigning a depth value to each pixel in the edge pixel image of the dynamic object to create a final image, wherein the depth value is taken from the depth data in the image provided by the camera. The distance from each pixel in the final image to each of the multiple control points on the robot is calculated, The minimum distance between the robot and the obstacle is determined from the calculated distance, A method for filling in gaps in depth data before assigning the depth value to each pixel, the method comprising filling in gaps in the image, the depth value being assigned to pixels in the image where the depth data is missing based on the pixel having the depth value closest to the camera within a predetermined adjacent distance.

5. The method according to claim 1, further comprising: determining whether the number of pixels in the final image is greater than a predetermined threshold; calculating the distance from each pixel in the final image to each of the plurality of control points on the robot if the number of pixels is greater than the threshold; and determining that there are no obstacles in the work space if the number of pixels in the final image is less than or equal to the predetermined threshold.

6. The method according to claim 1, wherein if the occlusion confirmation is not positive, the distance from the pixel to the control point is calculated as the actual three-dimensional distance using the sum of squares of differences in the coordinate formula.

7. The method according to claim 1, wherein each of the control points on the robot is defined as a point on the outer surface of the robot arm, along the axis of motion of the robot arm, or at the center of a joint.

8. The method according to claim 1, further comprising using the minimum distance between the robot and the obstacle in a collision avoidance robot motion planning algorithm in a robot control device.

9. The method according to claim 8, wherein the computer is the robot control device.

10. A method for calculating the minimum distance from a robot to an obstacle in the robot's workspace, To provide simultaneous images of the workspace from each of multiple three-dimensional (3D) cameras, Using a computer having a processor and memory, edge filtering is performed on each of the images to obtain an edge image having only pixels that define the edges of objects in the workspace. The robot and background objects are removed from each of the aforementioned edge images to obtain an edge pixel image of the dynamic object. The process involves assigning a depth value to each pixel in each of the edge pixel images of the dynamic object to create a final image, wherein the depth value is taken from the depth data in the image provided by the camera. If the number of pixels in each of the aforementioned final images is greater than or equal to a predetermined threshold, the distance from each pixel in the final image to each of the multiple control points on the robot is calculated. From the calculated distance, determine the minimum distance between the robot and the obstacle for each of the cameras, The largest of the minimum distances between the robot and obstacles determined above is selected and used in the robot motion programming. If the number of pixels in all of the final images is less than the predetermined threshold, it is determined that there are no obstacles in the workspace. Includes, A method comprising calculating the distance from each pixel in the final image to each of the plurality of control points on the robot, wherein the occlusion check is positive if the distance from the camera to the pixel is less than the distance from the camera to the control point, and if the occlusion check is positive, the distance from the pixel to the control point is calculated as the distance projected onto a plane parallel to the camera imaging plane.

11. A system for calculating the minimum distance from a robot to an obstacle in the robot's workspace, Multiple three-dimensional (3D) cameras, each providing a simultaneous image of the aforementioned workspace, A computer having a processor and memory, which communicates with the plurality of cameras, An edge filter is applied to each of the aforementioned images to obtain an edge image having only pixels that define the edges of objects in the workspace. The robot and background objects are removed from each of the aforementioned edge images to obtain an edge pixel image of the dynamic object. The process involves assigning a depth value to each pixel in each of the edge pixel images of the dynamic object to create a final image, wherein the depth value is taken from the depth data in the image provided by the camera. If the number of pixels in each of the aforementioned final images is greater than or equal to a predetermined threshold, the distance from each pixel in the final image to each of the multiple control points on the robot is calculated. From the calculated distance, determine the minimum distance between the robot and the obstacle for each of the cameras, The largest of the minimum distances between the robot and obstacles determined above will be selected and used. If the number of pixels in all of the final images is less than the predetermined threshold, it is determined that there are no obstacles in the workspace. The computer is configured to perform the following: Equipped with, The system includes calculating the distance from each pixel in the final image to each of the plurality of control points on the robot, wherein the occlusion check is positive if the distance from the camera to the pixel is less than the distance from the camera to the control point, and if the occlusion check is positive, the distance from the pixel to the control point is calculated as the distance projected onto a plane parallel to the camera imaging plane.

12. A system for calculating the minimum distance from a robot to an obstacle in the robot's workspace, One or more three-dimensional (3D) cameras, each providing simultaneous images of the aforementioned workspace, A computer having a processor and memory, which communicates with one or more cameras, An edge filter is applied to each of the aforementioned images to obtain an edge image having only pixels that define the edges of objects in the workspace. The robot and background objects are removed from each of the aforementioned edge images to obtain an edge pixel image of the dynamic object. The process involves assigning a depth value to each pixel in each of the edge pixel images of the dynamic object to create a final image, wherein the depth value is taken from the depth data in the image provided by the camera. If the number of pixels in each of the aforementioned final images is greater than or equal to a predetermined threshold, the distance from each pixel in the final image to each of the multiple control points on the robot is calculated. From the calculated distance, determine the minimum distance between the robot and the obstacle for each of the cameras, When two or more cameras are used, the largest of the minimum robot-obstacle distances determined above shall be selected and used. If the number of pixels in all of the final images is less than the predetermined threshold, it is determined that there are no obstacles in the workspace. The computer is configured to perform the following: Equipped with, The computer is configured to perform a system that fills in the depth data before assigning the depth value to each pixel, which includes assigning the depth value to pixels in the image where the depth data is missing based on the pixel having the depth value closest to the camera within a predetermined adjacent distance.

13. The system according to claim 11, wherein the plurality of cameras are located near the periphery of the workspace, and the images from each of the cameras are captured with the cameras oriented substantially horizontally.

14. The system according to claim 11, further comprising a robot control device that communicates with the computer and is configured to use the minimum distance between the robot and the obstacle in a collision avoidance robot motion planning algorithm.

15. The system according to claim 11, wherein the computer is a robot control device configured to use the minimum distance between the robot and the obstacle in a collision avoidance robot motion planning algorithm.

Images