Control device, robot system, object presence / absence determination mehtod, and program

Abstract

A control device comprising: an image acquisition unit which acquires a distance image captured by a visual sensor; and a determination unit which, on the basis of the distance image, determines whether or not an object has been placed within a range specified by predetermined input information.

Description

Control device, robot system, object presence / absence determination method, and program

[0001] The present disclosure relates to a control device for an industrial robot, a robot system, an object presence / absence determination method, and a program.

[0002] A robot system is known that performs a workpiece picking operation by detecting a workpiece using a visual sensor, using a robot to pick up a workpiece (such as a bulk pile of workpieces) that is placed in an undefined position, and placing it on a workbench, a conveying device, or the like.

[0003] For example, Patent Document 1 describes a handling system for transporting a workpiece when a plate-shaped metal workpiece obtained by fusion cutting or the like is subjected to grinding finishing.

[0004] Patent Document 2 describes a monitoring device provided in a robot system that performs a predetermined task on a workpiece. This monitoring device is equipped with a three-dimensional camera and determines whether an object is present within a restricted area based on the distance from the camera to multiple measurement points set on the workpiece surface.

[0005] JP 2007-021635 A JP 2018-195959 A

[0006] In a robot system that handles workpieces, such as by picking them up or transporting them, a situation may arise in which, for some reason, a workpiece placed by a previous operation remains in the location where it should be placed. When this situation occurs, there is a possibility that problems (such as a stoppage of the production cycle) may occur due to interference between the workpieces when the robot attempts to place them.

[0007] There is a demand for technology that can detect whether or not an object is present in an inspection location, thereby enabling reliable detection of defects in workpiece handling operations by robots and prevention of defects in advance.

[0008] One aspect of the present disclosure is a control device that includes an image acquisition unit that acquires a distance image captured by a visual sensor, and a judgment unit that determines, based on the distance image, whether an object is placed within a range specified by predetermined input information.

[0009] These and other objects, features and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments of the invention illustrated in the accompanying drawings.

[0010] 10 is a diagram illustrating a device configuration of a robot system including a robot control device according to an embodiment. FIG. 10 is a functional block diagram of a robot control device and a visual sensor control unit. FIG. 10 is a diagram illustrating a distance image acquired by a three-dimensional camera. FIG. 10 is a diagram illustrating a first example of a search range setting function performed by a search range setting unit. FIG. 10 is a diagram illustrating a second example of a search range setting function performed by a search range setting unit. FIG. 10 is a diagram illustrating a third example of a search range setting function performed by a search range setting unit. FIG. 10 is a diagram illustrating a fourth example of a search range setting function performed by a search range setting unit. FIG. 10 is a diagram illustrating an imaging range and a search range. FIG. 10 is a diagram illustrating a histogram for the first search range illustrated in FIG. 8. FIG. 10 is a diagram illustrating a histogram for the second search range illustrated in FIG. 8. FIG. 10 is a flowchart illustrating a general representation of an object presence / absence determination function executed by a robot control device. FIG. 10 is a flowchart illustrating specific operations of an object presence / absence determination process executed by a robot control device. FIG. 10 is a flowchart illustrating a case where an object presence / absence determination process is applied to a workpiece transfer process. FIG. 10 is a diagram illustrating a state in which a visual sensor captures a distance image when nothing is placed on a placement surface, and a histogram of the distance image acquired at that time. FIG. 1 is a diagram showing a state in which a visual sensor captures a distance image when a workpiece is placed on the placement surface, and a histogram of the distance image acquired at that time. FIG. 2 is a diagram showing an example of a graphical user interface for setting a threshold. FIG. 3 is a flowchart when an object presence / absence determination process is applied to the process of removing bulk workpieces. FIG. 4 is a diagram showing a state in which a visual sensor captures a distance image when there are many workpieces in a container, and a histogram of the distance image acquired at that time. FIG. 5 is a diagram showing a state in which a visual sensor captures a distance image when there is no object in the container, and a histogram of the distance image acquired at that time. FIG. 6 is a diagram showing a state in which a visual sensor captures a distance image when there is one workpiece remaining in the container, and a histogram of the distance image acquired at that time.

[0011] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, like components or functional parts are designated by like reference numerals. The scales of these drawings have been changed appropriately to facilitate understanding. Furthermore, the embodiment shown in the drawings is one example for implementing the present invention, and the present invention is not limited to the illustrated embodiment.

[0012] FIG. 1 is a diagram showing the equipment configuration of a robot system 100 including a robot control device 50 according to one embodiment. As shown in FIG. 1, the robot system 100 includes a robot 10 having a hand 33 mounted on the end of its arm, a robot control device 50 for controlling the robot 10, a teaching pendant 40 connected to the robot control device 50, a visual sensor 70 attached to the end of the arm of the robot 10, and a visual sensor control device 20 for controlling the visual sensor 70. The robot system 100 can, for example, detect an object 1 on a workbench 2 using the visual sensor 70 and handle the object 1 with a hand 33 mounted on the robot 10. The robot system 100 is further configured to determine, based on a range image captured by the visual sensor 70, whether an object is placed within a range specified by predetermined input information.

[0013] The robot system 100 may further include a storage device 80 for saving execution history data and other information. The storage device 80 is, for example, an external memory connected to the robot control device 50. Alternatively, the storage device 80 may be a storage device or an external computer connected to the robot control device 50 via a network. Note that while FIG. 1 shows a configuration in which the robot system 100 includes an independent device as the storage device 80, the function of the storage device 80 may be installed in the robot control device 50 or the teaching pendant 40. A configuration example in which the storage device 80 is connected to the teaching pendant 40 is also possible.

[0014] In this embodiment, the robot 10 is a vertical articulated robot, but other types of robots, such as a parallel link robot or a dual-arm robot, may be used depending on the purpose of the work. The robot 10 can perform a desired work using an end effector attached to the wrist. Figure 1 shows an example in which a hand 33 is used as the end effector.

[0015] The visual sensor 70 functions as a two-dimensional camera that captures grayscale and color images, and as a three-dimensional camera that captures distance images. The robot system 100 may be equipped with multiple visual sensors. Examples of the three-dimensional camera include a time-of-flight (TOF) camera that captures distance images using a time-of-flight method, or a stereo camera including two cameras. While FIG. 1 illustrates an example configuration in which the visual sensor 70 is mounted on the robot 10, the visual sensor 70 may also be a fixed camera that is fixed within the workspace.

[0016] The visual sensor control device 20 stores a model pattern of the object and can execute image processing to detect the object by pattern matching the image of the object in the captured image with the model pattern. It is assumed that the visual sensor 70 has been calibrated, and the visual sensor control device 20 stores calibration data that defines the relative positional relationship between the visual sensor 70 and the robot 10. This allows the position on the two-dimensional image captured by the visual sensor 70 to be converted into a position on a coordinate system (such as a robot coordinate system) fixed to the workspace.

[0017] In FIG. 1, the visual sensor control device 20 is configured as a device separate from the robot control device 50, but the function of the visual sensor control device 20 may be incorporated into the robot control device 50.

[0018] The robot control device 50 controls the operation of the robot 10 in accordance with an operation program or commands from the teaching pendant 40. The robot control device 50 may have a hardware configuration as a general computer having a processor 51 (FIG. 2), memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, an input / output interface, a network interface, etc.

[0019] The teaching pendant 40 is used as an operation terminal for teaching (creating a program) the robot 10 and for performing various settings. A teaching device configured as a tablet terminal or the like may be used as the teaching pendant 40. The teaching pendant 40 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, a display unit 41 ( FIG. 2 ), an input / output interface, a network interface, etc. The display unit 41 ( FIG. 2 ) has, for example, a liquid crystal display.

[0020] The visual sensor control device 20 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), storage device, operation unit, display unit, input / output interface, network interface, etc.

[0021] 2 is a functional block diagram of the robot control device 50 and the visual sensor control device 20. As shown in Fig. 2, the robot control device 50 includes an operation control unit 151, an image acquisition unit 152, a histogram creation unit 153, a determination unit 154, a threshold setting unit 155, an imaging range setting unit 156, a search range setting unit 157, a history image storage unit 158, a workpiece removal execution unit 159, and a workpiece transfer execution unit 160. These functional blocks may be realized by the processor 51 of the robot control device 50 executing software. The robot control device 50 further includes a storage unit 161.

[0022] The storage unit 161 is a storage device formed of, for example, a non-volatile memory or a hard disk drive, etc. The storage unit 161 stores a robot program for controlling the robot 10, a program (vision program) for performing image processing such as workpiece detection based on images captured by the visual sensor 70, various setting information, etc.

[0023] The operation control unit 151 controls the operation of the robot in accordance with the robot program or commands from the teaching pendant 40. The robot control device 50 is equipped with a servo control unit (not shown) that executes servo control for the servo motors of the respective axes in accordance with commands for the respective axes generated by the operation control unit 151.

[0024] The image acquisition unit 152 has a function of acquiring distance images and two-dimensional images captured by the visual sensor 70 from the visual sensor control device 20 .

[0025] The histogram creation unit 153 provides a function for creating a histogram, which is a frequency distribution relating to the distance information of each point in the acquired distance image. The histogram creation unit 153 may further have a function for displaying the created histogram on a display screen (for example, the display unit 41 of the teaching pendant 40).

[0026] The determination unit 154 provides a function of determining whether or not an object is placed within a range specified by predetermined input information, based on the distance image acquired by the image acquisition unit 152. Here, the range specified by the predetermined input information may be a search range in which the determination unit 154 performs a search for determination, or the imaging range of the visual sensor 70. For example, the determination unit 154 may be configured to determine whether or not an object is placed on the inspection target surface within the specified search range, based on the histogram created by the histogram creation unit 153.

[0027] The threshold setting unit 155 provides a function for setting a threshold for distinguishing between distance information of the inspection target surface on the histogram and distance information of other objects on the inspection target surface. The threshold setting unit 155 may have a function for automatically setting a threshold based on a distance image captured by the visual sensor 70. The threshold setting unit 155 may be configured to accept a threshold setting via a UI (user interface) screen. Alternatively, the threshold setting unit 155 may be configured to accept a threshold input from an external device. The threshold setting unit 155 may set a threshold according to information related to threshold setting (e.g., variables defining a threshold) described in a robot program (e.g., a workpiece removal program, a workpiece transport program, a vision program, etc.).

[0028] The imaging range setting unit 156 provides a function for setting the imaging range when the visual sensor 70 captures a distance image. The imaging range setting unit 156 may be configured to accept a designation of the imaging range via a UI screen. The imaging range setting unit 156 may also be configured to accept input of the imaging range from an external device. The imaging range setting unit 156 may set the imaging range (size, position, etc.) according to information regarding the designation of the imaging range (variables defining the imaging range, etc.) described in a robot program (e.g., workpiece removal program, workpiece transfer program, vision program, etc.). The imaging range setting unit 156 sends a signal to the visual sensor 70 (visual sensor control device 20) instructing it to capture an image within the set imaging range. Furthermore, when the position and orientation of the robot 10 are to be changed to move the imaging range, the imaging range setting unit 156 sends a command to the operation control unit 151 to change the position and orientation of the robot 10.

[0029] The search range setting unit 157 provides a function of setting, based on predetermined input information, a search range to be searched when the determination unit 154 determines whether an object is placed in the workspace. The predetermined input information may be, for example, information relating to the designation of the search range written in the robot program (such as variables that define the search range), setting information input via a UI (user interface), information input from an external device, etc. Details of setting the search range by the search range setting unit 157 will be described later.

[0030] The historical image storage unit 158 ​​provides a function of storing historical information including a historical image when the visual sensor 70 performs a process of detecting a workpiece, for example, in the storage device 80 .

[0031] The memory unit 161 stores a workpiece removal program for causing the robot 10 to perform the workpiece removal operation of the randomly stacked workpieces, and a workpiece transfer program for causing the robot 10 to remove the workpieces and transfer them to another location. The workpiece removal execution unit 159 is a functional block realized by the processor 51 executing the workpiece removal program. The workpiece removal execution unit 159 executes the randomly stacked workpiece removal operation in cooperation with the operation control unit 151 and the visual sensor control device 20. The workpiece transfer execution unit 160 is a functional block realized by the processor 51 executing the workpiece transfer program. The workpiece transfer execution unit 160 executes the workpiece transfer operation in cooperation with the operation control unit 151 and the visual sensor control device 20.

[0032] The visual sensor control device 20 includes an image processing unit 121 and a storage unit 122. The storage unit 122 is a storage device formed, for example, of a non-volatile memory. The storage unit 122 stores various data required for image processing, such as various settings used to generate distance images, calibration data, and workpiece model data. The image processing unit 121 executes various types of image processing, such as workpiece detection processing.

[0033] The distance image acquired by the visual sensor 70's function as a three-dimensional camera will be described with reference to FIG. 3 . FIG. 3 shows the state in which the visual sensor 70 captures images of three workpieces W1, W2, and W3 placed at different heights on a floor surface 90, and the captured distance image M1. A distance image is an image whose brightness varies depending on the height of objects within the image capture range. That is, the distance image represents a point cloud whose brightness varies depending on the distance from the camera. In the distance image M1, points are displayed in colors closer to white at higher heights and closer to black at lower heights. Therefore, in the distance image M1, the portion (point cloud) corresponding to workpiece W1 is displayed in the brightest color, the portion (point cloud) corresponding to workpiece W2 is displayed in the next brightest color, the portion (point cloud) corresponding to workpiece W3 is displayed in the next brightest color, and the portion (point cloud) corresponding to floor surface 90 is displayed in the darkest color.

[0034] The search range setting function of the search range setting unit 157 will be described in detail with reference to Figures 4 to 7. The search range setting function of the search range setting unit 157 includes the following (F1) to (F4). (F1) A function to set a search range specified on a captured image as a range defined on the real working space. (F2) A function to set a search range specified on a captured image in correspondence with the imaging range of the visual sensor. (F3) A function to set a search range in real space. (F4) A function to set a search range specified in real space in correspondence with the imaging range of the visual sensor.

[0035] In the above functions (F1) and (F2), the search range setting unit 157 accepts input for specifying the search range as a range on the image. For example, the search range setting unit 157 is configured to be able to accept, as input methods for specifying the search range as a range on the image, (a1) a method for specifying the search range as numerical information based on coordinates on the image, and (a2) an input method for specifying the search range on a GUI (graphical user interface).

[0036] In the above-described specification method (a1), the search range setting unit 157 accepts input specifying the search range E11 as numerical information based on coordinates on the distance image M11, as shown, for example, on the left side of FIG. 4 . In the example on the left side of FIG. 4 , the position of the search range E11 is specified as the coordinate values ​​((x, y) = (5, 2)) of the upper left corner and the widths in the x-axis and y-axis directions (Δx = 20, Δy = 10) in a coordinate system with the upper left corner of the distance image M11 as the origin. Note that, in this example, the search range is a rectangular region, and the position and size of the search range are specified as numerical information. However, the search range is not limited to a rectangle and may be specified as other shapes, such as a circle or a polygon. When specifying other shapes, the numerical information may include, for example, information specifying the geometric center, outer diameter, vertex positions, etc. The search range setting unit 157 may accept such a search range specification based on numerical information based on coordinates on the image as input from a robot program, or may accept a user operation to input numerical information values ​​on a UI. Such a UI screen may be provided on the display unit 41 of the teaching pendant 40.

[0037] In the above-described specification method (a2), the search range setting unit 157 provides a GUI that enables the operator to specify the position and size of a rectangular frame line indicating the search range on the distance image M11, as illustrated, for example, on the left side of FIG. 5 . Such a GUI may be displayed, for example, on the display unit 41 of the teaching pendant 40. The operator can specify the position and size of the frame indicating the search range E11 by moving the cursor C1 using a mouse or touch operation. Note that, although an example in which the search range is set on the GUI as a rectangular area has been shown, the search range may also be set on the GUI as another shape, such as a circle or a polygon. In specification method (a2), the operator can specify the search range through intuitive operations.

[0038] The search range setting unit 157 may be configured to accept a search range specification that simultaneously applies the above-described specification methods (a1) and (a2). In this case, the search range setting unit 157 accepts a specification in a procedure in which, for example, an operator sets a search range on a GUI and then precisely sets the search range using numerical information.

[0039] The above function (F1) will be described with reference to FIG. 4. Here, as an example, it is assumed that the above-mentioned designation method (a1) is used to designate the search range. The distance image M11 shown on the left side of FIG. 4 is a distance image of the area (image capture range R11) where the workpieces W1-W3 are placed on the installation surface 92 in the workspace. When the search range E11 is designated on the image using designation method (a1), the search range setting unit 157 sets the search range E11 designated on the image as a search range defined in real space (denoted by the same reference symbol E11 on the right side of FIG. 4), as shown on the right side of FIG. 4. In this case, the search range setting unit 157 can convert the search range on the image to the position of the search range in real space corresponding to the search range in the image by using calibration data, etc.

[0040] In this function (F1), the search range setting unit 157 determines the search range specified by the above-described specification method (a1) or (a2) as a range fixed in real space. Therefore, even if the position or orientation of the visual sensor 70 changes, the search range can remain fixed in real space.

[0041] The determination unit 154 determines whether an object is placed within a range corresponding to the search range E11 in real space based on the distance image. Since the search range is set as a fixed range in real space, it is possible to determine whether an object is placed within a fixed range in real space even if the position or orientation of the visual sensor 70 changes and the imaging range moves. This function (F1) is effective when the work location of the robot is fixed at a predetermined location.

[0042] The above function (F2) will be described with reference to FIG. 5 . Here, as an example, it is assumed that the above-described specification method (a2) is used as the method for specifying the search range. When the search range E11 is specified on the image, the search range setting unit 157 sets the search range E11 as a range corresponding to the image capture range R11 of the captured image M11, as shown on the right side of FIG. 5 . That is, the search range setting unit 157 sets the search range E11 as a range whose position is fixed relative to the image capture range R11 (see the right side of FIG. 5 ). Therefore, if the position or orientation of the visual sensor 70 changes and the image capture range R11 moves, the search range E11 moves to follow the image capture range R11.

[0043] In this function, the determination unit 154 determines whether an object is placed within a search range E11 set for the captured distance image M11. Because the search range is set in association with the captured image range, the range of the determination target on the captured image can always be fixed.

[0044] In the above functions (F3) and (F4), the search range setting unit 157 accepts input specifying the search range as a range in the real workspace. For example, the search range setting unit 157 is configured to be able to accept, as methods for specifying the search range as a range in real space, the following: (b1) a method for specifying the search range in real space by performing a touch-up operation on the robot, and (b2) a method for specifying the search range as position information in real space from an input source such as a robot program or user settings.

[0045] The above function (F3) will be described with reference to FIG. 6. Here, an example will be shown in which the designation method (b1) is used. As shown on the left side of FIG. 6, in this function (F3), the operator operates the robot 10 to touch up multiple points in the workspace with touch-up pins 35 attached to the tip of the arm of the robot 10, thereby designating a search range. In the example shown on the left side of FIG. 6, four points 301, 302, 302, and 304 corresponding to the four corner positions of the search range are designated on the installation surface 92 in the workspace by the touch-up operation. The three-dimensional positions of the four points 301-304 are obtained by the touch-up operation.

[0046] In this case, as shown on the right side of Figure 6, the search range setting unit 157 sets a rectangular range with four points 301-304 as its four corners as the search range E12 in real space. In this function (F3), the search range E12 is set as a fixed range within real space. The operator can set the search range to a desired position within the work space.

[0047] The determination unit 154 determines whether an object is placed within a range in the distance image that corresponds to the search range E12 in real space. When performing image processing such as determination on the image, the determination unit 154 obtains a search range on the image that corresponds to the search range E12 specified as a position in real space based on calibration data, etc. In this example, since the search range is defined in real space, it is possible to determine a fixed range in real space even if the position or orientation of the visual sensor 70 changes and the imaging range moves. This function (F3) is effective when the work location of the robot is fixed at a predetermined location.

[0048] The above function (F4) will be explained with reference to Fig. 7. Here, a case where the above-mentioned designation method (b1) is used will be exemplified. As in the case of the above-mentioned function (F3), the operator touches up the robot to designate multiple points in real space (see the left side of Fig. 7). As a result, three-dimensional coordinate values ​​representing the positions of the four corners of the search range are obtained.

[0049] The search range setting unit 157 sets the search range E12 in real space specified by the touch-up operation as a search range (shown with the same reference number E12 on the right side of FIG. 7 ) associated with the image capture range R11 of the visual sensor 70. Therefore, in the case of this function, the search range E12 is set as a position based on the image capture range R11, and even if the image capture range R11 moves due to a change in the position or orientation of the visual sensor, the search range moves accordingly. In other words, the position of the search range on the captured image can always be fixed.

[0050] In this function, the determination unit 154 determines whether or not an object is placed within a search range E12 set for the distance image of the imaging range R11. Because the search range is set in association with the imaging range, the range of the determination target on the captured image can always be fixed.

[0051] As described above, according to this embodiment, the search range can be specified either as a range on an image or as a range in real space. Therefore, the operator can specify an appropriate search range depending on various conditions, including the work content and the environmental conditions of the work space. Furthermore, as described above, the search range can be specified using various input methods, such as from the robot program, by user input via a UI, or by touching the robot. In this way, this embodiment provides a high degree of flexibility in the search range specification method.

[0052] The determination unit 154 determines whether an object is placed on the inspection target surface based on the distance image, with the search range set as described above as the determination target. As a specific example of the process by which the determination unit 154 determines whether an object is placed on the search range, the following describes an operation in which the histogram creation unit 153 determines whether an object is placed on the inspection target surface based on a frequency distribution (histogram) created from the distance image. Methods other than the frequency distribution (histogram) illustrated here may also be used to determine whether an object is placed on the inspection target surface.

[0053] The histogram creation unit 153 creates a histogram using distance (brightness) as a variable for an image whose brightness changes with distance, such as the distance image M1. When creating a histogram, the histogram creation unit 153 may be configured to search for a point cloud within a search range set by the search range setting unit 157 on the distance image. An example will be described using the distance image M1 acquired as shown in FIG. 3. As shown in FIG. 8, a search range E1 is specified on the distance image M1. FIG. 9 shows a histogram 201 created by the histogram creation unit 153 in this case. In the histogram 201 in FIG. 9, the horizontal axis (variable) represents the brightness of the points, and the vertical axis represents the sum of the number of points (frequency). In the search range E1, the floor surface 90 accounts for the largest proportion, and therefore the frequency of the floor surface 90, represented as the darkest point cloud, is the highest. The workpieces W1, W2, and W3, each with a different height, have the same upper surface area. Therefore, the frequencies of the brightness points representing the works W1, W2, and W3 are the same.

[0054] Referring to FIG. 9 , it can be seen that by identifying the brightness of the floor surface 90 as the surface to be inspected, it is possible to determine whether or not an object of a different height from the floor surface 90 is placed on the floor surface 90. In the example of FIG. 9 , point clouds of brightness different from that of the floor surface 90 are distributed in three locations, so it is possible to determine that three workpieces W1, W2, and W3 of different heights are placed on the floor surface 90. Regarding the identification of the brightness of the surface to be inspected (floor surface 90), the brightness of the surface to be inspected (a threshold value for determining whether or not the brightness is that of the surface to be inspected) may be set by user input. The brightness of the surface to be inspected (a threshold value for determining whether or not the brightness is that of the surface to be inspected) may also be automatically set by the determination unit 154. Details of how to set the brightness of the surface to be inspected (a threshold value for determining whether or not the brightness is that of the surface to be inspected) will be described later.

[0055] Consider the case where search range E2 is specified on distance image M1 as shown in Fig. 8. A histogram 202 created by histogram creation unit 153 in this case is shown in Fig. 10. Since search range E2 is an area that includes only floor surface 90, histogram 202 contains only one type of brightness with a frequency, and the frequency is also lower than in the case of search range E1.

[0056] By specifying a wide search range within the imaging range R1, such as search range E1, a histogram can be obtained for a wide range, allowing the presence or absence of an object to be confirmed within the wide range. On the other hand, specifying a relatively narrow search range, such as search range E2, is useful when it is desired to quickly determine the presence or absence of an object in a limited area. Furthermore, limiting the search range to a relatively narrow range can reduce the possibility of outliers being included in the histogram.

[0057] 8 also shows the imaging range R1 of the visual sensor 70. The imaging range setting unit 156 accepts designation of the size and position of the imaging range within the imaging range of the visual sensor 70. The imaging range setting unit 156 may be configured to provide a graphical user interface for setting at least one of the size and position of a rectangular image representing the imaging range on a screen such as that shown in FIG. 8. Such a user interface may be provided on the display unit 41 of the teaching pendant 40.

[0058] The histogram creation unit 153 may have a function to display, for example, the display unit 41 of the teaching operation panel 40, a histogram (histograms 201, 202, etc.) created for the specified range (imaging range or search range) as described above, in response to a predetermined operation on the operation unit of the teaching operation panel 40. Note that the histogram creation unit 153 may have a function to store distance information within the specified range (imaging range or search range) as numerical information (a numerical value table representing a histogram) in, for example, the storage unit 161, and to display, for example, on the display unit 41 of the teaching operation panel 40 in response to a predetermined operation on the operation unit of the teaching operation panel 40.

[0059] As described above, the robot control device 50 has a function of determining whether an object is placed within a search range specified by predetermined input information. FIG. 11 is a flowchart more generally describing this function (object presence / absence determination processing) performed by the determination unit 154. Note that this processing is executed under the control of the processor 51. As shown in FIG. 11, the image acquisition unit 152 acquires a distance image captured by the visual sensor 70 (step S1). Next, the determination unit 154 determines, based on the distance image, whether an object is placed within the search range specified based on the predetermined input information (step S2).

[0060] In this embodiment, the robot control device 50 is configured to determine whether an object is placed on the inspection surface (floor surface 90 in the case of histogram 201) based on the histogram described above. FIG. 12 is a flowchart illustrating a specific example of the object presence / absence determination process executed by the robot control device 50 (processor 51). First, the image acquisition unit 152 acquires a distance image captured by the visual sensor 70 within an imaging range including the inspection surface (step S101). Next, the histogram creation unit 153 creates a brightness histogram in the distance image (step S102). The histogram may be created for the search range set as described above. Then, the determination unit 154 determines whether an object is placed on the inspection surface within the search range based on the brightness histogram in the distance image (step S103).

[0061] Below, two examples will be given in which the object presence / absence determination process of FIG. 12 is applied to the handling of a workpiece by the robot 10, and the details of the object presence / absence determination process will be described.

[0062] The first embodiment is an example of operation when object presence / absence determination processing is applied to workpiece transfer work by the workpiece transfer execution unit 160. Fig. 13 is a flowchart when object presence / absence determination processing is applied to workpiece transfer processing. This processing is executed under the control of the processor 51 of the robot control device 50.

[0063] The workpiece transfer execution unit 160 causes the robot 10 to take out a workpiece from a supply device such as a belt conveyor (step S11). When placing the workpiece at its destination (transport device, workbench, etc.), the workpiece transfer execution unit 160 executes an object presence / absence determination process to confirm that no object is placed at the destination (step S12). Note that in this example, the robot 10 is taught to place the workpiece at its destination on the premise that there are no objects (obstacles) at the destination.

[0064] A specific example of the object presence / absence determination process executed in step S12 will be described with reference to Figures 14 and 15. Figure 14 shows the state in which the visual sensor 70 captures a distance image when nothing is placed on the destination placement surface 91, and also shows a histogram 211 of the distance image acquired at that time. Figure 15 shows the state in which the visual sensor 70 captures a distance image when a workpiece W5 is placed on the placement surface 91, and also shows a histogram 212 acquired at that time.

[0065] The robot control device 50 may previously acquire a histogram 211 acquired when nothing is placed on the placement surface 91 and store it in the storage unit 161. The determination unit 154 can determine whether an object is placed on the placement surface 91 by comparing the previously stored histogram 211 with the histogram 212 of the distance image acquired in step S12 during execution of the workpiece transfer process.

[0066] As shown in the histogram 211 in Fig. 14 , the threshold setting unit 155 provides a function of setting a threshold for determining whether or not there is a frequency of a height (brightness) other than the height (brightness) of the placement surface 91 in the distance image. Here, the threshold may be set as an upper limit value K1 and a lower limit value K2 for defining a range including the height (brightness) of the placement surface 91 as a reference, for example, as shown in Fig. 14 . The determination unit 154 can determine that an object is placed on the placement surface 91 when it is recognized that there is a point cloud of brightness outside the range defined by the upper limit value K1 and the lower limit value K2.

[0067] The thresholds (upper limit K1, lower limit K2) may be set by a user or may be automatically set by the threshold setting unit 155. The threshold setting unit 155 may automatically set the thresholds (upper limit K1 and lower limit K2) by the following procedure: r1) identifying a brightness having a frequency in a pre-stored histogram 211 as the brightness of the placing surface 91, and r2) setting the brightness obtained by adding a predetermined margin to the brightness of the placing surface 91 as the upper limit K1, and the brightness obtained by subtracting the predetermined margin from the brightness of the placing surface 91 as the lower limit K2.

[0068] Here, a method has been described in which the threshold setting unit 155 automatically sets a threshold based on a distance image acquired in advance when there are no objects on the surface to be inspected. As another method, for example, depending on the condition of the surface to be inspected, an algorithm may be adopted that determines that the surface to be inspected is a point cloud that satisfies the following conditions: (d1) the most frequent point in the distance image at the time of inspection, or (d2) the most frequent point and the darkest point in the distance image at the time of inspection. Algorithm (d1) is effective in situations where it is assumed that there are generally few objects placed on the surface to be inspected. Algorithm (d2) is effective when there are generally few objects placed on the surface to be inspected and the surface to be inspected is at the lowest position within the imaging range (or search range), such as a floor.

[0069] When the threshold setting unit 155 accepts user settings for the thresholds (upper limit K1, lower limit K2), a graphical user interface screen (GUI screen 350) such as that shown in FIG. 16 may be provided on the display unit 41 of the teaching pendant 40. The GUI screen 350 shown in FIG. 16 displays a histogram created by the histogram creation unit 153 based on a distance image captured without any objects on the surface to be inspected, and is configured to accept threshold settings on the screen. The GUI screen 350 shown in FIG. 16 is configured to allow the user to input the upper limit K1 and lower limit K2 of the thresholds, sandwiching the brightness N1 of the surface to be inspected, into numerical input fields 351 and 352. The user can set the desired upper limit K1 and lower limit K2 of the thresholds in the numerical input fields 351 and 352. By enabling the user to set the thresholds on a screen displaying the histogram of the surface to be inspected, the user can intuitively set the thresholds by referring to the histogram.

[0070] Assume that during inspection (i.e., in the object presence / absence determination process in step S12), a situation occurs in which a workpiece W5 remains on the placement surface 91, as shown in FIG. 15. In this case, a histogram 212 such as that shown on the right side of FIG. 15 is generated. The determination unit 154 then identifies the presence of a point cloud with brightness exceeding the upper threshold value K1 (i.e., the presence of a workpiece W5). In this case, it is determined in step S13 that an object has been placed on the surface to be inspected (S13: YES), the workpiece placement operation is interrupted, and the process proceeds to step S15.

[0071] On the other hand, if it is determined in step S13 that no object is placed on the inspection target surface (S13: NO), the workpiece transfer execution unit 160 executes the operation of placing the workpiece on the placement surface 91 (step S14). Then, the workpiece transfer execution unit 160 may continue the operation of picking up and transferring the next workpiece.

[0072] In step S15, the historical image storage unit 158 ​​stores a two-dimensional image, taken by the two-dimensional camera function of the visual sensor 70, of the state in which the workpiece W5 remains on the placement surface 91 as a historical image together with other historical information in the storage device 80. Then, the workpiece transfer execution unit 160 ends this process.

[0073] As described above, the workpiece transfer process according to this embodiment can reliably detect that there is nothing in the place where the workpiece is to be placed, thereby improving the safety of the workpiece placement operation. If there is an object (obstacle) in the place where the workpiece is to be placed, the robot can halt the workpiece placement operation. Therefore, the robot can place the workpiece in the place where the workpiece is placed, thereby preventing problems (such as a stop in the production cycle) caused by workpieces interfering with each other. Therefore, according to this embodiment, the workpiece transfer process can be performed even more safely and smoothly.

[0074] The historical image saved in step S15 can be used to understand the situation in which the workpiece W5 remains on the placement surface 91, analyze the cause, etc. This also makes it possible to take necessary measures.

[0075] During the workpiece transfer process, the histogram creation unit 153 may be configured to display a histogram based on the distance image acquired by the visual sensor 70 on a display screen (display unit 41) in response to a predetermined user operation. During the workpiece transfer process, the determination unit 154 may determine whether or not an object is placed on the inspection target surface within a set range (imaging range or search range).

[0076] A second embodiment will be described in which the object presence / absence determination process is applied to workpiece handling operations by the robot 10. The second embodiment is an example of operation when the object presence / absence determination process is applied to the process of picking up bulk workpieces by the workpiece pick-up execution unit 159. FIG. 17 is a flowchart when the object presence / absence determination process is applied to the process of picking up bulk workpieces. This process is executed under the control of the processor 51 of the robot control device 50. Here, it is assumed that an operation is executed to pick up workpieces W6 (only some of the workpieces are marked with reference numerals) that are piled up in a container 95 on the floor surface 90 as shown in FIGS. 18 and 19 .

[0077] The workpiece removal execution unit 159 executes the removal process of the bulk workpieces in accordance with the workpiece removal program stored in the storage unit 161 (step S21). Then, the workpiece removal execution unit 159 determines whether the removal of the bulk workpieces is completed based on a predetermined completion determination condition (step S22). The predetermined completion determination condition is, for example, that the workpiece W6 is not detected as a detection result of the detection process of the workpiece W6 executed based on a two-dimensional image of the inside of the container 95 captured by the visual sensor 70. If it is determined that the removal of the workpieces is not completed (S22: NO), the workpiece removal process continues.

[0078] When the workpiece removal execution unit 159 determines that the workpiece removal process is completed (S22: YES), an object presence / absence determination process is executed (step S23). In this object presence / absence determination process, it is determined whether or not the workpiece W6 remains in the container 95.

[0079] The object presence / absence determination process executed in step S23 will be described with reference to Figures 18 to 20. Figure 18 shows a situation in which the visual sensor 70 captures distance images when many workpieces W6 are present in the container 95, and a histogram 221 of the distance images acquired at that time. Figure 19 shows a situation in which the visual sensor 70 captures distance images when there are no objects in the container 95, and a histogram 222 of the distance images acquired at that time. Figure 20 shows a situation in which the visual sensor 70 captures distance images when one workpiece W6 remains in the container 95, and a histogram 223 of the distance images acquired at that time.

[0080] In this embodiment, a histogram 222 captured when there are no objects inside the container 95, as shown in FIG. 19, is stored in the storage unit 161 in advance. The threshold setting unit 155 identifies the height (brightness) of the bottom surface 95a of the container 95 based on this histogram 222 and determines the upper limit K1 and lower limit K2 of the thresholds based on this brightness. The identification of the brightness of the bottom surface 95a and the setting of the thresholds (upper limit K1 and lower limit K2) can be performed according to the above-described steps (r1) and (r2). As shown in FIG. 18, in a situation where many workpieces W6 are present inside the container 95, the histogram 221 will have many points distributed in the brightness region that exceeds the range determined by the upper limit K1 and lower limit K2 of the thresholds based on the bottom surface 95a. On the other hand, as shown in Figure 19, when there is no work W6 remaining in the container, there is no point group with a frequency in the area of ​​brightness that exceeds the range determined by the upper limit value K1 and lower limit value K2 of the threshold based on the bottom surface 95a.

[0081] If the result of the object presence determination process is that there is no point cloud with brightness exceeding the range determined by the upper limit K1 and lower limit K2 of the threshold, it is determined that there is no object on the bottom surface 95a as the inspection target surface (S24: NO). In this case, the removal of the bulk workpieces has been properly completed, and this process is terminated.

[0082] Assume that the situation inside the container 95 when the object presence determination process is executed is one in which a workpiece W6 remains, as shown in Figure 20. In this case, the determination unit 154 determines the presence or absence of an object on the inspection target surface using the histogram 223 shown in Figure 20. In the histogram 223, a distribution of point clouds is observed in an area of ​​brightness that exceeds the range determined by the upper limit K1 and lower limit K2 of the threshold values. Therefore, in this case, the determination unit 154 determines that an object is present on the bottom surface 95a, which is the inspection target surface (S24: YES).

[0083] In this case (S24: YES), the historical image storage unit 158 ​​captures a two-dimensional image of the state in which the workpiece W6 remains on the bottom surface 95a using the two-dimensional camera function of the visual sensor 70, and stores the image as a historical image together with other historical information in the storage device 80 (step S25). Then, the workpiece removal execution unit 159 ends this process.

[0084] As described above, according to the workpiece removal process of this embodiment, it is possible to reliably detect a malfunction in which the robot control device determines that workpiece removal is complete when a workpiece remains in the container.

[0085] The historical image saved in step S25 can be used to analyze the factors behind the workpiece removal execution unit 159 determining that removal is complete while the workpiece W6 remains. For example, as illustrated in FIG. 20 , in a situation where one side of the workpiece W6 remains in close contact with the inner surface of the container 95, it is possible that the workpiece W6 cannot be detected properly using a detection process that applies pattern matching to a two-dimensional image. On the other hand, as described above, the presence of the workpiece W6 can be appropriately determined using the object presence / absence determination process. Therefore, by saving the situation in which the workpiece W6 remains as a historical image as shown in FIG. 20 , it is possible to analyze the factors and take necessary measures.

[0086] In the second embodiment described above, the imaging range or search range may be adjusted in advance so as not to measure the side surface of the container 95. By setting the imaging range or search range in this way, the histogram will show the frequency of only the height due to the bottom surface 95a of the container 95 and the workpiece W6 (if the workpiece W6 is present), making it possible to more effectively determine the presence or absence of an object.

[0087] As described above, according to this embodiment, it is possible to reliably determine whether or not an object is placed in the location to be inspected, and it is possible to reliably prevent malfunctions from occurring in the handling of workpieces by a robot and to reliably detect the occurrence of malfunctions.

[0088] 2 in the above-described embodiment is merely an example, and various modifications of the functional arrangement are possible. For example, a configuration example is possible in which some of the functional blocks arranged in the robot control device 50 (e.g., the threshold setting unit 155, the imaging range setting unit 156, and the search range setting unit 157) are arranged on the teaching pendant 40 side.

[0089] The functions of the teaching pendant 40 (such as the functions of the display unit and operation unit as a user interface) may be included in the functions of the robot control device 50 and defined as a robot control device.

[0090] In the above embodiment, an example of an operation in which the determination unit 154 determines whether an object is placed within a search range using a histogram of a distance image has been described. However, the example of an operation in which the determination unit determines whether an object is placed within a search range is not limited to this. For example, the determination unit 154 may determine whether an object is placed within a specified search range using distance data for each point in the distance image. In this case, for example, if the presence of an object having a specific height based on the height of a reference plane within the search range is recognized, it may be determined that an object is placed within the search range (on the reference plane). Note that when the determination unit is configured in this manner, functions related to determination using a histogram (the histogram creation unit 153 and the threshold setting unit 155) can be omitted.

[0091] The functional blocks of the robot control device and the visual sensor control device shown in Figure 2 may be realized by the processors of these devices executing various software stored in a storage device, or may be realized by a configuration mainly consisting of hardware such as an ASIC (Application Specific Integrated Circuit).

[0092] The above-described object presence / absence determination process (FIGS. 11 and 12) can be executed on various information processing devices.

[0093] The programs for executing various processes such as the object presence / absence determination process (FIGS. 11 and 12), workpiece transfer process (FIG. 13), and workpiece removal process (FIG. 17) in the above-described embodiments can be recorded on various computer-readable recording media (e.g., semiconductor memory such as ROM, EEPROM, and flash memory, magnetic recording media, and optical disks such as CD-ROM and DVD-ROM).

[0094] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0095] The following supplementary notes are provided regarding the above-described embodiment and modified examples. (Supplementary Note 1) A control device (50) including an image acquisition unit (152) that acquires a distance image captured by a visual sensor (70) and a determination unit (154) that determines, based on the distance image, whether an object is placed within a range specified by predetermined input information. (Supplementary Note 2) The control device (50) according to Supplementary Note 1 further includes a search range setting unit (157) that sets the range specified by the predetermined input information as a search range. (Supplementary Note 3) The control device (50) according to Supplementary Note 2, wherein the predetermined input information is information that specifies the search range as a range on an image. (Supplementary Note 4) The control device (50) according to Supplementary Note 2, wherein the predetermined input information is information that specifies the search range as a range in real space. (Supplementary Note 5) The control device (50) according to any one of Supplements 1 to 4, wherein the predetermined input information is written in a robot program. (Supplementary Note 6) The control device (50) according to any one of Supplements 1 to 4, wherein the predetermined input information is information input via a user interface. (Supplementary Note 7) The control device (50) according to any one of Supplements 2 to 4, wherein the search range setting unit (157) sets the search range as a range defined in real space. (Supplementary Note 8) The control device (50) according to any one of Supplements 2 to 4, wherein the search range setting unit (157) sets the search range as a range associated with the imaging range of the visual sensor (70). (Supplementary Note 9) The control device (50) according to Supplementary Note 3, wherein the range on the image is represented as numerical information based on coordinates on the image. (Supplementary Note 10) The control device (50) according to Supplementary Note 3, wherein the search range setting unit (157) provides a graphical user interface for specifying the range on the image by graphical operations. (Appendix 11) The control device (50) described in Appendix 4, wherein the search range setting unit (157) acquires three-dimensional coordinates of multiple points obtained by causing the robot (10) to perform an action of touching up the multiple points in real space as information specifying the search range as a range in real space.(Supplementary Note 12) The control device (50) according to Supplementary Note 1, further comprising an imaging range setting unit (156) for setting the range specified by the predetermined input information as the imaging range to be captured by the visual sensor (70), wherein the imaging range setting unit (156) sends a signal to the visual sensor (70) instructing the visual sensor to capture an image within the set imaging range. (Supplementary Note 13) The control device (50) according to any one of Supplements 1 to 12, wherein the determination unit (154) determines whether or not an object is placed on the inspection target surface within the specified range. (Supplementary Note 14) The control device (50) according to Supplementary Note 13, wherein the determination unit (154) determines whether or not an object is placed on the inspection target surface based on a frequency distribution of distance information for each point in the acquired distance image. (Supplementary Note 15) The control device (50) according to Supplementary Note 14 further comprises a threshold setting unit (155) for setting a threshold for distinguishing between distance information of the inspection target surface in the frequency distribution and distance information of other objects on the inspection target surface, and the determination unit (154) uses the threshold to determine whether or not an object is placed on the inspection target surface. (Supplementary Note 16) The control device (50) according to Supplementary Note 15, wherein the threshold setting unit (155) sets the threshold based on a frequency distribution relating to distance information of each point in a distance image obtained when no object is placed on the inspection target surface. (Supplementary Note 17) The control device (50) according to Supplementary Note 15, wherein the threshold setting unit (155) is configured to provide a user interface that accepts setting of the threshold by a user operation. (Supplementary Note 18) The control device (50) according to Supplementary Note 17, wherein the user interface is configured as a graphical user interface including an image representing the frequency distribution of the distance image. (Supplementary Note 19) The control device (50) according to Supplementary Note 15, wherein the threshold setting unit (155) sets the threshold in accordance with threshold information written in a robot program. (Supplementary Note 20) The control device (50) according to any one of Supplementary Notes 14 to 19, further comprising a histogram creation unit (153) that generates an image representing the frequency distribution based on distance information of each point in the acquired distance image and displays the image on a display screen.(Supplementary Note 21) The control device (50) according to Supplementary Note 20, wherein the histogram creation unit (153) is configured to generate an image representing the frequency distribution within a specified range in the distance image. (Supplementary Note 22) The control device (50) according to any one of Supplements 14 to 21, further comprising a work transfer execution unit (160) for executing a work transfer task of removing a work from a supply device and placing it on a predetermined placement surface by the robot (10), and the determination unit (154) determines whether or not an object is present on the placement surface based on a distance image captured of the placement surface before the work transfer execution unit (160) places the work on the placement surface. (Supplementary Note 23) The control device (50) of any one of Supplements 14 to 21, further comprising a workpiece removal execution unit (159) for performing a removal operation of removing workpieces randomly stacked in a container by a robot (10), wherein the determination unit (154) determines whether or not an object is present on the bottom of the container based on a distance image capturing the bottom of the container when the workpiece removal execution unit (159) determines that workpiece removal is complete based on a predetermined completion determination condition. (Supplementary Note 24) The control device (50) of Supplementary Note 22 or 23, further comprising a historical image storage unit (158) for storing a historical image when the determination unit (154) determines that an object is present. (Supplementary Note 25) The control device of any one of Supplements 14 to 24, wherein the distance image is an image representing the distance information of each point by brightness, and the frequency distribution represents the brightness distribution of each point of the distance image. (Supplementary Note 26) A robot system (100) comprising a robot (10), a visual sensor (70), and a control device (50) according to any one of Supplements 1 to 25, wherein the control device (50) controls the robot (10). (Supplementary Note 27) A method executed on an information processing device, comprising: acquiring a distance image captured by the visual sensor (70); and determining, based on the distance image, whether or not an object is placed within a range specified by predetermined input information.(Appendix 28) A program for causing a computer processor to execute the steps of: acquiring a distance image captured by a visual sensor (70); and determining, based on the distance image, whether an object is placed within a range specified by predetermined input information.

[0096] REFERENCE SIGNS LIST 10 Robot 20 Visual sensor control device 33 Hand 40 Teaching operation panel 50 Robot control device 51 Processor 70 Visual sensor 80 Storage device 90 Floor surface 91 Placement surface 95 Container 95a Bottom surface 100 Robot system 121 Image processing unit 122 Storage unit 151 Operation control unit 152 Image acquisition unit 153 Histogram creation unit 154 Determination unit 155 Threshold setting unit 156 Imaging range setting unit 157 Search range setting unit 158 ​​History image storage unit 159 Workpiece removal execution unit 160 Workpiece transfer execution unit 161 Storage unit 350 GUI screen

Claims

1. an image acquisition unit that acquires a distance image captured by a visual sensor; a determination unit that determines, based on the distance image, whether an object is placed within a range specified by predetermined input information; A control device comprising:

2. The control device according to claim 1 , further comprising a search range setting unit that sets the range specified by the predetermined input information as a search range.

3. The control device according to claim 2 , wherein the predetermined input information is information that specifies the search range as a range on an image.

4. The control device according to claim 2 , wherein the predetermined input information is information that specifies the search range as a range in real space.

5. The control device according to claim 1 , wherein the predetermined input information is written in a robot program.

6. The control device according to claim 1 , wherein the predetermined input information is information input via a user interface.

7. The control device according to claim 2 , wherein the search range setting unit sets the search range as a range defined in real space.

8. The control device according to claim 2 , wherein the search range setting unit sets the search range as a range associated with an imaging range of the visual sensor.

9. The control device according to claim 3 , wherein the range on the image is expressed as numerical information based on coordinates on the image.

10. The control device according to claim 3 , wherein the search range setting unit provides a graphical user interface for specifying the range on the image by a graphical operation.

11. 5. The control device according to claim 4, wherein the search range setting unit acquires three-dimensional coordinates of a plurality of points obtained by causing the robot to perform an action of touching up the plurality of points in real space as information specifying the search range as a range in real space.

12. an imaging range setting unit for setting the range designated by the predetermined input information as an imaging range to be captured by the visual sensor; The control device according to claim 1 , wherein the imaging range setting unit sends a signal to the visual sensor instructing the visual sensor to capture an image within the set imaging range.

13. The control device according to claim 1 , wherein the determination unit determines whether or not an object is placed on the inspection target surface within the specified range.

14. The control device according to claim 13 , wherein the determination unit determines whether or not an object is placed on the inspection target surface based on a frequency distribution relating to distance information of each point in the acquired distance image.

15. a threshold setting unit for setting a threshold for distinguishing between distance information of the inspection target surface in the frequency distribution and distance information of other objects on the inspection target surface, The control device according to claim 14 , wherein the determination unit determines whether or not an object is placed on the inspection target surface using the threshold value.

16. The control device according to claim 15 , wherein the threshold setting unit sets the threshold based on a frequency distribution of distance information for each point in a distance image obtained when no object is placed on the inspection target surface.

17. The control device according to claim 15 , wherein the threshold setting unit is configured to provide a user interface that accepts the setting of the threshold through a user operation.

18. The control device according to claim 17 , wherein the user interface is configured as a graphical user interface including an image representing a frequency distribution of the distance image.

19. The control device according to claim 15 , wherein the threshold setting unit sets the threshold in accordance with threshold information described in a robot program.

20. The control device according to claim 14 , further comprising a histogram creation unit that generates an image representing the frequency distribution based on distance information of each point in the acquired distance image and displays the image on a display screen.

21. The control device according to claim 20 , wherein the histogram creation unit is configured to generate an image representing the frequency distribution in a specified range in the distance image.

22. a workpiece transfer execution unit for executing a workpiece transfer operation in which the robot takes out the workpiece from the supply device and places it on a predetermined placement surface; The control device according to claim 14 , wherein the determination unit determines whether or not an object is present on the placement surface based on a distance image captured of the placement surface before the workpiece transfer execution unit places the workpiece on the placement surface.

23. The robot further includes a workpiece removal execution unit for performing a removal operation of removing workpieces randomly stacked in the container by the robot, The control device according to claim 14, wherein the determination unit determines whether or not there is an object on the bottom surface of the container based on a distance image captured of the bottom surface of the container when the work removal execution unit determines that the work removal has been completed based on a predetermined completion determination condition.

24. The control device according to claim 22 , further comprising a historical image storage unit that stores a historical image when the determination unit determines that an object is present.

25. the distance image is an image that represents the distance information of each point by brightness, the frequency distribution represents a distribution of brightness at each point in the distance image; The control device according to claim 14.

26. Robots and A visual sensor; The control device according to any one of claims 1 to 4, The control device controls the robot.

27. A method executed on an information processing device, comprising: A distance image captured by a visual sensor is acquired, An object presence / absence determination method for determining whether or not an object is placed within a range specified by predetermined input information based on the distance image.

28. The computer processor A procedure for acquiring a range image captured by a visual sensor; a step of determining whether or not an object is placed within a range specified by predetermined input information based on the distance image; A program to execute.