Control device, robot system, object determination method, and program

Abstract

A control device is provided with: an image acquisition unit that acquires a distance image captured by a vision sensor; and a determination unit that determines, on the basis of the distance image, whether or not an object is placed within a range specified by the predetermined input information.

Description

Technical Field

[0001] The present disclosure relates to a control device for an industrial robot, a robot system, an object presence / absence determination method, and a program. Background Art

[0002] There is known a robot system that performs a workpiece taking-out operation of taking out a workpiece (such as a bulk workpiece) arranged at an indefinite position by a robot and placing it on a workbench or a conveying device by detecting the workpiece using a vision sensor.

[0003] For example, Patent Document 1 describes a handling system for workpiece transfer when performing grinding finish on a plate-shaped metal workpiece obtained by fusing or the like.

[0004] Patent Document 2 describes a monitoring device included in a robot system that performs a predetermined operation on a workpiece. The monitoring device includes a three-dimensional camera and determines whether an object exists in a restricted area based on the distances from the camera to a plurality of measurement points set on the surface of the workpiece.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2007-021635

[0008] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2018-195959 Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] In a robot system that performs handling such as taking out or transferring a workpiece, there may be a situation where, for example, due to some reason, a workpiece placed by a previous operation exists at a place where the workpiece should be arranged. In such a situation, when the robot is about to place a workpiece, it may cause a malfunction (such as a stop of the production cycle) due to interference between the workpieces.

[0011] There is a need for a technique that can detect whether an object exists in an inspection target area, thereby enabling reliable detection of malfunctions or prevention of malfunctions in workpiece handling operations based on a robot.

[0012] Means for Solving the Problems

[0013] One aspect of the present disclosure is a control device including: an image acquisition unit that acquires a distance image captured by a vision sensor; and a determination unit that determines whether an object is placed within a range specified by prescribed input information based on the distance image.

[0014] The objects, features, and advantages of the present invention, as well as other objects, features, and advantages, will become more apparent from the following detailed description of exemplary embodiments of the present invention shown in the accompanying drawings. Description of the Drawings

[0015] Figure 1 FIG. is a diagram showing the device structure of a robot system including a robot control device according to an embodiment.

[0016] Figure 2 FIG. is a functional block diagram of the robot control device and the vision sensor control unit.

[0017] Figure 3 FIG. is a diagram for explaining a distance image obtained by a three-dimensional camera.

[0018] Figure 4 FIG. is a diagram for explaining a first example of the setting function of the search range based on the search range setting unit.

[0019] Figure 5 FIG. is a diagram for explaining a second example of the setting function of the search range based on the search range setting unit.

[0020] Figure 6 FIG. is a diagram for explaining a third example of the setting function of the search range based on the search range setting unit.

[0021] Figure 7 FIG. is a diagram for explaining a fourth example of the setting function of the search range based on the search range setting unit.

[0022] Figure 8 FIG. is a diagram for explaining the shooting range and the search range.

[0023] Figure 9 FIG. shows Figure 8 a histogram of the first search range shown in the example.

[0024] Figure 10 FIG. shows Figure 8 a histogram of the second search range shown in the example.

[0025] Figure 11 FIG. is a flowchart generally showing the presence / absence object determination function executed by the robot control device.

[0026] Figure 12 FIG. is a flowchart showing the specific operation of the presence / absence object determination process executed by the robot control device.

[0027] Figure 13 FIG. is a flowchart when the presence / absence object determination process is applied to the workpiece transfer process.

[0028] Figure 14 This is a diagram showing the state of a distance image captured by a vision sensor when no object is placed on the placement surface and the histogram of the distance image obtained at this time.

[0029] Figure 15 This is a diagram showing the state of a distance image captured by a vision sensor when a workpiece is placed on the placement surface and the histogram of the distance image obtained at this time.

[0030] Figure 16 This is a diagram showing an example of a graphical user interface for setting a threshold value.

[0031] Figure 17 This is a flowchart when the object presence / absence determination process is applied in the process of taking out bulk workpieces.

[0032] Figure 18 This is a diagram showing the state of a distance image captured by a vision sensor when there are multiple workpieces in a container and the histogram of the distance image obtained at this time.

[0033] Figure 19 This is a diagram showing the state of a distance image captured by a vision sensor when there is no object in a container and the histogram of the distance image obtained at this time.

[0034] Figure 20 This is a diagram showing the state of a distance image captured by a vision sensor when there is one remaining workpiece in a container and the histogram of the distance image obtained at this time. Detailed implementation manners

[0035] Next, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings referred to, the same reference numerals are assigned to the same structural parts or functional parts. For easy understanding, the scales of these drawings are appropriately changed. In addition, the embodiments shown in the drawings are examples for implementing the present invention, and the present invention is not limited to the illustrated embodiments.

[0036] Figure 1 This is a diagram showing the device structure of a robot system 100 including a robot control device 50 according to an embodiment. As Figure 1As shown, the robot system 100 includes: a robot 10 with a manipulator 33 mounted at the front end of its arm; a robot control device 50 that controls the robot 10; a teaching operation panel 40 connected to the robot control device 50; a vision sensor 70 installed at the front end of the arm of the robot 10; and a vision sensor control device 20 that controls the vision sensor 70. The robot system 100 can, for example, detect an object 1 on a workbench 2 through the vision sensor 70 and handle the object 1 with the manipulator 33 mounted on the robot 10. The robot system 100 can determine whether an object is placed within a range specified by prescribed input information based on the distance image captured by the vision sensor 70.

[0037] The robot system 100 may further include: a storage device 80 for storing execution history data and other information. The storage device 80 is, for example, an external memory connected to the robot control device 50. Alternatively, the storage device 80 may be a storage device, an external computer, etc. that is network-connected to the robot control device 50. In addition, Figure 1 shows the structure when the robot system 100 has an independent device as the storage device 80, but the function of the storage device 80 may also be incorporated into the robot control device 50 or the teaching operation panel 40. There may also be a structural example in which the storage device 80 is connected to the teaching operation panel 40.

[0038] The robot 10 is a vertical multi-joint robot in this embodiment, but other types of robots such as a parallel link robot or a two-arm robot may also be used according to the operation purpose. The robot 10 can perform a desired operation through an end effector installed at the wrist. In Figure 1 shows an example of using the manipulator 33 as the end effector.

[0039] The vision sensor 70 has the functions of: a two-dimensional camera for capturing a grayscale image or a color image, and a three-dimensional camera for obtaining a distance image. Multiple vision sensors may be configured in the robot system 100. As the three-dimensional camera, for example, a TOF (Time of Flight) camera that captures a distance image by the time-of-flight method, or a stereo camera including two cameras can be used. In addition, Figure 1 shows a structural example in which the vision sensor 70 is mounted on the robot 10, but the vision sensor 70 may also be a fixed camera fixed within the operation space.

[0040] The visual sensor control device 20 stores the model pattern of the object and can execute image processing for detecting the object by performing pattern matching between the object image in the captured image and the model pattern. In addition, the visual sensor 70 is set to be calibrated, and the visual sensor control device 20 is set to hold calibration data that defines the relative positional relationship between the visual sensor 70 and the robot 10. Thereby, the position on the two-dimensional image captured by the visual sensor 70 can be converted into the position on the coordinate system (such as the robot coordinate system) fixed to the work space.

[0041] In Figure 1 this case, the visual sensor control device 20 is configured as a device different from the robot control device 50, but as a function of the visual sensor control device 20, it can also be mounted in the robot control device 50.

[0042] The robot control device 50 controls the movement of the robot 10 according to the action program or instructions from the teach pendant 40. The robot control device 50 may have a hardware structure of a general computer, which has a processor 51 ( Figure 2 ), a memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, an input / output interface, a network interface, etc.

[0043] The teach pendant 40 is used as an operation terminal for performing teaching (program production) of the robot 10 and various settings. As the teach pendant 40, a teaching device composed of a tablet terminal or the like can be used. The teach pendant 40 may have a hardware structure of a general computer, which has a processor, a memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, a display unit 41 ( Figure 2 ), an input / output interface, a network interface, etc. The display unit 41 ( Figure 2 ) has, for example, a liquid crystal display.

[0044] The visual sensor control device 20 may have a hardware structure of a general computer, which has a processor, a memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, a display unit, an input / output interface, a network interface, etc.

[0045] Figure 2 FIG. shows the functional block diagrams of the robot control device 50 and the visual sensor control device 20. As Figure 2As shown, the robot control device 50 includes: a motion control unit 151, an image acquisition unit 152, a histogram creation unit 153, a determination unit 154, a threshold setting unit 155, a shooting range setting unit 156, a search range setting unit 157, a historical image storage unit 158, a workpiece extraction execution unit 159, and a workpiece transfer execution unit 160. These functional blocks can also be implemented by a processor 51 of the robot control device 50 executing software. The robot control device 50 also has a storage unit 161.

[0046] The storage unit 161 is, for example, a storage device composed of a non-volatile memory or a hard disk device. The robot program for controlling the robot 10, a program for image processing such as workpiece detection based on the image captured by the vision sensor 70 (vision program), and various setting information are stored in the storage unit 161.

[0047] The motion control unit 151 controls the motion of the robot according to the robot program or an instruction from the teaching operation panel 40. The robot control device 50 includes: a servo control unit (not shown) that performs servo control of the servo motors for each axis according to the instruction for each axis generated by the motion control unit 151.

[0048] The image acquisition unit 152 has a function of acquiring a distance image or a two-dimensional image captured by the vision sensor 70 from the vision sensor control device 20.

[0049] The histogram creation unit 153 provides the following function: creating a frequency distribution, that is, a histogram, related to the distance information of each point in the acquired distance image. The histogram creation unit 153 may also have a function of displaying the created histogram on a display screen (for example, the display unit 41 of the teaching operation panel 40).

[0050] The determination unit 154 provides the following function: determining whether an object is placed within a range specified by prescribed input information based on the distance image acquired by the image acquisition unit 152. Here, the range specified by the prescribed input information may be a search range that the determination unit 154 searches for determination or the shooting range of the vision sensor 70. As an example, the determination unit 154 can determine whether an object is placed on the inspection target surface within the specified search range based on the histogram created by the histogram creation unit 153.

[0051] The threshold setting unit 155 provides a function for setting a threshold value used to identify the distance information of the inspection target surface on the histogram and the distance information of other objects in the inspection target surface. The threshold setting unit 155 may have a function of automatically setting the threshold value according to the distance image captured by the vision sensor 70. The threshold setting unit 155 may accept the setting of the threshold value via a UI (user interface) screen. Alternatively, the threshold setting unit 155 may also accept the input of the threshold value from an external device. The threshold setting unit 155 may set the threshold value according to the information related to threshold setting (such as variables defining the threshold value) described in a robot program (such as a workpiece picking program, a workpiece transfer program, a vision program, etc.).

[0052] The shooting range setting unit 156 provides a function for setting the shooting range when the vision sensor 70 captures a distance image. The shooting range setting unit 156 may accept the designation of the shooting range via a UI screen. The shooting range setting unit 156 may also accept the input of the shooting range from an external device. The shooting range setting unit 156 may set the shooting range (size, position, etc.) according to the information related to the designation of the shooting range (such as variables defining the shooting range) described in a robot program (such as a workpiece picking program, a workpiece transfer program, a vision program, etc.). The shooting range setting unit 156 issues a signal to the vision sensor 70 (vision sensor control device 20) instructing to perform shooting within the set shooting range. In addition, when changing the position and posture of the robot 10 in order to move the shooting range, the shooting range setting unit 156 issues an instruction to the motion control unit 151 to change the position and posture of the robot 10.

[0053] The search range setting unit 157 provides a function of setting a search range for searching when the determination unit 154 determines whether an object is placed in the work space according to specified input information. The specified input information may be, for example, the information related to the designation of the search range (such as variables defining the search range) described in a robot program, the setting information input via a UI (user interface), the information input from an external device, etc. The details of the setting of the search range based on the search range setting unit 157 will be described later.

[0054] The historical image storage unit 158 provides a function of storing historical information including historical images, such as the images captured when performing workpiece detection processing based on the vision sensor 70, in a storage device 80, for example.

[0055] Stored in the storage unit 161 are: a workpiece taking-out program for causing the robot 10 to perform a taking-out operation of bulk workpieces, and a workpiece transfer program for causing the robot 10 to take out a workpiece and transfer it to another place. The workpiece taking-out execution unit 159 is a functional block implemented by the processor 51 executing the workpiece taking-out program. The workpiece taking-out execution unit 159 cooperates with the motion control unit 151 and the vision sensor control device 20 to perform a taking-out operation of bulk workpieces. The workpiece transfer execution unit 160 is a functional block implemented by the processor 51 executing the workpiece transfer program. The workpiece transfer execution unit 160 cooperates with the motion control unit 151 and the vision sensor control device 20 to perform a transfer operation of the workpiece.

[0056] The vision sensor control device 20 includes an image processing unit 121 and a storage unit 122. The storage unit 122 is a storage device constituted by, for example, a non-volatile memory. The storage unit 122 stores various data required for image processing, such as various settings for generating a distance image, calibration data, and model data of a workpiece. The image processing unit 121 performs various image processing, such as detection processing of a workpiece.

[0057] Refer to Figure 3 A distance image obtained by the function of the vision sensor 70 as a three-dimensional camera will be described. Figure 3 The state of three workpieces W1, W2, and W3 with different heights placed on the ground 90 photographed by the vision sensor 70 and the photographed distance image M1 are shown. The distance image is an image in which the brightness changes according to the height of an object within the photographing range. That is, the distance image represents a point group in which the brightness of a point changes according to the distance from the camera. In the case of the distance image M1, each point is displayed in such a way that the higher the height, the closer to white the color becomes, and the lower the height, the closer to black the color becomes. Therefore, in the distance image M1, the brightness of the portion (point group) corresponding to the workpiece W1 is displayed in the brightest color, the portion (point group) corresponding to the workpiece W2 is displayed in the second brightest color, the portion (point group) corresponding to the workpiece W3 is displayed in the third brightest color, and the portion (point group) corresponding to the ground 90 is displayed in the darkest color.

[0058] Refer to Figures 4 to 7 Details of the setting function of the search range based on the search range setting unit 157 will be described. The setting function of the search range based on the search range setting unit 157 includes the following (F1) to (F4).

[0059] (F1) A function of setting a search range specified on a photographed image as a range determined in the actual work space.

[0060] (F2) A function of setting a search range specified on a photographed image corresponding to the photographing range of the vision sensor.

[0061] (F3) Function to set a search range in the actual space.

[0062] (F4) Function to set the search range specified in the actual space corresponding to the shooting range of the visual sensor.

[0063] In the above functions (F1) and (F2), the search range setting unit 157 accepts an input of specifying the search range as a range on the image. As an example, the search range setting unit 157 can accept the following methods as input methods for specifying the search range as a range on the image:

[0064] (a1) Method of specifying the search range as numerical information based on coordinates on the image;

[0065] (a2) Input method of specifying the search range on the GUI (Graphical User Interface).

[0066] In the above specified method (a1), the search range setting unit 157, for example Figure 4 as shown on the left, accepts an input of specifying the search range E11 as numerical information based on coordinates on the distance image M11. In Figure 4 the example on the left, as the position of the search range E11, the coordinate values of the upper left corner ((x, y) = (5, 2)) in the coordinate system with the upper left corner of the distance image M11 as the origin and the widths in the x-axis and y-axis directions (Δx = 20, Δy = 10) are specified. In addition, an example of the case where the search range is set as a rectangular area and the position and size of the search range are specified as numerical information is shown here, but the search range is not limited to a rectangle and can also be specified as other shapes such as a circle or a polygon. In the case of specifying as other shapes, as numerical information, for example, information specifying the geometric center, outer diameter, vertex positions, etc. can be used. The search range setting unit 157 can accept, for example, specifying the search range based on numerical information based on coordinates on the image like this as an input from a robot program, or can also accept a user operation of inputting a numerical information value on the UI. In addition, such a UI screen can be provided on the display unit 41 of the teaching operation panel 40.

[0067] In the above specified method (a2), the search range setting unit 157, for example Figure 5As shown in the example on the left, provide a GUI that can specify the position and size of a rectangular frame line representing the search range on the distance image M11. Such a GUI can be displayed, for example, on the display unit 41 of the teaching operation panel 40. The operator can move the cursor C1 through mouse operation or touch operation to specify the position and size of the frame representing the search range E11. In addition, an example of setting the search range as a rectangular area on the GUI is shown here, but the search range can also be set as other shapes such as a circle or a polygon on the GUI. In the specifying method (a2), the operator can specify the search range through an intuitive operation.

[0068] The search range setting unit 157 can accept the specification of the search range in a manner that simultaneously applies the above-mentioned specifying methods (a1) and (a2). In this case, the search range setting unit 157 accepts, for example, the specification of a process where the operator sets the search range on the GUI and then strictly sets the search range based on numerical information.

[0069] Refer to Figure 4 The above function (F1) will be described. Here, as an example, the above-mentioned specifying method (a1) is used as the method for specifying the search range. Figure 4 The distance image M11 shown on the left is a distance image of the area (shooting range R11) where the workpieces W1 - W3 are arranged on the setting surface 92 in the work space. When the search range E11 is specified on the image by the specifying method (a1), as Figure 4 shown on the right, the search range setting unit 157 sets the search range E11 specified on the image as the search range determined in the actual space (indicated by the same symbol E11 on the Figure 4 right). In this case, the search range setting unit 157 can convert it to the position of the search range in the actual space corresponding to the search range on the image by using calibration data or the like.

[0070] In this function (F1), the search range setting unit 157 determines the search range specified by the above-mentioned specifying method (a1) or (a2) as a fixed range in the actual space. Therefore, even when the position or posture of the vision sensor 70 changes, the search range can be kept fixed in the actual space.

[0071] The determination unit 154 makes a determination as to whether an object is placed by using the distance image and taking the range in the actual space corresponding to the search range E11 as the determination object. Since the search range is set as a fixed range in the actual space, even when the position or posture of the vision sensor 70 changes and the shooting range moves, the fixed range in the actual space can be used as the determination object. This function (F1) becomes an effective function when the working place of the robot is determined to be a specified position.

[0072] Refer to Figure 5 The above function (F2) will be described. Here, as an example, as a method for specifying a search range, the above-mentioned specifying method (a2) is used. When specifying a search range E11 on the image, as Figure 5 shown on the right, the search range setting unit 157 sets the search range E11 to a range corresponding to the shooting range R11 of the captured image M11. That is, the search range setting unit 157 sets the search range E11 to a range that is fixed in position relative to the shooting range R11 (refer to Figure 5 the right side). Therefore, when the position or posture of the vision sensor 70 changes and the shooting range R11 moves, the search range E11 moves following the shooting range R11.

[0073] In the case of this function, the determination unit 154 uses the search range E11 set for the captured distance image M11 as the determination object to determine whether an object is placed. Since the search range is set in association with the shooting range, it is possible to always fix the range of the determination object on the captured image.

[0074] In the above functions (F3) and (F4), the search range setting unit 157 accepts an input that designates the search range as a range in the actual work space. As an example, the search range setting unit 157 can accept the following methods as methods for designating the search range as a range in the actual space:

[0075] (b1) A method of performing a touch-up operation on the robot to specify a search range in the actual space; and

[0076] (b2) A method of designating a search range as position information in the actual space from an input source such as a robot program or user settings.

[0077] Refer to Figure 6 The above function (F3) will be described. Here, an example of using the specifying method (b1) is used. As Figure 6 shown on the left, in this function (F3), the operator operates the robot 10 to perform an operation of lightly touching (touching and lifting) a plurality of points in the work space using the touch pin 35 installed at the front end of the arm of the robot 10, thereby specifying the search range. In Figure 6 the example shown on the left, on the setting surface 92 in the work space, 4 points 301, 302, 302, and 304 corresponding to the four corner positions of the search range are specified by the touch operation. Through the touch operation, the three-dimensional positions of the four points 301-304 are obtained.

[0078] In this case, as Figure 6As shown on the right side, the search range setting unit 157 sets a rectangular range with four points 301 - 304 as the four corners as the search range E12 in the actual space. In this function (F3), the search range E12 is set as a range fixed within the actual space. The operator can set the search range at a desired position within the work space.

[0079] The determination unit 154 determines whether an object is placed by taking the range in the distance image corresponding to the search range E12 in the actual space as the determination object. In addition, when performing image processing such as determination on the image, the determination unit 154 obtains the search range on the image corresponding to the search range E12 specified as the position in the actual space based on calibration data, etc. In this example, since the search range is determined in the actual space, even when the position or posture of the vision sensor 70 changes and the shooting range moves, the range fixed in the actual space can be used as the determination object. This function (F3) becomes an effective function when the work place of the robot is determined to be a specified position.

[0080] Refer to Figure 7 The above function (F4) will be described. Here, the case of using the above - specified method (b1) is exemplified. The operator, in the same way as in the case of the above function (F3), performs a touch operation on the robot to specify multiple points in the actual space (refer to Figure 7 the left side). Thereby, the three - dimensional coordinate values representing the four - corner positions of the search range are obtained.

[0081] The search range setting unit 157 sets the search range E12 in the actual space specified by the touch operation as a search range corresponding to the shooting range R11 of the vision sensor 70 (denoted by the same E12 on the Figure 7 right side). Therefore, in the case of this function, the search range E12 is set at a position based on the shooting range R11. Even when the position or posture of the vision sensor changes and the shooting range R11 moves, the search range follows this movement. That is, the position of the search range on the captured image can always be fixed.

[0082] In the case of this function, the determination unit 154 determines whether an object is placed by taking the search range E12 set for the distance image of the shooting range R11 as the determination object. Since the search range is set in association with the shooting range, the range of the determination object on the captured image can always be fixed.

[0083] As described above, according to the present embodiment, regarding the specification of the search range, it can be specified as a range on the image, or it can also be specified as a range in the actual space. Therefore, the operator can specify an appropriate search range corresponding to various conditions and situations including the working content and the environmental conditions of the working space. In addition, as described above, as an input method for specifying the search range, a method of specifying from a robot program, a method of specifying through user input via a UI, a method of specifying by touching the robot, etc. can be adopted. Thus, the present embodiment can provide a high degree of freedom regarding the method of specifying the search range.

[0084] The determination unit 154 uses the search range set as described above as the determination object based on the distance image, and determines whether an object is placed. As a specific example of the process in which the determination unit 154 determines whether an object is placed in the search range, the following describes the case where the following operations are performed: based on the frequency distribution (histogram) created by the histogram creation unit 153 from the distance image, it is determined whether an object is placed on the inspection target surface. As a method for determining whether an object is placed, a method other than the frequency distribution (histogram) used in this example can be used.

[0085] The histogram creation unit 153 creates a histogram with distance (brightness) as a variable for an image in which the brightness changes according to the distance, such as the distance image M1. When creating a histogram, the histogram creation unit 153 can search for a point group within the search range set by the function of the search range setting unit 157 on the distance image to create a histogram. Taking the distance image M1 obtained as Figure 3 an example for explanation. As Figure 8 shown, it is assumed that a search range E1 is specified on the distance image M1. Figure 9 Fig. 201 shows the histogram created by the histogram creation unit 153 in this case. In Figure 9 the histogram 201, the horizontal axis (variable) represents the brightness of the points, and the vertical axis represents the sum of the number of points (frequency). In the search range E1, the proportion of the ground 90 is the largest, so the frequency of the ground 90, which represents the darkest point group, is the highest. The upper surfaces of the workpieces W1, W2, and W3 with different heights have the same area. Therefore, the frequencies of the point groups representing the respective brightnesses of the workpieces W1, W2, and W3 are the same.

[0086] Referring to Figure 9 it can be seen that by determining the brightness of the ground 90 as the inspection target surface, it is possible to determine whether an object with a height different from that of the ground 90 is placed on the ground 90. In Figure 9In the case of the example, point groups with different brightness from the ground 90 are distributed at three locations. Therefore, it can be determined that three workpieces W1, W2, and W3 with different heights are placed on the ground 90. Regarding the determination of the brightness of the inspection target surface (ground 90), the brightness of the inspection target surface (the threshold value for determining whether it is the brightness of the inspection target surface) can be set through user input. It can also be automatically set in the determination unit 154. The details of the method for setting the brightness of the inspection target surface (the threshold value for determining whether it is the brightness of the inspection target surface) will be described later.

[0087] As Figure 8 shown, consider the case where a search range E2 is specified on the distance image M1. Figure 10 Fig. 202 shows the histogram created by the histogram creation unit 153 in this case. Since the search range E2 is an area that only includes the ground 90, there is only one brightness value with a frequency in the histogram 202, and in addition, its frequency is lower than that in the case of the search range E1.

[0088] By setting a wide search range within the shooting range R1 like the search range E1, a histogram can be obtained for a wide range, and it can be confirmed whether there are objects within the wide range. On the other hand, when it is desired to quickly determine the presence or absence of an object for a limited area, it is useful to specify a relatively narrow search range like the search range E2. In addition, when the search range is limited to a relatively narrow range, the possibility that the histogram contains deviated values can be reduced.

[0089] Figure 8 The shooting range R1 of the vision sensor 70 is also shown. The shooting range setting unit 156 accepts the designation of the size / position of the shooting range within the range that the vision sensor 70 can shoot. The shooting range setting unit 156 can provide a graphical user interface for setting at least one of the size and position of a rectangular image representing the shooting range on a screen as Figure 8 shown. Such a user interface can also be provided on the display unit 41 of the teaching operation panel 40.

[0090] The histogram creation unit 153 may have the following functions: according to a specified operation on the operation unit of the teaching operation panel 40, for example, display on the display unit 41 of the teaching operation panel 40 a histogram (histograms 201, 202, etc.) created for the specified range (shooting range or retrieval range) as described above. In addition, the histogram creation unit 153 may have the following functions: save the distance information within the specified range (shooting range or retrieval range) in, for example, the storage unit 161 through numerical information (a numerical table representing the histogram), and display it on the display unit 41 of the teaching operation panel 40 according to a specified operation on the operation unit of the teaching operation panel 40.

[0091] As described above, the robot control device 50 has the following function: determine whether an object is placed within the retrieval range specified by the specified input information. Figure 11 This is a flowchart that more generally describes such a function (object presence / absence determination process) of the determination unit 154. In addition, this process is executed under the control of the processor 51. As Figure 11 shown, the image acquisition unit 152 acquires a distance image captured by the vision sensor 70 (step S1). Next, the determination unit 154 determines whether an object is placed within the retrieval range specified by the specified input information based on the distance image (step S2).

[0092] In the present embodiment, the robot control device 50 can determine whether an object is placed on the inspection target surface (the ground 90 in the case of histogram 201) based on the histogram as described above. Figure 12 This is a flowchart showing a specific operation example of the object presence / absence determination process executed by the robot control device 50 (processor 51). First, the image acquisition unit 152 acquires a distance image obtained by shooting a shooting range including the inspection target surface from the vision sensor 70 (step S101). Next, the histogram creation unit 153 creates a histogram of the brightness in the distance image (step S102). Regarding the histogram, it can be created within the retrieval range set as described above. Then, the determination unit 154 determines whether an object is placed on the inspection target surface within the retrieval range based on the histogram related to the brightness in the distance image (step S103).

[0093] Hereinafter, two examples Figure 12 of applying the object presence / absence determination process to the workpiece handling operation based on the robot 10 will be listed to explain the details of the object presence / absence determination process.

[0094] The first embodiment is an operation example when the object presence / absence determination process is applied to the workpiece transfer operation performed by the workpiece transfer execution unit 160. Figure 13This is a flowchart showing the application of the object presence / absence determination process in workpiece transfer processing. This process is executed under the control of the processor 51 of the robot control device 50.

[0095] The workpiece transfer execution unit 160 takes out a workpiece from a supply device such as a belt conveyor by the robot 10 (step S11). When the workpiece transfer execution unit 160 places the workpiece at the transfer destination (transfer device, workbench, etc.), in order to confirm that there is no object placed at the transfer destination, it executes the object presence / absence determination process (step S12). In addition, in this example, on the premise that there is no object (obstacle) at the transfer destination, the operation of the robot 10 to place the workpiece at the transfer destination is taught.

[0096] Refer to Figure 14 and Figure 15 A specific example of the object presence / absence determination process executed in step S12 will be described. Figure 14 This shows the state where the vision sensor 70 captures a distance image in a situation where no object is placed on the placement surface 91 at the transfer destination, and also shows the histogram 211 of the distance image obtained at this time. Figure 15 This shows the state where the vision sensor 70 captures a distance image in a state where the workpiece W5 is placed on the placement surface 91, and also shows the histogram 212 obtained at this time.

[0097] The robot control device 50 can pre-acquire the histogram 211 obtained in a state where no object is placed on the placement surface 91 and store it in the storage unit 161. The determination unit 154 can determine whether an object is placed on the placement surface 91 by comparing the pre-stored histogram 211 with the histogram 212 of the distance image obtained in step S12 during the execution of the workpiece transfer process.

[0098] As Figure 14 shown in the histogram 211, the threshold setting unit 155 provides a function for setting a threshold value used to determine whether there is a frequency of heights (brightnesses) other than the height (brightness) of the placement surface 91 in the distance image. Here, for example, Figure 14 as shown, the threshold value can be set to the upper limit value K1 and the lower limit value K2 for determining the height (brightness) range including the reference based on the height (brightness) of the placement surface 91. The determination unit 154 can determine that an object is placed on the placement surface 91 when it confirms the existence of a group of brightness points outside the range determined by the upper limit value K1 and the lower limit value K2.

[0099] This threshold value (upper limit value K1, lower limit value K2) can be set by the user or automatically set by the threshold setting unit 155. The threshold setting unit 155 can perform the automatic setting of the threshold value (upper limit value K1 and lower limit value K2) through the following process:

[0100] r1) Determine the brightness having a frequency in the pre-stored histogram 211 as the brightness of the mounting surface 91;

[0101] r2) Set the brightness obtained by adding a prescribed margin to the brightness of the mounting surface 91 as the upper limit value K1, and set the brightness obtained by subtracting the prescribed margin from the brightness of the mounting surface 91 as the lower limit value K2.

[0102] In addition, here, a method in which the threshold setting unit 155 automatically sets a threshold based on a distance image acquired in advance in a state where there is no object on the inspection target surface has been described. As another method, for example, an algorithm that determines a point group satisfying the following conditions as the inspection target surface may be adopted according to the condition of the inspection target surface.

[0103] (d1) Has the most frequency in the distance image at the time of inspection; or

[0104] (d2) Has the most frequency and is the darkest in the distance image at the time of inspection.

[0105] In a situation where it is assumed that items are basically not placed on the inspection target surface most of the time, the algorithm (d1) is effective. In a situation where items are basically not placed on the inspection target surface most of the time and the inspection target surface is at the lowest position within the shooting range (or retrieval range) such as on the ground, the algorithm (d2) is effective.

[0106] When accepting user setting of thresholds (upper limit value K1, lower limit value K2), the threshold setting unit 155 can provide Figure 16 a graphical user interface screen (GUI screen 350) as shown to the display unit 41 of the teaching operation panel 40. Figure 16 The GUI screen 350 shown displays the histogram created by the histogram creation unit 153 based on the distance image captured in a state where there is no object on the inspection target surface, and accepts the setting of thresholds on this screen. Figure 16 The GUI screen 350 shown is configured to be able to input the upper limit value K1 and the lower limit value K2 of the threshold into the numerical input fields 351, 352 with the brightness N1 of the inspection target surface in between. The user can set the desired upper limit value K1 and lower limit value K2 of the threshold in the numerical input fields 351, 352. In this way, by being able to set the threshold on the screen displaying the histogram of the inspection target surface, the user can intuitively set the threshold with reference to the histogram.

[0107] At the time of inspection (i.e., in the presence / absence of object determination process in step S12), assume a situation where Figure 15 as shown, a workpiece W5 remains on the mounting surface 91. In this case, generate as Figure 15A histogram 212 as shown on the right side. Then, the determination unit 154 determines that there is a group of points with a luminance exceeding the upper limit value K1 of the threshold value (i.e., the presence of the workpiece W5). In this case, in step S13, it is determined that an object is placed on the inspection target surface (S13: Yes), the operation of placing the workpiece is interrupted, and the process proceeds to step 15.

[0108] On the other hand, when it is determined in step S13 that no object is placed on the inspection target surface (S13: No), the workpiece transfer execution unit 160 executes the operation of placing the workpiece on the placement surface 91 (step S14). Then, the workpiece transfer execution unit 160 can continue to take out and transfer the next workpiece.

[0109] In step S15, the history image storage unit 158 stores, as a history image, a two-dimensional image obtained by photographing the state in which the workpiece W5 remains on the placement surface 91 through the two-dimensional camera function of the vision sensor 70, together with other history information, in the storage device 80. Then, the workpiece transfer execution unit 160 ends this process.

[0110] In this way, according to the workpiece transfer process of the present embodiment, it is possible to reliably detect that there is nothing on the place where the workpiece is placed. Therefore, the safety of the operation of placing the workpiece can be improved. Assuming that there are some objects (obstacles) on the place where the workpiece is placed, the operation of placing the workpiece by the robot can be interrupted. Therefore, when the robot performs the operation of placing the workpiece on the place where the workpiece is already placed, it is possible to prevent defects (such as the stop of the production cycle) caused by interference between workpieces. Therefore, according to the present embodiment, the workpiece transfer process can be performed more safely and smoothly.

[0111] The history image saved in step S15 can be used for grasping the state where the workpiece W5 remains on the placement surface 91, cause analysis, etc. In addition, necessary countermeasures can also be taken accordingly.

[0112] During the workpiece transfer process, the histogram creation unit 153 can display the histogram based on the distance image obtained by the vision sensor 70 on the display screen (display unit 41) according to a prescribed user operation. In the workpiece transfer process, the determination unit 154 can determine whether an object is placed on the inspection target surface within the set range (the photographing range or the retrieval range).

[0113] A second embodiment in which the presence / absence of object determination processing is applied to the workpiece handling operation based on the robot 10 will be described. The second embodiment is an operation example when the presence / absence of object determination processing is applied to the take-out process of bulk workpieces performed by the workpiece take-out execution unit 159. Figure 17This is a flowchart of the presence / absence object determination process applied in the process of removing bulk workpieces. This process is executed under the control of the processor 51 of the robot control device 50. Here, as Figure 18 and Figure 19 shown, the operation of removing the bulk workpiece W6 (only some of the workpieces are labeled) in the container 95 on the ground 90 is performed.

[0114] The workpiece removal execution unit 159 performs the removal process of the bulk workpiece according to the workpiece removal program stored in the storage unit 161 (step S21). Then, the workpiece removal execution unit 159 determines whether the removal of the bulk workpiece is completed according to the specified completion determination condition (step S22). The specified completion determination condition is, for example: performing the detection process of the workpiece W6 based on the two-dimensional image obtained by the vision sensor 70 photographing the inside of the container 95, and as the detection result, the workpiece W6 is not detected. In the case where it is determined that the removal of the workpiece is not completed (S22: No), the removal process of the workpiece is continued.

[0115] In the case where the workpiece removal execution unit 159 determines that the removal process of the workpiece is completed (S22: Yes), the presence / absence object determination process is performed (step S23). In this presence / absence object determination process, it is determined whether there is a workpiece W6 remaining in the container 95.

[0116] Refer to Figures 18 to 20 The presence / absence object determination process performed in step S23 will be described. Figure 18 It shows the state of the vision sensor 70 photographing the distance image in the state where there are multiple workpieces W6 in the container 95, and the histogram 221 of the distance image obtained at this time. Figure 19 It shows the state of the vision sensor 70 photographing the distance image in the state where there is no object in the container 95, and the histogram 222 of the distance image obtained at this time. Figure 20 It shows the state of the vision sensor 70 photographing the distance image in the state where there is one workpiece W6 remaining in the container 95, and the histogram 223 of the distance image obtained at this time.

[0117] In addition, in this embodiment, as Figure 19 shown, the histogram 222 photographed in the state where there is no object in the container 95 is pre-stored in the storage unit 161. The threshold setting unit 155 determines the height (brightness) of the bottom surface 95a of the container 95 based on this histogram 222, and determines the upper limit value K1 and the lower limit value K2 of the threshold based on this brightness. The brightness determination of the bottom surface 95a and the setting of the threshold (upper limit value K1 and lower limit value K2) can be performed according to the above processes (r1) and (r2). In addition, as Figure 18As shown, when there are multiple workpieces W6 in the container 95, in the histogram 221, multiple point groups are distributed in the luminance region exceeding the range determined by the upper limit value K1 and the lower limit value K2 of the threshold with respect to the bottom surface 95a. On the other hand, as Figure 19 shown, when there are no remaining workpieces W6 in the container, there are no point groups with frequencies in the luminance region exceeding the range determined by the upper limit value K1 and the lower limit value K2 of the threshold with respect to the bottom surface 95a.

[0118] When the result of the object presence / absence determination process is that there are no point groups with luminance exceeding the range determined by the upper limit value K1 and the lower limit value K2 of the threshold, it is determined that there is no object on the bottom surface 95a, which is the surface to be inspected (S24: No). In this case, the removal of the bulk workpieces is appropriately completed, and this process ends.

[0119] Assume that the condition inside the container 95 when performing the object presence / absence determination process is a condition where there are remaining workpieces W6 as Figure 20 shown. In this case, the determination unit 154 determines the presence / absence of an object on the surface to be inspected through the histogram 223 as Figure 20 shown. In the histogram 223, the distribution of point groups is confirmed in the luminance region exceeding the range determined by the upper limit value K1 and the lower limit value K2 of the threshold. Therefore, in this case, the determination unit 154 determines that there is an object on the bottom surface 95a, which is the surface to be inspected (S24: Yes).

[0120] At this time (S24: Yes), the historical image storage unit 158 saves, as a historical image, the two-dimensional image obtained by photographing the state where the workpiece W6 remains on the bottom surface 95a using the two-dimensional camera function of the visual sensor 70, together with other historical information, in the storage device 80 (step S25). Then, the workpiece removal execution unit 159 ends this process.

[0121] As described above, according to the workpiece removal process of the present embodiment, it is possible to reliably detect a defective situation where the robot control device determines that the workpiece removal is completed while there are remaining workpieces in the container.

[0122] The historical image saved in step S25 can be used to analyze the reason for the determination of completion of removal made by the workpiece removal execution unit 159 in the state where there are remaining workpieces W6. For example Figure 20 as shown in the example, in a situation where one side surface of the workpiece W6 is in close contact with the inner side surface of the container 95 and remains, it can be considered that the workpiece W6 cannot be detected well in the detection process of applying pattern matching to the two-dimensional image. On the other hand, as described above, according to the determination of the object presence / absence determination process, it is possible to appropriately determine the presence of the workpiece W6. Therefore, as Figure 20As shown, the state of the remaining workpiece W6 is saved as a historical image, whereby the above-mentioned cause can also be analyzed and necessary countermeasures can be taken.

[0123] In addition, in the above-described second embodiment, the imaging range or the search range can be adjusted in advance so as not to measure the side surface of the container 95. By setting the imaging range or the search range in this way, only the frequencies based on the height of the bottom surface 95a of the container 95 and the workpiece W6 (when the workpiece W6 exists) appear in the histogram, and the determination of the presence or absence of an object can be performed better.

[0124] As described above, according to the present embodiment, it is possible to reliably determine whether an object is placed at the inspection target location, and it is possible to reliably prevent the occurrence of an abnormal situation and detect the occurrence of an abnormal situation in the workpiece handling operation based on the robot.

[0125] In the above-described embodiment, Figure 2 The configuration of the functions in the functional block diagram shown is an example, and various modifications can be made to the functional configuration. For example, there can be a structural example in which a part of the functional blocks arranged in the robot control device 50 (for example, the threshold setting unit 155, the imaging range setting unit 156, and the search range setting unit 157) is arranged on the teach pendant 40 side.

[0126] The functions of the teach pendant 40 (functions such as the display unit and the operation unit as the user interface) can be included in the functions of the robot control device 50 and defined as the robot control device.

[0127] In the above-described embodiment, an example of the operation of the determination unit 154 using the histogram of the distance image to determine whether an object is placed in the search range has been described, but the example of the operation of the determination unit to determine whether an object is placed in the search range is not limited to this. For example, the determination unit 154 can use the distance data of each point in the distance image to determine whether an object is placed in the specified search range. In this case, for example, when it is confirmed that there is an object having a specific height based on the height of the reference plane in the search range, it can be determined that an object is placed in the search range (on the reference plane). In addition, when the determination unit is configured in this way, the functions related to the determination based on the histogram (the histogram creation unit 153 and the threshold setting unit 155) can be omitted.

[0128] Figure 2 The functional blocks of the robot control device and the vision sensor control device shown can be realized by the processor of these devices executing various software stored in the storage device, or can also be realized by a structure mainly composed of hardware such as an ASIC (Application Specific Integrated Circuit).

[0129] The above-described determination process for the presence or absence of an object can be executed on various information processing devices ( Figure 11 , Figure 12 ).

[0130] A program for executing the determination process for the presence or absence of an object ( Figure 11 , Figure 12 ), the workpiece transfer process ( Figure 13 ), the workpiece extraction process ( Figure 17 ), etc. in the above-described embodiment can be recorded in various computer-readable storage media (for example, semiconductor memories such as ROM, EEPROM, and flash memory, magnetic storage media, optical discs such as CD-ROM and DVD-ROM).

[0131] Although the present disclosure has been described in detail, the present disclosure is not limited to the above-described respective embodiments. These embodiments can be subjected to various additions, replacements, changes, partial deletions, etc. within the scope of not departing from the spirit of the present disclosure, or within the scope of not departing from the spirit of the present disclosure derived from the content recited in the claims and its equivalents. In addition, these embodiments can also be implemented in combination. For example, in the above-described embodiment, the order of each action and the order of each process are shown as an example, but are not limited thereto. In addition, the same applies to the cases where numerical values or mathematical expressions are used in the description of the above-described embodiment.

[0132] The following supplementary notes are further described regarding the above-described embodiment and modification examples.

[0133] (Supplementary Note 1)

[0134] A control device (50) includes: an image acquisition unit (152) that acquires a distance image captured by a vision sensor (70); and a determination unit (154) that determines whether an object is placed within a range specified by prescribed input information based on the distance image.

[0135] (Supplementary Note 2)

[0136] The control device (50) according to Supplementary Note 1, wherein

[0137] the control device (50) further includes: a search range setting unit (157) that sets the range specified by the prescribed input information as a search range.

[0138] (Supplementary Note 3)

[0139] The control device (50) according to Supplementary Note 2, wherein

[0140] the prescribed input information is information that designates the search range as a range on an image.

[0141] (Supplementary Note 4)

[0142] The control device (50) according to Supplementary Note 2, wherein,

[0143] The specified input information is: information specifying the search range as a range in the actual space.

[0144] (Supplementary Note 5)

[0145] The control device (50) according to any one of Supplementary Notes 1 to 4, wherein,

[0146] The specified input information is described in the robot program.

[0147] (Supplementary Note 6)

[0148] The control device (50) according to any one of Supplementary Notes 1 to 4, wherein,

[0149] The specified input information is: information input via the user interface.

[0150] (Supplementary Note 7)

[0151] The control device (50) according to any one of Supplementary Notes 2 to 4, wherein,

[0152] The search range setting unit (157) sets the search range as a range determined in the actual space.

[0153] (Supplementary Note 8)

[0154] The control device (50) according to any one of Supplementary Notes 2 to 4, wherein,

[0155] The search range setting unit (157) sets the search range as a range corresponding to the shooting range of the vision sensor (70).

[0156] (Supplementary Note 9)

[0157] The control device (50) according to Supplementary Note 3, wherein,

[0158] The range on the image is represented as numerical information based on the coordinates on the image.

[0159] (Supplementary Note 10)

[0160] The control device (50) according to Supplementary Note 3, wherein,

[0161] The search range setting unit (157) provides a graphical user interface for specifying the range on the image through graphical operations.

[0162] (Supplementary Note 11)

[0163] The control device (50) according to Note 4, wherein,

[0164] The search range setting unit (157) acquires the three-dimensional coordinates of a plurality of points obtained by causing the robot (10) to perform the following action as information specifying the search range as a range in the actual space, wherein the action is to cause the robot to lightly touch the plurality of points in the actual space.

[0165] (Note 12)

[0166] The control device (50) according to Note 1, wherein,

[0167] The control device (50) further includes: a shooting range setting unit (156) that sets the range specified by the specified input information as the shooting range for the visual sensor (70) to shoot, and the shooting range setting unit (156) issues to the visual sensor (70): a signal instructing the visual sensor to shoot within the set shooting range.

[0168] (Note 13)

[0169] The control device (50) according to any one of Notes 1 to 12, wherein,

[0170] The determination unit (154) determines whether an object is placed on the inspection target surface within the specified range.

[0171] (Note 14)

[0172] The control device (50) according to Note 13, wherein,

[0173] The determination unit (154) determines whether an object is placed on the inspection target surface based on the frequency distribution related to the distance information of each point in the acquired distance image.

[0174] (Note 15)

[0175] The control device (50) according to Note 14, wherein,

[0176] The control device (50) further includes: a threshold setting unit (155) that sets a threshold for identifying the following distance information, wherein the distance information refers to the distance information of the inspection target surface and the distance information of other objects in the inspection target surface in the frequency distribution, and the determination unit (154) uses the threshold to determine whether an object is placed on the inspection target surface.

[0177] (Note 16)

[0178] The control device (50) according to Note 15, wherein,

[0179] The threshold setting unit (155) sets the threshold based on the frequency distribution related to the distance information of each point of the distance image obtained in the state where no object is placed on the surface to be inspected.

[0180] (Note 17)

[0181] The control device (50) according to Note 15, wherein,

[0182] The threshold setting unit (155) is configured to provide a user interface that accepts the setting of the threshold based on a user operation.

[0183] (Note 18)

[0184] The control device (50) according to Note 17, wherein,

[0185] The user interface process is a graphical user interface including an image, and the image represents the frequency distribution of the distance image.

[0186] (Note 19)

[0187] The control device (50) according to Note 15, wherein,

[0188] The threshold setting unit (155) sets the threshold according to the threshold information described in the robot program.

[0189] (Note 20)

[0190] The control device (50) according to any one of Notes 14 to 19, wherein

[0191] The control device (50) further includes: a histogram creation unit (153) that generates an image representing the frequency distribution based on the distance information of each point in the acquired distance image and displays it on the display screen.

[0192] (Note 21)

[0193] The control device (50) according to Note 20, wherein,

[0194] The histogram creation unit (153) is configured to generate an image representing the frequency distribution within a specified range in the distance image.

[0195] (Note 22)

[0196] The control device (50) according to any one of Notes 14 to 21, wherein,

[0197] The control device (50) further includes: a workpiece transfer execution unit (160) configured to perform a workpiece transfer operation of picking up a workpiece from a supply device by a robot (10) and placing it on a specified placement surface.

[0198] The determination unit (154) determines whether there is an object on the placement surface based on a distance image obtained by photographing the placement surface before the workpiece transfer execution unit (160) places the workpiece on the placement surface.

[0199] (Appendix 23)

[0200] The control device (50) according to any one of Appendices 14 to 21, wherein

[0201] The control device (50) further includes: a workpiece extraction execution unit (159) configured to perform an extraction operation of extracting workpieces scattered in a container by a robot (10). When the workpiece extraction execution unit (159) makes a determination of completion of workpiece extraction based on specified completion determination conditions, the determination unit (154) determines whether there is an object on the bottom surface of the container based on a distance image obtained by photographing the bottom surface of the container.

[0202] (Appendix 24)

[0203] The control device (50) according to Appendix 22 or 23, wherein

[0204] The control device (50) further includes: a historical image storage unit (158) configured to store a historical image when the determination unit (154) determines that there is an object.

[0205] (Appendix 25)

[0206] The control device according to any one of Appendices 14 to 24, wherein

[0207] The distance image is an image representing the distance information of each point by brightness, and the frequency distribution represents the brightness distribution of each point of the distance image.

[0208] (Appendix 26)

[0209] A robot system (100) includes:

[0210] A robot (10); a vision sensor (70); and the control device (50) according to any one of Appendices 1 to 25, wherein the control device (50) controls the robot (10).

[0211] (Appendix 27)

[0212] A method for determining the presence or absence of an object, which is a method executed on an information processing device. In the method for determining the presence or absence of an object, a distance image captured by a vision sensor (70) is obtained; and based on the distance image, it is determined whether an object is placed within a range specified by prescribed input information.

[0213] (Appendix 28)

[0214] A program for causing a processor of a computer to execute the following processes: obtaining a distance image captured by a vision sensor (70); and based on the distance image, determining whether an object is placed within a range specified by prescribed input information.

[0215] Symbol Explanation

[0216] 10 Robot

[0217] 20 Vision Sensor Control Device

[0218] 33 Manipulator

[0219] 40 Teach Pendant

[0220] 50 Robot Control Device

[0221] 51 Processor

[0222] 70 Vision Sensor

[0223] 80 Storage Device

[0224] 90 Floor

[0225] 91 Placement Surface

[0226] 95 Container

[0227] 95a Bottom Surface

[0228] 100 Robot System

[0229] 121 Image Processing Unit

[0230] 122 Storage Unit

[0231] 151 Motion Control Unit

[0232] 152 Image Acquisition Unit

[0233] 153 Histogram Creation Unit

[0234] 154 Judgment Unit

[0235] 155 Threshold Setting Unit

[0236] 156 Shooting Range Setting Unit

[0237] 157 Retrieval range setting unit

[0238] 158 Historical image storage unit

[0239] 159 Workpiece removal execution unit

[0240] 160 Workpiece transfer execution unit

[0241] 161 Storage unit

[0242] 350 GUI screen

Claims

1. A control device, characterized in that, comprising: an image acquisition unit that acquires a distance image captured by a vision sensor; and a determination unit that determines whether an object is placed within a range specified by the prescribed input information based on the distance image.

2. The control device according to claim 1, wherein the control device further comprises: a search range setting unit that sets the range specified by the prescribed input information as a search range.

3. The control device according to claim 2, wherein the prescribed input information is information that specifies the search range as a range on an image.

4. The control device according to claim 2, wherein the prescribed input information is information that specifies the search range as a range in the actual space.

5. The control device according to any one of claims 1 to 4, wherein the prescribed input information is described in a robot program.

6. The control device according to any one of claims 1 to 4, wherein the prescribed input information is information input via a user interface.

7. The control device according to any one of claims 2 to 4, wherein the search range setting unit sets the search range as a range determined in the actual space.

8. The control device according to any one of claims 2 to 4, wherein the search range setting unit sets the search range as a range corresponding to the shooting range of the vision sensor.

9. The control device according to claim 3, wherein the range on the image is represented as numerical information based on the coordinates on the image.

10. The control device according to claim 3, wherein the search range setting unit provides a graphical user interface for specifying the range on the image through graphical operations.

11. The control device according to claim 4, wherein the search range setting unit acquires the three-dimensional coordinates of a plurality of points obtained by causing the robot to perform an action of lightly touching the plurality of points in the actual space as the information that specifies the search range as a range in the actual space.

12. The control device according to claim 1, wherein the control device further comprises: a shooting range setting unit that sets the range specified by the prescribed input information as the shooting range for the vision sensor to shoot, and the shooting range setting unit issues a signal to the vision sensor instructing the vision sensor to shoot within the set shooting range.

13. The control device according to any one of claims 1 to 12, wherein the determination unit determines whether an object is placed on the inspection target surface within the specified range.

14. The control device according to claim 13, wherein the determination unit determines whether an object is placed on the inspection target surface based on the frequency distribution related to the distance information of each point in the acquired distance image.

15. The control device according to claim 14, wherein The control device further includes: a threshold setting unit configured to set a threshold for identifying the following distance information, where the distance information refers to the distance information of the inspection object surface and the distance information of other objects in the inspection object surface in the frequency distribution. The determination unit uses the threshold to determine whether an object is placed on the inspection object surface.

16. The control device according to claim 15, wherein the threshold setting unit sets the threshold according to the frequency distribution related to the distance information of each point of the distance image obtained when no object is placed on the inspection object surface.

17. The control device according to claim 15, wherein the threshold setting unit is configured to provide a user interface that accepts setting of the threshold based on a user operation.

18. The control device according to claim 17, wherein the user interface is configured to be a graphical user interface including an image, and the image represents the frequency distribution of the distance image.

19. The control device according to claim 15, wherein the threshold setting unit sets the threshold according to the threshold information described in the robot program.

20. The control device according to any one of claims 14 to 19, wherein the control device further includes: a histogram creation unit that generates an image representing the frequency distribution based on the distance information of each point in the acquired distance image and displays it on a display screen.

21. The control device according to claim 20, wherein the histogram creation unit is configured to generate an image representing the frequency distribution within a specified range in the distance image.

22. The control device according to any one of claims 14 to 21, wherein the control device further includes: a workpiece transfer execution unit configured to execute a workpiece transfer operation of taking out a workpiece from a supply device by a robot and placing it on a specified placement surface. The determination unit determines whether there is an object on the placement surface based on a distance image obtained by photographing the placement surface before the workpiece transfer execution unit places the workpiece on the placement surface.

23. The control device according to any one of claims 14 to 21, wherein the control device further includes: a workpiece extraction execution unit configured to execute an extraction operation of extracting workpieces scattered in a container by a robot. When the workpiece extraction execution unit makes a completion determination of workpiece extraction according to specified completion determination conditions, the determination unit determines whether there is an object on the bottom surface of the container based on a distance image obtained by photographing the bottom surface of the container.

24. The control device according to claim 22 or 23, wherein the control device further includes: a historical image storage unit that stores a historical image when the determination unit determines that there is an object.

25. The control device according to any one of claims 14 to 24, wherein the distance image is an image representing the distance information of each point by brightness. The frequency distribution represents the luminance distribution of each point of the distance image.

26. A robot system, characterized in that, comprising: a robot; a vision sensor; and a control device according to any one of claims 1 to 25, wherein the control device controls the robot.

27. A method for determining the presence or absence of an object, which is a method executed on an information processing device, characterized in that, In the method for determining the presence or absence of an object, a distance image captured by a vision sensor is acquired; based on the distance image, it is determined whether an object is placed within a range specified by prescribed input information.

28. A program, characterized in that it causes a processor of a computer to execute the following processes: acquire a distance image captured by a vision sensor; and based on the distance image, determine whether an object is placed within a range specified by prescribed input information.

Images