Image processing device

Abstract

Provided is an image processing device comprising: an image acquisition unit that acquires image information of an object, such information obtained by a visual sensor; a rotationally symmetrical figure detection unit that detects an object having a rotationally symmetrical shape, on the basis of the acquired image information; and a detection angle output unit that, on the basis of the acquired image information, outputs an angle of the detected object with respect to a predetermined reference direction. When the frequency of rotational symmetry of the detected object is n, the detection angle output unit outputs the angle of the detected object with respect to the predetermined reference direction as an angle of 0 degrees or more and less than (360 / n) degrees.

Description

Image Processing Device

[0001] The present disclosure relates to an image processing device.

[0002] 2. Description of the Related Art A robot system is known that is configured to detect an object using a visual sensor and handle the object using a robot based on the detection result.

[0003] In this regard, Patent Document 1 describes a robot system that uses a camera to recognize the position and orientation of an item on a pallet and transfers the item to a transport conveyor.Regarding image processing, Patent Document 2 describes an imaging device that uses a camera to capture an image of a document placed on a pedestal and obtains the outline of the document from the captured image.

[0004] JP 2006-082811 A JP 2005-260691 A

[0005] In a robot system that handles objects based on detection results from a visual sensor, when handling an object such as a box, the position and orientation of the hand relative to the object when the hand grasps the object are generally taught in advance. During operation of the robot system, the robot operates to grasp the object at the taught position and orientation based on the object detection results from the visual sensor. In this case, if the surface of the object is a rotationally symmetrical shape such as a rectangle or a square, an object at the same position and orientation may be detected at different angles (rotation angles) depending on the number n of rotational symmetries. If such different detected angles are output for an object with the same position and orientation, multiple handling positions for the robot with respect to the object may be generated, potentially causing the robot to attempt to handle the object from an irrational direction. There is a need for an image processing device that can output the detected angle of an object so that the robot can properly handle even rotationally symmetric objects.

[0006] One aspect of the present disclosure is an image processing device comprising an image acquisition unit that acquires image information of an object obtained by a visual sensor, a rotationally symmetric figure detection unit that detects an object with a rotationally symmetric shape based on the acquired image information, and a detection angle output unit that outputs the angle of the detected object with respect to a predetermined reference direction based on the acquired image information, wherein the detection angle output unit outputs the angle of the detected object with respect to the predetermined reference direction as an angle greater than or equal to 0 degrees and less than (360 / n) degrees, where n is the number of times the detected object has rotational symmetry.

[0007] These and other objects, features and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments of the invention illustrated in the accompanying drawings.

[0008] 1 is a diagram illustrating a configuration of a robot system including an image processing device according to an embodiment. FIG. 1 is a functional block diagram relating to an image processing device and a robot control device. FIG. 2 is a diagram illustrating two types of gripping positions and postures that a hand can take when an object is rectangular. FIG. 3 is a diagram illustrating detection of an angle for a rectangular object. FIG. 4 is a diagram illustrating different gripping positions and postures that a hand can take when an object is square. FIG. 5 is a diagram illustrating detection of an angle for a square object. FIG. 6 is a flowchart illustrating object detection processing according to a first embodiment. FIG. 7 is a flowchart illustrating object transfer processing according to the first embodiment. FIG. 8 is a diagram illustrating criteria for determining a square. FIG. 9 is a diagram illustrating a gripping position and posture of a hand with respect to an object when a rectangular object is erroneously determined to be a square. FIG. 10 is a flowchart illustrating object detection processing according to a second embodiment. FIG. 11 is a flowchart illustrating object transfer processing according to the second embodiment. FIG. 12 is a flowchart illustrating detection processing for an image processing device to detect an object as an arbitrary rotationally symmetric figure.

[0009] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, like components or functional parts are designated by like reference numerals. The scales of these drawings have been changed appropriately to facilitate understanding. Furthermore, the embodiment shown in the drawings is one example for implementing the present invention, and the present invention is not limited to the illustrated embodiment.

[0010] FIG. 1 is a diagram showing the configuration of a robot system 100 including an image processing device 20 according to one embodiment. The image processing device 20 controls a visual sensor 70 and processes images captured by the visual sensor 70. As shown in FIG. 1, the robot system 100 includes a robot 10, a robot control device 50 that controls the robot 10, a teaching device 40 connected to the robot control device 50, the visual sensor 70, and the image processing device 20. The robot system 100 can, for example, detect an object 90 placed in a working area using the visual sensor 70 and handle the object 90 with a hand 11 mounted on the robot 10. Here, the hand 11 is assumed to be a suction-type hand, as an example.

[0011] In this embodiment, the robot 10 is a vertical articulated robot, but other types of robots such as a horizontal articulated robot, a parallel link robot, a dual-arm robot, etc. may be used depending on the purpose of the work. While Fig. 1 shows an example in which a hand 11 is attached to the wrist of the robot 10, various types of tools (end effectors) can be attached to the wrist of the robot 10 depending on the purpose of the work.

[0012] The visual sensor 70 may be a two-dimensional camera that captures grayscale or color images, or a three-dimensional sensor that can acquire three-dimensional position information of an object, such as a range image or a three-dimensional point cloud. Examples of three-dimensional sensors that can be used include a stereo camera that performs measurements using a stereo method, and a time-of-flight (TOF) camera that measures the distance to an object using a time-of-flight method. Here, it is assumed that the visual sensor 70 is a fixed camera fixed in the workspace so as to capture an image of the object 90 from above, as shown in FIG. 1 . Alternatively, the visual sensor 70 may be mounted on the hand of the robot 10.

[0013] The image processing device 20 has the function of executing various image processing operations for detecting an object, as will be described later. The visual sensor 70 is assumed to be calibrated, and internal parameters that associate a camera coordinate system fixed to the visual sensor 70 (imaging surface) with two-dimensional coordinates set on the image (imaging surface) and external parameters that associate a world coordinate system fixed to the workspace with the camera coordinate system are assumed to be known in the robot system 100. These calibration data may be stored in, for example, a storage unit within the image processing device 20. This allows the image processing device 20 to convert the position of an object on a two-dimensional image captured by the visual sensor 70 into a position on a coordinate system (such as a robot coordinate system) fixed to the workspace.

[0014] Although Figure 1 shows an example of a configuration in which the image processing device 20 is placed in the robot system 100 as a device separate from the robot control device 50, the functions of the image processing device 20 may also be incorporated into the robot control device 50.

[0015] The image processing device 20 may have a hardware configuration as a general computer including a processor 21, memory (ROM, RAM, non-volatile memory, etc.), a storage device 22, an operation unit 23, a display unit 24, an input / output interface, a network interface, etc. (See FIG. 2 ). The image processing device 20 may be configured as a PC (personal computer) or any other type of information processing device. The display unit 24 is, for example, a liquid crystal display. The operation unit 23 may include, for example, a keyboard and various pointing devices such as a mouse.

[0016] The robot control device 50 controls the operation of the robot 10 in accordance with an operation program or commands from the teaching device 40. The robot control device 50 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, an input / output interface, a network interface, etc.

[0017] The teaching device 40 is used as an operation terminal for teaching (creating a program) the robot 10 and for performing various settings. The teaching device 40 may be a teaching operation panel, or may be configured as a tablet terminal or the like. The teaching device 40 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, a display unit 41, an input / output interface, a network interface, etc. The display unit 41 is configured as, for example, a liquid crystal display.

[0018] 2 is a functional block diagram of the image processing device 20 and the robot control device 50. As shown in Fig. 2, the robot control device 50 includes an operation control unit 151. The operation control unit 151 controls the operation of the robot 10 in accordance with an operation program or in accordance with commands from the teaching device 40. The robot control device 50 includes a servo control unit (not shown) that executes servo control for the servo motors of the respective axes in accordance with commands for the respective axes generated by the operation control unit 151.

[0019] 2, the image processing device 20 includes a visual sensor control unit 121, an image acquisition unit 122, a rotationally symmetric figure detection unit 123, a detected position and orientation output unit 124, and a judgment criterion setting unit 125. As shown in Fig. 2, these functional blocks may be realized by the processor 21 executing software.

[0020] 2 illustrates a storage device 22, an operation unit 23, and a display unit 24 as hardware components of the image processing device 20. The storage device 22 is formed of, for example, a non-volatile memory, a hard disk drive, etc., and stores model data of the object, calibration data, and various other data required for controlling the visual sensor and for image processing.

[0021] The visual sensor control unit 121 controls the operation of the visual sensor 70. For example, the visual sensor control unit 121 can receive an operation command for the visual sensor 70 from the robot control device 50 and control the visual sensor 70. The image acquisition unit 122 acquires image information obtained by the visual sensor 70 capturing an image of an object. If the visual sensor 70 is a two-dimensional camera, the image information may be a captured image of the object, or if the visual sensor 70 is a three-dimensional sensor, the image information may be three-dimensional position information of the object.

[0022] The rotationally symmetric figure detection unit 123 provides a function for detecting an object as a rotationally symmetric figure from image information. The rotationally symmetric figure detection unit 123 may detect an object having rotational symmetry using the following pattern matching technique: (1) extracting an object (figure) by detecting edges or contours from a captured image; (2) matching the center of gravity of pre-stored model data representing a rotationally symmetric figure with the center of gravity of the extracted object (figure), and applying transformation processes such as rotation and scaling as necessary. If the model data matches the object (figure), the object (figure) is determined to be a rotationally symmetric figure. When detecting an object such as a box from a range image, a group of pixels having certain height information may be identified as the object, and an edge or contour may be extracted from the identified group of pixels to detect the figure as the object.

[0023] In addition, when performing the detection process for such objects, the rotationally symmetric figure detection unit 123 can apply various component image processing techniques such as filtering, binarization processing, edge detection, corner detection, geometric transformation processing, Hough transform for detecting straight lines, and generalized Hough transform for detecting more general figures.

[0024] The rotationally symmetric figure detection unit 123 may include a quadrilateral shape detection unit 130 for detecting a quadrilateral object, and a square determination unit 131 for determining whether the detected quadrilateral is a square. For example, the square determination unit 131 may determine that the detected rectangle is a square when the ratio of the length of the short side to the length of the long side of the rectangle is equal to or greater than a predetermined threshold.

[0025] The detected position and orientation output unit 124 provides a function to output the detected position, orientation, etc. of the object obtained by the object detection process. These detection results are provided to the robot control device 50 and used to control the robot 10 that handles the object. The detected position and orientation output unit 124 can also be defined as a detected angle output unit because it has a function to output a detected angle.

[0026] The judgment criterion setting unit 125 provides a function for setting judgment criteria used by the square judgment unit 131 to determine whether a detected quadrangle is a square. The judgment criterion setting unit 125 may have a function for setting the judgment criteria by input from an external device or by user input. The judgment criterion setting unit 125 may display a user interface screen for inputting the judgment criteria on the display unit 24 and accept the input of the judgment criteria via the user interface screen. The input judgment criteria may be stored in the storage device 22.

[0027] Before describing specific examples of object detection by the image processing device 20, we will explain the problem that when the surface of the object has a rotationally symmetric shape such as a rectangle or a square, an object in the same position and posture may be detected at different rotation angles depending on the number n of rotational symmetries, which may result in the robot attempting to handle the object from an irrational direction.

[0028] FIG. 3 shows an image 201 resulting from detection by the visual sensor 70 when the surface of the object is rectangular. A rectangular object 90 is detected in the image 201. FIG. 3 also shows a state in which a central axis A1 (longitudinal central axis) parallel to the long side of the rectangular object and a central axis A2 (shortitudinal central axis) parallel to the short side are detected. The direction (angle) of the object is detected as the angle of the central axis A1 (longitudinal central axis) relative to the X-axis of the XY coordinate system defined in the image. In this case, because a rectangle is a point-symmetric figure with its center as the reference, it may be detected as a direction rotated 180 degrees from the expected direction. That is, as shown in FIG. 3, the angle of the object may be detected as 0 degrees or 180 degrees.

[0029] Here, when the visual sensor 70 detects a rectangular object 90 in a state as shown in image 201, the operator assumes that the detection direction is 0 degrees and teaches the robot 10 so that the longitudinal direction of the object coincides with the longitudinal direction of the hand 11. That is, the instructor teaches the robot 10 assuming that the hand 11 will grasp the object 90 in the grasping position and posture indicated by reference numeral 301 in FIG. 3. The forward direction defined for the hand 11 is indicated by arrow F in FIG. The forward direction (arrow F) defined for the hand 11 is assumed to be the direction that coincides with the front of the robot 10 when the robot 10 is in the origin position and posture as shown in FIG.

[0030] The problem in this situation is that when the detected direction is output as 0 degrees, the robot 10 can grasp the object 90 with the hand 11 at a grasping position / posture 301 assumed by the instructor, whereas when the detected direction is output as an angle reversed by 180 degrees, the robot 10 attempts to grasp the object at a grasping position / posture 302 in which the hand 11 is rotated by 180 degrees. When the detected direction is reversed by 180 degrees, the robot 10 operates to grasp the object from an irrational direction, which can cause problems such as an increase in cycle time.

[0031] To avoid such problems, the detected position and orientation output unit 124 according to this embodiment outputs the detected direction of the object as an angle greater than or equal to 0 degrees and less than 180 degrees when the object is detected to be rectangular. For example, when the object 90 is detected as in the image 202 shown in FIG. 4 , the detected position and orientation output unit 124 outputs the angle θ1 (0°≦θ1<180°) as the detected direction of the object 90, and does not output the angle θ2 (θ2≧180°) as the detected direction. This enables the robot 10 to always pick up the object in the appropriate orientation.

[0032] Next, the problem of a square object will be described with reference to FIG. 5 . Here, the algorithm for detecting a square is as follows: First, the quadrilateral shape detection unit 130 detects the object as a rectangle, as described above, thereby detecting the central axis A1 in the long side direction and the central axis A2 in the short side direction; and The square determination unit 131 determines that the object is a square if the ratio of the length of the short side to the length of the long side is equal to or greater than a predetermined threshold. Assume that a square object 90a is detected as shown in image 211 in FIG. 5 . In the case of a square object, when the lengths of the long and short sides are detected from the image, depending on factors such as how the object is depicted in the image, the central axis A1 in the long side direction and the central axis A2 in the short side direction may be detected as shown in the detection result image 211a. In contrast, the central axis A1 in the long side direction and the central axis A2 in the short side direction may be detected as rotated 90 degrees as shown in the detection result image 211b. That is, even if a square object is in the same position and orientation, depending on how the image is captured, the central axis of the long side may be detected as being rotated by 90 degrees in calculations.

[0033] In this case, in the case of the detection result shown in image 211a, the detection direction of the object 90a is output as 0 degrees, whereas in the case of the detection result shown in image 211b, the detection direction of the same object 90a is output as 90 degrees.

[0034] The problem in this situation is that when the detected direction is 0 degrees, the robot 10 can grasp the object 90a with the hand 11 in a suitable grasping position and posture that was previously taught (see the grasping posture of the hand 11 indicated by reference numeral 311), whereas when the detected direction is rotated by 90 degrees, the robot 10 attempts to grasp the object 90a in a position and posture that is similar to a 90-degree rotated position and posture of the hand 11 (see the grasping posture of the hand 11 indicated by reference numeral 312). Note that the forward direction defined for the hand 11 is indicated by arrow F in Figure 5. If the detected direction is rotated by 90 degrees, the robot 10 may operate to grasp the object 90a from an irrational direction, which may result in problems such as an increase in cycle time.

[0035] To avoid such problems, when the detected position and orientation output unit 124 according to this embodiment determines that the object is a square, it does not distinguish between the calculated central axis A1 in the long side direction and the central axis A2 in the short side direction, and outputs the direction of either of these central axes (here, the angle with the X-axis) that is greater than or equal to 0 degrees and less than 90 degrees as the detection direction. For example, when an object 90a is detected as in the image 212 shown in FIG. 6 , the detected position and orientation output unit 124 outputs the angle θ11 (0°≦θ11<90°) as the detection direction of the object 90, but does not output the angle θ12 (θ12≧90°) as the detection direction. This enables the robot 10 to always pick up objects in the correct orientation.

[0036] Below, two embodiments will be described regarding the object detection process executed by the image processing device 20. The embodiments described here are examples of detecting an object having a rectangular or square surface, such as a box.

[0037] First Example Fig. 7 is a flowchart showing an object detection process according to a first example. The detection process in Fig. 7 is executed under the control of the processor 21. When this process starts, the judgment criterion setting unit 125 first accepts the setting of a judgment criterion that the square judgment unit 131 uses to determine that a quadrangular shape detected by the quadrangular shape detection unit 130 is a square (step S11). The judgment criterion setting unit 125 may accept the setting of the judgment criterion via a user interface presented on the display unit 24.

[0038] An example of the judgment criteria set here will be described with reference to FIG. 9 . When detecting a rectangular object 90, the quadrilateral shape detection unit 130 detects a central axis A1 parallel to the longitudinal direction and a central axis A2 parallel to the lateral direction, as shown in the detected image 221 in FIG. 9 . The square determination unit 131 then determines that the quadrilateral shape is a square if the ratio of the lateral length to the longitudinal length of the quadrilateral shape is equal to or greater than a predetermined threshold (or within a predetermined range). In step S11, the threshold (or value range) for this case is set. For example, as shown in FIG. 9 , if a threshold value of 0.8 is set as the judgment criteria, the square determination unit 131 determines that a quadrilateral shape is a square if the ratio of the short side length to the long side length is equal to or greater than 0.8. For reference, FIG. 9 also shows an image 222 of an object 90b whose short side length to the long side length is detected as 0.8 and an image 223 of an object 90c whose short side length to the long side length is detected as 1. If the objects supplied to the robot system are expected to be square, the threshold value may be set to as large a value as possible, taking into account variations in the shapes of the objects, for example.

[0039] Next, the visual sensor 70 captures an image of the object (step S12).

[0040] Next, the quadrilateral shape detection unit 130 detects quadrilateral shapes from the image (step S13). Here, the quadrilateral shape detection unit 130 may apply the following detection algorithm: (1) extract a target figure by detecting edges or contours from the captured image, and (2) determine that the target figure is a quadrilateral (a rectangle) when the target figure satisfies the following conditions: the target figure is a quadrilateral having two pairs of parallel, equal-length opposite sides and all right angles.

[0041] Furthermore, the quadrangular shape detection unit 130 obtains the length of the longitudinal direction and the length of the lateral direction of the detected quadrangular shape (step S14).

[0042] Next, the square determination unit 131 determines whether the detected quadrangular shape is a square or not using the determination criteria set in step S1 (step S15). If it is not determined to be a square (S15: NO), the object is determined to be a rectangle, and the process proceeds to step S16.

[0043] In step S16, the detected position and orientation output unit 124 outputs the detected position and angle of the object. In this case, the detected position and orientation output unit 124 outputs the detected angle of the object (for example, the angle between the central axis in the longitudinal direction and the X-axis of the image) as an angle greater than or equal to 0 degrees and less than 180 degrees. The detection results including the detected position and detected angle are provided to the robot control device 50.

[0044] If it is determined in step S15 that the object is a square (S15: YES), the process proceeds to step S17. In step S17, the detected position and orientation output unit 124 outputs the detected position and detected angle of the object. In this case, the detected position and orientation output unit 124 outputs, as the detected angle of the object, the angle formed by either the central axis in the longitudinal direction or the central axis in the lateral direction and the X-axis of the image, which is greater than or equal to 0 degrees and less than 90 degrees. The detection results including the detected position and detected angle are provided to the robot control device 50.

[0045] This type of detection processing can avoid the above-mentioned problems when the object is rectangular or square, and can output a detected angle that allows the robot 10 to properly grasp the object with the hand 11.

[0046] FIG. 8 is a flowchart outlining a transfer process for transferring an object based on the detection results of the detection process in FIG. 7 . This transfer process is executed under the control of the processor of the robot control device 50. As shown in FIG. 8 , the robot 10 grasps the object in accordance with the taught position of the hand relative to the object (the taught position in this case also includes the orientation) using the detected position and detected angle obtained from the detected position and orientation output unit 124 (step S21). As described above, the detected angle output from the detected position and orientation output unit 124 is limited to an angle greater than or equal to 0 degrees and less than 180 degrees if the object is rectangular, and to an angle greater than or equal to 0 degrees and less than 90 degrees if the object is square. Therefore, in step S21, the robot 10 can grasp the object in an appropriate position and orientation.

[0047] Next, the robot 10 places the object in another location according to the instruction (operation program) (step S22).

[0048] According to the first embodiment described above, the detected angle of a rectangular (point-symmetric) object is output as an angle greater than or equal to 0 degrees and less than 180 degrees. Furthermore, the detected angle of a square object is output as an angle greater than or equal to 0 degrees and less than 90 degrees. Therefore, according to the first embodiment, the handling direction of the robot can be limited to the intended direction for both rectangular and square objects, thereby shortening the cycle time. Furthermore, according to the first embodiment, the handling path of these objects by the robot can be limited to the intended path, thereby reducing the man-hours required for starting up the robot system.

[0049] Furthermore, if an attempt were made to limit the direction of handling of a rectangular or square object by a robot without using the detection process for limiting the detection angle as in the first embodiment described above, it would be necessary to add conditional branching processing based on the detected angle to the transport program, which would complicate the contents of the transport program. In this regard, the detection process as in the first embodiment eliminates the need to perform conditional branching processing based on the detected angle on the transport program side.

[0050] Second Example In the detection process according to the first example, in order to detect a square object as having a detection direction greater than or equal to 0 degrees and less than 90 degrees, the two central axes of the rectangular shape, the longitudinal and lateral directions, are not distinguished as to whether they are longitudinal or lateral directions. The angle between either central axis and the reference direction and greater than or equal to 0 degrees and less than 90 degrees is output as the detection angle. When such square-related processing is employed, for example, if an object is determined to be square despite being rectangular, depending on how the object is captured in the image, a situation may arise in which the robot (hand) grasps the object in a position and orientation in which the longitudinal and lateral directions are reversed relative to the originally intended position and orientation of the object. For example, this situation may arise when the surface of the object is a rectangle that is relatively close to a square. This point will be described with reference to FIG. 10 .

[0051] Assume that the robot 10 is taught to grasp the object 90d (a box having rectangular sides) 90d with the hand 11 in a position where the longitudinal direction of the hand 11 coincides with the longitudinal direction of the object 90d, as shown on the left side of FIG. 10 (hereinafter, the position and posture of the hand 11 in this case will be referred to as a grasping position and posture 321). In FIG. 10, the long and short sides of the object are indicated by symbols d1 and d2, respectively. If the detection process of the first embodiment erroneously determines that the object 90d is a square, the long and short sides become indistinguishable from each other. As a result, a situation may arise in which the hand 11 grasps the object 90d in a relative relationship with the object 90d and the hand 11 as shown on the right side of FIG. 10, i.e., with the long and short sides of the object 90d coinciding with the longitudinal direction of the hand 11 (hereinafter, the position and posture of the hand 11 in this case will be referred to as a grasping position and posture 322). That is, a situation may arise in which the long and short sides of the hand 11 do not coincide with the long and short sides of the object 90d.

[0052] When such a situation occurs, for example, when it is desired to arrange objects with their longitudinal directions aligned or when it is desired to arrange objects using the length of their long or short sides, it may be impossible to arrange the objects as intended. Therefore, in the second embodiment, when an object is determined to be a square, the robot control device 50 is notified of the orientation in which the hand 11 is holding the object. This allows the robot to arrange the objects in the intended arrangement even in a situation where a rectangular object is mistakenly determined to be a square.

[0053] FIG. 11 is a flowchart showing the object detection process according to the second embodiment. The detection process in FIG. 11 is executed under the control of the processor 21. Note that the same steps in FIG. 11 as those in the detection process according to the first embodiment (FIG. 7) are given the same step numbers, and their explanations will be omitted. In other words, the detection process according to the second embodiment corresponds to the detection process according to the first embodiment plus step S31. Therefore, the contents of step S31 will be explained here.

[0054] In step S15, it is assumed that the object is actually rectangular, but is determined to be a square through calculations depending on the way it shifts in the image, etc. (step S15: YES). In this case, the process proceeds to step S17, and the detected position and orientation output unit 124 outputs, as the detected angle of the object, either the central axis in the longitudinal direction or the central axis in the lateral direction, an angle between the central axis and the X-axis of the image and that is greater than or equal to 0 degrees and less than 90 degrees. Then, in step S31, the detected position and orientation output unit 124 outputs information indicating whether the central axis used in calculating the detected angle is the central axis in the longitudinal direction or the central axis in the lateral direction. Note that in calculating the detected angle (step S17), a process is performed in which the central axis in the lateral direction and the central axis in the longitudinal direction are used, without distinguishing between them, but whether the central axis used in calculating the detected direction is the central axis in the longitudinal direction or the central axis in the lateral direction is determined by the detected position and orientation output unit 124. The information output in steps S17 and S31 is provided to the robot control device 50.

[0055] FIG. 12 is a flowchart showing an overview of a transfer process for transferring an object based on the detection results of the detection process of FIG. 11. This transfer process is executed under the control of the processor of the robot control device 50. Here, the flow of the transfer process will be described in the case where the object is determined to be a square and the above information is output in step S31 of FIG. 11. The situation in which the object is determined to be a square includes the case where the object is determined to be a square despite being rectangular. When the object is determined to be a rectangle, the transfer process of FIG. 8 described above can be applied.

[0056] 11 , the robot 10 grasps an object according to the taught position of the hand relative to the object (the taught position in this case also includes the posture) using the detected position and detected angle obtained from the detected position and posture output unit 124 (step S41). If the robot 10 is taught so that the longitudinal direction of the object coincides with the longitudinal direction of the hand 11, and the object is determined to be a square despite being rectangular, the robot 10 may grasp the object with the longitudinal direction and lateral direction of the object reversed from those of the grasp position and posture 321, as shown in grasp position and posture 322 in FIG.

[0057] The robot controller 50 acquires information (information output in step S31) indicating whether the central axis in the longitudinal direction or the central axis in the lateral direction was used to detect the orientation (angle) of the object (step S42). Based on the acquired information and the teachings, for example, the robot controller 50 can determine whether the longitudinal direction of the hand 11 currently coincides with the longitudinal direction or the lateral direction of the object. Therefore, the robot 10 can place the object so that the longitudinal direction or the lateral direction of the object is in the intended direction (step S43).

[0058] According to this transfer process, the robot 10 can determine whether the longitudinal direction of the hand 11 coincides with the longitudinal direction or the lateral direction of the object, even in a situation where the robot 10 determines that the object is a square despite being rectangular, and grasps the object with the longitudinal direction and lateral direction reversed from the taught position and posture, as in the grasping position and posture 322. Therefore, even in such a situation, the robot 10 can, for example, align and place the objects so that the longitudinal directions of the objects are aligned.

[0059] In the first and second examples described above, when the quadrilateral shape detection unit 130 detects a rectangular object, the detected position and orientation output unit 124 outputs the detected angle as an angle greater than or equal to 0 degrees and less than 180 degrees. A rectangle is a point-symmetric figure, and a point-symmetric figure is a figure that overlaps with the original figure when rotated 180 degrees (i.e., a figure with two rotational symmetries), so this processing corresponds to processing that outputs the detected angle as an angle greater than or equal to 0 degrees and less than 360 / n degrees (where n is the number of rotational symmetries) when the object is a point-symmetric figure.

[0060] The rotationally symmetric figure detection unit 123 can be configured to detect various rotationally symmetric figures in addition to the rectangles and squares described above. For example, the rotationally symmetric figure detection unit 123 can also be configured to detect objects with rotational symmetry, such as triangles, parallelograms, rhombuses, stars, and regular hexagons. Even in the case of these rotationally symmetric figures, the rotationally symmetric figure detection unit 123 detects the object and detects the number of rotational symmetries by pattern matching using model data. The detected position and orientation output unit 124 can output the detected angle as an angle between 0 degrees and less than 360 / n degrees, where n is the number of rotational symmetries in the rotationally symmetric figures. For example, if the object is a rhombuses, and the long diagonal or short diagonal of the rhombuses is used as a reference for detecting the orientation (angle) of the object, it can be understood from the above-described embodiment that limiting the detected angle to an angle between 0 degrees and less than 360 / n degrees (where n is the number of rotational symmetries) is useful for preventing the robot from picking up the object from an unreasonable direction. Similarly, for other rotationally symmetric figures, when an axis parallel to a specific side, a specific diagonal, or the like is used as a criterion for detecting the direction (angle) of the rotationally symmetric figure, it is useful to limit the detected angle to an angle greater than or equal to 0 degrees and less than 360 / n degrees (where n is the number of rotationally symmetric figures) and output it.

[0061] 13 is a flowchart showing the detection process performed by the image processing device 20 to detect an object as a rotationally symmetric figure. First, the visual sensor 70 captures an image of the object (step S101). The rotationally symmetric figure detection unit 123 detects the object as a rotationally symmetric figure as described above (step S102). The detected position and orientation output unit 124 then outputs an angle equal to or greater than 0 degrees and less than 360 / n degrees (where n is the number of rotationally symmetric figures) as the direction (detected angle) of the detected object (step S103).

[0062] As described above, according to this embodiment, it is possible to limit the detection angle of an object having rotational symmetry, thereby enabling the robot to appropriately handle the object.

[0063] In the above-described embodiment, an example was described in which the rotationally symmetric figure detection unit 123 detects an object using a pattern matching technique based on a two-dimensional image or a range image. Alternatively, the rotationally symmetric figure detection unit 123 may be configured to detect an object as a rotationally symmetric figure using a learning function. In this case, the rotationally symmetric figure detection unit 123 may include a learning unit that performs learning using a large number of training images (annotated images) in which images of the object are associated with information indicating the object's features (contour position, angle, etc.) in the images. The learning unit may perform learning using supervised learning, a type of machine learning, as an example. Deep learning techniques may also be incorporated into the learning. The learning unit inputs an image and trains a neural network (NN) or a convolutional neural network (CNN) as an estimator using training data in which the feature positions (position, angle, etc.) identified in the image are correct. This allows for the construction of a learning model for estimating feature positions in any image containing an object.

[0064] The functional layout in the functional block diagram shown in Figure 2 is an example, and various modifications are possible regarding the functional layout. For example, there may be a configuration example in which at least some of the functional blocks in the image processing device 20 are arranged in the robot control device 50 or the teaching device 40. Alternatively, the entire function of the image processing device 20 may be incorporated into the robot control device 50 or the teaching device 40. In other words, the image processing device may be provided as an independent device within the system, or its function may be incorporated into the control device or the teaching device.

[0065] The configurations of the above-described embodiments can be applied to systems of various types of industrial machines that handle objects.

[0066] The functional blocks shown in the functional block diagram of Figure 2 may be realized by one or more processors of the image processing device or robot control device executing various software stored in a storage device, or may be realized by a configuration mainly based on hardware such as an ASIC (Application Specific Integrated Circuit).

[0067] The programs that execute various processes such as the detection process and the transport process in the above-described embodiments can be recorded on various computer-readable recording media (e.g., semiconductor memories such as ROM, EEPROM, and flash memory, magnetic recording media, and optical discs such as CD-ROM and DVD-ROM).

[0068] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0069] The following supplementary notes are provided regarding the above-described embodiment and modified examples. (Supplementary Note 1) An image processing device (20) comprising: an image acquisition unit (122) that acquires image information of an object obtained by a visual sensor (70); a rotationally symmetric figure detection unit (123) that detects an object having a rotationally symmetric shape based on the acquired image information; and a detected angle output unit (124) that outputs the angle of the detected object with respect to a predetermined reference direction based on the acquired image information, wherein the detected angle output unit (124) outputs the angle of the detected object with respect to the predetermined reference direction as an angle greater than or equal to 0 degrees and less than (360 / n) degrees, where n is the number of rotational symmetries of the detected object. (Supplementary Note 2) The image processing device (20) according to Supplementary Note 1, wherein the rotationally symmetric figure detection unit (123) determines whether the detected object is a point-symmetric figure based on the image information, and the detected angle output unit outputs the angle of the object determined to be a point-symmetric figure with respect to the predetermined reference direction as an angle greater than or equal to 0 degrees and less than 180 degrees. (Supplementary Note 3) The image processing device (20) according to Supplementary Note 1 or 2, wherein the rotationally symmetric figure detection unit (123) comprises: a quadrilateral shape detection unit (130) that detects a rectangular object based on the acquired image information; and a square determination unit (131) that determines whether the detected rectangular object is a square based on a predetermined determination criterion, and the detected angle output unit outputs an angle of the object determined to be a square with respect to the predetermined reference direction as an angle greater than or equal to 0 degrees and less than 90 degrees. (Supplementary Note 4) The image processing device (20) according to Supplementary Note 3, wherein the quadrilateral shape detection unit (130) detects long sides and short sides of the rectangular object, and the predetermined determination criterion used by the square determination unit (131) is that the ratio between the lengths of the long sides and the short sides of the detected rectangular object is within a predetermined range. (Supplementary Note 5) The image processing device (20) according to Supplementary Note 4, wherein the detected angle output unit (124) outputs, for an object determined to be a square, an angle that is greater than or equal to 0 degrees and less than 90 degrees, either an angle that a first axis parallel to the detected long side makes with respect to the predetermined reference direction or an angle that a second axis parallel to the detected short side makes with respect to the predetermined reference direction.(Supplementary Note 6) The image processing device (20) according to Supplementary Note 5, wherein the detected angle output unit (124) further outputs information indicating which of the first axis and the second axis was used to detect the angle. (Supplementary Note 7) The image processing device (20) according to any one of Supplementary Notes 3 to 6, further comprising a judgment criterion setting unit (125) that receives an input for setting the predetermined judgment criterion used by the square judgment unit (131).

[0070] REFERENCE SIGNS LIST 10 Robot 11 Hand 20 Image processing device 21 Processor 22 Storage device 23 Operation unit 24 Display unit 40 Teaching device 41 Display unit 70 Visual sensor 90 Object 100 Robot system 121 Visual sensor control unit 122 Image acquisition unit 123 Rotational symmetry figure detection unit 124 Detected position and orientation output unit 125 Judgment criterion setting unit 130 Quadrilateral shape detection unit 131 Square judgment unit 151 Operation control unit

Claims

1. An image processing apparatus comprising: an image acquisition unit that acquires image information of an object obtained by a visual sensor; a rotationally symmetric figure detection unit that detects a rotationally symmetric object based on the acquired image information; and a detection angle output unit that outputs an angle of the detected object with respect to a predetermined reference direction based on the acquired image information, wherein when the number of rotational symmetries of the detected object is n, the detection angle output unit outputs the angle of the detected object with respect to the predetermined reference direction as an angle of 0 degrees or more and less than (360 / n) degrees.

2. The image processing apparatus according to claim 1, wherein the rotationally symmetric figure detection unit determines whether the detected object is a point-symmetric figure based on the image information, and the detection angle output unit outputs the angle of the object determined to be a point-symmetric figure with respect to the predetermined reference direction as an angle of 0 degrees or more and less than 180 degrees.

3. The rotationally symmetric figure detection unit of claim 1 or 2, further comprising: a rectangular shape detection unit that detects a rectangular object based on the acquired image information; and a square determination unit that determines whether the detected rectangular object is a square based on a predetermined determination criterion, wherein the detection angle output unit outputs the angle of the object determined to be a square with respect to the predetermined reference direction as an angle of 0 degrees or more and less than 90 degrees.

4. The rectangular shape detection unit detects the long side and the short side of the rectangular object, and the predetermined determination criterion used by the square determination unit is that the ratio between the length of the long side and the length of the short side of the detected rectangular object is within a predetermined range.

5. For an object determined to be a square, the detection angle output unit outputs an angle of 0 degrees or more and less than 90 degrees among the angles formed by a first axis parallel to the detected long side with respect to the predetermined reference direction, or an angle formed by a second axis parallel to the detected short side with respect to the predetermined reference direction.

6. The image processing apparatus according to claim 5, wherein the detection angle output unit further outputs information indicating which of the first axis and the second axis is used for detecting the angle.

7. The image processing apparatus according to any one of claims 3 to 6, further comprising a determination criterion setting unit that receives an input for setting the predetermined determination criterion used by the square determination unit.

Images