Object detection device and model pattern evaluation device

Abstract

The present invention provides an object detection device capable of detecting a symmetrical object represented by an image with high precision. The object detection device comprises: a memory (33) storing a model pattern (300) representing a plurality of predetermined features located at mutually different positions of the object when the object is observed from a predetermined direction; a feature extraction unit (41) extracting the plurality of predetermined features from the image representing the object; and a comparison unit (42) calculating a degree of consistency while changing at least one of the relative position or angle, relative orientation, and relative size of the model pattern (300) relative to the image. The degree of consistency represents the degree of consistency between the plurality of predetermined features of the model pattern (300) and the plurality of predetermined features extracted from the region corresponding to the model pattern (300) on the image. It is determined that the object is represented in an area on the image corresponding to the model pattern (300) when the degree of consistency is above a predetermined threshold value, the predetermined features stored in the storage unit (33) include a focus feature, and the focus feature can be used to detect the position of the object on the image in a specific direction, or can be used to detect the angle in the rotation direction with a predetermined point on the image as the rotation center. When the focus feature of the model pattern (300) is consistent with the predetermined feature on the image, the comparison unit (42) increases the contribution in the calculation of the consistency to calculate the consistency compared to a case where the predetermined features other than the focus feature of the model pattern (300) are consistent with the predetermined features on the image.

Description

Technical Field

[0001] The present invention relates to, for example, an object detection device that detects an object represented in an image, and a model pattern evaluation device that evaluates a model pattern used to detect the object. Background Art

[0002] In the past, a technology has been proposed for detecting an area representing an object from an image representing the object. In such a technology, in particular, in order to have a high degree of symmetry in shape, such as the wheel of an automobile, a technology has been proposed for highly accurately positioning an object having a shape whose position and posture cannot be accurately determined without paying attention to a specific part. For example, the positioning method described in Patent Document 1 sets a first area surrounding a standard pattern and a second area that gives features to the position and posture of the standard pattern in a standard image of a standard product serving as an inspection object. Moreover, in this positioning method, the features extracted from the first area are searched from the inspection object image to roughly determine the position and posture of the standard pattern in the inspection object image (first search process), and at least one of the roughly determined position and posture is carefully determined by searching the inspection object image for the features extracted from the second area (second search process).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-96749 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] However, in the technique described in Patent Document 1, the second area is set by the user operating the console. In this method, the second area representing a position or posture is set based on the user's subjective judgment. Therefore, an area that is not actually suitable for representing a position or posture may be set as the second area due to user error.

[0008] In one aspect, an object is to provide an object detection device capable of detecting a symmetrical object represented in an image with high accuracy.

[0009] Means for solving problems

[0010] According to one embodiment, an object detection device is provided. The object detection device includes: a storage unit storing a model pattern representing a plurality of predetermined features located at different positions on a target object when the target object is viewed from a predetermined direction; a feature extraction unit extracting the plurality of predetermined features from an image representing the target object; and a comparison unit calculating a degree of consistency while changing at least one of a relative position or angle, a relative orientation, and a relative size of the model pattern relative to the image. The degree of consistency indicates the degree of consistency between the plurality of predetermined features of the model pattern and a plurality of predetermined features extracted from a region on the image corresponding to the model pattern, wherein the region on the image corresponding to the model pattern is determined to represent the target object when the degree of consistency is equal to or greater than a predetermined threshold. The predetermined features stored in the storage unit include a focus feature that can be used to detect the position of the target object on the image in a specific direction or to detect an angle in a rotational direction about a predetermined point on the target object on the image. When the focus feature of the model pattern matches the predetermined feature on the image, the comparison unit increases the contribution of the model pattern to the degree of consistency calculation compared to when a predetermined feature other than the focus feature of the model pattern matches the predetermined feature on the image.

[0011] According to another embodiment, a model pattern evaluation device is provided. The model pattern evaluation device includes: an evaluation unit that calculates an evaluation value representing the geometric distribution of multiple features included in a model pattern used to detect an object from an image representing the object; and a focus feature setting unit that sets, based on the evaluation value, a first feature that can be used to detect the position of the object in a specific direction on the image, or a second feature that can be used to detect the angle in a rotation direction about a predetermined point on the object on the image as a focus feature.

[0012] Effects of the Invention

[0013] According to one aspect, a symmetrical object represented in an image can be detected with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic structural diagram of a robot system according to one embodiment.

[0015] Figure 2 This is a schematic diagram of the control device.

[0016] Figure 3A This is a diagram showing an example of the shape of a workpiece viewed from a predetermined direction.

[0017] Figure 3B This is a diagram showing an example of a workpiece model pattern.

[0018] Figure 4 This is a functional block diagram of a processor included in a control device, which is related to processing for evaluating a model pattern and movable part control processing including object detection processing.

[0019] Figure 5 It means evaluation Figure 3B Schematic diagram of the method's position detection capability of the model pattern.

[0020] Figure 6 This is a schematic diagram showing another example of a model pattern.

[0021] Figure 7 Yes Figure 6 Schematic diagram of the evaluation method of the angle detection capability of the model pattern shown.

[0022] Figure 8 This is a flowchart showing the overall flow of processing for evaluating position detection capability and angle detection capability and setting focus features based on the evaluation.

[0023] Figure 9 This is a flowchart showing a process in which the position detection capability evaluation unit evaluates the position detection capability of a model pattern.

[0024] Figure 10 3 is a flowchart showing a process in which the angle detection capability evaluation unit 38b evaluates the angle detection capability of the model pattern.

[0025] Figure 11A This is a schematic diagram illustrating the comparison between the model pattern and the image.

[0026] Figure 11B This is a schematic diagram illustrating the comparison between the model pattern and the image.

[0027] Figure 11C This is a schematic diagram illustrating the comparison between the model pattern and the image.

[0028] Figure 11D This is a schematic diagram illustrating the comparison between the model pattern and the image.

[0029] Figure 11E This is a schematic diagram illustrating the comparison between the model pattern and the image.

[0030] Figure 12 This is an operational flow chart of the movable part control process including the object detection process.

[0031] Figure 13 This is an operational flowchart of a movable part control process including an object detection process, and shows a process for calculating the degree of coincidence by giving a higher weight to a focus feature of a model pattern than to features other than the focus feature. DETAILED DESCRIPTION

[0032] The following describes a model pattern evaluation device and an object detection device according to embodiments of the present invention with reference to the accompanying drawings. The object detection device detects a target object (hereinafter referred to as the target object) from an image representing the target object, obtained by capturing the image with an imaging unit. The object detection device matches the image with a model pattern representing the target object's appearance when viewed from a predetermined direction while varying the relative positional relationship between the image and the model pattern. The device then calculates a degree of consistency, representing the degree of agreement between multiple features defined for the entire model pattern and multiple features extracted from a comparison region on the image corresponding to the model pattern.

[0033] For example, a feature point group extracted from a model image containing an object to be detected is stored as a model pattern. When the degree of consistency between the feature point group extracted from the image obtained by the shooting unit and the feature point group of the model pattern exceeds a threshold, it is determined that the detection of the object is successful.

[0034] When calculating the degree of consistency between features of a model pattern and features in an image representing an object, depending on the shape of the object to be detected using the model pattern, it may be difficult to align the position or angle (posture) of the model pattern relative to the object represented by the image. For example, when the object is shaped like an elongated rectangle, or when the object has a symmetry such as rotational symmetry, it may be difficult to align the position or angle of the model pattern with the object represented in the image. In the following description, the ability to detect the position of an object represented in an image is referred to as position detection capability, and the ability to detect the angle of an object represented in an image is referred to as angle detection capability.

[0035] The model pattern evaluation device of this embodiment calculates an evaluation value representing the geometric distribution of multiple features included in the model pattern, and based on the evaluation value, determines the direction in which the breadth of the feature position distribution is the largest as a specific direction with low position detection capability. In addition, based on the evaluation value, the model pattern evaluation device determines the distribution of features in a rotation direction with a predetermined point as the rotation center, and determines this rotation direction as a direction with low angle detection capability. The focus feature setting unit 37 extracts features distributed in directions different from these specific directions or rotation directions and sets them as focus features on the model pattern. By using the focus features included in the model pattern, the object detection device detects the position or angle of the target object with high precision from the image representing the target object.

[0036] The following describes an example in which a model pattern evaluation device and an object detection device are incorporated into a robot system. In this example, an imaging unit attached to a movable portion of the robot captures a workpiece, the target object, being the robot's workpiece, thereby generating an image representing the target object. Furthermore, a control device of the robot, incorporating the object detection device, detects the target object from the image and controls the movable portion based on the detection result.

[0037] Figure 1 This is a schematic diagram of a robot system 1 equipped with a model pattern evaluation device and an object detection device according to one embodiment. Robot system 1 includes a robot 2; a control device 3 that controls robot 2; and a camera 4 attached to a movable portion of robot 2 for imaging a workpiece 10, an example of a target object. Robot system 1 is an example of an automatic machine.

[0038] The robot 2 includes a base 11, a rotating table 12, a first arm 13, a second arm 14, a wrist 15, and a tool 16. The rotating table 12, the first arm 13, the second arm 14, the wrist 15, and the tool 16 are each an example of a movable portion. The rotating table 12, the first arm 13, the second arm 14, and the wrist 15 are each supported by shafts provided on joints to which they are attached, and are moved by servo motors driving these shafts. The workpiece 10 is transported, for example, by a belt conveyor. While the workpiece 10 is within a predetermined range, the robot 2 performs a predetermined operation on the workpiece 10.

[0039] The base 11 serves as a foundation when the robot 2 is installed on the ground. The rotating table 12 is mounted on the top surface of the base 11 via a joint 21 so as to be rotatable about an axis (not shown) provided perpendicular to one surface of the base 11.

[0040] One end of the first arm 13 is mounted on the rotating table 12 via a joint 22 provided on the rotating table 12. Figure 1 As shown, the first arm 13 is rotatable via a joint 22 about an axis (not shown) provided parallel to the surface of the base 11 on which the rotating table 12 is mounted.

[0041] One end of the second arm 14 is attached to the first arm 13 via a joint 23 provided on the other end of the first arm 13 on the opposite side of the joint 22. Figure 1 As shown, the second arm 14 is rotatable via a joint 23 about an axis (not shown) provided parallel to the surface of the base 11 on which the rotating table 12 is mounted.

[0042] The wrist 15 is attached to the distal end of the second arm 14 on the opposite side of the joint 23 via the joint 24. The wrist 15 has the joint 25 and is bendable at the joint 25 about an axis (not shown) parallel to the axes of the joint 22 and the joint 23. Furthermore, the wrist 15 is also rotatable in a plane perpendicular to the longitudinal direction of the second arm 14 about an axis (not shown) parallel to the longitudinal direction of the second arm 14.

[0043] The tool 16 is attached to the front end of the wrist 15 on the side opposite the joint 24. The tool 16 includes a mechanism or device for performing operations on the workpiece 10. For example, the tool 16 may include a laser for machining the workpiece 10 or a servo gun for welding the workpiece 10. Alternatively, the tool 16 may include a hand mechanism for gripping the workpiece 10 or a component to be assembled with the workpiece 10.

[0044] The camera 4 is an example of an imaging unit and is, for example, mounted on the tool 16. Alternatively, the camera 4 may be mounted on another movable unit, such as the wrist 15 or the second arm 14. When the robot 2 performs work on the workpiece 10, the camera 4 faces the workpiece 10 so that the workpiece 10 is included within the imaging range of the camera 4. Furthermore, the camera 4 captures the imaging range including the workpiece 10 at each predetermined imaging cycle, thereby generating an image representing the workpiece 10. Each time the camera 4 generates an image, it outputs the generated image to the control device 3 via the communication line 5.

[0045] The control device 3 is connected to the robot 2 via a communication line 5 and receives information indicating the operating status of the servo motors driving the axes of the joints of the robot 2, images from the camera 4, and the like from the robot 2 via the communication line 5. The control device 3 then controls the servo motors based on the received information and images, as well as the robot 2's operating instructions received from a higher-level control device (not shown) or preset, thereby controlling the position and posture of the robot 2's movable parts.

[0046] Figure 2 3 is a schematic diagram of the control device 3. The control device 3 includes a communication interface 31, a drive circuit 32, a memory 33, and a processor 34. The control device 3 may also include a user interface such as a touch panel (not shown).

[0047] The communication interface 31 includes, for example, a communication interface for connecting the control device 3 to the communication line 5 and a circuit for executing processing related to the transmission and reception of signals via the communication line 5. Furthermore, the communication interface 31 receives information indicating the operating status of the servo motor 35, such as a measured value of the rotation amount from an encoder for detecting the rotation amount of the servo motor 35 as an example of a driving unit, from the robot 2 via the communication line 5, and transmits the information to the processor 34. Figure 2 , a single servo motor 35 is shown as a representative example, but the robot 2 may also include a servo motor for each joint to drive the axis of the joint.

[0048] The drive circuit 32 is connected to the servo motor 35 via a current supply cable, and supplies power corresponding to the torque, rotation direction, or rotation speed generated in the servo motor 35 to the servo motor 35 under the control of the processor 34 .

[0049] The memory 33 is an example of a storage unit, and includes, for example, a readable and writable semiconductor memory and a read-only semiconductor memory. The memory 33 may include a storage medium such as a semiconductor memory card, a hard disk, or an optical storage medium, and a device for accessing the storage medium.

[0050] The memory 33 stores various computer programs for controlling the robot 2, which are executed by the processor 34 of the control device 3. The memory 33 also stores information used to control the movement of the robot 2 during operation. Furthermore, the memory 33 stores information indicating the operating status of the servo motor 35 obtained from the robot 2 during its operation. Furthermore, the memory 33 stores various data used in object detection processing. This data includes, for example, a model pattern of the workpiece 10 used for detection of the workpiece 10, camera parameters representing information related to the camera 4, such as the focal length, mounting position, and orientation of the camera 4, and images obtained from the camera 4.

[0051] In this embodiment, the model pattern shows a plurality of predetermined features of the workpiece 10 at different positions when the workpiece 10 is viewed from a predetermined direction (e.g., vertically above). In addition, the workpiece 10 may have a symmetrical shape when viewed from the predetermined direction, such as a linearly symmetrical or rotationally symmetrical shape.

[0052] Figure 3A is a diagram showing an example of the shape of the workpiece 10 viewed from a predetermined direction. Figure 3B 1 is a diagram showing an example of a model pattern of a workpiece 10. Figure 3A As shown in FIG. 1 , in this example, the workpiece 10 has an elongated rectangular outline when viewed from a predetermined direction, and has a shape in which a protrusion 10a is provided in the center in the longitudinal direction. Figure 3B As shown, in the model pattern 300 , a plurality of points 301 on the contour of the workpiece 10 are respectively set as features.

[0053] These multiple features 301 are obtained, for example, by processing a reference image obtained by photographing the workpiece 10 from a predetermined direction using a feature extraction unit 38 of a processor 34, described later. The model pattern 300 is represented, for example, by a binary image in which pixels containing each feature 301 have different values ​​from pixels containing other features. Images such as standardized correlation images themselves can also be used as model patterns (templates). Furthermore, to facilitate processing of the model pattern 300, it is preferable to appropriately transform the image coordinate system in the reference image into a model pattern coordinate system that extracts only the region surrounding the model pattern 300 in the reference image.

[0054] In addition to obtaining features 301 from an image obtained by photographing the workpiece 10, features 301 can also be obtained by converting the three-dimensional data of the workpiece 10 into two-dimensional data. In this case, for example, a three-dimensional local coordinate system is defined with its origin on the imaging plane of camera 4. The target object represented by CAD data is virtually arranged in this local coordinate system. A three-dimensional point group is then set at predetermined intervals on the contour lines of the CAD data. If necessary, the contour lines used as the model pattern 300 can also be specified from the CAD data. The three-dimensional point group represented by the CAD data is then projected onto the imaging plane of camera 4 to obtain a two-dimensional point group in the image coordinate system. If the direction of brightness and darkness is specified in the CAD data, the direction of the brightness gradient can also be added to the two-dimensional point group in the image coordinate system. The direction of brightness and darkness refers to the direction from a light area to a dark area, or vice versa, with the contour line as the boundary. When the model pattern 300 is obtained by photographing the target object, the brightness information of the pixels includes the direction of brightness and darkness. When projecting the three-dimensional point group represented by CAD data onto the shooting surface of the camera 4, since the position of the two-dimensional point group on the image coordinate system includes the influence of the aberrations of the camera 4 such as field curvature, it is preferable to obtain the aberration characteristics of the camera 4 in advance in a manner that reflects these aberrations on the position of the two-dimensional point group and perform corresponding calibration.

[0055] The processor 34 is an example of a control unit and includes, for example, a CPU (Central Processing Unit) and its peripheral circuits. Furthermore, the processor 34 may also include a processor for numerical calculations. Furthermore, the processor 34 controls the entire robot system 1. Furthermore, the processor 34 executes processing for evaluating the model pattern and controlling the movable part, including object detection.

[0056] Figure 4This is a functional block diagram of the processor 34 related to the processing for evaluating the model pattern and the movable unit control processing, including object detection. The processor 34 includes an evaluation unit 36, a focus feature setting unit 37, a feature extraction unit 38, a comparison unit 39, and a movable unit control unit 40. These components of the processor 34 are, for example, functional modules implemented by a computer program executed on the processor 34. Alternatively, these components may be implemented as dedicated arithmetic circuits incorporated into the processor 34.

[0057] Among these components of processor 34, the processing performed by evaluation unit 36 ​​and focus feature setting unit 37 corresponds to model pattern evaluation processing, while the processing performed by feature extraction unit 38 and comparison unit 39 corresponds to object detection processing. In the following description, the model pattern evaluation processing performed by evaluation unit 36 ​​and focus feature setting unit 37 will be described first, followed by the object detection processing performed by feature extraction unit 38 and comparison unit 39.

[0058] When calculating the degree of consistency between the features of the model pattern and the features on the object image, if some features of the model pattern are not taken into account, the position or angle of the model pattern relative to the object represented by the object image may not be correctly determined. Figure 3A 、 Figure 3B In the shape shown, there is a protrusion 10a on the contour of the workpiece 10. Figure 3A 、 Figure 3B In the case of the shape shown in FIG, if the features are matched between the model pattern 300 and the object image, even if Figure 3B In the region 310 surrounded by the dotted line, the model pattern and the target image do not match, but the degree of match may be recognized as high as the pattern as a whole. In such a case, the position and angle (posture) of the model pattern relative to the target image may be erroneously detected.

[0059] Therefore, in this embodiment, the evaluation unit 36 ​​calculates an evaluation value representing the geometric distribution of multiple features included in the model pattern. Based on the evaluation value, the evaluation unit 36 ​​identifies the direction in which the breadth of the feature position distribution is the largest as a specific direction with low position detection capability. Furthermore, based on the evaluation value, the evaluation unit 36 ​​identifies at least a portion of the multiple features distributed along a rotational direction centered around a predetermined point as a direction with low angle detection capability. The feature-of-interest setting unit 37 sets features distributed in directions different from these specific directions or rotational directions as features of interest. Thus, the model pattern represents the features of the workpiece 10 and also represents features of interest with high position or angle detection capability.

[0060] Specifically, using the algorithm described below, the evaluation unit 36 ​​geometrically evaluates the position detection capability or angle detection capability of the features represented by the model pattern. The feature-of-interest setting unit 37 extracts features with high position detection capability or angle detection capability and sets them as the feature-of-interest for the model pattern. Position detection capability is evaluated by the position detection capability evaluation unit 36a of the evaluation unit 36, while angle detection capability is evaluated by the angle detection capability evaluation unit 36b of the evaluation unit 36. While the evaluation unit 36 ​​performs both position and angle detection capability evaluations in parallel, it is also possible to evaluate only one of these capabilities.

[0061] First, the processing performed by the position detection capability evaluation unit 36a will be described. In the following, the case where each feature is a point on the outline of the model pattern, that is, the case where each feature is a feature point, will be described as an example.

[0062] Reference Figure 3A The workpiece 10 has a simple rectangular shape except for the protrusion 10a, and lacks shape changes along the longitudinal direction of the rectangle. Therefore, it is difficult to detect the position of the workpiece 10 in the longitudinal direction in the image representing the workpiece 10. Therefore, the position detection capability evaluation unit 36a Figure 3B In the model pattern 300 shown, features distributed in a direction different from the longitudinal direction are set in the model pattern 300 as features of interest.

[0063] The position detection capability evaluation unit 36a calculates the position detection capability of the model pattern 300 based on the edge angle distribution of the contour of the model pattern 300. If the angle distribution is uniform in two orthogonal directions, the position detection capability evaluation unit 36a determines that the position detection capability of the model pattern 300 is high. If the edge angle distribution is biased in one direction, the position detection capability of the model pattern 300 is determined to be low.

[0064] Figure 5 It means evaluation Figure 3B The position detection capability evaluation unit 36a calculates the angle θ representing the normal vector that passes through each feature point P_i set in the model pattern 300 and is orthogonal to the direction in which the adjacent feature points are arranged (the direction of the contour line of the model pattern 300). iThe covariance matrix Q of the distribution is then calculated. Then, the two eigenvalues ​​of the covariance matrix Q and the eigenvectors corresponding to these eigenvalues ​​are calculated. In addition, various feature points can be used as feature points, but in this embodiment, edge points on the contour of the model pattern 300 obtained by edge detection are used as feature points. Edge points are, for example, points obtained as points with a large brightness gradient in an image and are points on the contour of an object obtained based on the brightness gradient. As a method for detecting edge points, a well-known method is applied. In addition, a normal vector is used here, but a tangent vector can also be used.

[0065] Specifically, if Figure 5 As shown, the normal vector of each of the N feature points P_i is defined as follows.

[0066] [Formula 1]

[0067]

[0068] If the angle of each normal vector is set to θ i , the normal vector at the feature point P_i is expressed as follows.

[0069] [Formula 2]

[0070]

[0071] The angle θ of the normal vector i (Edge angle) itself is periodic and difficult to be treated as an evaluation object. Therefore, the data input to the evaluation unit 36 ​​as the evaluation object is not directly used as the angle θ i , but use the normal vector (cosθ as above i , sinθ i ). In addition, in order to similarly evaluate input data having different 180° directions and to avoid difficulty in evaluating the magnitude of the eigenvalue due to the center of gravity shift of the model pattern 300, it is preferable to input two vectors, (cosθ, sinθ) and (-cosθ, -sinθ), as evaluation objects for one feature point into the evaluation unit 36.

[0072] In addition, the average of N normal vectors is expressed by the following formula (1).

[0073] [Formula 3]

[0074]

[0075] Furthermore, the covariance matrix Q of the normal vector is expressed by the following equation (2).

[0076] [Formula 4]

[0077]

[0078] Furthermore, if the intrinsic value of the covariance matrix Q is set to λ j , set the eigenvector to e j , then the following formula (3) holds.

[0079] [Formula 5]

[0080]

[0081] When a set of one-dimensional data is considered as the input data to be evaluated, the covariance matrix Q corresponds to the variance of the input data. If the model pattern 300 is considered as a set of two-dimensional coordinate values ​​(x, y), the eigenvector e of the covariance matrix Q of the normal vector is i and the intrinsic value λ i Equivalent to the variance of the distribution of the normal vector, it represents the direction of the breadth of the distribution of the normal vector and the size of the breadth (variance). In this embodiment, the covariance matrix Q is obtained based on the 2-dimensional value, so the number of eigenvectors and eigenvalues ​​is 2 respectively. Moreover, the larger eigenvalue and the eigenvector corresponding to the larger eigenvalue represent the size of the breadth of the distribution of the normal vector and the direction in which the breadth of the distribution is the largest. In addition, the small eigenvalue and the eigenvector corresponding to the small eigenvalue represent the size of the breadth of the distribution in the direction orthogonal to the eigenvector corresponding to the large eigenvalue and its direction. In addition, the direction in which the breadth of the position distribution of the feature is the largest becomes the direction of the eigenvector corresponding to the large eigenvalue.

[0082] Therefore, the two eigenvalues ​​are approximately equal in size. The closer the ratio of the two eigenvalues ​​is to 1, the more stably the normal vectors of the feature points constituting the model pattern 300 are distributed in the directions of the two orthogonal eigenvectors, and the higher the position detection capability of the model pattern 300 is.

[0083] On the other hand, when the two eigenvalues ​​deviate in magnitude, and the ratio of the smaller eigenvalue to the larger eigenvalue approaches zero, the normal vectors of the feature points constituting model pattern 300 are distributed skewed toward the direction of the eigenvector corresponding to the smaller eigenvalue. In this case, positional alignment with the direction of the eigenvector corresponding to the larger eigenvalue, i.e., the direction with the widest feature position distribution, becomes difficult, and the position detection capability of model pattern 300 decreases. As described above, principal component analysis using normal vector values ​​calculates the ratio of the principal component of the normal vector to the direction orthogonal to the principal component, thereby calculating the position detection capability of model pattern 300.

[0084] Therefore, the ratio obtained by dividing the small eigenvalue (min(λ1, λ2)) by the large eigenvalue (max(λ1, λ2)) is defined as the evaluation value A1 of the position detection capability of the model pattern 300. The evaluation value A1 is defined by the following formula (4).

[0085] A1=min(λ1, λ2) / max(λ1, λ2)…(4)

[0086] The evaluation value A1 takes a value from 0 to 1.0. The closer the evaluation value A1 is to 1, the greater the angle θ of the normal vector. i The more stable the distribution of i The smaller the deviation, the higher the position detection capability of the model pattern 300. For example, when the model pattern 300 is a perfect circle, the value of the evaluation value A1 is 1, and the position detection capability is the highest. In addition, when the model pattern 300 is an ellipse, the direction of the major axis of the ellipse is the direction of the eigenvector corresponding to the larger eigenvalue of the covariance matrix Q of the normal vector, and the direction of the minor axis is the direction of the eigenvector corresponding to the smaller eigenvalue. The longer the major axis is relative to the minor axis, the more the distribution of the normal vector is biased towards the direction of the minor axis, so the position detection capability of the model pattern 300 in the direction of the major axis becomes lower. In addition, when the model pattern 300 is a straight line, the value of the evaluation value A1 is 0, and the position detection capability is the lowest. Therefore, the evaluation value A1 represents the geometric distribution of multiple features contained in the model pattern. In addition, the covariance matrix Q, the eigenvalue, and the eigenvector are also evaluation values ​​representing the geometric distribution of multiple features contained in the model pattern.

[0087] The position detection capability evaluation unit 36a compares the evaluation value A1 with a preset threshold value. If the evaluation value A1 is less than the preset threshold value, the unit determines that the position detection capability of the model pattern 300 is low, that is, the model pattern 300 does not have position detection capability. The position detection capability evaluation unit 36a then determines that the direction of the eigenvector corresponding to the larger eigenvalue is a direction that does not have position determination capability.

[0088] exist Figure 5 In the example shown, since the normal vectors of most feature points P_i are Figure 5 Since the normal vectors are oriented in the direction of arrow D1, the breadth of their distribution is smallest in the direction of arrow D1. On the other hand, the normal vectors are oriented in the direction orthogonal to the direction of arrow D1. Therefore, the direction of the eigenvector corresponding to the smaller eigenvalue of the two eigenvectors is approximately in the direction of D1. Furthermore, the direction of the eigenvector corresponding to the larger eigenvalue of the two eigenvectors is approximately orthogonal to the direction of D1. Because the eigenvector in the direction orthogonal to the direction of arrow D1 has a larger eigenvalue, the position detection capability evaluation unit 36a determines that the position detection capability in the direction orthogonal to the direction of arrow D1 is low.

[0089] Next, the processing performed by the angle detection capability evaluation unit 36b will be described. Figure 63 is a schematic diagram showing another example of the mold pattern 300. Here, the mold pattern 300 corresponding to the workpiece 10 having a line-symmetrical shape, whose outline is substantially circular when viewed from a predetermined direction and whose part is a concave notch.

[0090] The workpiece 10 is, for example, a round bar having an end face provided with a key groove 10b when viewed from the axial direction. The workpiece 10 has a simple circular shape except for the key groove 10b, and lacks shape changes in the rotation direction with the center of the circle as the rotation center. Therefore, it is difficult to detect the angle of the workpiece 10 in the rotation direction in the image representing the workpiece 10. Therefore, the angle detection capability evaluation unit 36b is Figure 6 In the model pattern 300 shown, features distributed in a direction different from the rotation direction are set in the model pattern 300 as features of interest.

[0091] Figure 7 Yes Figure 6 Schematic diagram of the evaluation method of the angle detection capability of the model pattern 300 shown. Figure 7 The rectangular area R shown in the figure is used to evaluate the normal of the feature points in the rectangular area R. Figure 6 The angle detection capability of the model pattern 300 is shown.

[0092] The angle detection capability evaluation unit 36b first obtains a normal vector that passes through each feature point P_i constituting the model pattern 300 and is orthogonal to the direction in which the adjacent feature points are arranged (the direction of the contour line of the model pattern 300), and sets a normal line L extending in the direction of the normal vector. Figure 7 In the figure, only some of the feature points are shown with normal vectors ( Figure 7 However, the normal vector and normal L are set for all feature points within the rectangular area R.

[0093] The angle detection capability evaluation unit 36b sets the point where the most lines intersect within the rectangular area R as the rotation center O. When the normals of multiple feature points intersect at the rotation center O, these feature points are points on the circumference of the circle centered on the rotation center O, and are feature points with low angle detection capability in the direction of rotation with the rotation center O as the rotation center. On the other hand, feature points whose normals L do not pass through the rotation center O are not points on the circumference of the circle centered on the rotation center O, but are feature points with high angle detection capability in the direction of rotation with the rotation center O as the rotation center. Figure 7 In the example, since the normal line L does not pass through the rotation center O, the feature point P_c on the contour line corresponding to the keyway 10b has a high angle detection capability. On the other hand, since the normal line L passes through the rotation center O, the feature point P_r on the circumference corresponding to the outer peripheral surface of the workpiece 10 has a low angle detection capability.

[0094] Therefore, whether the angle detection capability of the model pattern 300 is low can be determined based on the number of feature points, such as feature point P_r, whose normal lines L pass through the rotation center O. The greater the number of feature points whose normal lines L pass through the rotation center O, the closer the shape of the outline of the model pattern 300 (the shape of the feature point distribution) becomes to a circle, and the lower the angle detection capability of the model pattern 300. Therefore, the angle detection capability evaluation unit 36b calculates the ratio of the number of normal lines L intersecting at the rotation center O to the total number of feature points. The angle detection capability evaluation unit 36b uses this ratio as the evaluation value A2. If the evaluation value A2 is greater than a preset threshold, the angle detection capability of the model pattern 300 is determined to be low, that is, the model pattern 300 does not have angle detection capability.

[0095] In addition, if Figure 7 As shown, if the outline of model pattern 300 includes a perfect circle, theoretically, multiple normal lines L intersect at a single point. However, if errors are included in the positions of feature points, the normal lines L may not intersect at a single point. Furthermore, if the outline of model pattern 300 is not a perfect circle or has slight irregularities, the normal lines L may not intersect at a single point. Therefore, the angle detection capability evaluation unit 36b may cluster the multiple intersection points within rectangular region R based on their density, determine the centroid of the cluster with the highest concentration of intersection points, and define a region R1 at a predetermined distance from this centroid as the center of rotation.

[0096] Evaluation value A2 takes a value between 0 and 1.0. The closer evaluation value A2 is to 0, the lower the ratio of the number of normal lines L intersecting at the rotation center O to the total number of feature points. The feature points are not distributed along the circumference, thus indicating that model pattern 300 is not rotationally symmetrical, and the angle detection capability of model pattern 300 is higher. Conversely, the closer evaluation value A2 is to 1, the higher the ratio of the number of normal lines L intersecting at the rotation center O to the total number of feature points. The feature points are distributed along the circumference, thus reducing the angle detection capability of model pattern 300. For example, if the normal lines L of all feature points in model pattern 300 intersect at a single point, the evaluation value is 1. In this case, model pattern 300 is a perfect circle, and the angle detection capability of model pattern 300 is the lowest. Therefore, evaluation value A2 represents the geometric distribution of the multiple features included in the model pattern.

[0097] As described above, the angle detection capability evaluation unit 36b evaluates the geometric distribution of the features of the model pattern 300 based on the evaluation value A2. When the evaluation value A2 is larger than the threshold, it can be determined that at least some of the features are distributed along the rotation direction around the rotation center O.

[0098] As described above, the position and angle detection capabilities of the model pattern 300 are evaluated. If the evaluation values ​​A1 and A2 determine that the position and angle detection capabilities of the model pattern 300 are low, the focus feature setting unit 37 sets features distributed in a direction different from the direction in which the position or angle detection capabilities are low as focus features. More specifically, the focus feature setting unit 37 sets features on the contour of the model pattern 300 in a direction different from the direction in which the position or angle detection capabilities are low as focus features.

[0099] exist Figure 5 In the example, for example, a feature point P_c on the contour line of the model pattern 300 corresponding to the protrusion 10a is set as a focus feature. Preferably, a feature on the contour of the model pattern 300 in a direction perpendicular to the direction with low position detection capability is set as a focus feature. Alternatively, a feature on the contour of the model pattern 300 in a direction within a predetermined angular range relative to the direction perpendicular to the direction with low position detection capability may be set as a focus feature. For example, a feature on the contour of the model pattern 300 in a direction within a range of ±30° relative to the direction perpendicular to the direction with low position detection capability may be set as a focus feature.

[0100] In addition, Figure 7 In the example, for example, a feature point P_c on the contour line of the model pattern 300 corresponding to the keyway 10b is set as a feature of interest. Preferably, a feature on the contour of the model pattern 300 in a direction orthogonal to the direction with low angle detection capability is set as a feature of interest. In addition, a feature on the contour of the model pattern 300 in a direction within a predetermined angular range relative to the direction orthogonal to the direction with low angle detection capability may be set as a feature of interest. For example, a feature on the contour of the model pattern 300 in a direction within a range of ±30° relative to the direction orthogonal to the direction with low angle detection capability may be set as a feature of interest. In addition, a feature point P_i whose normal line L does not pass through the rotation center O may be set as a feature of interest.

[0101] The features of interest can be Figure 5 as well as Figure 7The feature setting unit 37 may be set to the feature point itself, as in the feature point P_c shown in FIG. 1 , or to an area that is a collection of feature points. For example, the feature setting unit 37 may extract a plurality of feature points distributed in a direction different from the direction in which the position detection capability or angle detection capability is low, and set the area on the model pattern containing these plurality of feature points as the feature of interest. In addition, the feature setting unit 37 may extract a plurality of feature points distributed in a direction different from the direction in which the position detection capability or angle detection capability is low, calculate the center of gravity based on the positions of the plurality of extracted feature points, cluster the feature points located within a predetermined range from the center of gravity, and thereby set the feature of interest as a collection of the plurality of feature points.

[0102] As described above, the feature of interest is set in the model pattern as a feature having a position detection capability or an angle detection capability. The model pattern in which the feature of interest is set is stored in the memory 33 .

[0103] As described above, by geometrically evaluating the model pattern, it is determined that the direction in which the position or angle is difficult to detect relative to the object represented by the object image is determined, and the focus feature that can stably detect the position and angle is set. As a result, it is not necessary for a person to visually judge the model pattern to determine the direction in which the position detection capability or angle detection capability is low, and it is possible to suppress the selection of features on the model pattern that should not be focused on due to careless mistakes of a person. In particular, if the model pattern becomes complex, it is difficult for a person to visually determine the features that should be focused on. According to this embodiment, the geometric distribution of multiple features contained in the model pattern is evaluated by a model pattern evaluation device, so even if the model pattern has a complex shape, it is possible to reliably set a focus feature that can stably detect the position and angle of the object represented by the object image.

[0104] Then, based on Figures 8 to 10 , the processing performed by the evaluation unit 36 ​​and the focus feature setting unit 37 of the processor 34 of the control device 3 is described. Figure 8 This is a flowchart showing the overall flow of processing for evaluating position detection capability and angle detection capability and setting focus features based on the evaluation.

[0105] First, the position detection capability evaluation unit 36a evaluates the position detection capability of the model pattern (step S100) and determines whether the position detection capability of the model pattern is low (step S102). In addition, as described above, the position detection capability evaluation unit 36a compares the evaluation value A1 with a pre-set threshold value, and when the evaluation value A1 is a value smaller than the pre-set threshold value, it is determined that the position detection capability of the model pattern is low. Then, when the position detection capability of the model pattern is low, the focus feature setting unit 37 sets the features distributed in a direction different from the direction of low position detection capability as focus features based on the direction of low position detection capability (step S104). In addition, when the position detection capability of the model pattern is high, step S104 is not processed and step S106 is entered.

[0106] Next, the angle detection capability evaluation unit 36b evaluates the angle detection capability of the model pattern (step S106) and determines whether the angle detection capability of the model pattern is low (step S108). In addition, as described above, the angle detection capability evaluation unit 36b compares the evaluation value A2 with a pre-set threshold value, and when the evaluation value A2 is a value larger than the pre-set threshold value, it is determined that the angle detection capability of the model pattern is low. Then, when the angle detection capability of the model pattern is low, the focus feature setting unit 37 sets the feature distributed in a direction different from the direction of low angle detection capability as the focus feature based on the direction of low angle detection capability (step S110). The processing ends after step S110. In addition, when the angle detection capability of the model pattern is high, the processing of step S108 is not performed and the processing ends.

[0107] Figure 9 This is a process in which the position detection capability evaluation unit 36a evaluates the position detection capability of the model pattern ( Figure 8 Flowchart of step S100). First, calculate the covariance matrix Q of the normal vectors of the feature point group constituting the model pattern (step S200). Then, compare the sizes of the two eigenvalues ​​λ1 and λ2 of the covariance matrix Q (step S202). Next, determine whether the evaluation value A1 (min(λ1, λ2) / max(λ1, λ2)) as the ratio of the eigenvalues ​​λ1 and λ2 is less than the set threshold (step S204). When the evaluation value A1 is less than the threshold, it is determined that the position detection capability of the model pattern is low (step S206). The processing ends after step S206. In addition, when the evaluation value A1 is above the threshold in step S204, the position detection capability of the model pattern is high, so the processing of step S206 is not performed and the processing ends.

[0108] Figure 10 This is a process in which the angle detection capability evaluation unit 36b evaluates the angle detection capability of the model pattern ( Figure 8First, as shown in step S106 of FIG. Figure 7 As shown, a rectangular region R surrounding the model pattern is set (step S300). Next, normal lines are set for the contours of each feature point that constitutes the model pattern (step S302). Next, the point where the most normal lines intersect is set as the rotation center (step S304).

[0109] Next, a determination is made as to whether the ratio of the number of normal lines passing through the center of rotation to the total number of feature points is greater than a set threshold (step S306). If the condition in step S306 is met, the process proceeds to step S308, where it is determined that the model pattern's angular position detection capability is low. The process ends after step S308. If the condition in step S306 is not met, the model pattern's angular position detection capability is high, and the process ends without performing step S308.

[0110] Next, a description will be given of a process in which the feature extraction unit 38 and the matching unit 39 calculate the degree of coincidence between the features of the model pattern and the features on the target image representing the target object using the model pattern in which the focus features are set as described above.

[0111] The feature extraction unit 38 extracts a plurality of features of the same type as those represented by the model pattern from a series of time-series images obtained by the camera 4. The feature extraction unit 38 only needs to perform the same processing on each image, so the processing for one image will be described below.

[0112] When the features are characteristic points located on the contour of the workpiece 10, the feature extraction unit 38 applies an edge detection filter, such as a Sobel filter, to each pixel in the image, extracting pixels with an edge strength greater than a predetermined value as features. Alternatively, the feature extraction unit 38 may extract pixels representing corners detected by applying a Harris corner detection filter to the image as features. Alternatively, the feature extraction unit 38 may extract pixels detected by applying a SIFT algorithm to the image as features.

[0113] For each image, the feature extraction unit 38 notifies the comparison unit 39 of the position of each feature extracted from the image. For example, the feature extraction unit 38 generates a binary image as data representing the position of each feature, in which pixels representing the extracted feature have different values ​​from other pixels, and passes this binary image to the comparison unit 39.

[0114] The comparison unit 39 compares the features extracted from each of the time-series images obtained by the camera 4 with the model pattern, thereby detecting the workpiece 10 from the image. The comparison unit 39 only needs to perform the same processing on each image, so the processing for one image will be described below.

[0115] For example, the comparison unit 39 reads the model pattern from the memory 33 and compares the model pattern with the image of interest, thereby detecting an area representing the workpiece 10 on the image. For example, the comparison unit 39 sets multiple comparison areas on the image that are compared with the model pattern by changing the relative positional relationship of the model pattern with respect to the image. In addition, the change in relative positional relationship is performed, for example, by changing at least one of the relative position (or angle) of the model pattern with respect to the image, the relative orientation of the model pattern with respect to the image, and the relative size (scale) of the model pattern with respect to the image. For example, such a change is performed by transforming the position and posture of the features constituting the model pattern into the position and posture observed from the image coordinate system using the homogeneous transformation matrix described in Japanese Patent Application Laid-Open No. 2017-91079. For example, it can also be performed by applying an affine transformation to the model pattern. Then, for each comparison area, the comparison unit 39 calculates the overall consistency between the comparison area and the model pattern, which overall consistency represents the degree of consistency between the multiple features set for the entire model pattern and the multiple features extracted from the comparison area. Furthermore, the comparison unit 39 calculates a partial degree of agreement indicating the degree of agreement between the attention feature set in the model pattern and the feature extracted from the image for each comparison region.

[0116] The comparison unit 39 can calculate the overall degree of coincidence and the partial degree of coincidence according to the following equation, for example.

[0117] Overall consistency = the number of features set for the model pattern whose distance to any of the features extracted from the comparison area of ​​the target image is less than a predetermined value / the total number of features set for the model pattern

[0118] Partial consistency = the number of attention features in the model pattern whose distance to any of the features in the target image is less than a predetermined value / the total number of attention features

[0119] Furthermore, when each feature is represented by a point such as a point on a contour (i.e., when each feature is a feature point), the distance between two features in the above-mentioned calculation formulas for overall consistency and partial consistency can be set as the Euclidean distance between the feature points. Furthermore, when each feature is a straight line or a curve, the distance between two features can be set as the average of the Euclidean distances between a plurality of predetermined positions (e.g., the end points and the midpoint) of the two straight lines or curves serving as the features.

[0120] Alternatively, the comparison unit 39 may calculate the overall and partial consistency for each feature of the model pattern, such that the shorter the distance to the nearest feature in the comparison area, the greater the overall and partial consistency, for example, according to the following equation.

[0121] Overall consistency = Σ n＝1 N (1 / (dn+1)) / N

[0122] Partial agreement = Σ m＝1 M (1 / (dm+1)) / M

[0123] Here, dn is the minimum distance from the nth feature set for the model pattern to any of the features extracted from the comparison area of ​​the target image, and N represents the total number of features set for the model pattern. Similarly, dm is the minimum distance from the mth feature of interest in the model pattern to any of the features extracted from the comparison area of ​​the target image, and M represents the total number of features of interest included in the model pattern.

[0124] The comparison unit 39 determines that the workpiece 10 is represented in the comparison area of ​​interest when the overall consistency is greater than a predetermined overall consistency threshold and the partial consistency is greater than a predetermined partial consistency threshold. Here, the feature of interest is a feature distributed in a direction different from the direction in which the position detection capability is low. Therefore, when the partial consistency is greater than the predetermined partial consistency threshold, it is possible to accurately determine that the workpiece 10 is represented in the comparison area of ​​interest even in the direction in which the position detection capability is low. Similarly, the feature of interest is a feature distributed in a direction different from the rotation direction in which the angle detection capability is low. Therefore, when the partial consistency is greater than the predetermined partial consistency threshold, it is possible to accurately determine that the workpiece 10 is represented in the comparison area of ​​interest even in the direction in which the angle detection capability is low.

[0125] Figures 11A to 11E 300 is a diagram illustrating the comparison between the model pattern 300 and the image. Figure 6 The comparison between the model pattern 300 and the image is described below. Figure 11A As shown in FIG. 5 , the workpiece 10 shown in the image 500 has a substantially circular outline with a portion of the outline being a concave notch (keyway 10 b). Furthermore, a plurality of features 501 are extracted along the outline of the workpiece 10. In contrast, as shown in FIG. Figure 11B As shown, in the model pattern 300 , a plurality of features 301 are also set along the contour of the workpiece 10 , and a focus feature 320 is set in a portion of the contour that is a concave notch.

[0126] like Figure 11CAs shown, in image 500, comparison region 530 is set to include workpiece 10, and the angle of model pattern 300 matches the angle of workpiece 10 on image 500. In this case, across the entire model pattern 300, the features 301 set for the model pattern substantially match the features 501 extracted from image 500, and the feature of interest 320 substantially matches the feature 501 corresponding to the feature of interest 320 extracted from image 500. Therefore, since both the overall and partial coincidence degrees are high, it is determined that workpiece 10 is represented in comparison region 530, and it can be seen that the angle of workpiece 10 on image 500 is the same as that of model pattern 300.

[0127] On the other hand, Figure 11D As shown, when comparison region 540 deviates from the region representing workpiece 10, the features 301 set for model pattern 300 and the features 501 extracted from image 500 do not match with respect to the entire comparison region. Furthermore, feature of interest 320 does not match with the features 501 extracted from image 500. As a result, both the overall degree of agreement and the degree of partial agreement become low. Therefore, comparison region 540 is determined to be different from the region representing workpiece 10.

[0128] And, as Figure 11E As shown, while comparison region 550 includes workpiece 10, the angle of workpiece 10 in image 500 is set to differ from the angle of model pattern 300 being compared. In this case, model pattern 300 is aligned to roughly match the contour of workpiece 10. Consequently, more of the multiple features 501 are close to any of the features 301, resulting in a relatively high overall degree of agreement. However, because the position of the concave notch in workpiece 10 is offset from the position of feature of interest 320 in model pattern 300, the partial degree of agreement is low. As a result, workpiece 10 is not detected in comparison region 550.

[0129] Thus, the comparison unit 39 detects the region showing the workpiece 10 on the image based on both the overall and partial coincidences calculated from the entire model pattern. Therefore, the angle of the workpiece 10 can be accurately detected for the rotation direction with low angle detection capability.

[0130] In addition, Figures 11A to 11E In the figure, the Figure 6 The model pattern 300 shown is an example of comparing with the image, but in Figure 5When the model pattern 300 shown is compared with the image, the comparison unit 39 also detects the area representing the workpiece 10 on the image based on the overall consistency and partial consistency calculated based on the entire model pattern. Therefore, even in a direction with low position detection capability, the position of the workpiece 10 can be accurately detected.

[0131] In addition, the overall consistency threshold and the partial consistency threshold may be the same or different from each other. In addition, when there are multiple areas with set focus features in the model pattern, the value of the partial consistency threshold applied to each of the multiple areas may be different, or the value of the partial consistency threshold applied to the multiple areas may be the same.

[0132] In addition, a plurality of model patterns may be prepared in advance. In this case, the direction of observing the workpiece 10 may also be different for each model pattern. In this case, the comparison unit 39 calculates the overall consistency and the partial consistency according to the comparison area for each of the plurality of model patterns in the same manner as described above. Then, the comparison unit 39 determines that in the comparison area corresponding to the position of the model pattern where the sum of the overall consistency and the partial consistency is the maximum, the overall consistency is above the overall consistency threshold, and the partial consistency is above the partial consistency threshold, the workpiece 10 observed from the direction represented by the model pattern is represented.

[0133] Once the position of the workpiece 10 on the image is determined, the comparison unit 39 detects the position of the workpiece 10 in real space based on that position. The position of each pixel on the image corresponds to the orientation as viewed from the camera 4 on a one-to-one basis. Therefore, the comparison unit 39 can, for example, determine the orientation from the camera 4 toward the workpiece 10 as corresponding to the center of gravity of the area where the workpiece 10 is displayed on the image. Furthermore, the comparison unit 39 can calculate an estimated distance from the camera 4 to the workpiece 10 by multiplying the ratio of the area of ​​the workpiece 10 on the image to the area of ​​the area where the workpiece 10 is displayed on the image, when the distance from the camera 4 to the workpiece 10 is a predetermined reference distance, by the reference distance. Therefore, the comparison unit 39 can detect the position of the workpiece 10 in the camera coordinate system based on the position of the camera 4, based on the orientation from the camera 4 toward the workpiece 10 and the estimated distance.

[0134] Furthermore, the comparison unit 39 can determine the actual rotation amount of the workpiece 10 relative to the orientation of the workpiece 10 represented by the model pattern when viewed from a predetermined direction, based on the orientation of the model pattern in the comparison region determined to represent the workpiece 10. Therefore, the comparison unit 39 can determine the posture of the workpiece 10 based on this rotation amount. Furthermore, the comparison unit 39 can determine the posture of the workpiece 10 represented by the camera coordinate system by rotating the posture of the workpiece 10 in the image by the difference between the predetermined orientation specified for the model pattern and the orientation corresponding to the center of gravity of the region representing the workpiece 10 from the camera 4.

[0135] Each time the position of the workpiece 10 in the actual space is obtained, the comparison unit 39 outputs the position of the workpiece 10 to the movable unit control unit 40 .

[0136] The movable part control unit 40 controls the movable parts of the robot 2 based on the position and posture of the workpiece 10 detected by the comparison unit 39. For example, the movable parts of the robot 2 are controlled so that the tool 16 of the robot 2 moves to a position where it can perform work on the workpiece 10. In this case, the movable part control unit 40 controls the movable parts of the robot 2 so that, in the image generated by the camera 4, the workpiece 10 is represented at a predetermined position and a predetermined size corresponding to the position where the tool 16 performs work on the workpiece 10. In this case, the movable part control unit 40 can control the movable parts of the robot 2 according to a method such as the position-based method or the feature-based method that controls the robot based on an image of an object obtained by a camera. For such a method, please refer to Hashimoto, "Vision and Control," Control Division Meeting of the Japan Society of Measurement and Automatic Control, Walker & Co., Kyoto, pp. 37-68, 2001.

[0137] Figure 12 This is an operational flowchart of the movable unit control process including the object detection process. The processor 34 executes the movable unit control process according to the operational flowchart below each time an image is acquired from the camera 4. The processes of steps S401 to S407 in the operational flowchart below are included in the object detection process.

[0138] The feature extraction unit 38 extracts features of the appearance of different locations of the multiple workpieces 10 from the image (step S401). The comparison unit 39 sets a comparison area on the image to be compared with the model pattern (step S402). The comparison unit 39 calculates the overall consistency Sa and the partial consistency Sp between the model pattern and the comparison area (step S403). The comparison unit 39 then determines whether the overall consistency Sa is greater than the overall consistency threshold Tha and whether the partial consistency Sp is greater than the partial consistency threshold Thp (step S404).

[0139] If the overall degree of agreement Sa is less than the overall degree of agreement threshold Tha, or the partial degree of agreement Sp is less than the partial degree of agreement threshold Thp ("No" in step S404), the comparison unit 39 determines that the comparison area does not contain a workpiece 10 that is oriented in the same direction as the model pattern. The comparison unit 39 then changes the comparison area by changing at least one of the relative position, orientation, and scale of the model pattern relative to the image (step S405). The comparison unit 39 then repeats the process from step S103 onward.

[0140] On the other hand, if the overall degree of consistency Sa is greater than or equal to the overall degree of consistency threshold Tha and the partial degree of consistency Sp is greater than or equal to the partial degree of consistency threshold Thp ("Yes" in step S404), the comparison unit 39 determines that a workpiece 10 having the same orientation as that of the compared model pattern is present in the comparison area (step S406). The comparison unit 39 then detects the position and posture of the workpiece 10 in real space based on the position and orientation of the workpiece 10 in the image (step S407).

[0141] The movable part control unit 40 controls the movable part of the robot 2 so that the tool 16 moves to a position where the workpiece 10 can be worked on based on the position and posture of the workpiece 10 in real space (step S408 ).

[0142] After step S408, the processor 34 terminates the movable unit control process. Furthermore, if the workpiece 10 is not detected in the image even after repeating steps S403 to S405 a predetermined number of times or more, the comparison unit 39 may determine that the workpiece 10 is not shown in the image. In this case, the movable unit control unit 40 may stop the movable unit.

[0143] Next, as a method for calculating the degree of consistency between the model pattern 300 with the focus feature set and the target image, a method for calculating the degree of consistency by weighting the focus feature of the model pattern 300 higher than the weights of features other than the focus feature will be described. In this case, the comparison unit 39 calculates the degree of consistency according to the following equation (5).

[0144] [Formula 6]

[0145]

[0146] In formula (5), N is the total number of feature points on the model pattern. j and the image corresponding to P j The characteristic point Q j The distance is set to |P j ~Q j |, then define δ as follows j .

[0147] [Formula 7]

[0148]

[0149] In the formula (5), W is a weight coefficient. As an example, the feature point P on the model pattern 300 is j In the case of a focus feature, let W = 1.2, and the feature point P on the model pattern 300 j If it is not a feature of interest, W is set to 0.8.

[0150] The comparison unit 39 calculates the value of any feature point P on the model pattern 300 based on the equation (5). j The feature point P corresponding to the object image j The characteristic point Q j The distance between |P j -Q j When the value D is less than the predetermined value, it is determined to be a feature point P. j With feature point Q j Consistent, using weight coefficient W j The value of (δ j =1) to calculate the consistency. On the other hand, at the distance |P j ~Q j | When it is greater than the predetermined value D, the comparison unit 39 determines that it is a feature point P j With feature point Q j Inconsistent, weight coefficient W is not used j The value of (δ j =0), to calculate the consistency.

[0151] Furthermore, in equation (5), the characteristic point P on the model pattern 300 is j is the weight coefficient W when the feature is of interest j Set as the feature point P j The weight coefficient W is the factor other than the focus feature. j Therefore, the more consistent the focus feature set for the model pattern 300 is with the feature in the target image, the higher the degree of consistency calculated according to formula (5). In the calculation of the degree of consistency, the contribution of the focus feature with position detection capability or angle detection capability is higher than the contribution of features other than the focus feature. Therefore, even if the model pattern 300 does not have position detection capability or angle detection capability, the degree of consistency in the focus feature is mainly utilized to calculate the degree of consistency between the model pattern 300 and the target object represented by the target image with high precision.

[0152] Figure 13This is a flowchart of the movable unit control process, illustrating the process for calculating the degree of coincidence by assigning a higher weight to the feature of interest of model pattern 300 than to features other than the feature of interest. Processor 34 executes the movable unit control process according to the flowchart below each time an image is acquired from camera 4. The processes in steps S501 through S507 in the flowchart below are included in the object detection process.

[0153] The feature extraction unit 38 extracts the features of the appearance of the multiple workpieces 10 at different positions from the image (step S501). The comparison unit 39 sets a comparison area on the image to be compared with the model pattern (step S502). The comparison unit 39 calculates the consistency S between the model pattern and the comparison area (step S503). At this time, the comparison unit 39 uses the formula (5) to make the feature point P on the model pattern 300 j is the weight coefficient W when the feature is of interest j Greater than feature point P j The weight coefficient W is the factor other than the focus feature. j , and calculates the degree of consistency S. Then, the comparison unit 39 determines whether the degree of consistency S is equal to or greater than a threshold value Th (step S504).

[0154] If the degree of coincidence S is less than the threshold value Th ("No" in step S504), the comparison unit 39 determines that the workpiece 10 corresponding to the model pattern 300 is not represented in the comparison area. The comparison unit 39 then changes the comparison area by changing at least one of the relative position, orientation, and scale of the model pattern 300 relative to the image (step S505). The comparison unit 39 then repeats the process from step S503 onward.

[0155] On the other hand, if the degree of coincidence S is greater than or equal to the threshold value Th ("Yes" in step S504), the comparison unit 39 determines that a workpiece 10 corresponding to the compared model pattern 300 is represented in the comparison area (step S506). The processing after step S506 is performed in the same manner as the processing after step S406 in FIG. 11.

[0156] As described above, the model pattern evaluation device calculates an evaluation value representing the geometric distribution of multiple features included in the model pattern. Based on the evaluation value, it identifies the direction in which the feature position distribution is the widest as a specific direction with low angle detection capability. Furthermore, based on the evaluation value, the model pattern evaluation device determines that at least some of the multiple features are distributed along a rotational direction about a predetermined point, and defines this rotational direction as a direction with low angle detection capability. Features distributed in directions different from these specific directions or rotational directions are then set as focus features in the model pattern. The object detection device compares the model pattern with the target image and, based on the degree of consistency between the focus features and features extracted from the image, determines whether the position or angle of the model pattern matches the target object represented in the target image. Therefore, even when another object with an overall shape similar to the target object is shown in the image, the object detection device can prevent the misdetection of the other object as the target object and can accurately detect the target object represented in the image. In particular, the object detection device can accurately detect the target object from an image, even when the target object has a simple, linear shape or a symmetrical shape.

[0157] (Variation)

[0158] The object detection device can also be used for purposes other than controlling an automatic machine. For example, the object detection device can also be used to determine whether the workpiece 10 conveyed on a belt conveyor is good or bad. In this case, the camera 4 can be fixedly installed so that a portion of the conveying path of the workpiece 10 is included in the imaging range of the camera 4. In addition, the object detection device can be provided with a Figure 2 The device has the same structure as the control device 3 shown in FIG. However, the drive circuit 32 may be omitted. In this case, a model pattern representing a qualified product of the workpiece 10 is stored in a memory possessed by the object detection device. In this case, the processor of the object detection device can perform the processing of the feature extraction unit 38 and the processing of the comparison unit 39 on the image obtained by the camera 4. Then, when the comparison unit 39 detects the workpiece 10 from the image obtained by the camera 4 by comparing it with the model pattern, it can be determined that the workpiece 10 is a qualified product. On the other hand, in the case where the comparison unit 39 cannot detect the workpiece 10 from the image, it can also be determined that the workpiece 10 located within the shooting range of the camera 4 when the image was generated is a defective product. Then, the processor of the object detection device can display the quality judgment result on the display device, or can notify the quality judgment result to other devices connected via the communication interface.

[0159] According to this modification, the object detection device can determine whether the inspection object is good or bad even when the position and posture of the inspection object are not determined.

[0160] Furthermore, the computer program for executing the processing of each unit included in the processor 34 of the control device 3 may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0161] In addition, Figure 4 2 shows an example in which the model pattern evaluation device and the object detection device are configured by one processor 34 of the control device 3 included in the robot system 1 . However, the model pattern evaluation device and the object detection device may be configured as separate bodies.

[0162] For example, if the shape of the model pattern is predetermined, a model pattern evaluation device, separate from the object detection device, can be used to pre-evaluate the model pattern's position detection and angle detection capabilities and pre-set a region of interest within the model pattern. In this case, the model pattern evaluation device, separate from the control device 3, pre-sets the region of interest for the model pattern. The model pattern with the region of interest set is stored in the memory 33 of the control device 3 included in the robot system 1.

[0163] On the other hand, it is also possible to assume that, for example, each time an image of a target object is captured by the camera 4, the model pattern 300 is generated sequentially, the position detection capability and angle detection capability of the generated model pattern are evaluated to set the focus feature, and the target object is detected sequentially using the model pattern. In such a case, Figure 4 As shown, preferably, one processor 34 includes components of the model pattern evaluation process consisting of an evaluation unit 36 ​​and a focus feature setting unit 37 , and components of the object detection process consisting of a feature extraction unit 38 and a comparison unit 39 .

[0164] All examples and specific terms listed herein are intended to help the reader understand the concept of the present invention and the advancement of the technology and should be interpreted as not being limited to the structure of any example in this specification related to the advantages and disadvantages of the present invention, such specific examples and conditions. Although the embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

[0165] Explanation of symbols

[0166] 1 Robotic system,

[0167] 2 robots,

[0168] 3 control devices,

[0169] 4 cameras,

[0170] 5 communication lines,

[0171] 6 Object detection device,

[0172] 11 base,

[0173] 12 rotating tables,

[0174] 13 First Arm,

[0175] 14 Second Arm,

[0176] 15 wrists,

[0177] 16 tools,

[0178] 21-25 joints,

[0179] 31 communication interfaces,

[0180] 32 drive circuit,

[0181] 33 Memory,

[0182] 34 processors,

[0183] 35 servo motors,

[0184] 36 Evaluation Department,

[0185] 36a Position Detection Capability Evaluation Department,

[0186] 36b Angle detection capability evaluation department,

[0187] 37 Focus on feature setting department,

[0188] 38 Feature Extraction Unit,

[0189] 39 control part,

[0190] 40 Movable part control unit.

Claims

1. An object detection device, characterized in that: have: a storage unit storing a model pattern representing a plurality of predetermined features located at mutually different positions of the target object when the target object is viewed from a predetermined direction; a feature extraction unit configured to extract a plurality of predetermined features from an image representing the target object; as well as a comparison unit that calculates a degree of consistency while changing at least one of a relative position or angle, a relative orientation, and a relative size of the model pattern with respect to the image, the degree of consistency indicating a degree of consistency between the plurality of predetermined features of the model pattern and the plurality of predetermined features extracted from the region on the image corresponding to the model pattern, and determines that the target object is represented in the region on the image corresponding to the model pattern when the degree of consistency is greater than a predetermined threshold value; The predetermined features stored in the storage unit include focus features, and the focus features can be used to detect the position of the object on the image in a specific direction when the position detection capability of the object in the specific direction of the model pattern is low, or can be used to detect the angle of the object on the image in the rotation direction with the predetermined point of the model pattern as the rotation center when the angle detection capability of the object in the rotation direction with the predetermined point of the model pattern as the rotation center is low. When the focus feature of the model pattern is consistent with the predetermined feature on the image, the comparison unit calculates the degree of consistency by setting the weight coefficient of the focus feature in the calculation of the degree of consistency to a larger value than when the predetermined feature other than the focus feature of the model pattern is consistent with the predetermined feature on the image.

2. The object detection device according to claim 1, wherein The object detection device further comprises: an evaluation unit that calculates an evaluation value representing a geometric distribution of the plurality of predetermined features included in the model pattern; and A focus feature setting unit sets at least a part of the plurality of predetermined features as the focus feature based on the evaluation value.

3. A model pattern evaluation device, characterized in that: have: an evaluation unit that calculates, with respect to a model pattern used to detect a target object from an image showing the target object, an evaluation value indicating a geometric distribution of a plurality of features included in the model pattern; as well as A focus feature setting unit, based on the evaluation value, sets as a focus feature a first feature among the multiple features that can be used to detect the position of the object object on the image in the specific direction when the position detection capability of the object object in the specific direction of the model pattern is low, or a second feature that can be used to detect the angle of the object object on the image in the rotation direction with the predetermined point of the model pattern as the rotation center when the angle detection capability of the object object in the rotation direction with the predetermined point of the model pattern as the rotation center is low.

4. The model pattern evaluation device according to claim 3, characterized in that The evaluation unit determines, based on the evaluation value, a direction in which the width of the position distribution of the feature is the largest as the specific direction. The focus feature setting unit sets, as the first feature, the feature distributed in a direction different from the specific direction among the plurality of features.

5. The model pattern evaluation device according to claim 3 or 4, characterized in that: The evaluation unit obtains a normal vector relative to the direction of arrangement of adjacent features for each of the multiple features, and calculates a covariance matrix representing the distribution of the normal vector as the evaluation value. When the ratio of the smaller of the two eigenvalues ​​of the covariance matrix to the larger eigenvalue is below a first threshold, the direction of the eigenvector corresponding to the larger eigenvalue is set to the specific direction.

6. The model pattern evaluation device according to claim 3, characterized in that The evaluation unit determines, based on the evaluation value, that a portion of the feature is distributed along the rotation direction with the predetermined point as the rotation center. The feature-of-interest setting unit sets the feature distributed in a direction different from the rotation direction as the feature of interest.

7. The model pattern evaluation device according to claim 6, characterized in that The evaluation unit obtains a normal line relative to the direction in which adjacent features are arranged for each of the multiple features, at least a portion of the normal lines intersect at the predetermined point, and the ratio of the number of normal lines that do not intersect at the predetermined point to the total number of features is set as the evaluation value. When the ratio is greater than a second threshold value, it is determined that a portion of the feature is distributed along the rotation direction with the predetermined point as the rotation center.

8. The model pattern evaluation device according to claim 7, characterized in that The focus feature setting unit sets the feature whose normal lines do not intersect at the predetermined point as the focus feature.

Images