Hole location detection system and hole location detection method

Abstract

The system detects the position of holes formed in manufactured parts with high precision. [Solution] A hole position detection system that detects the position of a hole by template matching using an image of a part and a template image of a hole formed in the part comprises a camera that images a part with a hole, a storage unit that stores a template image of the hole generated by cutting out a partial region image including the hole from a reference image obtained by imaging a part at a reference position with the camera, and a detection device that identifies a first displacement amount indicating the amount of movement of the part from a reference position on a first plane substantially perpendicular to the optical axis of the camera from the image of the part captured by the camera, converts the first displacement amount into a second displacement amount in a direction parallel to the second plane and a third displacement amount in a direction perpendicular to the second plane based on the plane angle between the second plane where the hole is located and the first plane, corrects the template image based on the second and third displacement amounts, and detects the position of the hole using the corrected template image.

Description

Technical Field

[0005] ,

[0001] The present disclosure relates to a hole position detection system and a hole position detection method for detecting the position of holes formed in manufactured parts.

Background Art

[0002] Conventionally, in the field of product manufacturing, devices for detecting the position of holes formed in parts have been used. For example, the position of a hole is detected in order to perform predetermined processing around the hole of a part. Patent Document 1 discloses a system that performs processing such as changing the brightness and binarizing an image of a part taken, and detecting the hole position on the processed image. By making the hole clearly appear in the image through image processing, the hole position can be accurately detected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above conventional technology, the hole position is detected by using a template matching technique in which an image of a part with a hole is scanned with a template image of a hole taken in advance. However, when the position of the part to be imaged changes, the hole position may not be accurately detected. For example, if the shape of the hole appearing in the captured image changes because the position and orientation of the part placed on the part table change, it may not match the shape of the hole appearing in the template image, and there is a possibility that the hole position cannot be accurately detected.

[0005] To solve this problem, precisely placing parts on the parts tray is time-consuming. Designing the parts tray to precisely position parts compromises its versatility. Using sensors to precisely detect part positions on the tray increases the cost of the sensors and related equipment. Therefore, there was a need for a simpler method that could detect hole positions with high accuracy while suppressing the effects of changes in part position.

[0006] This disclosure has been made in view of the problems of the prior art described above, and one of its purposes is to provide a hole position detection system and a hole position detection method that can suppress the influence of changes in the position of a part when detecting the position of a hole formed in a part. [Means for solving the problem]

[0007] The hole position detection system according to this disclosure is a hole position detection system that detects the position of a hole by template matching using an image of a part and a template image of a hole formed in the part, and comprises a camera that images a part with a hole, a storage unit that stores a template image of the hole generated by cutting out a partial region image including the hole from a reference image obtained by imaging a part at a reference position with the camera, and a detection device that identifies a first displacement amount indicating the amount of movement of the part from the reference position on a first plane substantially perpendicular to the optical axis of the camera from the image of the part captured by the camera, converts the first displacement amount into a second displacement amount in a direction parallel to the second plane and a third displacement amount in a direction perpendicular to the second plane based on the plane angle between the second plane where the hole is located and the first plane, corrects the template image based on the second and third displacement amounts, and detects the position of the hole using the corrected template image.

[0008] In the above configuration, the storage unit further stores information relating to the detection area of ​​the hole, which is set in a predetermined region including the hole on the reference image, and the detection device may correct the detection area based on the second displacement amount and the third displacement amount, and then scan the region on the captured image corresponding to the corrected detection area with the corrected template image to detect the position of the hole.

[0009] In the above configuration, the detection device sets up a first coordinate system consisting of orthogonal X1 and Y1 axes set on the surface of the part and a Z1 axis set corresponding to the optical axis of the camera which is substantially perpendicular to the surface of the part, and identifies the movement of the part by the amount of displacement in the X1 axis direction, the amount of displacement in the Y1 axis direction, and the displacement angle around the Z1 axis in the first coordinate system. On the surface where the hole is located, a second coordinate system is set up consisting of an X2 axis corresponding to the major axis direction of the hole which appears as an ellipse in the captured image, a Y2 axis corresponding to the minor axis direction, and a Z2 axis passing through the center of the hole and perpendicular to the surface where the hole is located, and identifies the movement of the part by the amount of displacement in the X1 axis direction in the first coordinate system. Based on the position, the displacement in the Y1 axis direction, and the displacement angle around the Z1 axis, the movement of the hole accompanying the movement of the part is determined by the displacement in the X2 axis direction, the displacement in the Y2 axis direction, and the displacement in the Z2 axis direction in the second coordinate system. Based on the distance from the camera to the hole and the displacement of the hole in the Z2 direction, the size of the hole as depicted in the template image may be corrected. Based on the displacement in the X2 axis direction, the major axis of the hole as depicted in the template image as an ellipse may be corrected. Based on the displacement in the Y2 axis direction, the minor axis of the hole as depicted in the template image as an ellipse may be corrected.

[0010] In the above configuration, the storage unit further stores information relating to the detection area of ​​the hole, which is set in a predetermined region including the hole on the reference image, and the detection device may correct the size of the detection area based on the distance from the camera to the hole and the amount of displacement of the hole in the Z2 direction, and correct the position of the detection area based on the displacement in the X2 axis direction and the displacement in the Y2 axis direction.

[0011] In the above configuration, the detection device may calculate a correlation coefficient at each position while scanning the captured image with the corrected template image, rotate the captured image, scan the rotated captured image with the corrected template image to calculate a correlation coefficient at each position, and detect the position of the hole based on the comparison result of the multiple correlation coefficients calculated.

[0012] The hole position detection method according to this disclosure is a hole position detection method that detects the position of a hole by template matching using an image of a part and a template image of a hole formed in the part, and includes the steps of: generating a template image of the hole by cutting out a partial region image including the hole from a reference image obtained by imaging a part at a reference position with a camera; imaging a part with a hole with a camera; identifying a first displacement amount indicating the amount of movement of the part from the reference position on a first plane substantially perpendicular to the optical axis of the camera from the image of the part captured by the camera; converting the first displacement amount into a second displacement amount in a direction parallel to the second plane and a third displacement amount in a direction perpendicular to the second plane based on the plane angle between the second plane where the hole is located and the first plane; correcting the template image based on the second and third displacement amounts; and detecting the position of the hole using the corrected template image. [Effects of the Invention]

[0013] According to the hole position detection system and hole position detection method described herein, by detecting changes in the position of a part from an image of the part having a hole, and correcting the hole template image according to the direction and amount of movement of the part, the position of a hole formed in the part can be detected with high accuracy. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 is a schematic diagram illustrating the overview of the hole position detection system according to this embodiment. [Figure 2] Figure 2 is a block diagram showing an example configuration of a hole location detection system. [Figure 3]Figure 3 is a flowchart showing an example of the processing flow performed by the hole location detection system. [Figure 4] Figure 4 illustrates a method for converting the displacement amount due to the movement of a part in the part coordinate system to the displacement of a hole in the hole coordinate system. [Figure 5] Figure 5 illustrates a method for converting the displacement of a component due to rotation in the component coordinate system to the displacement of a hole in the hole coordinate system. [Figure 6] Figure 6 is a schematic diagram illustrating the method for correcting the template image. [Modes for carrying out the invention]

[0015] Hereinafter, embodiments of the hole position detection system and hole position detection method according to this disclosure will be described with reference to the attached drawings. Figure 1 is a schematic diagram illustrating the overview of the hole position detection system 1 according to this embodiment.

[0016] The hole position detection system 1 includes a detection device 10 and a camera 100. Although not shown in Figure 1, a part 300 with holes 310 (310a, 310b) is placed on a part stand, and the part 300 is illuminated with light from a light source above, while the part 300 is imaged by the camera 100. The part stand is an example of a support stand for supporting the part 300. The part stand is configured to position the part 300 at a predetermined reference position, but the position of the part 300 may move slightly from the reference position. For example, the position of the part 300 may shift on the part stand, or the part 300 may rotate.

[0017] The hole position detection system 1 detects the position of the hole 310 by template matching, using an image 402 of the part 300 and a template image 500 of the hole 310 formed in the part 300. The template image 500 can be generated by cutting out a partial region image including the hole 310 from a reference image 401 obtained by imaging the part 300 at a reference position with the camera 100.

[0018] When the hole position detection system 1 detects the hole position, it identifies a first displacement amount indicating the amount of movement from the reference position of the component 300 on a first plane substantially perpendicular to the optical axis of the camera 100 from the image 402 of the component 300 captured by the camera 100. Based on the surface angle between the second plane with the hole 310 and the first plane, it converts the first displacement amount into a displacement amount in a direction parallel to the second plane and a displacement amount in a direction perpendicular to the second plane, and corrects the template image 500 based on the converted displacement amount. Here, the displacement amount referred to includes at least one of the displacement amount due to the translational movement of the component 300 and the displacement amount (displacement angle) due to the rotational movement of the component 300.

[0019] The hole position detection system 1 identifies how much the hole 310 is displaced in a direction parallel to the plane where the hole 310 is located and how much it is displaced in a perpendicular direction from the displacement amount of the component 300, and corrects the template image 500 based on the identified displacement amount of the hole 310. For example, if the template image 500 is displaced in a direction away from the camera 100, it is reduced, and if it is displaced in a direction approaching the camera 100, it is enlarged. Also, when the major or minor diameter of the hole 310 that appears elliptical in the captured image 402 by the camera 100 changes with the displacement of the hole 310, the template image 500 is corrected according to this change. Thus, in the hole position detection system 1, when the size and shape of the hole 310 imaged by the camera 100 change with the movement of the component 300, the template image 500 is corrected first according to this change, and the corrected template image 510 is used for detecting the hole position, so that the hole position can be detected with high precision.

[0020] Hereinafter, as shown in FIG. 1, the description will continue assuming that the optical axis of the camera 100 passes through the point P1 which is substantially the center of the component 300 at the reference position, and the XYZ orthogonal coordinate system is set such that the surface of the component 300 including the point P1 is the XY plane and the Z axis coincides with the optical axis of the camera 100.

[0021] The shape of component 300, the types, shapes, numbers, etc. of holes 310 formed in component 300 are not particularly limited. However, in the case of component 300 having a plurality of through-holes 310 (310a, 310b), the description will continue by taking as an example the case of detecting the position of through-hole 310a formed in a plane inclined with respect to the XY plane. In the present embodiment, the upper surface of component 300, that is, the surface imaged by camera 100, is described as the surface of component 300.

[0022] As shown in FIG. 1, camera 100 images component 300 from above (S1). Detection device 10 specifies the displacement amount of component 300 in the captured image due to the positional deviation of component 300 from the captured image 402 by camera 100 (S2).

[0023] Detection device 10 stores a reference image 401 obtained by imaging component 300 at a reference position, that is, a state without positional deviation. The template image 500 of through-hole 310a used by detection device 10 for detecting the hole position is an image obtained by cutting out a partial region including through-hole 311a shown in this image 401 from reference image 401.

[0024] In the captured image 402 shown in FIG. 1, component 302 shown in this image is indicated by a solid line, and the position corresponding to component 301 in reference image 401 is indicated by a dashed line. Detection device 10 compares the position of component 301 shown in reference image 401 with the position of component 302 shown in captured image 402, and specifies the deviation amount from the reference position, that is, displacement amounts dx, dy, and da.

[0025] Displacement amount dx indicates the displacement amount in the X-axis direction, and displacement amount dy indicates the displacement amount in the Y-axis direction. Displacement amount da indicates the rotation angle around an axis parallel to the Z-axis passing through point P2 indicating the approximate center of component 302 shown in captured image 402. Point P2 indicates the position after movement due to the positional deviation of point P1 of component 300 at the reference position.

[0026] For example, if the position of part 300 in real space is displaced by Dx in the positive X-axis direction and by Dy in the positive Y-axis direction, as shown in Figure 1, point P1 of part 301 in the reference image 401 will be displaced by dx in the positive X-axis direction and by dy in the positive Y-axis direction, becoming point P2 of part 302 in the captured image 402. Furthermore, if part 300 rotates by Da (displacement angle) around an axis parallel to the Z-axis passing through point P2 in real space, as shown in Figure 1, part 302 in the captured image 402 will rotate by da (displacement angle) around point P2. Note that since part 300 is placed on a part stand and positioned aiming for the reference position, displacement in the Z-axis direction, rotation around the X-axis, and rotation around the Y-axis do not occur, or if they do, they occur only slightly. For this reason, the hole position detection system 1 uses only the displacement amounts dx, dy, and da for position detection.

[0027] The displacement amounts dx, dy, and da on the image caused by the misalignment of part 300 can be determined using template matching technology. For example, the detection device 10 scans a template image, which is a partial region containing part 301 cut out from a reference image 401, on the captured image 402 while changing its position and angle, to search for a position and angle that matches part 302 in the captured image 402. Having determined the position and angle of part 302 as seen in the captured image 402, the detection device 10 calculates the displacement amounts dx, dy, and da from this position and angle, and the position and angle of part 301 as seen in the reference image 401, as shown in Figure 1. However, the method for determining the displacement amounts dx, dy, and da is not particularly limited. For example, edge detection technology may be used to detect a part of the outer edge of part 301 in the reference image 401, and the corresponding outer edge may also be detected in the captured image 402, and the displacement amounts dx, dy, and da may be calculated from the positional relationship of these outer edges.

[0028] The detection device 10 detects the hole position using template matching technology. If there is no misalignment of the part 300, the detection device 10 scans the region on the captured image 402 corresponding to the detection area 201 set in the reference image 401 with the template image 500 of the through hole 310a obtained from the reference image 401 to detect the hole position. That is, if dx, dy, and da are approximately 0 (zero), the hole position is detected using the template image 500 as is. While scanning the captured image with the template image 500, the detection device 10 calculates the correlation coefficient between the captured image and the template image 500 at each position, and detects the position where the correlation coefficient shows the maximum value as the position of the through hole 310a.

[0029] On the other hand, if there is a misalignment of part 300, the detection device 10 identifies the amount of displacement of the through-hole 310a in directions parallel and perpendicular to the surface on which the through-hole 310a is formed, before detecting the hole position (S3). The through-hole 310a is formed on a plane that forms an angle with the XY plane, that is, a plane inclined from the XY plane. For this reason, the amount of displacement of part 300 on the XY plane and the amount of displacement of the through-hole 310a in the direction parallel to the surface on which the through-hole 310a is formed will be different values. In addition, a misalignment of part 300 may also cause a displacement of the through-hole 310a in the direction perpendicular to the surface on which the through-hole 301a is formed.

[0030] For example, in the example shown in the upper left of FIG. 1, when the position of the component 300 is displaced by a displacement amount L1 in the positive X-axis direction on the XY plane, the displacement amount L2 of the through hole 310a in the direction parallel to the surface with the through hole 310a is smaller than the displacement amount L1 (L2 < L1). Further, due to the positional deviation of the component 300, a displacement of the displacement amount L3 has occurred in the direction perpendicular to the surface on which the through hole 310a is formed. The detection device 10 specifies the displacement amount of the through hole 310a in the direction parallel to the surface on which the through hole 310a is formed and the displacement amount in the direction perpendicular to the surface on which the through hole 310a is formed based on the displacement amounts dx, dy, and da specified above. In the figure in the upper left of FIG. 1, only the displacement in the X-axis direction is shown for simplicity of explanation. Actually, the displacement amount of the through hole 310a is specified taking into account the displacement in the Y-axis direction and the rotation angle around the axis parallel to the Z-axis. The method for specifying the displacement amount will be described later.

[0031] The detection device 10 that has specified the displacement amount of the through hole 310a corrects the template image 500 (S4). The detection device 10 corrects the template image 500 based on how the shape and size of the through hole 310a shown in the captured image 402 change from the shape and size of the through hole 310a shown in the reference image 401 according to the displacement amount of the through hole 310a.

[0032] Even when a circular through hole 310a is formed on the surface of the component 300, the surface with the through hole 310a is inclined from the XY plane perpendicular to the optical axis direction (Z-axis direction) of the camera 100, so the through hole 310a shown in the captured image 402 is elliptical. The detection device 10 specifies how the lengths of the major and minor diameters of this elliptical shape change according to the displacement amount of the through hole 310a, and corrects the template image 500 based on the specification result.

[0033] For example, as shown in the upper left of Figure 1, when a displacement L3 occurs in a direction perpendicular to the surface on which the through-hole 310a is located, the distance from the camera 100 to the through-hole 310a changes. If the distance from the camera 100 to the through-hole 310a increases, the detection device 10 reduces the size of the template image 500 according to the amount of change, and if the distance decreases, it enlarges the size of the template image 500 according to the amount of change.

[0034] Furthermore, when the through-hole 310a is displaced in a direction parallel to the plane on which the through-hole 310a is located, the elliptical shape of the through-hole 310a as seen in the captured image 402 deforms, for example, into a shape that is enlarged or reduced in the direction of displacement. The detection device 10 identifies the amount of change in the major axis and the amount of change in the minor axis of the elliptical shape of the through-hole 310a as seen in the template image 500, and corrects the template image 500 based on the identification result. The correction is performed by at least one of the following in response to the displacement of the through-hole 310a: enlargement in the major axis direction, reduction in the major axis direction, enlargement in the minor axis direction, and reduction in the minor axis direction.

[0035] In the example shown in Figure 1, the corrected template image 510 is an image in which the template image 500 has been scaled down in the direction of the short axis of the elliptical through-hole 310a, and the overall size has also been reduced.

[0036] The detection device 10, having identified the displacement of the through-hole 310a, also corrects the detection area 201 (S5). The detection device 10 corrects the detection area 201 set in the reference image 401 according to the displacement of the through-hole 310a.

[0037] As shown in the upper left of Figure 1, the detection device 10, having identified the displacement amount L2 in the direction parallel to the plane containing the through-hole 310a, moves the detection area 201 set in the reference image 401 by a length corresponding to this displacement amount L2 in the direction in which the displacement occurred. For example, the device determines the displacement amount of the through-hole 310a in two directions: one parallel to the major axis and the other parallel to the minor axis of the elliptical through-hole 310a in the reference image 401. The detection area 201 set in the reference image 401 is then moved according to the displacement amount in each direction to become the detection area 202 of the captured image 402.

[0038] As shown in the upper left of Figure 1, if a displacement L3 occurs in a direction perpendicular to the surface containing the through-hole 310a, and the distance from the camera 100 to the through-hole 310a changes, the detection device 10 may enlarge or reduce the size of the detection area 201 set in the reference image 401 in accordance with the change in distance. If the distance from the camera 100 to the through-hole 310a increases, the size of the detection area 201 should be reduced in accordance with the amount of change, and if the distance decreases, the size of the detection area 201 should be enlarged in accordance with the amount of change. If the part 300 is rotating, the detection area 201 may be rotated in accordance with this rotation, but if the detection area 202 is moved in directions parallel to the major axis and minor axis of the elliptical through-hole 310a, the through-hole 310a will fit within the detection area 202, so the detection area 201 does not need to rotate.

[0039] Having completed the correction of the template image 500 and the detection area 201, the detection device 10 scans the area on the captured image 402 corresponding to the corrected detection area 202 with the corrected template image 510 to detect the position of the through hole 310a (S6).

[0040] Specifically, the system scans the detection area 202 set in the captured image 402 with the corrected template image 510, calculating the correlation coefficient between the corrected template image 510 and the captured image 402 at each position. The position where the correlation coefficient shows the maximum value is then detected as the position of the through-hole 310a.

[0041] The detection device 10 performs rotation correction by rotating the captured image 402 (S7). The detection device 10 detects the position of the through hole 310a by scanning the image after rotation correction with the corrected template image 510 (S8). The detection device 10 rotates the captured image 402, but does not rotate the template image 510. Even if the part 302 captured in the captured image 402 is rotating, the position of the through hole 310a can be detected with high accuracy by rotating the captured image 402 to the same angle as the part 301 captured in the reference image 401.

[0042] The direction and angle of rotation correction performed by the detection device 10 can be set in advance. For example, if the part 302 shown in the captured image 402 may rotate both clockwise and counterclockwise around an axis passing through point P2, the captured image 402 should be rotated in both clockwise and counterclockwise directions accordingly, and the through-hole 310a should be detected in the area corresponding to the detection area 202 for the image rotated in each direction. For example, if the part 302 shown in the captured image 402 may rotate only in either a clockwise or counterclockwise direction due to the support method of the part 300, the through-hole 310a should be detected for the image rotated in that direction.

[0043] Regarding the rotation angle, the image can be rotated by a predetermined angle within the range of possible rotations of the component 302 captured in the image 402, and the through-hole 310a can be detected for each rotated image. For example, if the component 300 may rotate up to 20 degrees, the image can be rotated by a predetermined angle for a predetermined number of times, such as 2 degrees 10 times or 5 degrees 4 times.

[0044] For each of the rotation-corrected images, the region corresponding to the detection area 202 is scanned with the template image 510, and the correlation coefficient between the corrected template image 510 and the image is calculated at each position. The position where the correlation coefficient shows the maximum value is then detected as the position of the through-hole 310a.

[0045] Even when part 300 is rotating, the image of the detection area 202 is rotated multiple times by predetermined angles based on a preset rotation direction and rotation angle range. As a result, the image of the through hole 312a in one of the rotated images matches the template image 510, and the correlation coefficient shows its maximum value.

[0046] The detection device 10 compares the correlation coefficient obtained when detecting the through-hole 310a in the captured image 402 without rotation with the correlation coefficient obtained when detecting the through-hole 310a in the image after each rotation while rotating the image multiple times in a predetermined direction by a predetermined angle, and determines the detection result of the through-hole 310a based on the comparison result. The detection position of the through-hole 310a in the image where the correlation coefficient showing the maximum value is obtained is determined as the position of the through-hole 310a of the part 300.

[0047] The detection device 10 may notify the determined location of the through-hole 310a. For example, the detection device 10 may notify the detection of the through-hole 310a by displaying it on a display unit or by playing an audible signal. The detection device 10 may also display the captured image 402 on a display device and indicate the location of the detected through-hole 310a on the image. For example, the location of the through-hole 310a may be indicated by a figure such as an arrow or a circle, or the area corresponding to the through-hole 310a may be indicated by a color different from the surrounding color. The detection device 10 may also output information indicating the detection result to an external device.

[0048] Figure 2 is a block diagram showing an example configuration of the hole position detection system 1. The detection device 10 includes a control unit 11, a storage unit 12, an operation unit 13, and a display unit 14. The control unit 11 includes a camera control unit 11a, a displacement amount identification unit 11b, a detection area correction unit 11c, a template image correction unit 11d, and a hole position detection processing unit 11e.

[0049] The memory unit 12 is a non-volatile memory device that stores information related to the control of the camera 100, information related to the hole position detection process, etc. The operation unit 13 can receive input of various information related to the hole position detection process. The display unit 14 can display various information such as images captured by the camera 100 and the hole position detection results.

[0050] The control unit 11 controls each part of the detection device 10. For example, a program corresponding to the control unit 11 is pre-stored in the storage unit 12 or a dedicated storage device, and the functions and operation of the control unit 11 are realized when this program is executed by hardware such as a CPU. The control unit 11 can control each part based on information input and output via the operation unit 13 and information stored in the storage unit 12. The functions and operation of the hole position detection system 1 described in this embodiment are realized by the control unit 11.

[0051] This will be explained in detail using the flowchart shown in Figure 3. The camera control unit 11a controls the camera 100 to image the component 300 (step S11). The captured image is stored in the storage unit 12.

[0052] The displacement amount identification unit 11b identifies the displacement amount of the part 300 on the captured image (step S12). As explained in Figure 1, the displacement amount identification unit 11b identifies the displacement amounts dx, dy, and da of the part 300 captured in the image.

[0053] The displacement amount determination unit 11b determines the displacement amount and displacement angle of part 300 in the part coordinate system (step S13). From the displacement amount and displacement angle of part 300 in the part coordinate system, the displacement amount of the through hole 310a in the hole coordinate system is determined by the displacement amount of the through hole 310a in the hole coordinate system (step S14). Here, the part coordinate system is a Cartesian coordinate system with the origin at point P where the optical axis of the camera 100 intersects the surface of part 300, as explained in Figure 1. The hole coordinate system is a Cartesian coordinate system with the origin at the center of the through hole 310a when part 300 is in the reference position.

[0054] As shown in Figure 4(a), the component coordinate system is defined as an X1Y1Z1 orthogonal coordinate system where the surface of component 300 is the X1Y1 plane, and the direction of the optical axis passing through point P1 on the surface of component 300, i.e., the X1Y1 plane, is the Z1 direction. The hole coordinate system is defined as an X2Y2Z2 orthogonal coordinate system where the surface with the through hole 310a is the X2Y2 plane, and the direction passing through the center of the through hole 310a and perpendicular to the X2Y2 plane is the Z2 direction. The X2 axis direction and the Y2 axis direction are set to correspond to the major axis direction and minor axis direction of the through hole 310a, which is captured as an elliptical shape in the image 402 taken by the camera 100, respectively. The X2 axis direction is the major axis direction, and the Y2 axis direction is the minor axis direction.

[0055] For example, if there is no rotation of part 300 (da=0) and only the displacement amounts dx and dy of part 300 are identified on the captured image 402, the displacement amount identification unit 11b identifies the displacement amounts Mx and My of part 300 in real space that correspond to the displacement amounts dx and dy on the image. As shown in Figure 4(a), when point P1, which indicates the approximate center of part 300, moves to point P2, the displacement amount in the X1 axis direction is Mx and the displacement amount in the Y1 axis direction is My.

[0056] As shown in Figure 4(a), when the center position of the through hole 310a moves from point P11 to P12 as the part 300 moves, the displacement amounts mx and my in the X2Y2 plane of the hole coordinate system, which correspond to the displacement amounts Mx and My in the part coordinate system, become the displacement amounts mx and my shown in Figure 4(b). Similarly, the displacement amount mz in the direction perpendicular to the X2Y2 plane of the hole coordinate system, which corresponds to the displacement amounts Mx and My in the part coordinate system, becomes the displacement amount mz shown in Figure 4(c). The displacement amount identification unit 11b calculates the displacement amounts mx, my, and mz in the hole coordinate system from the displacement amounts Mx and My in the part coordinate system by performing a coordinate transformation between the part coordinate system and the hole coordinate system.

[0057] Similarly, if, for example, there is no movement of part 300 (dx=dy=0) and only the displacement angle da (displacement amount da) of part 300 is identified on the captured image 402, the displacement amount identification unit 11b identifies the displacement amounts mx, my, and mz in the hole coordinate system that occur as a result of the rotation of part 300.

[0058] For example, as shown in Figure 5(a), suppose that part 300 rotates by an angle Ma around the Z1 axis in real space, and the center of the through hole 310a moves from point P11 to P13. The displacement amount determination unit 11b converts the displacement angle Ma in the part coordinate system into displacement amounts mx, my, and mz in the hole coordinate system. When part 300 rotates as shown in Figure 5(a), the displacement amounts mx and my in the X2Y2 plane of the hole coordinate system become the displacement amounts mx and my shown in Figure 5(b). Similarly, the displacement amount mz in the direction perpendicular to the X2Y2 plane of the hole coordinate system becomes the displacement amount mz shown in Figure 5(c). The displacement amount determination unit 11b calculates the displacement amounts mx, my, and mz in the hole coordinate system from the displacement angle Ma in the part coordinate system by performing a coordinate transformation between the part coordinate system and the hole coordinate system.

[0059] Similarly, if part 300 is both moved and rotated, the position of the center of the through-hole 310a in the hole coordinate system moves in accordance with the movement and rotation of part 300 in the part coordinate system. The displacement amount identification unit 11b converts the amount of displacement of the center position of the through-hole 310a due to the movement and rotation of part 300 in the part coordinate system into a displacement amount in the hole coordinate system through coordinate transformation. As this method of converting a displacement amount in one coordinate system to a displacement amount in the other coordinate system by coordinate transformation between two coordinate systems is conventionally known, a detailed explanation will be omitted. For example, the displacement amount in the part coordinate system can be converted to a displacement amount in the hole coordinate system based on the angle and distance between corresponding coordinates in the two coordinate systems.

[0060] After the displacement amount identification unit 11b identifies the displacement amounts mx, my, and mz of the through hole 310a in the hole coordinate system due to the movement and rotation of the part 300, the detection area correction unit 11c corrects the detection area 201 (Figure 3, step S15).

[0061] For example, the detection area correction unit 11c moves the detection area 201, which was set based on the reference image 401, on the captured image 402 by an amount corresponding to the displacement mx in the direction corresponding to the X2 axis direction of the hole coordinate system. Similarly, the detection area correction unit 11c moves the detection area 201 on the captured image 402 by an amount corresponding to the displacement my in the direction corresponding to the Y2 axis direction of the hole coordinate system.

[0062] The detection area correction unit 11c may enlarge or reduce the rectangle representing the detection area 201 according to the displacement amount mz in the hole coordinate system. For example, if the distance from the camera 100 to the through hole 310a is La, and the displacement amount mz is a displacement in the negative Z2 direction away from the camera 100, the vertical and horizontal dimensions of the rectangular detection area 201 are reduced by La / (La+mz). If the displacement amount mz is a displacement in the positive Z2 direction closer to the camera 100, the vertical and horizontal dimensions of the detection area 201 are enlarged by (La+mz) / La.

[0063] Thus, as explained in Figure 1, the detection area correction unit 11c sets the detection area 202 on the captured image 402 at the corrected position and with the corrected size.

[0064] After the displacement amount identification unit 11b identifies the displacement amounts mx, my, and mz of the through hole 310a in the hole coordinate system, the template image correction unit 11d corrects the template image 500 (Figure 3, step S16). The template image correction unit 11d corrects the major axis, minor axis, and area of ​​the through hole 310a, which is depicted as an elliptical shape in the template image 500.

[0065] Figure 6 is a schematic diagram illustrating the correction method for the template image 500. Figure 6(a) is a diagram illustrating the method of correcting the elliptical shape 511 representing the through-hole 310a in the template image 500 based on the displacement amount mz of the through-hole 310a in the hole coordinate system. Figure 6(b) is a diagram illustrating the method of correcting the elliptical shape 521 representing the through-hole 310a in the template image 500 based on the displacement amounts mx and my of the through-hole 310a in the hole coordinate system.

[0066] Once the displacement amount mz of the through-hole 310a in the hole coordinate system is obtained due to the movement and rotation of part 300, the template image correction unit 11d performs a correction as shown in Figure 6(a) based on the distance d from the center P11 of the through-hole 310a of part 300 at the reference position, i.e., the origin in the hole coordinate system, to the camera 100, and the displacement amount mz.

[0067] If the center of the through-hole 310a moves from the reference point P11 to point P14, away from the camera 100, and the amount of displacement is mz, the template image correction unit 11d corrects the through-hole 310a, which was projected as an ellipse 511 on the template image 500 at the reference point P11, to a reduced ellipse 512. The ratio of the major axis Lx11 of the ellipse 511 before correction to the major axis Lx12 of the ellipse 512 after correction, and the ratio of the minor axis Ly11 of the ellipse 511 before correction to the minor axis Ly12 of the ellipse 512 after correction are corrected to match the ratio of distance (d+mz) to distance d (Lx12 / Lx11=Ly12 / Ly11=(d+mz) / d).

[0068] Once the displacement mx of the through-hole 310a in the hole coordinate system due to the movement and rotation of part 300 is obtained, the template image correction unit 11d corrects the template image 500 using the data 601 that has been prepared in advance in the storage unit 12.

[0069] The data 601 shown in Figure 6(b) shows the relationship between the displacement mx in the X2 axis direction and the changes in the major and minor axes, when the surface angle between the X1Y1 plane in the part coordinate system shown in Figures 4 and 5 and the X2Y2 plane in the hole coordinate system is angle Fa. In other words, it shows the relationship between the displacement of the through hole 310a in the X2 axis direction due to the displacement of part 300, when the surface angle between the surface where point P of part 300 is located and the surface where the through hole 310a is located is angle Fa, and the changes in the minor and major axes of the through hole 310a, which are imaged as an ellipse by the camera 100.

[0070] The relationship between the displacement mx and the changes in the minor and major axes varies depending on the surface angle of the face on which the through-hole 310a is located. Therefore, if there is another through-hole 310b whose hole position is to be detected in addition to the through-hole 310a, separate data will be prepared corresponding to the surface angle of the face on which this through-hole 310b is located. For example, data for each surface angle can be prepared in the storage unit 12, and the data corresponding to the surface angle of the face on which the through-hole 310 whose hole position is to be detected is used. Alternatively, a calculation formula can be prepared for each surface angle, with the displacement of the through-hole 310 as the input value and the changes in the minor and major axes as the output values, and the displacement of the through-hole 310 can be input into this calculation formula to calculate the changes in the minor and major axes.

[0071] The example shown in Figure 6(b) illustrates that when the displacement in the hole coordinate system is value mx, the change in the minor and major axes of the through-hole 310a, which appears as an ellipse in the captured image 402, is α%. Therefore, the template image correction unit 11d corrects the template image 500 based on the change amount α, as shown in Figure 6(b).

[0072] Specifically, the template image correction unit 11d corrects the through hole 310a, which is projected as an elliptical shape 521 on the template image 500 at the reference position, to an elliptical shape 522 that is enlarged by α% in the major axis direction and the minor axis direction, respectively. The ratio of the major axis Lx21 of the elliptical shape 521 before correction to the major axis Lx22 of the elliptical shape 522 after correction, and the ratio of the minor axis Ly21 of the elliptical shape 521 before correction to the minor axis Ly22 of the elliptical shape 522 after correction are corrected to be (1+α) (Lx22 / Lx21=Ly22 / Ly21=1+α). For example, if the amount of change is +5%, the lengths of the major axis Lx22 and minor axis Ly22 after correction are corrected to be 105% of the lengths of the major axis Lx21 and Ly21 before correction, respectively.

[0073] The template image correction unit 11d also corrects the template image 500 based on the displacement amount my, similar to the displacement amount mx described in Figure 6(b). The correction method is the same and therefore will not be explained, but data or calculation formulas showing the relationship between the displacement amount my and the change in the length of the major axis and minor axis are prepared in advance in the storage unit 12 for each surface angle. The template image correction unit 11d acquires the change in the major axis and minor axis corresponding to the displacement amount my, and based on the change, enlarges or reduces the major axis and minor axis of the elliptical shape projected onto the template image 500.

[0074] In the example shown in Figure 6(a), an example was described in which the template image 500 is corrected using the displacement amount mz in the Z2 axis direction of the hole coordinate system. Since the displacement amount of part 300 is sufficiently small compared to the distance from camera 100 to part 300, it is acceptable to consider the displacement amount mz as the displacement amount in the optical axis direction of camera 100. However, it is also possible to correct the displacement amount mz in the Z2 axis direction to the displacement amount in the direction of camera 100 as viewed from the position of the through hole 310a after displacement, and then correct the template image 500 using the corrected displacement amount.

[0075] Furthermore, in the example shown in Figure 6(b), for the sake of simplicity, an example was explained in which the major and minor axes of the elliptical shape of the template image 500, which depicts the through-hole 310a, are similarly corrected using the change amounts corresponding to the displacement amount mx in the X2 axis direction and the displacement amount my in the Y2 axis direction. However, in reality, the major and minor axes are processed separately, and different corrections are performed on them. For example, since the rate of change differs between the major and minor axes depending on the displacement amount and direction of displacement of the through-hole 310a, data showing the relationship between the displacement amount and the rate of change of the diameter is prepared separately for the major and minor axes, and the major and minor axes are corrected separately using each set of data.

[0076] In this way, the template image correction unit 11d corrects the major and minor axes of the through-hole 310a, which is projected as an ellipse on the template image 500, by enlarging or shrinking them as necessary, based on the displacement amounts mx, my, and mz of the through-hole 310a in the hole coordinate system, thereby correcting the area of ​​the ellipse shape.

[0077] Once the correction of the detection area 201 and the correction of the template image 500 are completed, the hole position detection processing unit 11e performs a process (Step S17 in Figure 3) to detect the position of the through hole 310a by scanning the corrected detection area 202 set in the captured image 402 with the corrected template image 510, as explained in Figure 1.

[0078] Specifically, the hole position detection processing unit 11e scans the region corresponding to the corrected detection area 202 in the captured image 402 with the corrected template image 510, and calculates the correlation coefficient between the corrected template image 510 and the captured image 402 at each position. The hole position detection processing unit 11e detects the position where the correlation coefficient shows the maximum value as the position of the through hole 310a in the captured image 402.

[0079] Next, as explained in Figure 1, the hole position detection processing unit 11e rotates the captured image 402 by predetermined angles within a preset rotation direction and rotation angle range. The hole position detection processing unit 11e rotates the captured image 402 (step S19) and repeatedly performs the process (step S17) to detect the position of the through hole 310a in the rotated image until all rotation corrections are completed (step S18; No).

[0080] Specifically, the hole position detection processing unit 11e calculates a correlation coefficient between the corrected template image 510 and the rotated image at each position, while scanning the region corresponding to the detection area 202 with the corrected template image 510 for each image obtained by rotating the captured image 402 by a predetermined angle in a predetermined rotation direction within a predetermined angular range. The hole position detection processing unit 11e detects the position where the correlation coefficient shows the maximum value as the position of the through hole 310a in each image obtained by rotating and correcting the captured image 402.

[0081] For example, in the example shown in Figure 1, the hole position detection processing unit 11e rotates the captured image 402 around the central axis of the elliptical shape 312a representing the through hole 311a captured in the captured image 402, and calculates the correlation coefficient at each position while scanning the rotated image with the corrected template image 510. The hole position detection processing unit 11e may also rotate the captured image 402 around a point P2 that indicates the approximate center of the part 302 captured in the captured image 402. In this case as well, the correlation coefficient can be obtained at each position while scanning the region in the rotated captured image 402 corresponding to the rotated detection area 202 with the corrected template image 510.

[0082] Thus, the hole position detection processing unit 11e, which has detected the position of the through-hole 310a using the captured images 402 before and after rotation, compares the correlation coefficients when detecting the position of the through-hole 310a in each image to determine the position of the through-hole 310a (Figure 3, step S20).

[0083] If part 300 is not rotating, the correlation coefficient will show its maximum value in the captured image 402 before rotation correction is performed. If part 300 is rotating, among the multiple images obtained by rotating the captured image 402 by a predetermined angle in a predetermined direction of rotation, the correlation coefficient will show its maximum value in the image that is in the same state as when part 300 is not rotating, after rotation correction corresponding to the rotation direction and rotation angle of part 300 has been performed. The hole position detection processing unit 11e compares the correlation coefficients of each image before and after rotation correction and determines the position of the through hole 310a in the image showing the maximum correlation coefficient as the final through hole 310a. That is, the position showing the maximum correlation coefficient among the correlation coefficients calculated at each position while scanning each image before and after rotation correction with the template image 510 is detected as the position of the through hole 310a.

[0084] The hole position detection processing unit 11e, which has determined the position of the through hole 310a, reports the result as described above (step S21). For example, information indicating that the position of the through hole 310a has been detected, or information indicating the position of the through hole 310a, is displayed on the screen of the display unit 14. For example, information indicating the detected position of the through hole 310a is reported to other processing units included in the control unit 11, and another process is executed using the detected position. Alternatively, the detection device 10 may be equipped with a communication unit, and information indicating the detected position of the through hole 310 may be reported to an external device via the communication unit, and the external device may execute a process using the detected position.

[0085] As described above, in the hole position detection system 1 according to this embodiment, the detection device 10 can determine the displacement of part 300 by using the captured image 402 obtained by imaging part 300 with the camera 100. The displacement is determined by dividing it into displacement amounts Mx and My, which indicate translational movement on a plane parallel to the surface of part 300, and displacement amount (displacement angle) Ma, which indicates rotation of part 300, in a Cartesian coordinate system with the origin being the approximate center position of part 300 on the optical axis of the camera 100. The detection device 10 converts the determined displacement amounts Mx, My, and Ma in the part coordinate system into displacement amounts mx, my, and mz in the hole coordinate system set on the surface where the through hole 310a is located. Since the hole coordinate system is set to a three-axis Cartesian coordinate system with the major and minor axes of the through hole 310a, which appears as an ellipse in the captured image, displacement amounts mx and my in the major and minor axis directions, and displacement amount mz, which is related to the change in distance from the camera 100 to the through hole 310a, are obtained. The detection device 10 corrects the major and minor axes of the template image 500 obtained from the reference image 401 by enlarging or shrinking them as necessary, based on the displacement amounts mx, my, and mz. The detection device 10 also corrects the position of the detection area 201 set in the reference image 401 and changes the size of the detection area 201 based on the displacement amounts mx, my, and mz. In this way, by correcting the template image 500 and the detection area 201 in accordance with the positional displacement of the part 300, the position of the through hole 310a can be detected with high accuracy even when the positional displacement of the part 300 occurs.

[0086] In this embodiment, an example is described in which a correlation coefficient is calculated at each position while scanning an image showing a through hole and multiple images obtained by rotating and correcting the said image using a template image, and the position where the correlation coefficient showing the maximum value is obtained is detected as the position of the through hole. The method for detecting through holes is not limited to this, and for example, a predetermined threshold may be prepared, and for images in which the correlation coefficient is less than the threshold, the detection result in that image may be excluded from the comparison of correlation coefficients in advance. For example, if a correlation coefficient exceeding a predetermined threshold is obtained in the image showing the through hole before rotation correction is performed, the image may not be rotated, that is, steps S18 and S19 shown in Figure 3 may not be performed.

[0087] In this embodiment, an example was described in which an image showing a through hole is rotated, and the position of the through hole is searched using a template image of the through hole on the rotated image. However, it is also possible to rotate the template image of the through hole instead of the image showing the through hole, and then search for the position of the through hole using the rotated template image.

[0088] In this embodiment, an example of detecting the position of a through hole has been described, but the process of detecting the position of a through hole includes the process of detecting whether or not a through hole exists. As described above, when the detection area is corrected based on the captured image and the template image is corrected, and the detection of the presence or absence of a through hole is performed using the corrected detection area and template image, this is included in the execution of through hole position detection by the hole position detection system 1.

[0089] The configuration of the hole position detection system 1 shown in this embodiment is functionally schematic, and the configuration of the hole position detection system 1 is not physically limited to this configuration. The form of distribution and integration of each device is not limited to the example described above, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0090] While embodiments of the hole position detection system and hole position detection method according to this disclosure have been described above with reference to the drawings, the configuration and operation of the hole position detection system and each of its constituent devices are not limited to the above embodiments, and may be implemented in various forms with improvements, changes, and modifications based on the knowledge of those skilled in the art, without departing from the spirit of the invention. [Industrial applicability]

[0091] As described above, the hole position detection system and hole position detection method described herein are useful for detecting the position of holes formed in manufactured parts with high accuracy. [Explanation of symbols]

[0092] 1. Hole position detection system 10 Detection device 11 Control Unit 12 Storage section 13 Control section 14 Display section 100 Cameras

Claims

1. A hole position detection system that detects the position of a hole by template matching using an image of the part and a template image of a hole formed in the part, A camera that images parts with holes, A storage unit that stores a template image of the hole, which is generated by cutting out a partial region image including the hole from a reference image obtained by capturing a part at a reference position with the camera, A detection device that, from an image of a part captured by the camera, identifies a first displacement amount indicating the amount of movement of the part from the reference position on a first plane substantially perpendicular to the optical axis of the camera, converts the first displacement amount into a second displacement amount in a direction parallel to the second plane and a third displacement amount in a direction perpendicular to the second plane based on the plane angle between the second plane where the hole is located and the first plane, corrects the template image based on the second and third displacement amounts, and detects the position of the hole using the corrected template image. A hole position detection system characterized by comprising the following features.

2. The storage unit further stores information relating to the detection area of ​​the hole, which is set in a predetermined region including the hole on the reference image. The detection device corrects the detection area based on the second and third displacement amounts, and scans the region on the captured image corresponding to the corrected detection area with the corrected template image to detect the position of the hole. The hole position detection system according to claim 1, characterized in that it is as described above.

3. The detection device is A first coordinate system is established, consisting of orthogonal X1 and Y1 axes set on the surface of the part, and a Z1 axis set corresponding to the optical axis of the camera which is substantially perpendicular to the surface of the part. The movement of the part is determined by the amount of displacement in the X1 axis direction, the amount of displacement in the Y1 axis direction, and the displacement angle around the Z1 axis in the first coordinate system. On the surface with the hole, a second coordinate system is established consisting of an X2 axis corresponding to the major axis of the hole, which appears as an elliptical shape in the captured image, a Y2 axis corresponding to the minor axis, and a Z2 axis passing through the center of the hole and perpendicular to the surface with the hole. Based on the amount of displacement in the X1 axis direction, the amount of displacement in the Y1 axis direction, and the displacement angle around the Z1 axis in the first coordinate system, the movement of the hole accompanying the movement of the part is determined by the amount of displacement in the X2 axis direction, the amount of displacement in the Y2 axis direction, and the amount of displacement in the Z2 axis direction in the second coordinate system. Based on the distance from the camera to the hole and the amount of displacement of the hole in the Z2 direction, the size of the hole as seen in the template image is corrected. Based on the displacement in the X2 axis direction, the major axis of the hole, which is depicted as an ellipse in the template image, is corrected. Based on the displacement in the Y2 axis direction, the minor axis of the hole, which is depicted as an ellipse in the template image, is corrected. The hole position detection system according to claim 1, characterized in that it is as described above.

4. The storage unit further stores information relating to the detection area of ​​the hole, which is set in a predetermined region including the hole on the reference image. The detection device is Based on the distance from the camera to the hole and the amount of displacement of the hole in the Z2 direction, the size of the detection area is corrected. The position of the detection area is corrected based on the displacement in the X2 axis direction and the displacement in the Y2 axis direction. The hole position detection system according to claim 3, characterized in that it is as described above.

5. The detection device is The correlation coefficient is calculated at each position while scanning the captured image with the corrected template image, and the correlation coefficient is calculated at each position while rotating the captured image and scanning the rotated captured image with the corrected template image. The position of the hole is detected based on the comparison results of multiple calculated correlation coefficients. The hole position detection system according to claim 1, characterized in that it is as described above.

6. A hole position detection method that detects the position of a hole by template matching using an image of the part and a template image of a hole formed in the part, A step of generating a template image of the hole by extracting a partial region image including the hole from a reference image obtained by capturing an image of the part at a reference position with a camera, The process involves taking an image of a part with a hole using a camera, A step of determining a first displacement amount, which indicates the amount of movement of the part from the reference position on a first plane substantially perpendicular to the optical axis of the camera, from an image of the part captured by the camera, A step of converting the first displacement amount into a second displacement amount in a direction parallel to the second plane and a third displacement amount in a direction perpendicular to the second plane, based on the plane angle between the second plane having the hole and the first plane, A step of correcting the template image based on the second displacement and the third displacement, A step of detecting the position of the hole using the corrected template image. A hole position detection method characterized by including the following:

Images