Position / focus misalignment detection system

Abstract

It accurately detects camera misalignment and focus errors. [Solution] This misalignment / focus error detection system includes a detection unit that detects whether or not the camera is misaligned in an image taken by the camera of a marker whose contour shape is a circle, and a focus error determination unit that, when the detection unit detects that there is no misalignment of the camera, calculates the contrast ratio of a focus confirmation target included in the image taken by the camera and determines the focus error according to the contrast ratio.

Description

Technical Field

[0005] ,

[0001] This disclosure relates to a misalignment / focus detection system for detecting misalignment / misfocus of a camera. relates to.

Background Art

[0002] An inspection apparatus for inspecting a subject, for example, a defect inspection apparatus for inspecting a defect of an object (e.g., a casting), is equipped with a camera for imaging an image (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The camera mounted on the defect inspection apparatus needs to be accurately aligned with the subject and accurately focused on the object. If there is misalignment of the camera, the object cannot be accurately measured, and even when the inspection results are learned using machine learning, problems such as an increase in the variation of the number of learning images occur. The same applies to defocus. If there is defocus, fine defects of the object cannot be clearly imaged, and the determination accuracy deteriorates. In view of the above problems, an object of the present disclosure is to provide a misalignment / focus detection system that enables accurate detection of misalignment / misfocus of a camera.

Means for Solving the Problems

[0005] The positional misalignment / focus misalignment detection system according to this disclosure includes a detection unit that detects whether or not the camera is misaligned in an image captured by the camera of a marker whose contour shape is a circle, and a focus misalignment determination unit that, when the detection unit detects that there is no camera misalignment, calculates the contrast ratio of a focus confirmation target included in the image captured by the camera and determines the focus misalignment according to the contrast ratio. With this system, since the focus misalignment is determined after it has been determined that there is no camera misalignment, the focus misalignment can also be determined accurately, and the camera's positional misalignment and focus misalignment can be reduced to enable high-precision measurement of the target object. [Brief explanation of the drawing]

[0006] [Figure 1] This is a schematic diagram showing an example of the configuration of a positional displacement detection system according to one embodiment. [Figure 2] This is a schematic diagram illustrating the method for determining the contour shape of marker image 30. [Figure 3] This is a schematic diagram illustrating a method for determining camera positional shifts based on marker images. [Figure 4] This is a schematic diagram explaining distortion correction. [Figure 5] This is a schematic diagram explaining how to determine if the image is out of focus. [Figure 6] This flowchart shows an example of processing by a positional displacement detection system according to one embodiment. [Modes for carrying out the invention]

[0007] This embodiment will be described below with reference to the attached drawings. In the attached drawings, functionally identical elements may be indicated by the same number. The attached drawings show embodiments and implementation examples in accordance with the principles of this disclosure, but they are for the purpose of understanding this disclosure and are not to be used in any way to restrict the interpretation of this disclosure. The descriptions in this specification are merely typical examples and do not limit the claims or applications of this disclosure in any way.

[0008] While this embodiment is described in sufficient detail for those skilled in the art to implement the disclosure, it is important to understand that other implementations and forms are possible, and that the configuration and structure can be modified and various elements replaced without departing from the scope and spirit of the technical idea of ​​this disclosure. Therefore, the following description should not be construed as limiting to this.

[0009] Figure 1 is a schematic diagram showing an example of the configuration of a positional misalignment / focus misalignment detection system 1 according to one embodiment. The lower part of Figure 1 also shows an example of the observation field of view on the surface 5 where the object W, which is the target of positional misalignment / focus misalignment determination, is placed.

[0010] The misalignment / focus error detection system 1 is a system that detects misalignment / focus errors of a camera 2, for example, mounted on an industrial robot. The misalignment / focus error detection system 1 may be a computer equipped with a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc., which analyzes the captured image data Im taken by the camera 2.

[0011] Camera 2 is mounted, for example, on the housing of an industrial robot and moves in three-dimensional space with 6 degrees of freedom. These 6 degrees of freedom include 3 degrees of freedom in which Camera 2 moves along the X, Y, and Z axes, respectively, and 3 degrees of freedom in which Camera 2 rotates around the X, Y, and Z axes, respectively.

[0012] Camera 2 stops at a predetermined shooting position, performs a focusing operation using its internal focusing mechanism, and then photographs the object W while stationary. Based on the image of the object W captured by Camera 2, the object W is inspected (e.g., defect inspection). The object W is, for example, a part manufactured by a casting method.

[0013] In inspections using camera 2, if camera 2's position is misaligned from the predetermined shooting position when it moves to and stops at a predetermined shooting position, this misalignment will cause a shift in the image captured by camera 2. This image shift can lead to a decrease in inspection accuracy. When machine learning is used in inspections using camera 2, image shifts can also lead to an increase in learning variations. The position / focus shift detection system 1 reduces the position and focus shifts of camera 2, thereby improving inspection accuracy, reducing learning variations in machine learning, and enabling more accurate inspections.

[0014] As shown in Figure 1, the misalignment / focus error detection system 1 is connected to the camera 2 via wired or wireless means for communication. The camera 2 includes an optical system 21 and an imaging unit 22 such as a CCD that captures an image from the optical system 21. The misalignment / focus error detection system 1 receives image data Im of the marker 3 captured by the camera 2 as input.

[0015] Marker 3 is a figure used to detect the positional displacement of camera 2. Marker 3 consists of, for example, four markers, each with a circular outline. However, the number of markers 3 is not limited to four; there may be two or more. Marker 3 may be affixed to the placement surface 5 of the object W at a predetermined calibration position, or it may be engraved on the placement surface 5. The predetermined calibration position may be the same as the predetermined shooting position. Camera 2 photographs the marker 3 placed at the predetermined calibration position.

[0016] Furthermore, the optical system 21 includes a reticle plate (not shown) and is configured to project focus confirmation targets P and a centering reticle C into the observation field. Multiple focus confirmation targets P are projected into the observation field, and one example is a stripe-shaped target with the vertical direction of the observation field as its longitudinal direction. The centering reticle C, for example, is a cross-shaped target projected near the center of the observation field. The arrangement and shape of the focus confirmation targets P are not particularly limited, but for example, as shown in Figure 1, four focus confirmation targets P can be arranged along the direction of the cross of the cross-shaped centering reticle C. Two focus confirmation targets P aligned in the direction of the horizontal line of the cross function as targets for confirming focus in the meridional direction (the concentric direction of the lenses of the optical system 21), while two focus confirmation targets P aligned in the direction of the vertical line of the cross function as targets for confirming focus in the sagittal direction (the radial direction from the center of the lenses of the optical system 21).

[0017] The misalignment / focus error detection system 1 comprises a correction unit 11, a judgment unit 12, a detection unit 13, a focus error determination unit 14, and an output unit 15. The correction unit 11, judgment unit 12, detection unit 13, and focus error determination unit 14 can be implemented by computer programs stored in a computer or by special-purpose circuits.

[0018] The correction unit 11 corrects the distortion of the captured image data Im, and also has the function of further processing the distortion-corrected image with binarization and black and white inversion. The determination unit 12 determines whether the contour shape of the marker image 30 is circular or elliptical in the captured image Im of the marker 3 captured by the camera 2, which has a circular contour shape. As shown in Figure 2, the detection unit 13 uses the positions of the two focal points 32 of the ellipse in matrix calculations when the contour shape of the marker image 30 is elliptical. The focus shift determination unit 14 determines whether or not the object W is out of focus based on the grayscale ratio of the focus confirmation material P. The output unit 15 outputs the detection results from the detection unit 13 and the focus shift determination unit 14 to an external device.

[0019] In FIG. 2, the marker image 30a is an image of the marker 3 whose contour shape in the captured image is a circle. On the other hand, the marker image 30b is an image of the marker 3 whose contour shape in the captured image is an ellipse. Note that the marker image 30 is a collective notation for the marker image 30a and the marker image 30b.

[0020] When the optical axis of the optical system 21 is substantially orthogonal to the placement surface 5 of the object W on which the marker 3 is placed (when the camera 2 is not rotating with respect to the X-axis and the Y-axis as the respective rotation centers), in the captured image data Im, a marker image 30a having a circular contour shape is obtained. On the other hand, when the camera 2 is rotating with at least one of the X-axis and the Y-axis as the rotation center and the optical axis of the optical system 21 is not orthogonal to the placement surface 5, an elliptical marker image 30b is obtained. The center of gravity 31 is the center of gravity of each of the marker image 30a and the marker image 30b. The foci 32 are the two foci of the marker image 30b.

[0021] When determination is made using only the position of the center of gravity 31 of the marker image 30, although the position shift / focus shift detection system can detect the position shift of the camera 2 in the plane, it cannot detect the position shift of the camera 2 in the three-dimensional space (for example, the rotational position shift of the camera 2 with respect to the X-axis, the Y-axis, and the Z-axis as the respective rotation centers). As a result, there is a risk that the detection accuracy of the position shift of the camera 2 will be low.

[0022] Therefore, in the positional misalignment / focus misalignment detection system 1 of this embodiment, the determination unit 12 determines whether the contour shape of the marker image 30 is a circle or an ellipse. If the contour shape of the marker image 30 is an ellipse, the detection unit 13 uses the positions of the two foci 32 of the ellipse in matrix calculations. As a result, the positional misalignment / focus misalignment detection system 1 can detect not only the positional misalignment of the camera 2 in the plane but also the positional misalignment of the camera 2 in three-dimensional space, thus enabling highly accurate detection of the camera 2's positional misalignment. In other words, the positional misalignment / focus misalignment detection system 1 of this embodiment can detect complex positional misalignments of the camera 2 with six degrees of freedom. By reducing the positional misalignment of the camera 2, the accuracy of inspections using the camera 2 is improved. Furthermore, when machine learning is used for inspections using the camera 2, the number of learning variations is reduced, and the inspection accuracy by machine learning can be improved.

[0023] The determination unit 12 determines whether or not there is a positional shift of the camera 2 in three-dimensional space by determining whether the contour shape of the marker image 30 is a circle or an ellipse. For example, the determination unit 12 extracts the contour of the marker image 30 by image processing and detects the maximum amount of deviation from a circle in the extracted contour shape. The maximum amount of deviation from a circle can be the difference between the maximum width of the contour shape and the diameter of the circle, etc. The determination unit 12 determines that the contour shape of the marker image 30 is a circle if the maximum amount of deviation from a circle is less than a predetermined threshold. If the deviation from a circle is greater than or equal to a predetermined threshold, the determination unit 12 determines that the contour shape of the marker image 30 is an ellipse. However, the method of determining whether it is a circle or an ellipse is not limited to this.

[0024] The detection unit 13 performs a matrix calculation based on the positions of the two focal points 32 of the ellipse, for example, the marker image 30b, if the contour shape of the marker image 30 is elliptical. This matrix calculation is, for example, an affine transform or an inverse affine transform. Affine transforms and inverse affine transforms are coordinate transformations that use matrices to perform scaling, rotation, translation, etc., of an image.

[0025] Refer to Figure 3 to explain the details of the camera 2 displacement detection method. Figure 3 shows six states P1 to P6 of camera 2 displacement. In Figure 3, the reference image 30S virtually represents the position of the ideal marker image 30 with no displacement. The dashed line in the marker image 30 represents the marker image with displacement.

[0026] State P1 is a state in which the marker image 30 has shifted position relative to the reference image 30S in both the X and Y directions. State P2 is a state in which the marker image 30 has shifted position relative to the reference image 30S in both the X and Z directions.

[0027] State P3 is a state in which the marker image 30 has shifted position relative to the reference image 30S in both the Y and Z directions. In states P2 and P3, the camera 2 shifts position in the Z direction, which changes the imaging magnification by the optical system 21, causing the marker image 30 to appear larger relative to the reference image 30S.

[0028] State P4 is a state in which the marker image 30 is rotated and displaced relative to the reference image 30S with the Z-axis as the center of rotation. State P5 is a state in which the marker image 30 is rotated and displaced relative to the reference image 30S with the Y-axis as the center of rotation. Due to the rotation with the Y-axis as the center of rotation, the contour shape of the marker image 30 is an ellipse with the Y-direction as the major axis. State P6 is a state in which the marker image 30 is rotated and displaced relative to the reference image 30S with the X-axis as the center of rotation. Due to the rotation with the X-axis as the center of rotation, the contour shape of the marker image 30 is an ellipse with the X-direction as the major axis.

[0029] The detection unit 13 performs matrix calculations based on the marker images 30 in each of the states P1 to P6, or in a combined state of states P1 to P6. This allows the detection unit 13 to detect the positional displacement of the camera 2 in the plane and the positional displacement of the camera 2 in three-dimensional space. The detection unit 13 outputs the camera 2 positional displacement detection result to an external device via the output unit 15. Note that the state of camera 2's positional displacement is not limited to states P1 to P6 as shown in Figure 4, but may be various states.

[0030] If the contour shape of the marker image 30 is a circle, the detection unit 13 performs a matrix calculation to perform an affine transform or inverse affine transform based on the position of the centroid 31 of the circle, for example, the marker image 30a. This allows the detection unit 13 to detect the positional displacement of the camera 2 in the plane. If the contour shape of the marker image 30 is a circle, the detection unit 13 may also calculate the rotational displacement of the camera 2 with the Z-axis as the rotation center. Furthermore, if the contour shape of the marker image 30 is a circle, the detection unit 13 can treat the rotational displacement of the camera 2 with the X-axis and Y-axis as rotation centers as approximately zero.

[0031] On the other hand, the image data Im captured by camera 2 may exhibit distortion as shown in Figure 5, depending on the specifications of the optical system 21. In Figure 4, the rectangular image 40 shown by the dashed line is a captured image without distortion. The barrel-shaped image 41 shown by the solid line is an image in which the rectangular image 40 has been deformed into a barrel shape due to negative distortion. The pincushion-shaped image 42 shown by the solid line is an image in which the rectangular image 40 has been deformed into a pincushion shape due to positive distortion. For example, the wider the field of view of the optical system 21, the greater the distortion. Distortion can also be described as image distortion.

[0032] When the contour shape of the marker image 30 is deformed due to distortion, the accuracy of determining whether the contour shape of the marker image 30 is a circle or an ellipse decreases. This decrease in determination accuracy may reduce the accuracy of detecting the positional shift of the camera 2. In particular, when a wide-angle lens or fisheye lens with large distortion is used as the optical system 21, the decrease in detection accuracy may become more pronounced. Fisheye lenses have a large effect of distortion, and may incorrectly identify areas as out of focus even if they are not actually out of focus.

[0033] Therefore, in the position / focus misalignment detection system 1 of this embodiment, the correction unit 11 corrects the distortion of the captured image Im. Furthermore, the position / focus misalignment detection system 1, using the determination unit 12, determines whether the contour shape of the marker image 30 is a circle or an ellipse in the captured image data Im from which the distortion has been corrected by the correction unit 11. The correction unit 11 can use various distortion correction algorithms as the distortion correction process.

[0034] By using distortion-corrected captured image data Im, the determination unit 12 can accurately determine whether the contour shape of the marker image 30 is a circle or an ellipse. This enables the detection of positional displacement of the camera 2 in both a planar and three-dimensional space, allowing for highly accurate detection of the camera 2's positional displacement. In particular, when using a wide-angle lens or fisheye lens with significant distortion as the optical system 21, the decrease in detection accuracy is suppressed, and the accuracy of positional displacement detection can be increased.

[0035] Referring to Figure 5, the operation of the focus deviation detection unit 14 will be explained. The focus deviation detection unit 14 detects the intensity of the focus confirmation target P, calculates the intensity ratio C, and detects whether or not there is a focus deviation based on whether or not the intensity ratio C is greater than a predetermined threshold. The intensity ratio C can be calculated, for example, as C = (Imax - Imin) / (Imax + Imin) when the maximum value of the intensity of the focus confirmation target P is calculated to be Imax and the minimum value is calculated to be Imin.

[0036] As mentioned earlier, the focus-checking target P is a striped mark along the vertical direction of the observation field. When the image is in focus, the contrast between the black and white areas increases (the difference between Imax and Imin increases), and the contrast ratio C also increases. Conversely, when the image is out of focus, the contrast ratio C decreases. By determining the relationship between the contrast ratio and the threshold, it is possible to determine whether or not the image is out of focus.

[0037] Figure 6 is a flowchart illustrating an example of processing by the misalignment / focus error detection system 1 according to one embodiment. The misalignment / focus error detection system 1 starts the processing shown in Figure 6 when it receives a captured image Im from the camera 2 as a starting condition.

[0038] First, in step S1, the correction unit 11 corrects the distortion of the captured image Im input from the camera 2. Next, in step S2, the captured image data Im, after distortion correction, is binarized and inverted in black and white using image processing. The correction unit 11 passes the binarized and inverted captured image data Im to the judgment unit 12.

[0039] Next, in step S3, the determination unit 12 determines whether the contour shape of the marker image 30 included in the captured image data Im is an ellipse or a circle, according to the method described above. If it is determined to be an ellipse (Yes in step S3), the detection unit 13 determines the positions of the two focal points 32 in the ellipse of the marker image 30 (step S4).

[0040] Next, in step S5, the detection unit 13 uses the positions of the two foci 32 of the ellipse in a matrix operation. This allows the detection unit 13 to detect not only the positional displacement of the camera 2 in the plane, but also the positional displacement of the camera 2 in three-dimensional space. The detection unit 13 passes the detection results of the positional displacement of the camera 2 in the plane and the positional displacement of the camera 2 in three-dimensional space to the output unit 15.

[0041] On the other hand, if it is determined in step S3 that the marker image 30 is not an ellipse (step S3 No.), the process proceeds to step S11.

[0042] In step S11, it is determined whether or not there is a misalignment of camera 2 according to the positional relationship between the centering reticle C and the marker image 30. If it is determined that the misalignment is less than a predetermined threshold (No. in step S11), the industrial robot is adjusted / corrected, and other corrections are made to the captured image data Im as appropriate, and measurements are taken based on the captured image data Im of the object W.

[0043] On the other hand, if it is determined that the camera 2's misalignment is greater than a predetermined threshold (Yes in step S11), the captured image of the placement surface 5 is acquired again and distortion correction is performed (step S13). Here, binarization and black and white inversion as in step S2 are not performed in order to acquire the grayscale ratio, which will be described later. Subsequently, the focus misalignment determination unit 14 acquires the grayscale information of the focus confirmation target P and calculates the grayscale ratio C (step S14). If the grayscale ratio C is greater than or equal to the threshold, it can be determined that there is no focus misalignment, and conversely, if it is less than the threshold, it can be determined that there is a focus misalignment (step S15). In addition to the grayscale ratio C, the spatial frequency of the image of the focus confirmation target P can also be calculated and used to determine the focus misalignment.

[0044] As described above, according to this embodiment, distortion correction is performed on the captured image data Im, and then positional displacement is determined based on the marker image 30, so the positional displacement of the camera 2 can be accurately determined. Furthermore, after it is determined that there is no positional displacement, the focus deviation is determined based on the focus confirmation target P, so the focus deviation can also be accurately determined. Therefore, according to this embodiment, the positional displacement and focus deviation of the camera 2 can be reduced, and the measurement of the target object W can be performed with high accuracy.

[0045] This disclosure is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are described in detail for the purpose of explaining this disclosure clearly, and are not necessarily limited to having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations. [Explanation of Symbols]

[0046] 1…Position deviation detection system 2…Camera 3… Marker P...Focus confirmation target C... Reticle for centering 11...Correction section 12…Judgment department 13...Detection unit 14…Focus detection unit 15…Output section 21...Optical system 22…IMG Department 30, 30a, 30b... Marker images 30S…Reference image 31...center of gravity 32…Focus

Claims

1. A position / focus misalignment detection system for detecting camera focus misalignment, A detection unit that detects whether or not the camera is misaligned, When the detection unit detects that there is no positional misalignment of the camera, the focus misalignment determination unit calculates the contrast ratio of the focus confirmation target included in the image captured by the camera and determines the focus misalignment according to the contrast ratio. A positional misalignment / focus misalignment detection system having the following features.

2. The system further includes a determination unit that determines whether the contour shape of the marker image is a circle or an ellipse in the image captured by the camera of a marker whose contour shape is a circle. The positional misalignment / focus misalignment detection system according to claim 1, wherein the detection unit uses the positions of the two foci of the ellipse in matrix calculations when the contour shape of the marker image is an ellipse, and detects whether or not the camera is misaligned when the contour shape of the marker image is a circle.

3. It has a correction unit that corrects the distortion of the captured image, The positional / focus misalignment detection system according to claim 2, wherein the determination unit determines whether the contour shape of the marker image of the marker is a circle or an ellipse in the captured image in which distortion has been corrected by the correction unit.

4. The positional misalignment / focus misalignment detection system according to claim 3, wherein the correction unit, when it detects from the detection unit that there is no positional misalignment of the camera, acquires the captured image again, and after correction by the correction unit, calculates the contrast ratio of the focus confirmation target.

Images