Positional deviation amount calculation method for detecting positional deviation of imaging device

Abstract

To provide a technique for accurately performing an inspection using a photographed image even when a positional deviation of an imaging device with respect to an inspection facility occurs.SOLUTION: A positional deviation amount calculation method comprises steps of: causing an imaging device to image a reference plate having a frame part arranged so as to match an imaging range of the imaging device and a recognition pattern part arranged at a predetermined position on the inner side of the frame part to acquire a photographed image; cutting out a region with the predetermined width along an edge of the photographed image and determining the presence / absence of a positional deviation with an occupancy area ratio of the frame part in the cut-out region; and dividing the photographed image being the division object into a plurality of unit regions in the predetermined size and comparing the position in the photographed image of the maximum ratio region being the unit region where the occupancy area ratio of the recognition pattern part becomes maximum out of the plurality of divided unit regions with the position in a master image to calculate a positional deviation amount.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a positional deviation amount calculation method for detecting positional deviation of an imaging device. [Background technology]

[0002] In the appearance inspection of articles such as castings, defects in the inspected article may be determined by comparing the feature amount of the captured image of the inspected article with the feature amount of a reference image. Patent Document 1 discloses an appearance inspection method for detecting defects in the inspected article by moving an imaging device attached to an orthogonal robot to capture images of the inspected article from multiple points, and comparing an inspection image created by subjecting the captured images to noise removal processing with a reference image created by subjecting an image of a defect-free non-defective article to noise removal processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2001-337044 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the appearance inspection method described in Patent Document 1, due to the operation of the Cartesian robot, etc., the positional deviation of the robot relative to the inspection equipment and the positional deviation of the imaging device relative to the robot may occur. These positional deviations lead to the positional deviation of the imaging device relative to the inspection equipment. For this reason, in the appearance inspection in which the feature amount of the captured image is compared with the feature amount of the reference image as described above, the positional deviation of the imaging device is detected as a feature amount, and there is a risk that defects of the inspected product are not properly detected. In addition, in the appearance inspection using machine learning, in the teaching stage, due to the positional deviation of the imaging device described above, the data set may vary, and as a result, sufficient judgment accuracy may not be obtained. In the inspection stage, due to the positional deviation of the imaging device described above, there is a risk that a part of the inspected product does not fall within the imaging area, and the entire inspected product may not be inspected. [Means for solving the problem]

[0005] According to one embodiment of the present disclosure, there is provided a method for calculating a positional deviation of an imaging device, the method for calculating a positional deviation of an imaging device comprising the steps of: capturing an image of a reference plate having a frame portion arranged in accordance with an imaging range of the imaging device and a recognition pattern portion arranged at a predetermined position inside the frame portion, by the imaging device to obtain an image; cutting out an area of ​​a predetermined width along an edge of the captured image, and judging the presence or absence of the positional deviation based on an occupied area ratio of the frame portion in the cut-out area; and dividing the captured image into a plurality of unit areas of a predetermined size as a division target, and calculating the amount of the positional deviation by comparing a position in the captured image of a maximum ratio area, which is a unit area having a maximum occupied area ratio of the recognition pattern portion among the divided plurality of unit areas, with a position in the master image of the maximum ratio area when a master image obtained by imaging the reference plate in a state without the positional deviation is used as the division target. According to the positional deviation amount calculation method of this embodiment, a captured image is acquired, an area of ​​a predetermined width is cut out along the edge of the captured image, and the presence or absence of a positional deviation is determined based on the occupied area ratio of the frame part in the cut out area, so that the presence or absence of a positional deviation can be accurately specified. In addition, the positional deviation amount is calculated by comparing the position in the captured image of the maximum proportion area among the unit areas obtained by dividing the captured image as the division target with the position in the master image of the maximum proportion area among the unit areas obtained by dividing the master image as the division target, so that the amount of positional deviation can be accurately calculated. From these points, according to the positional deviation amount calculation method of the above embodiment, even if a positional deviation occurs in the imaging device relative to the inspection equipment, the presence or absence of a positional deviation can be accurately specified, and the amount of the positional deviation can be accurately specified, so that the amount of the positional deviation can be used to perform an inspection using the captured image with high accuracy. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is an explanatory diagram illustrating a schematic configuration of a position deviation calculation device. [Diagram 2] FIG. [Diagram 3] 11 is a flowchart showing a procedure for a position deviation amount calculation process. [Figure 4] FIG. 11 is an explanatory diagram showing a binarized image and an area cut out in step S30. [Diagram 5] FIG. 11 is an explanatory diagram for explaining image processing (X-axis direction) in step S50. [Figure 6] FIG. 11 is an explanatory diagram for explaining image processing (Z-axis direction) in step S50. [Figure 7] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 8] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 9] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 10] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 11]FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 12] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 13] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 14] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 15] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 16] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. [Figure 17] FIG. 11 is an explanatory diagram of another embodiment of the recognition pattern unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] A. Embodiment: A1. Equipment configuration: Fig. 1 is an explanatory diagram showing a schematic configuration of a position deviation calculation device 100. Fig. 1 shows arrows representing three mutually orthogonal coordinate axes, namely, X, Y, and Z axes. In the following description, when specifying a direction, positive and negative signs are used in combination to indicate the direction, with "+" indicating the positive direction indicated by the arrow and "-" indicating the negative direction opposite to the direction indicated by the arrow.

[0008] The positional deviation calculation device 100 includes a robot arm 10, an imaging device 20, a reference plate 30, and a processing unit 40. The robot arm 10, the imaging device 20, and the processing unit 40 function as a part of the positional deviation calculation device 100, and are also used as functional units arranged in a manufacturing line of a predetermined type of workpiece, which captures an image of the workpiece as an inspection target and performs a visual inspection of the workpiece based on the captured image. The "predetermined type of workpiece" may be, for example, a casting. Specifically, a casting having a complex structure such as an engine block may be included. Such a casting may be damaged, such as chipped, during the casting process, and the presence or absence of such damage is determined by visual inspection. The "predetermined type of workpiece" may be any type of product that requires visual inspection, not limited to a casting.

[0009] The positional deviation amount calculation device 100 judges whether or not there is a positional deviation (hereinafter, simply referred to as "positional deviation" unless otherwise specified) of the imaging device 20 with respect to the inspection facility, and if there is a positional deviation, specifies the amount of the positional deviation. Here, "inspection facility" means a facility in which the robot arm 10 in the above-mentioned manufacturing line may be installed. As will be described later, the robot arm 10 moves significantly in a short time in order to capture images of the workpiece to be inspected in various directions and postures. Due to vibrations or the like caused at that time, the robot arm 10 may be misaligned with respect to the inspection facility, and the imaging device 20 may be misaligned with respect to the robot arm 10. These positional deviations cause a positional deviation of the imaging device 20 with respect to the inspection facility. Therefore, the positional deviation amount calculation device 100 specifies whether or not there is such a positional deviation, and specifies the amount of the positional deviation, as described above.

[0010] The robot arm 10 is a multi-joint robot equipped with rotating parts J1 to J6. The imaging device 20 is attached to the arm tip 11 of the robot arm 10. By controlling the rotating parts J1 to J6 of the robot arm 10, the imaging device 20 can capture images in various directions and attitudes. In Fig. 1, the rotation axis of the rotating part J6 is parallel to the Y axis.

[0011] The reference plate 30 is installed parallel to the XZ plane at a position where it can be imaged by the imaging device 20. When the imaging device 20 images the reference plate 30, the imaging device 20 faces the reference plate 30 and images it from a predetermined position that is an imaging distance d away along the Y-axis direction. The imaging distance d is a distance that is set in consideration of the size of the element of the imaging device 20, the focal length, etc.

[0012] FIG. 2 is a plan view of the reference plate 30. The reference plate 30 is composed of a plate-like member having a rectangular shape in a plan view. In the reference plate 30, a frame portion 31 and a recognition pattern portion 32 are arranged on the surface in the -Y direction. The base color of the reference plate 30 is white, and the frame portion 31 and the recognition pattern portion 32 are black. The color of the frame portion 31 and the recognition pattern portion 32 may be any color other than black as long as it is different from the base color of the reference plate 30. In this embodiment, both the frame portion 31 and the recognition pattern portion 32 are drawn on the plate-like member. Note that the frame portion 31 and the recognition pattern portion 32 may be formed as separate bodies from the plate-like member using a member such as resin and bonded to the surface of the plate-like member in the -Y direction.

[0013] The frame portion 31 has a square shape arranged to match the imaging range of the imaging device 20. The frame portion 31 is arranged so that the entire frame portion 31 falls within the imaging range of the imaging device 20 when an image is captured without any positional deviation of the imaging device 20. In other words, the imaging position of the imaging device 20 is set to a position where the entire frame portion 31 falls within the captured image when an image of the reference plate 30 is captured without any positional deviation of the imaging device 20.

[0014] The recognition pattern section 32 is used to calculate the amount of misalignment in the misalignment amount calculation process described later. The recognition pattern section 32 is disposed at a predetermined position inside the frame section 31. In this embodiment, the recognition pattern section 32 is composed of scaled straight lines 321 and 322 that are perpendicular to each other. The scaled straight line 321 is disposed along the Z axis direction at the center position in the X axis direction within the rectangular area defined by the frame section 31. The scaled straight line 322 is disposed along the X axis direction at the center position in the Z axis direction within the rectangular area defined by the frame section 31. The main straight line of the two scaled straight lines 321 and 322 is also called a "center line". The thickness of the center line is set so that when an image is captured by the imaging device 20 from a predetermined position, the number of pixels of the width of the center line in the captured image is approximately the same as the number of pixels of the width of a "unit area" in the misalignment amount calculation process described later.

[0015] The interval between the scale lines of each of the scaled straight lines 321 and 322 is set in consideration of the calculation accuracy of the amount of misalignment. By adding the scale lines to the center line, it becomes easier for an operator to visually check the amount of misalignment calculated by the misalignment amount calculation process described later.

[0016] The processing unit 40 shown in Fig. 1 is configured by a computer having a CPU and a memory. The CPU of the processing unit 40 executes a positional deviation amount calculation program stored in advance in the memory to perform a positional deviation amount calculation process described below. In this embodiment, the processing unit 40 also performs a visual inspection of the workpiece in addition to the positional deviation amount calculation process. Note that the positional deviation amount calculation process and the visual inspection of the workpiece may be performed by different processing units.

[0017] A2. Calculation process of the amount of positional deviation of the imaging device 20: 3 is a flowchart showing the procedure of the positional deviation amount calculation process. The positional deviation amount calculation process is started by inputting an execution command to the processing unit 40 by an operator. The input of such an execution command may be incorporated in the program in advance so that the execution is automatically performed every time inspection of a predetermined number of workpieces is completed. Also, for example, in the case where the manufacturing line is set to start every morning and stop every evening, the positional deviation amount calculation process may be incorporated in the program in advance so that the calculation is automatically performed at the start of the line every morning.

[0018] In step S10, the imaging device 20 captures an image of the reference plate 30 from a predetermined position, and obtains a captured image. As described above, if there is no positional deviation, the captured image includes the entire frame portion 31 and the recognition pattern portion 32.

[0019] In step S20, a binarized image 50 of the captured image is created. The binarization process can reduce the influence of the imaging environment, such as the reflection of lighting, on the captured image. The threshold value of the binarization process is set in consideration of the imaging environment. If it is not necessary to consider the influence of the imaging environment on the captured image, the binarization process may be omitted.

[0020] A step of determining whether or not there is a positional deviation in the binarized image 50 is performed. The step of determining whether or not there is a positional deviation is made up of steps S30 to S42 in the flowchart. Each of steps S30 to S42 will be described in detail below.

[0021] In step S30, a region 51 of a predetermined width is cut out along an edge of the binarized image 50 by image processing. FIG. 4 is an explanatory diagram showing the binarized image 50 and the region 51 cut out in step S30. In step S30, a total of four regions 51 are cut out, which are shown hatched in FIG. 4. As described above, the region 51 located inward from each edge of the binarized image 50 by a predetermined width is cut out, so that each region 51 has a rectangular (rectangular) appearance in a plan view. In step S30, after the region 51 is cut out, the occupied area ratio of the white region, which is the portion excluding the frame portion 31 and the recognition pattern portion 32, in the cut out region 51 is calculated.

[0022] The proportion of the area occupied by the white region in the cut-out region 51 varies depending on the position of the frame portion 31 in the captured image. For example, if the imaging device 20 is misaligned in the -Z direction, the position of the frame portion 31 in the captured image will be shifted in the +Z direction compared to when the imaging device 20 is not misaligned. Since the frame portion 31 in the captured image may have linear distortion caused by lens distortion or the like, if the amount of misalignment of the imaging device 20 is equal to or greater than a certain amount, a portion of the +Z side edge of the frame portion 31 may not be included in the imaging range. As a result, the area occupied by the frame portion 31 in the cut-out region 51 decreases, and the proportion of the area occupied by the white region increases.

[0023] Furthermore, when the imaging device 20 is misaligned in the -Y direction, the center position of the frame portion 31 in the captured image remains the same as when the imaging device 20 is not misaligned. However, since the imaging device 20 captures an image from a position farther away from the reference plate 30 than the normal position, the frame portion 31 in the captured image becomes smaller than normal. This reduces the area occupied by the frame portion 31 in the cut-out region 51, and increases the proportion of the area occupied by the white region.

[0024] 3, in step S40, if the proportion of the area occupied by the white region in any of the cut-out regions 51 is smaller than a predetermined threshold, it is determined that there is no positional deviation (step S41). After completion of step S41, the positional deviation amount calculation process ends. In contrast, if the proportion of the area occupied by the white region in any of the cut-out regions 51 is equal to or greater than the threshold, it is determined that there is positional deviation (step S42). The threshold is set in consideration of the thickness of the frame portion 31, the imaging distance d, etc.

[0025] If it is determined that there is no positional deviation (step S41), a signal indicating that there is no positional deviation is sent from the processing unit 40 to a programmable logic controller (PLC) of the inspection equipment. If it is determined that there is a positional deviation (step S42), an alarm indicating that there is a positional deviation is sent from the processing unit 40 to the PLC of the inspection equipment.

[0026] After step S42 is completed, a step of calculating the amount of positional deviation of the imaging device 20 is performed. The step of calculating the amount of positional deviation is made up of steps S50 to S70 in the flowchart. Each of steps S50 to S70 will be described in detail below.

[0027] In step S50, the binarized image 50 is divided into a plurality of unit regions of a predetermined size along the X-axis direction and the Z-axis direction by image processing. Figures 5 and 6 are explanatory diagrams for explaining the image processing in step S50. First, the division along the X-axis direction will be explained below.

[0028] 5, the binarized image 50 is divided into N equal rectangular unit areas along the X-axis direction by image processing. The number of pixels in the X-axis direction of a unit area is 1 / N of the number of pixels in the X-axis direction of the binarized image 50. The number of pixels in the Z-axis direction of a unit area is the same as the number of pixels in the Z-axis direction of the binarized image 50. The number of divisions N is a natural number that is set in advance depending on the detection accuracy of the amount of positional deviation, the number of pixels of the imaging device 20, etc. The unit areas are assigned area numbers from 1 to N.

[0029] As shown in FIG. 3, in step S60, the occupied area ratio of the white area, which is the portion excluding the recognition pattern portion 32, in the unit area is calculated, and the area number of the unit area having the smallest occupied area ratio of the white area among unit areas 1 to N is obtained. This unit area is the unit area in which the occupied area ratio of the black area, which is the recognition pattern portion 32, in the unit area is the largest, and is hereinafter referred to as the "maximum proportion area." The area number of the maximum proportion area is also referred to as the "maximum proportion area number." In FIG. 5, the maximum proportion area is a hatched area, which is an area including the center line of the scaled straight line 321. In this embodiment, the maximum proportion area number in the X-axis direction is "i."

[0030] Similar to the X-axis direction, the binarized image 50 is equally divided into N rectangular unit areas along the Z-axis direction by image processing, as shown in Fig. 6. The number of pixels in the X-axis direction of each unit area is the same as the number of pixels in the X-axis direction of the binarized image 50. The number of pixels in the Z-axis direction of each unit area is 1 / N of the number of pixels in the Z-axis direction of the binarized image 50.

[0031] For the unit area divided along the Z axis, the maximum proportion area is also identified, and the maximum proportion area number is obtained. In Fig. 6, the maximum proportion area is the hatched area, which is the area including the center line of the scaled straight line 322. In this embodiment, the maximum proportion area number in the Z axis direction is "j".

[0032] As shown in FIG. 3, in step S70, the amount of positional deviation in the binarized image 50 is calculated by comparing the binarized image 50 with the master image. The master image is an image obtained by capturing an image of the reference plate 30 in a state where there is no positional deviation of the imaging device 20. In the processing unit 40, the maximum percentage area numbers in the X-axis direction and the Z-axis direction, which are acquired by performing the above-mentioned image processing on the master image as a division target, are specified in advance through experiments or the like and set. By comparing the maximum percentage area number in the binarized image 50 with the maximum percentage area number in the master image in each of the X-axis direction and the Z-axis direction, the amount of positional deviation in the binarized image 50 can be calculated in pixel units. After completion of step S70, the positional deviation calculation process ends.

[0033] In this embodiment, the amount of misalignment in pixels obtained by the above-described misalignment amount calculation process is used as follows. First, the obtained amount of misalignment in pixels is converted into an amount of misalignment in mm (millimeters) taking into consideration the imaging distance d, the focal length of the lens of the imaging device 20, and the like. Next, based on the amount of misalignment in mm, an operator corrects the position of the imaging device 20. At this time, the calculated amount of misalignment in mm may be displayed on a display device (not shown) connected to the processing unit 40.

[0034] According to the method for calculating the amount of misalignment of the imaging device 20 in the embodiment described above, a captured image is acquired, a region 51 of a predetermined width is cut out along the edge of the acquired captured image, and the presence or absence of misalignment is determined based on the occupied area ratio of the frame portion 31 in the cut-out region 51, so that the presence or absence of misalignment can be accurately specified. In addition, the amount of misalignment is calculated by comparing the position in the captured image of the maximum proportion area among the unit areas obtained by dividing the captured image as a division target with the position in the master image of the maximum proportion area among the unit areas obtained by dividing the master image as a division target, so that the amount of misalignment can be accurately calculated. From these points, according to the method for calculating the amount of misalignment of the embodiment described above, even if a misalignment of the imaging device 20 occurs with respect to the inspection equipment, the presence or absence of misalignment can be accurately specified, and the amount of the misalignment can be accurately specified, so that the inspection using the captured image can be performed accurately by utilizing the amount of misalignment.

[0035] B. Other embodiments: (B1) In the above embodiment, the frame portion 31 has a square shape, but the present disclosure is not limited thereto. The frame portion 31 may have any shape as long as it indicates the imaging range of the imaging device 20, and may have, for example, a dot-shaped external shape arranged to match the imaging range of the imaging device 20, or a curved external shape indicating each corner of the imaging range.

[0036] (B2) In the above embodiment, the recognition pattern section 32 is composed of the scaled straight lines 321 and 322 that are orthogonal to each other, but the present disclosure is not limited to this. The recognition pattern section 32 may be in any position and any shape as long as it is inside the frame section 31. For example, the recognition pattern section 32 may be in the shape of a black circle arranged at a position that does not coincide with the center of the frame section 31, as in the recognition pattern sections 32A to 32D in Figs. 7 to 10. Also, the recognition pattern section 32 may be in the shape of a black circle arranged at a position that coincides with the center of the frame section 31, as in the recognition pattern section 32E in Fig. 11. Also, the recognition pattern section 32 may be in the shape of a black diamond, as in the recognition pattern sections 32F and 32G in Figs. 12 and 13. Also, the recognition pattern section 32 may be in the shape of a black triangle having a center that coincides with the center of the frame section 31, as in the recognition pattern sections 32H and 32I in Figs. 14 and 15. Also, the recognition pattern section 32 may be in the shape of a black star having a center that coincides with the center of the frame section 31, as in the recognition pattern section 32J in Fig. 16. Also, for example, as in recognition pattern portion 32K in FIG. 17, there may be two sets of lines each consisting of two parallel lines spaced a predetermined distance from each other, with one set of lines parallel to the X-axis and the other set of lines parallel to the Z-axis.

[0037] (B3) In the above embodiment, what is calculated in step S30 is the occupied area ratio of the white area, which is the portion within cut-out area 51 excluding frame portion 31 and recognition pattern portion 32, but the present disclosure is not limited to this. What is calculated in step S30 may be the occupied area ratio of the black area, which is frame portion 31 and recognition pattern portion 32, within cut-out area 51. In such a configuration, if the occupied area ratio of the black area is smaller than a threshold value in step S40, it may be determined that there is a positional deviation.

[0038] (B4) In the above embodiment, the maximum proportion region is identified as the region in the unit region in which the proportion of the area occupied by the white region, which is the portion excluding the recognition pattern portion 32, is the smallest, but the present disclosure is not limited to this. The maximum proportion region may be identified as the region in the unit region in which the proportion of the area occupied by the black region, which is the recognition pattern portion 32, is the largest.

[0039] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0040] 10... robot arm, 11... arm tip, 20... imaging device, 30... reference plate, 40... processing unit, 31... frame unit, 32... recognition pattern unit, 32A to 32K... other embodiments of the recognition pattern unit, 50... binarized image, 51... area, 100... positional deviation calculation device, 321, 322... scaled straight line, d... imaging distance, J1 to J6... rotation unit

Claims

[Claim 1] A method for calculating a positional deviation amount for detecting a positional deviation of an imaging device, comprising the steps of: a step of capturing an image of a reference plate having a frame portion arranged in accordance with an imaging range of the imaging device and a recognition pattern portion arranged at a predetermined position inside the frame portion, by the imaging device; a step of cutting out an area of ​​a predetermined width along an edge of the captured image, and determining whether or not there is a positional deviation based on an occupancy area ratio of the frame portion within the cut-out area; a step of dividing the captured image into a plurality of unit areas of a predetermined size as division targets, and calculating the amount of the positional deviation by comparing a position in the captured image of a maximum proportion area, which is a unit area among the divided unit areas in which the occupied area proportion of the recognition pattern portion is maximum, with a position in the master image of the maximum proportion area when a master image obtained by imaging the reference plate in a state without the positional deviation is used as the division target; A position deviation calculation method comprising:

Images