Defect inspection device

Abstract

According to the invention, high-precision defect inspection can be efficiently carried out. The defect inspection apparatus detects a defect on a workpiece surface based on a captured image obtained by capturing an image of the workpiece surface while changing an illumination state in which the workpiece surface is illuminated, the defect inspection apparatus including: a measurement unit that measures a position of the workpiece; a conversion unit that associates the position of the surface of the workpiece with the positions on the plurality of captured images on the basis of the position of the workpiece measured by the measurement unit and the correspondence between the position of the surface of the workpiece model based on the three-dimensional data of the workpiece model and the imaging model and the positions on the respective captured images captured in the plurality of illumination states; an acquisition unit that, on the basis of the correspondence between the position of the workpiece surface and the positions on the plurality of captured images, acquires brightness information in a plurality of illumination states from the corresponding captured images at each position of the workpiece surface; and a defect detection unit that detects a defect on the surface of the workpiece on the basis of the brightness information acquired by the acquisition unit.

Description

Technical Field

[0001] The present invention relates to a defect inspection device. Background Art

[0002] In the past, visual inspection of appearance was performed for quality control in the manufacturing process of various products. Regarding visual inspection, there are many inspection items, and on the other hand, inspectors are required to have skilled inspection skills, so it is difficult to achieve efficiency. Therefore, technology to automate visual inspection is required (for example, Patent Documents 1 to 3).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2010-216870

[0004] Patent Document 2: Japanese Patent Application Publication No. 2023-032374

[0005] Patent Document 3: Japanese Patent Application Publication No. 2013-195368

[0006] When inspecting defects on the surface of a vehicle body passing through a production line, a correction (motion correction) is performed on the captured image obtained by capturing the surface of the vehicle body, which is accompanied by the movement of the vehicle body. Therefore, it is necessary to obtain time series data of pixel values ​​at the same part of the surface of the vehicle body. When positional shift occurs in the time series data, the time series data becomes composed of data of pixel values ​​at different parts of the surface of the vehicle body, and the detection accuracy of defects is greatly reduced. The surface of the vehicle body is a complex curved surface shape, so in order to obtain time series data of pixel values ​​at the same part of the surface of the vehicle body, a higher-precision motion correction is required.

[0007] For high-precision motion correction, it is necessary to perform motion correction in frames in the images obtained by photographing the surface of the vehicle body using multiple cameras. As a result, a lot of labor is required to set up multiple cameras when introducing a defect inspection system.

[0008] In the invention described in Patent Document 1, the surface to be inspected is a substantially flat steel plate, and the invention does not assume that the surface to be inspected has a complex curved shape.

[0009] In the invention described in Patent Document 2, since the inspection object is a small object such as one grasped by an arm, it is not assumed that correction is performed in accordance with the movement of the vehicle body flowing through the production line.

[0010] In the invention described in Patent Document 3, since the inspection object does not move, it is not assumed that correction is performed in accordance with the movement of the vehicle body flowing through the production line. Summary of the invention

[0011] An object of the present invention is to provide a defect inspection device capable of efficiently performing high-precision defect inspection.

[0012] In order to solve the above-mentioned problem, the defect inspection device described in the technical solution 1 detects defects on the surface of the workpiece based on an image obtained by photographing the surface of the workpiece while changing the illumination state of the surface of the workpiece, wherein the defect inspection device comprises:

[0013] A measuring unit for measuring the position of a workpiece;

[0014] a conversion unit that establishes a correspondence between the position of the workpiece surface and the positions on the plurality of captured images based on the position of the workpiece measured by the measurement unit and the correspondence between the position of the workpiece model surface of the captured model and the positions on the captured images captured under the plurality of illumination conditions based on the three-dimensional data of the workpiece model and the captured model;

[0015] an acquisition unit, which acquires brightness information under a plurality of illumination states from the corresponding captured images at each position on the surface of the workpiece according to the correspondence between the position on the surface of the workpiece and the positions on the plurality of captured images; and

[0016] The defect detection unit detects defects on the surface of the workpiece based on the brightness information acquired by the acquisition unit.

[0017] The invention described in claim 2 is a defect inspection device described in claim 1.

[0018] A calculation unit is provided, wherein the calculation unit calculates a feature quantity in the brightness information.

[0019] The defect detection unit detects the defect based on the feature amount calculated by the calculation unit.

[0020] The invention described in claim 3 is a defect inspection device described in claim 1,

[0021] The conversion unit establishes a correspondence between the position of the surface of the workpiece model and the position on the captured image by performing a coordinate conversion between the three-dimensional coordinates of the workpiece model and the image coordinates as the coordinates of the captured image based on the three-dimensional data of the workpiece model and the captured model, and establishes a correspondence between the position of the surface of the workpiece model and the positions on the captured image by correcting the correspondence between the position of the surface of the workpiece model and the positions on the captured image based on the position of the workpiece.

[0022] The invention described in claim 4 is that in the defect inspection device described in claim 1,

[0023] The conversion unit corrects the coordinates of the workpiece model based on the position of the workpiece, and associates the position of the workpiece surface with the positions on the plurality of captured images based on the three-dimensional data of the workpiece model after the coordinate correction and the captured model.

[0024] The invention described in claim 5 is a defect inspection device described in claim 4.

[0025] The conversion unit corrects the coordinates of the workpiece model based on the position of the workpiece, and establishes a correspondence between the position of the workpiece surface and the positions on the plurality of captured images based on the three-dimensional data of the workpiece model after the coordinate correction and the ray tracing of the illumination light irradiated onto the workpiece surface as the captured model.

[0026] The invention described in claim 6 is a defect inspection device described in claim 1.

[0027] The conversion unit generates a table indicating correspondence between workpiece coordinates indicating positions on the workpiece surface and image coordinates indicating positions on the plurality of captured images.

[0028] The invention described in claim 7 is a defect inspection device described in claim 6.

[0029] The table stores the correspondence between the workpiece coordinates and the image coordinates only for a portion of the image coordinates.

[0030] The conversion unit establishes a correspondence between the workpiece coordinates interpolated based on the correspondence with other image coordinates and the image coordinates not stored in the table, wherein the other image coordinates are stored in the table and include image coordinates closest to the image coordinates not stored in the table.

[0031] The invention described in claim 8 is a defect inspection device described in claim 1,

[0032] The imaging model includes information on the focal length, position, and posture of an imaging optical system of an imaging unit that captures the image.

[0033] The invention described in claim 9 is a defect inspection device described in claim 1.

[0034] The conversion unit associates workpiece coordinates indicating a position on the workpiece surface with image coordinates indicating a position on the captured image, and associates the predetermined workpiece coordinates with the image coordinates for which no coordinates corresponding to the workpiece surface exist.

[0035] The invention described in claim 10 is a defect inspection device described in claim 2,

[0036] The calculation unit generates a sequence of the brightness information by arranging the brightness information under the different lighting conditions in the order of change of the lighting conditions, and calculates two or more feature quantities based on the sequence.

[0037] The defect detection unit generates a multi-channel image by inputting the two or more feature quantities into different multiple channels one by one, and detects the features of defects appearing in the multi-channel image through a machine learning model.

[0038] The invention described in claim 11 is a defect inspection device described in claim 2,

[0039] The calculation unit converts the brightness information into a vector of a predetermined dimension by interpolation or compression, and calculates the feature amount in the vector.

[0040] The invention described in claim 12 is a defect inspection device described in claim 11,

[0041] The calculation unit applies a predetermined weight to the vector.

[0042] The invention described in claim 13 is a defect inspection device described in claim 2.

[0043] The calculation unit generates a sequence of the brightness information by arranging the brightness information under the different lighting conditions in the order of change of the lighting conditions, and calculates a feature value based on the sequence.

[0044] The defect detection unit generates a feature quantity image based on the feature quantity, and detects features of defects appearing in the feature quantity image through a machine learning model.

[0045] The invention described in claim 14 is a defect inspection device described in claim 1.

[0046] The conversion unit associates the position of the workpiece surface with the positions on the plurality of captured images only for pixels having pixel values ​​equal to or greater than a predetermined threshold value or within a predetermined range among the pixels of the captured images.

[0047] The invention described in claim 15 is a defect inspection device described in claim 1.

[0048] The measuring unit measures the posture of the workpiece.

[0049] The conversion unit establishes correspondence between the position of the workpiece surface and the positions on the plurality of captured images based on the position and posture of the workpiece measured by the measurement unit and the correspondence between the position of the workpiece model surface and the positions on each captured image captured under a plurality of lighting conditions.

[0050] The invention described in claim 16 is a defect inspection device described in claim 2,

[0051] The defect detection unit performs a predetermined process on the feature amount, and adjusts the detection sensitivity of the defect by adjusting a threshold value in the predetermined process.

[0052] The invention described in claim 17 is a defect inspection device described in claim 6,

[0053] In the above table, the variables related to the defect detection sensitivity are associated with the above workpiece coordinates.

[0054] The defect detection unit adjusts the detection sensitivity of the defect corresponding to the predetermined workpiece coordinates by adjusting the variable.

[0055] The invention described in claim 18 is a defect inspection device described in claim 1.

[0056] The conversion unit obtains light reflection characteristics from light information obtained by ray tracing of the illumination light irradiated on the workpiece surface, and associates the light reflection characteristics respectively corresponding to the positions on the workpiece model surface with the positions on the workpiece model surface.

[0057] The invention described in claim 19 is a defect inspection device described in claim 1.

[0058] The conversion unit sets an inspection target region, an inspection non-target region, or both of them on the three-dimensional data of the workpiece model.

[0059] The invention described in claim 20 is a defect inspection device described in claim 10 or 13,

[0060] The output of the above machine learning model represents an estimate of the accuracy of the above defects,

[0061] The defect detection unit adjusts the detection sensitivity of the defect by adjusting a threshold value in the estimated value.

[0062] The invention described in claim 21 is a defect inspection device described in claim 1,

[0063] The conversion unit corrects the imaging model based on position information and posture information of the imaging unit measured using a reference map after the imaging unit that captures the imaging image is installed.

[0064] According to the present invention, defect inspection can be performed efficiently and with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a block diagram of the defect inspection device of this embodiment.

[0066] Figure 2A It is a side view showing the structure of the inspection unit.

[0067] Figure 2B This is a front view showing the structure of the inspection unit.

[0068] Figure 3 This is a flowchart showing an example of the flow of the reference position registration process.

[0069] Figure 4 This is a diagram showing an example of the imaging range of the workpiece by the imaging unit.

[0070] Figure 5 FIG. 1 is a diagram showing an example of continuously captured images of a workpiece.

[0071] Figure 6 This is a flowchart showing an example of the flow of defect inspection processing.

[0072] Figure 7 This is a diagram showing an example of the reference position and the measurement position of the edge in the continuously captured images of the workpiece.

[0073] Figure 8 This is a diagram showing an example of normalizing time series data of pixel values.

[0074] Fig. 9 This is a diagram showing an example of generating a synthetic image from time-series data of pixel values.

[0075] Fig.10 This is a diagram showing an example of dividing a synthesized image into tiles.

[0076] Fig.11A This is a diagram showing an example of a convex defect.

[0077] Fig. 11B This is a diagram showing an example of a concave defect.

[0078] Fig. 11C This is a diagram showing an example of a linear object.

[0079] Fig.12It is a side view showing the structure of the inspection unit according to Modification 1.

[0080] Fig.13 This is a diagram showing an example of measuring the distance to a workpiece by the laser rangefinder according to Modification 4.

[0081] Explanation of the reference numerals: 1...defect inspection device; 10, 10A...inspection unit; 11...conveying unit; 111...conveyor belt; 12, 12A...shooting unit; 13, 13A...illumination unit; 131...light source; 14...detection unit; 15...position detection unit; 16A...robot arm; 17...laser rangefinder; 20...processing device; 21...control unit; 22...storage unit; 22a...program; 23...communication unit; 24...operation unit; 25...display unit; A...inspection start position; D...trolley; E1, E11, E2, E22...edge; H...shooting range; P1, P2...synthetic image; W...workpiece. DETAILED DESCRIPTION

[0082] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the contents shown in the drawings.

[0083] <Structure of defect inspection device>

[0084] The structure of the defect inspection device according to this embodiment will be described.

[0085] Figure 1 It is a block diagram of the defect inspection device 1 according to this embodiment.

[0086] like Figure 1 As shown, the defect inspection device 1 includes an inspection unit 10 and a processing device 20 . Figure 2A is a side view showing the structure of the inspection unit 10, Figure 2B 1 is a front view showing the structure of the inspection unit 10. Figure 2A , Figure 2B In the figure, the X-axis direction is a direction parallel to the conveying direction of the workpiece W, which is the inspection object in the defect inspection process described later, and the Y-axis direction is the width direction of the workpiece W. The X-axis direction and the Y-axis direction are two horizontal directions orthogonal to each other, and the vertical direction orthogonal to the X-axis and the Y-axis is set as the Z-axis direction.

[0087] The defect inspection device 1 is a device for inspecting surface defects in a workpiece W. In the present embodiment, the workpiece W is a vehicle body. The inspected portion of the workpiece W is the painted surface of the vehicle body, and the defect inspection device 1 inspects surface defects of the painted surface of the vehicle body. Generally speaking, the surface of the vehicle body is subjected to base treatment, metal coating, transparent coating, etc., to form a coating layer with a multi-layer structure, but due to the influence of foreign matter in the coating, etc., concave-convex defects are generated in the top transparent layer. In the present embodiment, it is applied to the inspection of such defects, but the workpiece W is not limited to the vehicle body, and may also be an object other than the vehicle body. The inspected portion may also be a surface other than the painted surface.

[0088] The inspection unit 10 includes a transport unit 11 , an imaging unit 12 , an illumination unit 13 , a detection unit 14 , a position detection unit 15 , and the like.

[0089] The transport unit 11 transports the workpiece W placed on the carriage D in the positive direction of the X axis at a predetermined speed using the conveyor belt 111 .

[0090] The lighting unit 13 is provided in the middle of the conveyor belt 111 in the longitudinal direction (X-axis direction).

[0091] When viewed from the conveying direction (X-axis direction) of the workpiece W, the lighting unit 13 is as shown in FIG. Figure 2B That is, the lighting unit 13 is formed to surround the peripheral surface except the lower surface of the workpiece W.

[0092] The lighting unit 13 includes a light source 131 for illuminating the workpiece W. The light source 131 is, for example, an LED (Light Emitting Diode).

[0093] The illumination unit 13 illuminates the peripheral surface of the workpiece W except for the lower surface thereof in a linear manner using the light source 131 .

[0094] The photographing unit 12 includes: Figure 2B As shown in FIG. 1 , two imaging units 12 are provided to be able to image the workpiece W from the positive direction side of the Z axis. The imaging unit 12 includes: Figure 2B As shown in FIG. 1 , two imaging units 12 are provided to be able to image the workpiece W from the positive direction side of the Y axis. The imaging unit 12 includes: Figure 2B As shown, the two imaging units 12 are provided so as to be able to image the workpiece W from the negative side in the Y-axis direction.

[0095] The imaging unit 12 continuously images the workpiece W that is moved at a predetermined speed by the conveying unit 11 and illuminated by the illumination unit 13 , using each portion in the circumferential direction of the workpiece W as an inspection site.

[0096] The detector 14 is a sensor that detects that the workpiece W conveyed by the conveyor 111 has arrived at the inspection start position A. The detector 14 is connected to the processing device 20 via wiring (not shown) and outputs the detection result to the processing device 20. The detector 14 can also detect that the cart D conveyed by the conveyor 111 has arrived at the inspection start position A.

[0097] The position detector 15 detects the travel distance information of the workpiece W conveyed by the conveyor 111 from the inspection start position A using an encoder or the like. The position detector 15 is connected to the processing device 20 via wiring (not shown) and outputs the detected travel distance information of the workpiece W to the processing device 20 .

[0098] As described above, in the inspection unit 10, the imaging unit 12 and the illumination unit 13 are fixed and the workpiece W is moved by the conveying unit 11, but the present invention is not limited thereto. The imaging unit 12 and the illumination unit 13 may be moved relative to the workpiece W which is not moved.

[0099] The processing device 20 is, for example, a personal computer. The processing device 20 is connected to the conveying unit 11 and the imaging unit 12 of the inspection unit 10 via wiring (not shown) to control the operation of the conveying unit 11 and the imaging unit 12. The processing device 20 may be connected to the lighting unit 13 via wiring to control the operation of the lighting unit 13.

[0100] The processing device 20 includes a control unit 21 , a storage unit 22 , a communication unit 23 , an operation unit 24 , and a display unit 25 .

[0101] The control unit 21 includes, for example, a processor such as a CPU (Central Processing Unit) or an MPU (MicroProcessing Unit) and a memory such as a RAM (Random Access Memory). The control unit 21 implements various processes including defect inspection of the workpiece W by executing a program 22a stored in a memory such as the RAM, a storage unit 22, and the like.

[0102] The control unit 21 controls the conveying unit 11 of the inspection unit 10 to convey the workpiece W in the positive direction of the X-axis.

[0103] The control unit 21 controls the imaging unit 12 of the inspection unit 10 to continuously image the surface of the workpiece W at predetermined intervals.

[0104] The control unit 21 determines the imaging timing of the imaging unit 12 based on the detection result of the detection unit 14 of the inspection unit 10 and the travel distance information of the workpiece W detected by the position detection unit 15 .

[0105] The storage unit 22 includes, for example, any storage module such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), a ROM (Read Only Memory), and a RAM. The storage unit 22 stores, for example, system programs, application programs, and various data. Specifically, the storage unit 22 stores a program 22a for executing a defect inspection process of a workpiece W, and the like.

[0106] The storage unit 22 stores a first coordinate conversion table for converting coordinates (image coordinates) in the image captured by the imaging unit 12 and workpiece model two-dimensional coordinates. The workpiece model two-dimensional coordinates are two-dimensional coordinates based on the workpiece W in the workpiece model.

[0107] The communication unit 23 includes, for example, a communication module including a NIC (Network Interface Card), a receiver, and a transmitter, etc. The communication unit 23 communicates various information and data with external devices connected via a network such as the Internet.

[0108] The operation unit 24 includes, for example, a mouse, a keyboard, a switch, a button, etc. The operation unit 24 may be, for example, a touch panel integrated with the display unit 25, or an interface for receiving voice input. The operation unit 24 receives instructions corresponding to various input operations from the user, and converts the received instructions into operation signals and outputs them to the control unit 21.

[0109] The display unit 25 is, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, etc. The display unit 25 performs display based on display data output from the control unit 21 .

[0110] <First coordinate conversion table>

[0111] Next, a first coordinate conversion table for converting image coordinates and workpiece model two-dimensional coordinates will be described.

[0112] The control unit 21 of the processing device 20 generates a first coordinate conversion table based on the three-dimensional data of the workpiece model and the ray tracing (shooting model) of the illumination light irradiated on the surface of the workpiece W based on the ray simulation software using the information of the shooting optical system. The three-dimensional data of the workpiece model is composed of shape data representing the three-dimensional shape of the workpiece W. The information of the shooting optical system is, for example, the focal length, position, posture, etc. of the shooting unit 12. The ray simulation software may also be CG (Computer Graphics) software.

[0113] Specifically, the control unit 21 establishes a correspondence between the coordinates on the captured image (image coordinates) and the 3D coordinates on the workpiece model (workpiece model three-dimensional coordinates) for each frame of the captured image for all captured images captured when performing the defect inspection process. Thus, the control unit 21 establishes a correspondence between the position on the captured image and the position on the surface of the workpiece model. The image coordinates are represented by (X, Z). The workpiece model three-dimensional coordinates are represented by (CX, CY, CZ).

[0114] The control unit 21 associates coordinates other than the workpiece model three-dimensional coordinates with image coordinates where there are no coordinates corresponding to the workpiece model three-dimensional coordinates. The coordinates other than the workpiece model three-dimensional coordinates are, for example, (CX, CY, CZ) = (0, 0, 0).

[0115] The control unit 21 sets the area excluded from the inspection target area as an area outside the inspection target on the workpiece model. The control unit 21 establishes a correspondence between the coordinates other than the three-dimensional coordinates of the workpiece model and the image coordinates corresponding to the area excluded from the inspection target area. The control unit 21 may also set the inspection target area, the area outside the inspection target, or both on the three-dimensional data of the workpiece model.

[0116] Next, the control unit 21 generates a first coordinate conversion table by converting the three-dimensional coordinates of the workpiece model into the two-dimensional coordinates of the workpiece model, and stores the first coordinate conversion table in the storage unit 22 .

[0117] Thus, the first coordinate conversion table for converting the image coordinates into the two-dimensional coordinates of the workpiece model is represented by the following equation (1).

[0118] Formula (1) (MX, MZ) = f (frame, X, Z)

[0119] “MX” and “MZ” represent coordinate positions in the two-dimensional coordinate system of the workpiece model defined by the X-axis and the Z-axis that are orthogonal to each other.

[0120] “X” and “Z” represent coordinate positions in a two-dimensional plane defined by an X-axis and a Z-axis that are orthogonal to each other, that is, in an image captured by the imaging unit 12 .

[0121] Frame is a frame number in the continuously captured images captured by the imaging unit 12 .

[0122] <Operation of defect inspection device>

[0123] Next, the operation of the defect inspection device 1 according to the present embodiment will be described.

[0124] Before executing the defect inspection process on the workpiece W, the control unit 21 of the processing device 20 executes Figure 3 The reference position registration process shown.

[0125] (Reference position registration process)

[0126] The control unit 21 controls the conveying unit 11 to convey the workpiece W in the positive direction of the X axis. Next, the control unit 21 determines the imaging timing of the imaging unit 12 based on the detection result of the detection unit 14 and the travel distance information of the workpiece W detected by the position detection unit 15. Next, the control unit 21 uses the imaging unit 12 to continuously image the workpiece W being conveyed at a predetermined interval at the determined imaging timing, and obtains continuous imaged images of the workpiece W (step A1).

[0127] Figure 4 An example of an imaging range H of the workpiece W by the imaging unit 12 is shown. Figure 5 An example of the continuously captured images acquired in step A1 is shown.

[0128] like Figure 5 As shown, the control unit 21 uses the imaging unit 12 to perform imaging so that the imaging ranges of the imaging images of the preceding and following frame numbers mostly overlap. Thus, the control unit 21 obtains a plurality of imaging images whose imaging ranges are continuously shifted in the conveying direction of the workpiece W.

[0129] Next, the control unit 21 extracts images captured near the start of shooting and including characteristic edges from among the continuous shooting images of the workpiece W acquired in step A1. Furthermore, the control unit 21 extracts images captured near the end of shooting and including characteristic edges from among the continuous shooting images of the workpiece W acquired in step A1.

[0130] exist Figure 5 In the example shown, the control unit 21 extracts the image of the frame number “ 1 ” including the characteristic edge E1 , and the image of the frame number “ 4 ” including the characteristic edge E2 .

[0131] Next, the control unit 21 extracts the edge region from the extracted image (step A2 ).

[0132] Next, the control unit 21 obtains the information related to the edge area extracted in step A2, namely the reference edge information (step A3). The edge information includes: the frame number of the image including the edge and the shape information of the edge, namely the position information of the edge. The reference edge information includes the position information of the edge serving as the reference, namely the reference position information.

[0133] The control unit 21 may acquire edge information from the entire image including the extracted edge instead of extracting the edge region from the extracted image.

[0134] Next, the control unit 21 stores the reference edge information acquired in step A3 in the storage unit 22 to register the reference position information (step A4 ).

[0135] Next, the defect inspection process (see Figure 6 ) process is described below.

[0136] The defect inspection device 1 detects defects on the surface of the workpiece W based on a captured image obtained by capturing the surface of the workpiece W while changing the illumination state of the surface of the workpiece W by executing a defect inspection process. Changing the illumination state of the surface of the workpiece W includes: the workpiece W moves relative to the illumination unit 13, thereby changing the position of the bright band relative to the surface of the workpiece W. The bright band is a portion illuminated by the illumination unit 13 in the captured image.

[0137] (Defect inspection process)

[0138] The control unit 21 of the processing device 20 continuously images the workpiece W being conveyed at predetermined intervals using the imaging unit 12 , similarly to step A1 of the reference position registration process, and acquires continuously captured images of the workpiece W (step B1 ).

[0139] Next, the control unit 21 uses the reference edge information stored in step A4 as template data and performs template matching on the continuously captured images of the workpiece W acquired in step B1. Thus, the control unit 21 detects edges from the continuously captured images of the workpiece W (step B2).

[0140] Next, the control unit 21 acquires edge information of the edge detected in step B2 (step B3). The edge information of the edge detected in step B2 includes the position information of the edge, that is, the measurement position information.

[0141] In step B3 , the control unit 21 acquires the measurement position information of the edge to measure the position of the workpiece W. The control unit 21 functions as a measuring unit.

[0142] Next, the control unit 21 compares the reference position information registered in step A4 with the measured position information of the edge acquired in step B3. The control unit 21 thereby calculates the position correction amount which is the difference between the reference position indicated by the reference position information and the measured position indicated by the measured position information (step B4).

[0143] The difference between the reference position and the measured position is caused by a deviation in the conveying direction (X-axis direction) of the workpiece W in the captured image, a deviation in the vehicle height direction (Z-axis direction) of the workpiece W in the captured image, etc. The difference between the reference position and the measured position may also be caused by enlargement, reduction, rotation in the XY plane, etc. of the workpiece W in the captured image.

[0144] When the carriage D is not placed horizontally on the conveyor 111 or the workpiece W is not placed horizontally on the carriage D, etc., deviations occur in the conveyance direction and the vehicle height direction of the workpiece W in the captured image.

[0145] When there is a deviation in the height (width in the Z-axis direction) of the carriage D, a deviation in the vehicle height direction of the workpiece W in the captured image occurs.

[0146] When the detector 14 detects that the carriage D has reached the inspection start position A and the position of the workpiece W relative to the carriage D is deviated from the reference position, a deviation in the conveyance direction of the workpiece W in the captured image occurs.

[0147] Figure 7 An example of the reference position and the measurement position of the edge in the continuously captured images of the workpiece W is shown.

[0148] exist Figure 7 In the example shown, the difference between the reference position of edge E1 in the image of frame number "1" and the measured position of edge E11 is (X, Z) = (8pix, 3pix). The difference between the reference position of edge E2 in the image of frame number "4" and the measured position of edge E22 is (X, Z) = (2pix, 0pix).

[0149] In this case, the control unit 21 sets the position correction amount in the image of frame number "1" to (X, Z) = (8pix, 3pix) in step B4. The control unit 21 sets the position correction amount in the image of frame number "4" to (X, Z) = (2pix, 0pix).

[0150] Next, the control unit 21 calculates the position correction amounts in the images of frame numbers "2", "3", and "5" by linear interpolation based on the position correction amounts in the image of frame number "1" and the image of frame number "4". The control unit 21 may calculate the position correction amount based on the difference between the reference position and the measured position in all the images of the continuously captured images.

[0151] Next, the control unit 21 corrects the first coordinate conversion table for performing coordinate conversion between the image coordinates and the workpiece model two-dimensional coordinates based on the position correction amount calculated in step B4 (step B5 ).

[0152] In step B5 , when the position correction amount is set to (X, Z)=(ΔX, ΔZ), the control unit 21 corrects the first coordinate conversion table according to the following equation (2).

[0153] Formula (2) (MX, MZ) = f (frame, X-ΔX, Z-ΔZ)

[0154] For example, the position correction amount in the image of the frame number "1" is (X, Z) = (8pix, 3pix), so the corrected first coordinate conversion table corresponding to the image of the frame number "1" is expressed by the following formula (3).

[0155] Formula (3) (MX, MZ) = f (1, X-8pix, Z-3pix)

[0156] For example, the position correction amount in the image of the frame number "4" is (X, Z) = (2pix, 0pix), so the corrected first coordinate conversion table corresponding to the image of the frame number "4" is expressed by the following formula (4).

[0157] Formula (4) (MX, MZ) = f (4, X-2pix, Z-0pix)

[0158] The first coordinate conversion table corrected in step B5 is a second coordinate conversion table for converting image coordinates into workpiece coordinates. That is, the second coordinate conversion table shows the correspondence between workpiece coordinates indicating positions on the surface of the workpiece W and image coordinates indicating positions on a plurality of captured images.

[0159] As described above, the control unit 21 generates a second coordinate conversion table based on the position of the workpiece W measured in step B3 and the first coordinate conversion table. Thus, the control unit 21 establishes a correspondence between the position of the workpiece surface (workpiece coordinates) and the positions on the plurality of captured images (image coordinates). The control unit 21 functions as a conversion unit. The first coordinate conversion table indicates the correspondence between the position of the workpiece model surface based on the three-dimensional data of the workpiece model and the captured model and the position on each captured image captured under a plurality of lighting conditions.

[0160] As described above, the control unit 21 establishes a correspondence between coordinates other than the three-dimensional coordinates of the workpiece model and image coordinates where there are no coordinates corresponding to the three-dimensional coordinates of the workpiece model. Therefore, the control unit 21 establishes a correspondence between the predetermined workpiece coordinates and the image coordinates where there are no coordinates corresponding to the workpiece surface in the second coordinate conversion table. The predetermined workpiece coordinates are, for example, (0, 0).

[0161] Next, the control unit 21 extracts pixels having pixel values ​​greater than or equal to a predetermined value (eg, 40) in all the captured images of the workpiece W acquired in step B1 (step B6 ). Pixels having pixel values ​​greater than or equal to the predetermined value in the captured images are pixels near the bright band.

[0162] Next, the control unit 21 performs coordinate conversion of converting the image coordinates of the pixels extracted in step B6 into workpiece coordinates based on the second coordinate conversion table (step B7). In other words, the control unit 21 establishes a correspondence between the position of the workpiece surface (workpiece coordinates) and the position (image coordinates) on the plurality of captured images only for pixels having a pixel value above a predetermined threshold or within a predetermined range among the pixels of the captured images.

[0163] The second coordinate conversion table of this embodiment stores the correspondence between the image coordinates and the workpiece coordinates for only a portion of all pixels in the captured image (e.g., 1 / 16 of all pixels). The control unit 21 interpolates the pixels for which the correspondence is not stored using the correspondence of other pixels stored. That is, the control unit 21 establishes a correspondence between the workpiece coordinates interpolated based on the correspondence of other image coordinates stored in the second coordinate conversion table and the image coordinates (unstored coordinates) not stored in the second coordinate conversion table. The other image coordinates include the image coordinates closest to the unstored coordinates, which are the image coordinates of the four points that form a rectangle surrounding the unstored coordinates.

[0164] Next, the control unit 21 arranges the pixel values ​​in each workpiece coordinate after the coordinate conversion in step B7 in the order of the frame number of the captured image, and acquires time series data of the pixel values ​​in each workpiece coordinate (step B8 ).

[0165] In other words, the control unit 21 obtains brightness information under multiple lighting conditions from the corresponding captured images at each position (each workpiece coordinate) on the workpiece surface according to the correspondence between the position (workpiece coordinate) on the workpiece surface and the position (image coordinate) on the multiple captured images. The control unit 21 functions as an acquisition unit. The brightness information includes pixel values.

[0166] Next, if Figure 8 As shown, the control unit 21 normalizes the time series data of the pixel values ​​in each workpiece coordinate acquired in step B8 into 32 data. Figure 8 The example shown is a case where a defect (convex defect) protruding from the vehicle body surface occurs in the vehicle body surface.

[0167] When the time series data of the pixel values ​​in each workpiece coordinate acquired in step B8 is not 32, the control unit 21 converts the acquired time series data of the pixel values ​​into 32 data by interpolating or compressing them.

[0168] Next, the control unit 21 divides the 32 data corresponding to each workpiece coordinate into 8 data, that is, divides the time series data of the pixel value into four regions, and calculates the representative value of the 8 data contained in each region. The representative value of the 8 data is the feature value such as the average value, median, mode, etc. of the 8 data. Thus, the control unit 21 compresses the time series data of the pixel value in each workpiece coordinate by calculating the four representative values ​​corresponding to the four regions respectively (step B9).

[0169] That is, the control unit 21 generates a sequence of brightness information (time series data) by arranging the brightness information under different lighting conditions in the order of change of the lighting conditions (time series order), and calculates two or more feature quantities based on the sequence. In other words, the control unit 21 calculates the feature quantities in the brightness information. The control unit 21 functions as a calculation unit.

[0170] Next, the control unit 21 inputs the four representative values ​​corresponding to the four regions calculated in step B9 into the four channels of R (Red), G (Green), B (Blue), and A (Alpha). Thus, the control unit 21 generates a color image of the time series data of the pixel values, that is, a composite image (step B10). Figure 8 In the example shown, the control unit 21 inputs the representative value in the region R1 to the A (Alpha) channel. The control unit 21 inputs the representative value in the region R2 to the B (Blue) channel. The control unit 21 inputs the representative value in the region R3 to the G (Green) channel. The control unit 21 inputs the representative value in the region R4 to the R (Red) channel.

[0171] In step B10 , the control unit 21 may generate a composite image of the time series data of pixel values ​​by inputting representative values ​​in the divided regions of the time series data of pixel values ​​into the three channels of R, G, and B one by one.

[0172] As described above, by displaying the time series data of pixel values ​​using a color image, the temporal changes in the pixel values ​​can be easily recognized by the human eye, thereby making it easy to perform annotation work for machine learning in an inspection model (described later).

[0173] Fig. 9 An example of a synthesized image P1 is shown when a convex defect occurs on the vehicle body surface.

[0174] like Fig. 9 As shown, in the synthesized image P1 , a shift in timing, that is, a phase, at which a mirror image of the light source 131 of the illumination unit 13 appears appears as a difference in color.

[0175] exist Fig. 9In the example shown, the Green portion Pg shown by Green in the synthetic image P1 is a region where no surface defect occurs. The Red portion Pr shown by Red in the synthetic image P1 is a region where the mirror image of the light source 131 appears earlier than the Green portion Pg due to a convex defect. The Blue portion Pb shown by Blue in the synthetic image P1 is a region where the mirror image of the light source 131 appears later than the Green portion Pg due to a convex defect.

[0176] Next, the control unit 21 divides the synthesized image generated in step B10 (step B11 ). Fig.10 An example of a synthesized image P2 generated by executing step B10 for all workpiece coordinates is shown.

[0177] In step B11, the control unit 21 divides the entire synthetic image P2 into, for example, 640×640 pix-sized tiles T. The control unit 21 divides the synthetic image P2 so that the tiles T overlap by, for example, 30%.

[0178] Next, the control unit 21 inputs the image blocks T divided in step B11 into the inspection model, and obtains the output result of the inspection model, that is, the estimated value indicating the accuracy of the defect. Next, the control unit 21 detects the surface defect by judging that the surface defect has occurred when the estimated value is above a preset threshold value (step B12), and ends the defect inspection process. The inspection model is a machine learning model that has previously performed machine learning on the defects on the surface of the vehicle body, for example, a machine learning model using TensorFlow.

[0179] That is, the control unit 21 detects defects on the surface of the workpiece W based on the brightness information acquired in step B8. The control unit 21 functions as a defect detection unit.

[0180] The control unit 21 of this embodiment generates a multi-channel image (synthetic image) by inputting two or more feature quantities in the brightness information into different multiple channels one by one, and detects the characteristics of defects appearing in the multi-channel image through a machine learning model (inspection model).

[0181] The control unit 21 may execute the following processing on the inspection model.

[0182] Specifically, the control unit 21 can also adjust the detection sensitivity of the defect by adjusting the threshold of the estimated value of the output result of the inspection model, that is, the accuracy of the defect. The control unit 21 can set the threshold uniformly for all blocks T, or set the threshold for each workpiece coordinate in the second coordinate conversion table. In other words, when the variable related to the detection sensitivity of the defect is associated with the workpiece coordinate in the second coordinate conversion table, the control unit 21 adjusts the detection sensitivity of the defect corresponding to the specified workpiece coordinate by adjusting the variable.

[0183] In this case, the control unit 21 may increase the detection sensitivity of defects in the workpiece coordinates corresponding to the conspicuous parts of the vehicle body, or decrease the detection sensitivity of defects in the workpiece coordinates corresponding to the inconspicuous parts or noisy parts of the vehicle body.

[0184] Figures 11A to 11C This figure shows an example of defects on the vehicle body surface that the inspection model has learned in advance.

[0185] Fig.11A The example shown is a convex defect in which a foreign object 330 is mixed into a coating layer 320 formed on a blank material 310 of a vehicle body, and the surface of the coating layer 320 protrudes more than other parts.

[0186] In the synthetic image in this case, the Green portion Pg is a region where no surface defect occurs. The Red portion Pr is a region where the mirror image of the light source 131 appears earlier than the Green portion Pg due to a convex defect. The Blue portion Pb is a region where the mirror image of the light source 131 appears later than the Green portion Pg due to a convex defect.

[0187] Fig. 11B The example shown is a concave defect in which a coating layer 320 is partially dented in the coating layer 320 because the coating layer 320 is not uniformly attached to the vehicle body due to foreign matter 330 or the like after the coating layer 320 is applied to the vehicle body blank 310 .

[0188] In the synthetic image in this case, the Green portion Pg is a region where no surface defect occurs. The Red portion Pr is a region where the mirror image of the light source 131 appears later than the Green portion Pg due to a concave defect. The Blue portion Pb is a region where the mirror image of the light source 131 appears earlier than the Green portion Pg due to a concave defect.

[0189] Fig. 11C In the example shown, due to the mixing of linear foreign matter 330 into a coating layer 320 formed on a blank material 310 of a vehicle body, the surface of the coating layer 320 has linear objects protruding linearly compared to other parts.

[0190] In the synthetic image in this case, the Green portion Pg is an area where no surface defects occur. The Red portion Pr is a linear area where the mirror image of the light source 131 appears earlier than the Green portion Pg due to the linear object. The Blue portion Pb is a linear area where the mirror image of the light source 131 appears later than the Green portion Pg due to the linear object.

[0191] The defects on the body surface pre-learned by the inspection model include not only convex defects, concave defects, and linear objects, but also scratches, discoloration, bulges, sagging, and other poor painting.

[0192] <Modification>

[0193] The present embodiment is described above, but the specific structure is not limited to the above-mentioned present embodiment, and can be changed within the scope of the main purpose of the invention. The following describes a modified example of the present embodiment. In the modified example, the same figure mark is marked for the same structure as the above-mentioned embodiment, and its description is omitted.

[0194] (Variant 1)

[0195] Fig.12 It is a side view showing the structure of the inspection unit 10A according to the first modification.

[0196] The inspection unit 10A according to the first modification includes an imaging unit 12A, an illumination unit 13A, a robot arm 16A, and the like.

[0197] The imaging unit 12A and the lighting unit 13A are mounted on the front end of the robot arm 16A.

[0198] The robot arm 16A is a multi-joint robot and can arrange the imaging unit 12A and the lighting unit 13A at a predetermined distance set with respect to the surface of the curved workpiece W under the control of the control unit 21 .

[0199] The control unit 21 of the processing device 20 of the modification 1 controls the imaging unit 12A to image the surface of the workpiece W in the stopped state while the illumination unit 13A is on. The control unit 21 controls the robot arm 16A to move the imaging unit 12A and the illumination unit 13A to image multiple locations on the surface of the workpiece W. Thus, the control unit 21 acquires the imaged images of the multiple locations on the surface of the workpiece W.

[0200] (Variant 2)

[0201] The control unit 21 of the processing device 20 according to the second modification executes the following processing instead of steps B1 to B3 of the above-mentioned defect inspection processing.

[0202] Specifically, the control unit 21 uses the plurality of imaging units 12 to continuously image the workpiece W being transported at predetermined intervals, and acquires a plurality of continuously imaged images of the workpiece W. The plurality of imaging units 12 include an imaging unit 12 that images the workpiece W from the positive direction of the Z axis, an imaging unit 12 that images the workpiece W from the positive direction of the Y axis, and an imaging unit 12 that images the workpiece W from the negative direction of the Y axis.

[0203] Next, the control unit 21 performs template matching on the continuous captured images of the workpiece W corresponding to each imaging unit 12 using the reference edge information stored in step A4 as template data. Thus, the control unit 21 detects edges from the continuous captured images of the workpiece W corresponding to each imaging unit 12.

[0204] Next, the control unit 21 acquires edge information of edges detected from the continuously captured images corresponding to the respective imaging units 12. The edge information of the detected edges includes measurement position information of the detected edges.

[0205] The control unit 21 acquires the measured position information of the edge from the continuously captured images corresponding to each imaging unit 12, thereby measuring the position and posture of the workpiece W. Thus, the position and posture of the workpiece W can be measured more accurately.

[0206] The control unit 21 of variant example 2 establishes a correspondence between the position of the workpiece surface and the position on multiple captured images based on the measured position and posture of the workpiece W and the correspondence between the position of the workpiece model surface and the position on each captured image captured under multiple lighting conditions (first coordinate conversion table), and generates a second coordinate conversion table.

[0207] (Variant 3)

[0208] The system 1 of the modification 3 includes a position measuring unit (not shown) which cooperates with the processing device 20 to execute steps B1 to B5 of the defect inspection process. The structure of the position measuring unit is the same as that of the inspection unit 10. The position measuring unit is provided upstream of the inspection unit 10 in the conveying direction of the workpiece W.

[0209] In Modification 3, the control unit 21 of the processing device 20 conveys the workpiece W as an inspection object to pass through the position measuring unit, and uses the position measuring unit to perform steps B1 to B5 of the defect inspection process on the workpiece W. Thus, the control unit 21 generates the second coordinate conversion table by correcting the first coordinate conversion table.

[0210] Next, the control unit 21 conveys the workpiece W through the inspection unit 10 and performs step B1 and steps B6 to B12 of defect inspection processing on the workpiece W. In step B6, the control unit 21 converts the image coordinates into workpiece coordinates using a pre-generated second coordinate conversion table and performs subsequent processing.

[0211] Thus, since the coordinate conversion can be performed using the second coordinate conversion table generated in advance, the processes after step B6 can be executed immediately after the workpiece W is imaged in the defect inspection process.

[0212] (Variant 4)

[0213] The inspection unit 10 of the modification 4 includes, in addition to the conveying unit 11, the imaging unit 12, the lighting unit 13, the detection unit 14, and the position detection unit 15. Fig.13 A plurality of laser rangefinders 17 are shown.

[0214] The laser distance meter 17 is provided on the positive Z-axis side of the workpiece W. The laser distance meter 17 measures the distance to the surface of the workpiece W near the front end, middle, and rear end of the workpiece W in the X-axis direction and outputs the measurement result to the processing device 20 .

[0215] The control unit 21 of the processing device 20 according to the fourth modification executes the following processing instead of the above-mentioned reference position registration processing.

[0216] Specifically, the control unit 21 measures the distance to the surface of the workpiece W using the plurality of laser range finders 17 and acquires the measurement result.

[0217] Next, the control unit 21 stores the distance information to the surface of the workpiece W in the storage unit 22 , thereby registering the distance information to the surface of the workpiece W as the reference position information.

[0218] The control unit 21 of the processing device 20 according to the modification 4 executes the following processing instead of steps B2 to B4 of the above-mentioned defect inspection processing.

[0219] Specifically, the control unit 21 measures the distance to the surface of the workpiece W using the plurality of laser range finders 17 and acquires the measurement result.

[0220] Next, the control unit 21 compares the reference position information stored in the storage unit 22 with the distance information (measurement position information) to the surface of the workpiece W as the measurement result. Thus, the control unit 21 calculates the difference between the reference position W0 indicated by the reference position information and the measurement position indicated by the distance information to the surface of the workpiece W, that is, the position correction amount.

[0221] For example, when the reference position in the Z-axis direction is 1500 mm and the measurement position in the Z-axis direction is 1490 mm, the control unit 21 calculates the position correction amount in the Z-axis direction as −10 mm.

[0222] This makes it possible to measure the spatial position of the workpiece W. This makes it possible to measure the difference between the reference position based on the rotation and the measurement position in the XY plane of the workpiece W.

[0223] The control unit 21 of the processing device 20 according to the fourth modification may execute the following processing before the defect inspection processing on the workpiece W.

[0224] Specifically, the control unit 21 causes the imaging unit 12 to continuously image the workpiece W being conveyed at predetermined intervals, thereby acquiring continuously captured images of the workpiece W.

[0225] Next, the control unit 21 calculates the length per 1 pix of the surface of the workpiece W in the captured image of the workpiece W. The control unit 21 may calculate the length per 1 pix of the surface of the workpiece W based on the design value of the workpiece W without using the captured image of the workpiece W.

[0226] In this case, the control unit 21 converts the position correction amount, which is the difference between the reference position W0 indicated by the reference position information and the measured position indicated by the distance information to the surface of the workpiece W, into pix. For example, when the position correction amount is -10 mm and the length of the surface of the workpiece W per 1 pix is ​​1 mm, the control unit 21 calculates the position correction amount as -10 pix.

[0227] (Variant 5)

[0228] In step B5 of the defect inspection process of the above embodiment, the control unit 21 corrects the coordinate conversion table for converting the image coordinates and the workpiece coordinates based on the position correction amount calculated in step B4, but the present invention is not limited to this.

[0229] The control unit 21 of the processing device 20 in the modification 5 may also perform the following processing in step B5 of the defect inspection processing. Specifically, the control unit 21 converts the image coordinates into the two-dimensional coordinates of the workpiece model in all the captured images of the workpiece W acquired in step B1 according to the first coordinate conversion table stored in the storage unit 22. Next, the control unit 21 corrects each coordinate after the coordinate conversion based on the position correction amount calculated in step B4.

[0230] (Variant 6)

[0231] The control unit 21 of the processing device 20 in the sixth modification generates a third coordinate conversion table for converting the two-dimensional coordinates of the workpiece model into image coordinates based on the three-dimensional data of the workpiece model and the imaging model.

[0232] The third coordinate conversion table for converting the workpiece model two-dimensional coordinates into image coordinates is represented by the following equation (5).

[0233] Formula (5) (C, frame, X, Z) = f－1 (MX, MY)

[0234] “C” is identification information for identifying the imaging unit 12 .

[0235] The control unit 21 in the sixth variation can obtain the image coordinates (C, frame, X, Z) corresponding to the workpiece coordinates by inputting the workpiece coordinates (MX, MY) in the composite image generated in step B10 of the defect inspection process into the above equation (5).

[0236] (Variant 7)

[0237] The control unit 21 of the processing device 20 in the modification 7 may also generate a second coordinate conversion table for converting the image coordinates and the workpiece coordinates through the following processing when performing the above-mentioned defect inspection processing. Specifically, the control unit 21 corrects the three-dimensional data of the workpiece model using the measured position information obtained in step B3 of the defect inspection processing. Next, the control unit 21 generates the second coordinate conversion table based on the corrected three-dimensional data of the workpiece model and the photographed model.

[0238] In other words, the control unit 21 corrects the coordinates of the workpiece model (the three-dimensional coordinates of the workpiece model) based on the position of the workpiece W measured in step B3. Next, the control unit 21 establishes a correspondence between the position of the workpiece surface and the positions on the plurality of captured images based on the three-dimensional data of the workpiece model after the coordinate correction and the captured model.

[0239] The control unit 21 may acquire light reflection characteristics from light information obtained by ray tracing of illumination light irradiated on the surface of the workpiece W, and associate the light reflection characteristics with the position of the workpiece model surface (workpiece model three-dimensional coordinates).

[0240] (Variant 8)

[0241] The control unit 21 of the processing device 20 in the modification 8 may not compress the time series data of the pixel values ​​in each coordinate in the step B9 of the above-mentioned defect inspection process. In this case, the control unit 21 generates a synthetic image based on the time series data of 32 pixel values ​​as a 32-dimensional vector in step B10. As a result, the amount of information in the synthetic image increases compared to the case where the time series data of the pixel values ​​in each workpiece coordinate is compressed, so the detection accuracy of surface defects can be improved.

[0242] In this case, the control unit 21 may also give weights to the time series data of the 32 pixel values ​​after normalization. Specifically, the control unit 21 gives a larger weight to the 8 data at the first time and the 8 data at the last time, which correspond to the vicinity of the boundary between the bright band illuminated by the illumination unit 13 and the dark band not illuminated, than to the 16 data near the center, among the 32 data. Thus, the features in the time series data of the pixel values ​​can be made conspicuous near the boundary between the bright band and the dark band, which are features of surface defects.

[0243] (Variant 9)

[0244] The control unit 21 of the processing device 20 in the ninth modification may not compress the time series data of the pixel value in each workpiece coordinate in step B9 of the above-mentioned defect inspection process.

[0245] In this case, the control unit 21 calculates the feature amount for the time series data of 32 pixel values ​​which are 32-dimensional vectors in each workpiece coordinate in step B10 .

[0246] The feature quantity in the time series data of pixel values ​​of Modification Example 9 is a phase difference with the time series data of reference pixel values, a difference between the maximum value and the minimum value in the pixel values, a standard deviation in the pixel values, and the like.

[0247] Next, the control unit 21 generates a grayscale image of a single channel by calculating the feature quantity in the time series data of the pixel value. The control unit 21 may also perform differential processing on the generated grayscale image to emphasize the features of the defective part. The control unit 21 may further perform binarization processing on the grayscale image to extract the defective part. The control unit 21 may also adjust the threshold value in the binarization processing as a variable, so as to adjust the detection sensitivity of the surface defects. In other words, the control unit 21 may also perform a prescribed processing on the feature quantity in the time series data of the pixel value, and adjust the detection sensitivity of the defect by adjusting the threshold value in the prescribed processing.

[0248] Next, the control unit 21 inputs the generated grayscale image into the inspection model to detect surface defects.

[0249] As described above, the control unit 21 of the modification 9 generates a sequence of brightness information (time series data) by arranging the brightness information under different lighting conditions in the order of change of the brightness state (time series order), and calculates a feature quantity based on the sequence. Next, the control unit 21 generates a feature quantity image (grayscale image) based on the feature quantity, and detects the characteristics of the defects appearing in the feature quantity image through the machine learning model.

[0250] (Variant 10)

[0251] The control unit 21 of the processing device 20 according to the modification 10 may execute the following processing before executing the defect inspection processing on the workpiece W.

[0252] Specifically, after installing a plurality of imaging units 12 , the control unit 21 causes the imaging units 12 to image a reference pattern (chessboard) disposed at a predetermined position, thereby acquiring a plurality of captured images including the reference pattern.

[0253] Next, the control unit 21 calculates the position and posture of the imaging unit 12 using a stereo matching method in a plurality of captured images including the reference image.

[0254] Next, the control unit 21 corrects the imaging model based on the calculated position and posture of the imaging unit 12 .

[0255] The control unit 21 may measure the focal length and distortion of the lens characteristics of the imaging unit 12 in addition to the position and posture of the imaging unit 12, and calibrate the imaging model based on the focal length and distortion. This can calibrate the error between the imaging model and the measured position, the deviation of the lens of the imaging unit 12, etc.

[0256] <Effect>

[0257] As described above, the defect inspection device 1 of the present embodiment is a defect inspection device that detects defects on the workpiece surface based on an image obtained by imaging the workpiece surface while changing the illumination state of the workpiece surface.

[0258] The defect inspection device 1 includes a measuring unit (control unit 21 ) that measures the position of the workpiece W.

[0259] The defect inspection device 1 includes a conversion unit (control unit 21) which establishes a correspondence between the position of the workpiece surface and the position on multiple captured images based on the position of the workpiece W measured by the measuring unit and the correspondence between the position of the workpiece model surface based on the three-dimensional data of the workpiece model and the captured model and the position on each captured image captured under multiple lighting conditions.

[0260] The defect inspection device 1 includes an acquisition unit (control unit 21) that acquires brightness information under multiple lighting conditions from corresponding captured images at each position on the workpiece surface based on the correspondence between the position on the workpiece surface and the positions on the multiple captured images.

[0261] The defect inspection device 1 includes a defect detection unit (control unit 21 ) that detects defects on the surface of the workpiece based on the brightness information acquired by the acquisition unit.

[0262] This makes it possible to detect defects on the surface of the vehicle bodies passing through the production line with high accuracy. Therefore, defect inspection can be performed efficiently and with high accuracy.

[0263] This eliminates the need to spend a lot of labor on setting up multiple cameras when introducing a defect inspection system.

[0264] The defect inspection device 1 of the present embodiment includes a calculation unit (control unit 21 ) that calculates a feature amount in brightness information.

[0265] The defect detection unit (control unit 21) detects a defect based on the feature amount calculated by the calculation unit.

[0266] This makes it possible to detect defects on the surface of the vehicle body with high accuracy.

[0267] In the defect inspection device 1 of the present embodiment, the conversion unit (control unit 21) performs coordinate conversion between the three-dimensional coordinates of the workpiece model and the image coordinates as the coordinates of the captured image based on the three-dimensional data of the workpiece model and the captured model. Thus, the conversion unit establishes a correspondence between the position of the workpiece model surface and the position on the captured image. The conversion unit corrects the correspondence between the position of the workpiece model surface and the position on the captured image based on the position of the workpiece. Thus, the conversion unit establishes a correspondence between the position of the workpiece model surface and the position on the captured image.

[0268] This allows the position of the workpiece surface to be accurately associated with the positions on the plurality of captured images, thereby making it possible to perform a highly accurate defect inspection on the surface of the vehicle body.

[0269] In the defect inspection device 1 of this embodiment, the conversion unit (control unit 21) corrects the coordinates of the workpiece model based on the position of the workpiece, and establishes a correspondence between the position of the workpiece surface and the position on multiple captured images based on the three-dimensional data of the workpiece model after the corrected coordinates and the shooting model.

[0270] This allows the position of the workpiece surface to be accurately associated with the positions on the plurality of captured images, thereby making it possible to perform a highly accurate defect inspection on the surface of the vehicle body.

[0271] In the defect inspection device 1 of the present embodiment, the conversion unit (control unit 21) corrects the coordinates of the workpiece model based on the position of the workpiece, and establishes a correspondence between the position of the workpiece surface and the position on multiple captured images based on the three-dimensional data of the workpiece model after the corrected coordinates and the ray tracing of the illumination light irradiated onto the workpiece surface as a shooting model.

[0272] This allows the position of the workpiece surface to be accurately associated with the positions on the plurality of captured images, thereby making it possible to perform a highly accurate defect inspection on the surface of the vehicle body.

[0273] In the defect inspection device 1 of the present embodiment, the conversion unit (control unit 21) generates a table (second coordinate conversion table) indicating correspondence between workpiece coordinates indicating positions on the workpiece surface and image coordinates indicating positions on a plurality of captured images.

[0274] Thus, the workpiece coordinates and the image coordinates can be easily converted using the second coordinate conversion table.

[0275] In the defect inspection device 1 of the present embodiment, the table (second coordinate conversion table) stores the correspondence between the workpiece coordinates and the image coordinates only for a part of the image coordinates.

[0276] The conversion unit (control unit 21) associates the workpiece coordinates interpolated based on the association with other image coordinates stored in the table and including image coordinates closest to the image coordinates not stored in the table with the image coordinates not stored in the table.

[0277] Therefore, the second coordinate conversion table does not need to store the correspondence between the workpiece coordinates and the image coordinates for all the image coordinates in the captured image.

[0278] In the defect inspection device 1 of the present embodiment, the imaging model includes information on the focal length, position, and posture of the imaging optical system of the imaging unit 12 that captures the imaging image.

[0279] This allows the position of the workpiece surface to be accurately associated with the positions on the plurality of captured images, thereby making it possible to perform a highly accurate defect inspection on the surface of the vehicle body.

[0280] In the defect inspection device 1 of this embodiment, the conversion unit (control unit 21) establishes a correspondence between the workpiece coordinates indicating the position of the workpiece surface and the image coordinates indicating the position on the captured image. The conversion unit establishes a correspondence between the predetermined workpiece coordinates and the image coordinates where there are no coordinates corresponding to the workpiece surface.

[0281] Thereby, even if there are no image coordinates corresponding to the workpiece surface, coordinate conversion into workpiece coordinates is possible.

[0282] In the defect inspection device 1 of the present embodiment, the calculation unit (control unit 21) generates a sequence of brightness information by arranging brightness information under different lighting conditions in the order of change of the brightness state, and calculates two or more feature quantities based on the sequence.

[0283] The defect detection unit (control unit 21) inputs two or more feature quantities into different multiple channels one by one to generate a multi-channel image. The defect detection unit detects the features of defects appearing in the multi-channel image using a machine learning model.

[0284] Thus, by generating a color image as a multi-channel image, temporal changes in pixel values ​​can be easily recognized by the human eye. Thus, annotation work for machine learning in the inspection model can be easily performed.

[0285] In the defect inspection device 1 of the present embodiment, the calculation unit (control unit 21) converts the brightness information into a vector of a predetermined dimension by interpolation or compression, and calculates the feature amount in the vector.

[0286] The calculation unit applies a predetermined weight to the vector.

[0287] This makes it possible to make features in the time series data of pixel values ​​stand out near the boundary between a bright band and a dark band where features of surface defects are easily revealed.

[0288] In the defect inspection device 1 of the present embodiment, the calculation unit (control unit 21) generates a sequence of brightness information by arranging brightness information under different lighting conditions in the order of change of the brightness state, and calculates one feature amount based on the sequence.

[0289] The defect detection section (control section 21 ) generates a feature quantity image based on the feature quantity, and detects the features of a defect appearing in the feature quantity image through a machine learning model.

[0290] As a result, compared with the case where time-series data of pixel values ​​in each workpiece coordinate is compressed, the amount of information in the synthesized image increases, and thus the detection accuracy of surface defects can be improved.

[0291] In the defect inspection device 1 of this embodiment, the conversion unit (control unit 21) establishes correspondence between the position of the workpiece surface and the positions on multiple captured images only for pixels in the captured images having pixel values ​​greater than a specified threshold or within a specified range.

[0292] This makes it possible to perform defect inspection on the surface of the vehicle body in the captured image based only on pixels near the bright band, which is a portion necessary for defect inspection.

[0293] In the defect inspection device 1 of the present embodiment, the measuring unit (control unit 21 ) measures the posture of the workpiece.

[0294] The conversion unit (control unit 21) establishes a correspondence between the position of the workpiece surface and the position on multiple captured images based on the position and posture of the workpiece measured by the measuring unit and the correspondence between the position of the workpiece model surface and the position on each captured image captured under multiple lighting conditions.

[0295] This allows the position of the workpiece surface to be accurately associated with the positions on the plurality of captured images, thereby making it possible to perform a highly accurate defect inspection on the surface of the vehicle body.

[0296] In the defect inspection device 1 of the present embodiment, the defect detection unit (control unit 21) performs a predetermined process on the feature amount, and adjusts the detection sensitivity of the defect by adjusting the threshold value in the predetermined process.

[0297] Thereby, in the defect inspection of the vehicle body surface, it is possible to set an appropriate defect detection sensitivity.

[0298] In the defect inspection device 1 of the present embodiment, in the table (second coordinate conversion table), variables related to the detection sensitivity of defects and workpiece coordinates are associated with each other.

[0299] The defect detection unit (control unit 21) adjusts the detection sensitivity of the defect corresponding to the predetermined workpiece coordinates by adjusting the variables.

[0300] Thereby, in defect inspection on the vehicle body surface, it is possible to set appropriate defect detection sensitivity for each workpiece coordinate.

[0301] In the defect inspection device 1 of the present embodiment, the conversion unit (control unit 21) obtains light reflection characteristics from light information obtained by ray tracing of the illumination light irradiated onto the workpiece surface, and establishes a correspondence between the light reflection characteristics corresponding to the positions on the workpiece model surface and the positions on the workpiece model surface.

[0302] Thus, in defect inspection of the vehicle body surface, it is possible to set appropriate defect detection sensitivity according to the light reflection characteristics corresponding to the position on the workpiece model surface.

[0303] In the defect inspection device 1 of the present embodiment, the conversion unit (control unit 21) sets the inspection target area, the non-inspection target area, or both on the three-dimensional data of the workpiece model.

[0304] This makes it possible to perform defect inspection in a desired area on the surface of the vehicle body.

[0305] In the defect inspection device 1 of the present embodiment, the machine learning model outputs an estimated value indicating the accuracy of the defect.

[0306] The defect detection unit (control unit 21) adjusts the detection sensitivity of defects by adjusting the threshold value in the estimated value.

[0307] Thereby, in the defect inspection of the vehicle body surface, it is possible to set an appropriate defect detection sensitivity.

[0308] In the defect inspection device 1 of the present embodiment, the conversion unit (control unit 21) corrects the imaging model based on the position information and posture information of the imaging unit 12 measured using the reference map after the imaging unit 12 that captures the image is installed.

[0309] This allows the position of the workpiece surface to be accurately associated with the positions on the plurality of captured images, thereby making it possible to perform a highly accurate defect inspection on the surface of the vehicle body.

[0310] As mentioned above, although embodiment of this invention was demonstrated concretely, this invention is not limited to the said embodiment, It can change in the range which does not deviate from the summary.

[0311] For example, in the above-mentioned embodiment, the case where the workpiece W as the inspection object is a vehicle body is illustrated and described, but the present invention is not limited thereto. The present invention can also be applied to inspection objects other than vehicle bodies to efficiently perform high-precision defect inspections.

[0312] Industrial Applicability

[0313] The present invention can be used in a defect inspection device.

Claims

1. A defect inspection device for detecting defects on a workpiece surface based on an image obtained by photographing the workpiece surface while changing the illumination state of the workpiece surface, in, The defect inspection device comprises: A measuring unit for measuring the position of a workpiece; a conversion unit that establishes a correspondence between the position of the workpiece surface and the positions on the plurality of captured images based on the position of the workpiece measured by the measurement unit and the correspondence between the position of the workpiece model surface of the three-dimensional data of the workpiece model and the captured model and the positions on the captured images captured under the plurality of illumination conditions; an acquisition unit, which acquires brightness information under a plurality of illumination states from the corresponding captured images at each position on the workpiece surface according to the correspondence between the position on the workpiece surface and the positions on the plurality of captured images; as well as The defect detection unit detects a defect on the surface of the workpiece based on the brightness information acquired by the acquisition unit.

2. The defect inspection device according to claim 1, in, A calculation unit is provided, wherein the calculation unit calculates a feature quantity in the brightness information, The defect detection unit detects the defect based on the feature amount calculated by the calculation unit.

3. The defect inspection device according to claim 1, in, The conversion unit establishes a correspondence between the position of the workpiece model surface and the position on the captured image by performing coordinate conversion between the three-dimensional coordinates of the workpiece model and the image coordinates as the coordinates of the captured image based on the three-dimensional data of the workpiece model and the captured model, and establishes a correspondence between the position of the workpiece model surface and the positions on the captured image by correcting the correspondence between the position of the workpiece model surface and the positions on the captured image based on the position of the workpiece.

4. The defect inspection device according to claim 1, in, The conversion unit corrects the coordinates of the workpiece model based on the position of the workpiece, and associates the position of the workpiece surface with the positions on the plurality of captured images based on the three-dimensional data of the workpiece model after the coordinate correction and the captured model.

5. The defect inspection device according to claim 4, in, The conversion unit corrects the coordinates of the workpiece model based on the position of the workpiece, and establishes a correspondence between the position of the workpiece surface and the positions on the plurality of captured images based on the three-dimensional data of the workpiece model after the coordinate correction and ray tracing of the illumination light irradiated onto the workpiece surface as the captured model.

6. The defect inspection device according to claim 1, in, The conversion unit generates a table indicating correspondence between workpiece coordinates indicating positions on the workpiece surface and image coordinates indicating positions on the plurality of captured images.

7. The defect inspection device according to claim 6, in, The table stores the correspondence between the workpiece coordinates and the image coordinates only for a portion of the image coordinates, The conversion unit associates workpiece coordinates interpolated based on the association with other image coordinates stored in the table including image coordinates closest to the image coordinates not stored in the table with the image coordinates not stored in the table.

8. The defect inspection device according to claim 1, in, The imaging model includes information on a focal length, a position, and a posture of an imaging optical system of an imaging unit that captures the image.

9. The defect inspection device according to claim 1, in, The conversion unit associates workpiece coordinates indicating the position of the workpiece surface with image coordinates indicating the position on the captured image, and associates the predetermined workpiece coordinates with the image coordinates for which there are no coordinates corresponding to the workpiece surface.

10. The defect inspection device according to claim 2, in, The calculation unit generates a sequence of the brightness information by arranging the brightness information under different lighting conditions in the order of change of the lighting conditions, and calculates two or more feature quantities based on the sequence. The defect detection unit generates a multi-channel image by inputting the two or more feature quantities into different multiple channels one by one, and detects features of defects appearing in the multi-channel image through a machine learning model.

11. The defect inspection device according to claim 2, in, The calculation unit converts the brightness information into a vector of a predetermined dimension by interpolation or compression, and calculates the feature amount in the vector.

12. The defect inspection device according to claim 11, in, The calculation unit performs a predetermined weighting on the vector.

13. The defect inspection device according to claim 2, in, The calculation unit generates a sequence of the brightness information by arranging the brightness information under different lighting conditions in the order of change of the lighting conditions, and calculates a feature value based on the sequence. The defect detection unit generates a feature quantity image based on the feature quantity, and detects features of a defect appearing in the feature quantity image through a machine learning model.

14. The defect inspection device according to claim 1, in, The conversion unit associates the position of the workpiece surface with the positions on the plurality of captured images only for pixels having pixel values ​​equal to or greater than a predetermined threshold or within a predetermined range among the pixels of the captured images.

15. The defect inspection device according to claim 1, in, The measuring unit measures the posture of the workpiece. The conversion unit establishes correspondence between the position of the workpiece surface and the positions on the plurality of captured images based on the position and posture of the workpiece measured by the measurement unit and the correspondence between the position of the workpiece model surface and the positions on the captured images captured under a plurality of lighting conditions.

16. The defect inspection device according to claim 2, in, The defect detection unit performs a predetermined process on the feature amount, and adjusts the detection sensitivity of the defect by adjusting a threshold value in the predetermined process.

17. The defect inspection device according to claim 6, in, In the table, variables related to the defect detection sensitivity are associated with the workpiece coordinates. The defect detection unit adjusts the detection sensitivity of the defect corresponding to the predetermined workpiece coordinates by adjusting the variable.

18. The defect inspection device according to claim 1, in, The conversion unit obtains light reflection characteristics from light information obtained by ray tracing of illumination light irradiated on the workpiece surface, and associates the light reflection characteristics respectively corresponding to positions on the workpiece model surface with the positions on the workpiece model surface.

19. The defect inspection device according to claim 1, in, The conversion unit sets an inspection target region, an inspection non-target region, or both of them on the three-dimensional data of the workpiece model.

20. The defect inspection device according to claim 10 or 13, in, The machine learning model outputs an estimate of the accuracy of the defect, The defect detection unit adjusts the detection sensitivity of the defect by adjusting a threshold value in the estimated value.

21. The defect inspection device according to claim 1, in, The conversion unit corrects the imaging model based on position information and posture information of the imaging unit measured using a reference map after the imaging unit that captures the imaging image is set.

Images