Defect inspection device

Abstract

The present invention makes it possible to efficiently perform highly accurate defect inspection.　This defect inspection device 1 comprises: a measurement section (control section 21) that measures a position of a workpiece W by detecting a defect on a surface of the workpiece on the basis of captured images of the surface of the workpiece captured while varying the illumination condition for illuminating the surface of the workpiece, and measures the position of the workpiece W; a conversion section (control section 21) that, on the basis of the position of the workpiece W measured by the measurement section and the correspondence relationship between the position of a workpiece model surface based on three-dimensional data of a workpiece model and an imaging model, and positions on the respective captured images captured under the plurality of illumination conditions, associates the position of the surface of the workpiece with the positions in the plurality of captured images; an acquisition section (control section 21) that, on the basis of the association between the position of the surface of the workpiece and the positions on the plurality of captured images, acquires luminance information under the plurality of illumination conditions for each position on the surface of the workpiece from a corresponding one of the captured images; and a defect detection section (control section 21) that detects a defect on the surface of the workpiece on the basis of the luminance information acquired by the acquisition section.

Description

Defect Inspection Equipment

[0001] The present invention relates to a defect inspection device.

[0002] Conventionally, visual inspection of the appearance of various products has been performed for quality control in the manufacturing process. Visual inspection involves a large number of inspection items, and requires highly skilled inspection skills from inspectors, making it difficult to improve efficiency. For this reason, there is a demand for technology to automate visual inspection (see, for example, Patent Documents 1 to 3).

[0003] JP 2010-216870 A JP 2023-032374 A JP 2013-195368 A

[0004] When inspecting for defects on the surface of a car body as it travels down a production line, the captured images of the car body surface are corrected (motion correction) to accompany the movement of the car body. This makes it necessary to acquire time-series data of pixel values ​​at the same location on the car body surface. If there is a positional shift in this time-series data, the time-series data will be composed of pixel value data from different locations on the car body surface, significantly reducing the accuracy of defect detection. Because the surface of a car body has a complex curved shape, more accurate motion correction is required to acquire time-series data of pixel values ​​at the same location on the car body surface. High-accuracy motion correction requires motion correction on a frame-by-frame basis for captured images of the car body surface taken by multiple cameras. As a result, a great deal of effort is required to set up multiple cameras when introducing a defect inspection system.

[0005] The invention described in Patent Document 1 does not assume that the surface to be inspected is a generally flat steel plate, and does not assume that the surface to be inspected has a complex curved shape. The invention described in Patent Document 2 does not assume that corrections will be made in accordance with the movement of car bodies as they move along the production line, because the object to be inspected is small enough to be grasped by an arm. The invention described in Patent Document 3 does not assume that corrections will be made in accordance with the movement of car bodies as they move along the production line, because the object to be inspected does not move.

[0006] An object of the present invention is to provide a defect inspection apparatus that can efficiently perform highly accurate defect inspection.

[0007] In order to solve the above problem, the defect inspection device described in claim 1 is a defect inspection device that detects defects on the surface of a workpiece based on captured images of the workpiece surface while changing the lighting conditions for illuminating the surface of the workpiece, and comprises: a measurement unit that measures the position of the workpiece; a conversion unit that associates the position of the workpiece surface with positions on multiple captured images based on the position of the workpiece measured by the measurement unit, and the correspondence between the position of the workpiece model surface based on the three-dimensional data of the workpiece model and the captured model, and the positions on each captured image captured under multiple lighting conditions; an acquisition unit that acquires luminance information under multiple lighting conditions at each position on the workpiece surface from the corresponding captured images by correlating the position of the workpiece surface with the positions on the multiple captured images; and a defect detection unit that detects defects on the surface of the workpiece based on the luminance information acquired by the acquisition unit.

[0008] The invention of claim 2 is the defect inspection device of claim 1, further comprising a calculation unit that calculates a feature amount in the luminance information, and the defect detection unit detects the defect based on the feature amount calculated by the calculation unit.

[0009] The invention described in claim 3 is a defect inspection device described in claim 1, wherein the conversion unit corresponds the position of the work model surface to a position on the captured image by performing a coordinate conversion between the three-dimensional coordinates of the work model and image coordinates, which are the coordinates of the captured image, based on the three-dimensional data of the work model and the captured model, and corresponds the position of the work surface to a position on the multiple captured images by correcting the correspondence between the position of the work model surface and the position on the captured image based on the position of the work.

[0010] The invention described in claim 4 is a defect inspection device described in claim 1, wherein the conversion unit corrects the coordinates of the work model based on the position of the work, and corresponds the position of the work surface to positions on the multiple captured images based on the three-dimensional data of the work model whose coordinates have been corrected and the captured model.

[0011] The invention described in claim 5 is a defect inspection device described in claim 4, wherein the conversion unit corrects the coordinates of the work model based on the position of the work, and corresponds the position of the work surface to positions on the multiple captured images based on the three-dimensional data of the work model whose coordinates have been corrected and ray tracing of illumination light irradiated onto the work surface as the captured model.

[0012] The invention described in claim 6 is the defect inspection device described in claim 1, wherein the conversion unit generates a table showing the correspondence between work coordinates indicating the position of the work surface and image coordinates indicating positions on the multiple captured images.

[0013] The invention described in claim 7 is a defect inspection device described in claim 6, wherein the table stores correspondences between the work coordinates and the image coordinates only for some of the image coordinates, and the conversion unit associates the image coordinates not stored in the table with work coordinates interpolated based on the correspondences corresponding to other image coordinates stored in the table and including image coordinates closest to the image coordinates not stored in the table.

[0014] According to an eighth aspect of the present invention, in the defect inspection apparatus of the first aspect, the imaging model includes information on a focal length, a position, and an attitude of an imaging optical system of an imaging unit that captures the captured image.

[0015] The invention described in claim 9 is a defect inspection device described in claim 1, wherein the conversion unit associates work coordinates indicating the position of the work surface with image coordinates indicating a position on the captured image, and associates the predetermined work coordinates with the image coordinates for which no coordinates corresponding to the work surface exist.

[0016] The invention described in claim 10 is a defect inspection device described in claim 2, wherein the calculation unit generates a sequence of the brightness information by arranging the brightness information under different lighting conditions in the order in which the lighting conditions change, and calculates two or more feature amounts from the sequence; and the defect detection unit generates a multi-channel image by inputting the two or more feature amounts one by one into a plurality of different channels, and detects features of defects that appear in the multi-channel image using a machine learning model.

[0017] According to an eleventh aspect of the present invention, in the defect inspection apparatus according to the second aspect, the calculation unit converts the luminance information into a vector of a predetermined number of dimensions by interpolating or compressing it, and calculates the feature amount in the vector.

[0018] A twelfth aspect of the present invention provides the defect inspection apparatus according to the eleventh aspect, wherein the calculation section assigns a predetermined weight to the vector.

[0019] The invention of claim 13 is the defect inspection device of claim 2, wherein the calculation unit generates a sequence of the brightness information by arranging the brightness information under different lighting conditions in the order of change in the lighting conditions, and calculates one feature from the sequence; and the defect detection unit generates a feature image based on the feature, and detects features of defects that appear in the feature image using a machine learning model.

[0020] The invention described in claim 14 is a defect inspection device described in claim 1, wherein the conversion unit associates the position of the work surface with positions on the multiple captured images only for pixels of the captured images that have pixel values ​​above a predetermined threshold or within a predetermined range.

[0021] The invention described in claim 15 is a defect inspection device described in claim 1, wherein the measurement unit measures the posture of the workpiece, and the conversion unit corresponds the position of the workpiece surface with positions on the multiple captured images based on the position and posture of the workpiece measured by the measurement unit and the correspondence between the position of the workpiece model surface and the positions on each captured image taken under multiple lighting conditions.

[0022] According to a sixteenth aspect of the present invention, in the defect inspection apparatus of the second aspect, the defect detection unit performs a predetermined process on the feature amount and adjusts a threshold value in the predetermined process to adjust defect detection sensitivity.

[0023] The invention described in claim 17 is a defect inspection device described in claim 6, wherein in the table, the work coordinates are associated with variables related to defect detection sensitivity, and the defect detection unit adjusts the detection sensitivity of the defect corresponding to the specified work coordinates by adjusting the variables.

[0024] The invention described in claim 18 is a defect inspection device described in claim 1, wherein the conversion unit acquires light reflection characteristics from light ray information obtained by ray tracing of illumination light irradiated onto the work surface, and associates the light reflection characteristics that correspond to each position on the work model surface.

[0025] According to a nineteenth aspect of the present invention, in the defect inspection device of the first aspect, the conversion unit sets an inspection target area, a non-inspection target area, or both, on the three-dimensional data of the workpiece model.

[0026] The invention described in claim 20 is a defect inspection device described in claim 10 or 13, wherein the machine learning model outputs an estimated value representing the likelihood of the defect, and the defect detection unit adjusts the defect detection sensitivity by adjusting a threshold value for the estimated value.

[0027] The invention of claim 21 is the defect inspection device of claim 1, wherein the conversion unit corrects the imaging model based on position information and orientation information of the imaging unit that captures the captured image, measured using a reference chart after installation of the imaging unit.

[0028] According to the present invention, highly accurate defect inspection can be carried out efficiently.

[0029] 1 is a block diagram of a defect inspection device according to the present embodiment. FIG. 2 is a side view showing the configuration of an inspection unit. FIG. 3 is a front view showing the configuration of an inspection unit. FIG. 4 is a flowchart showing an example of the flow of a reference position registration process. FIG. 5 is a diagram showing an example of an imaging range for a workpiece by an imaging unit. FIG. 6 is a diagram showing an example of successively captured images of a workpiece. FIG. 7 is a flowchart showing an example of the flow of a defect inspection process. FIG. 8 is a diagram showing an example of a reference position and a measurement position of an edge in successively captured images of a workpiece. FIG. 9 is a diagram showing an example of normalizing time-series data of pixel values. FIG. 10 is a diagram showing an example of generating a composite image from time-series data of pixel values. FIG. 11 is a diagram showing an example of dividing a composite image into tiles. FIG. 12 is a diagram showing an example of a convex defect. FIG. 13 is a diagram showing an example of a concave defect. FIG. 14 is a diagram showing an example of a thread defect. FIG. 15 is a side view showing the configuration of an inspection unit according to Modification 1. FIG. 16 is a diagram showing an example of measuring the distance to a workpiece using a laser rangefinder according to Modification 4.

[0030] Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to those shown in the drawings.

[0031] <Configuration of Defect Inspection Apparatus> The configuration of the defect inspection apparatus according to this embodiment will be described. FIG. 1 is a block diagram of the defect inspection apparatus 1 according to this embodiment. As shown in FIG. 1, the defect inspection apparatus 1 includes an inspection unit 10 and a processing device 20. FIG. 2A is a side view showing the configuration of the inspection unit 10, and FIG. 2B is a front view showing the configuration of the inspection unit 10. In FIGS. 2A and 2B, the X-axis direction is a direction parallel to the transport direction of the workpiece W, which is the inspection target in the defect inspection process described below, and the Y-axis direction is the width direction of the workpiece W. The X-axis direction and the Y-axis direction are two horizontal directions perpendicular to each other, and the vertical direction perpendicular to the X-axis and Y-axis is the Z-axis direction.

[0032] The defect inspection device 1 is a device for inspecting surface defects on a workpiece W. In this embodiment, the workpiece W is a car body. The inspected portion of the workpiece W is the painted surface of the car body, and the defect inspection device 1 inspects the painted surface of the car body for surface defects. Generally, the car body surface is subjected to a primer treatment, a metallic paint, a clear paint, etc., to form a multi-layered paint film layer, but uneven defects occur in the top clear layer due to the influence of foreign matter during painting, etc. In this embodiment, it is applied to the inspection of such defects, but the workpiece W is not limited to a car body and may be something other than a car body. The inspected portion may be a surface other than a painted surface.

[0033] The inspection unit 10 includes a transport unit 11, an imaging unit 12, an illumination unit 13, a detection unit 14, a position detection unit 15, etc. The transport unit 11 transports the workpiece W placed on the cart D in the positive direction of the X-axis at a predetermined speed by a conveyor 111.

[0034] The illumination unit 13 is provided at the middle of the conveyor 111 in the longitudinal direction (X-axis direction). When viewed from the transport direction (X-axis direction) of the workpiece W, the illumination unit 13 is formed in a gate shape as shown in the front view of FIG. 2B. In other words, the illumination unit 13 is formed so as to surround the circumferential surface of the workpiece W except for the bottom surface. The illumination unit 13 includes a light source 131 that illuminates the workpiece W. The light source 131 is, for example, an LED (Light Emitting Diode). The illumination unit 13 linearly illuminates the circumferential surface of the workpiece W except for the bottom surface thereof using the light source 131.

[0035] The imaging unit 12 includes two imaging units 12 installed so as to be able to image the workpiece W from the positive direction of the Z axis, as shown in Fig. 2B. The imaging unit 12 includes two imaging units 12 installed so as to be able to image the workpiece W from the positive direction of the Y axis, as shown in Fig. 2B. The imaging unit 12 includes two imaging units 12 installed so as to be able to image the workpiece W from the negative direction of the Y axis, as shown in Fig. 2B. The imaging units 12 continuously image the workpiece W, which is moved at a predetermined speed by the transport unit 11 and illuminated by the illumination unit 13, with each part of the workpiece W in the circumferential direction as an inspection area.

[0036] The detection unit 14 is a sensor that detects that the workpiece W being transported by the conveyor 111 has reached the inspection start position A. The detection unit 14 is connected to the processing device 20 via wiring (not shown), and outputs the detection result to the processing device 20. The detection unit 14 may also detect that the cart D being transported by the conveyor 111 has reached the inspection start position A.

[0037] The position detection unit 15 detects, by an encoder or the like, information on the travel distance of the workpiece W being transported by the conveyor 111 from the inspection start position A. The position detection unit 15 is connected to the processing device 20 via wiring (not shown), and outputs the detected travel distance information of the workpiece W to the processing device 20.

[0038] As described above, in the inspection unit 10, the imaging unit 12 and the illumination unit 13 are fixed, and the workpiece W is moved by the transport unit 11, but this is not limiting. The imaging unit 12 and the illumination unit 13 may move relative to the workpiece W that is stationary.

[0039] The processing device 20 is, for example, a personal computer. The processing device 20 is connected to the transport unit 11 and the imaging unit 12 of the inspection unit 10 via wiring (not shown), and controls the operations of the transport unit 11 and the imaging unit 12. The processing device 20 may also be connected to the illumination unit 13 via wiring, and controls the operation of the illumination unit 13. The processing device 20 includes a control unit 21, a storage unit 22, a communication unit 23, an operation unit 24, and a display unit 25.

[0040] The control unit 21 includes, for example, a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) and a memory such as a RAM (Random Access Memory). The control unit 21 executes a program 22a stored in a memory such as the RAM, the storage unit 22, etc., to perform various processes including defect inspection of the workpiece W. The control unit 21 controls the transport unit 11 of the inspection unit 10 to transport the workpiece W in the positive direction of the X-axis. The control unit 21 controls the imaging unit 12 of the inspection unit 10 to continuously capture images of the surface of the workpiece W at predetermined intervals. The control unit 21 determines the timing of imaging by the imaging unit 12 based on the detection results of the detection unit 14 of the inspection unit 10 and the progress distance information of the workpiece W detected by the position detection unit 15.

[0041] The storage unit 22 includes any storage module, such as a hard disk drive (HDD), a solid state drive (SSD), a read only memory (ROM), and a RAM. The storage unit 22 stores, for example, a system program, an application program, and various data. Specifically, the storage unit 22 stores a program 22a for executing a defect inspection process for the workpiece W. The storage unit 22 stores a first coordinate conversion table that converts coordinates (image coordinates) in the captured image captured by the imaging unit 12 and two-dimensional coordinates of the workpiece model. The two-dimensional coordinates of the workpiece model are two-dimensional coordinates that are set based on the workpiece W in the workpiece model.

[0042] The communication unit 23 includes, for example, a communication module including a network interface card (NIC), a receiver, and a transmitter, etc. The communication unit 23 communicates various information and data with external devices connected via a network such as the Internet.

[0043] The operation unit 24 includes, for example, a mouse, a keyboard, switches, buttons, etc. The operation unit 24 may be, for example, a touch panel integrally combined with the display unit 25, or an interface that accepts voice input. The operation unit 24 accepts instructions corresponding to various input operations from the user, converts the accepted instructions into operation signals, and outputs the operation signals to the control unit 21.

[0044] The display unit 25 is, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, etc. The display unit 25 performs display based on display data output from the control unit 21.

[0045] <First Coordinate Conversion Table> Next, the first coordinate conversion table for converting image coordinates and workpiece model two-dimensional coordinates will be described. The control unit 21 of the processing device 20 generates the first coordinate conversion table based on three-dimensional data of the workpiece model and ray tracing (image model) of illumination light irradiated onto the surface of the workpiece W using ray simulation software that uses information about the imaging optical system. The three-dimensional data of the workpiece model consists of shape data representing the three-dimensional shape of the workpiece W. The information about the imaging optical system includes, for example, the focal length, position, and orientation of the imaging unit 12. The ray simulation software may be CG (Computer Graphics) software. Specifically, for all captured images captured during the defect inspection process, the control unit 21 associates coordinates on the captured image (image coordinates) with 3D coordinates on the workpiece model (workpiece model three-dimensional coordinates) for each frame of the captured image. This allows the control unit 21 to associate positions on the captured image with positions on the workpiece model surface. Image coordinates are expressed as (X, Z). The workpiece model three-dimensional coordinates are represented by (CX, CY, CZ). The control unit 21 associates image coordinates that do not correspond to the workpiece model three-dimensional coordinates with coordinates that are outside the workpiece model three-dimensional coordinates. Coordinates that are outside the workpiece model three-dimensional coordinates are, for example, (CX, CY, CZ) = (0, 0, 0). The control unit 21 sets areas to be excluded from the inspection target area as non-inspection areas on the workpiece model. The control unit 21 associates coordinates that are outside the workpiece model three-dimensional coordinates with image coordinates that correspond to areas to be excluded from the inspection target area. The control unit 21 may set inspection target areas, non-inspection areas, or both, on the three-dimensional data of the workpiece model.

[0046] Next, the control unit 21 generates a first coordinate conversion table by converting the workpiece model three-dimensional coordinates into workpiece model two-dimensional coordinates, and stores the table in the storage unit 22. As a result, the first coordinate conversion table for converting the image coordinates into the workpiece model two-dimensional coordinates is expressed by the following formula (1): Formula (1) (MX, MZ) = f(frame, X, Z) "MX" and "MZ" represent coordinate positions in the workpiece model two-dimensional coordinates defined by the X-axis and Z-axis which are orthogonal to each other. "X" and "Z" represent coordinate positions in the captured image captured by the imaging unit 12, which is a two-dimensional plane defined by the X-axis and Z-axis which are orthogonal to each other. "frame" is the frame number in the continuous captured images captured by the imaging unit 12.

[0047] <Operation of Defect Inspection Apparatus> Next, a description will be given of the operation of the defect inspection apparatus 1 according to this embodiment. Before executing the defect inspection process on the workpiece W, the control unit 21 of the processing device 20 executes the reference position registration process shown in FIG.

[0048] (Reference Position Registration Process) The control unit 21 controls the transport unit 11 to transport the workpiece W in the positive direction of the X-axis. Next, the control unit 21 determines the timing for capturing images by the imaging unit 12 based on the detection results of the detection unit 14 and the progress distance information of the workpiece W detected by the position detection unit 15. Next, the control unit 21 captures images of the workpiece W being transported continuously at predetermined intervals using the imaging unit 12 at the determined capturing timing to acquire consecutive captured images of the workpiece W (step A1). FIG. 4 shows an example of the imaging range H of the workpiece W captured by the imaging unit 12. FIG. 5 shows an example of the consecutive captured images acquired in step A1. As shown in FIG. 5, the control unit 21 captures images using the imaging unit 12 so that the imaging ranges of captured images with previous and next frame numbers largely overlap. As a result, the control unit 21 acquires multiple captured images in which the imaging ranges of the captured images are consecutively shifted in the transport direction of the workpiece W.

[0049] Next, the control unit 21 extracts, from the continuous images of the workpiece W acquired in step A1, images that were captured near the start of image capture and that include characteristic edges. The control unit 21 also extracts, from the continuous images of the workpiece W acquired in step A1, images that were captured near the end of image capture and that include characteristic edges. In the example shown in FIG. 5 , the control unit 21 extracts the image with frame number "1" that includes characteristic edge E1 and the image with frame number "4" that includes characteristic edge E2. Next, the control unit 21 extracts edge regions from the extracted images (step A2).

[0050] Next, the control unit 21 acquires reference edge information, which is information about the edge region extracted in step A2 (step A3). The edge information includes the frame number of the image including the edge and edge position information, which is shape information of the edge. The reference edge information includes reference position information, which is position information of the reference edge. The control unit 21 may acquire edge information from the entire image including the extracted edge without extracting the edge region from the extracted image. Next, the control unit 21 registers the reference position information by storing the reference edge information acquired in step A3 in the storage unit 22 (step A4).

[0051] Next, the flow of the defect inspection process (see FIG. 6 ) in which the defect inspection device 1 inspects surface defects of the workpiece W will be described. By executing the defect inspection process, the defect inspection device 1 detects defects on the surface of the workpiece W based on captured images of the surface of the workpiece W while changing the illumination state illuminating the surface of the workpiece W. Changing the illumination state illuminating the surface of the workpiece W includes changing the position of a bright zone on the surface of the workpiece W due to the workpiece W moving relative to the illumination unit 13. The bright zone is the portion of the captured image illuminated by the illumination unit 13.

[0052] (Defect Inspection Processing) Similar to step A1 of the reference position registration processing, the control unit 21 of the processing device 20 continuously captures images of the workpiece W being transported at predetermined intervals using the imaging unit 12 to acquire consecutively captured images of the workpiece W (step B1). Next, the control unit 21 performs template matching on the consecutively captured images of the workpiece W acquired in step B1 using the reference edge information saved in step A4 as template data. As a result, the control unit 21 detects edges from the consecutively captured images of the workpiece W (step B2). Next, the control unit 21 acquires edge information of the edges detected in step B2 (step B3). The edge information of the edges detected in step B2 includes measurement position information, which is position information of the edges. In step B3, the control unit 21 measures the position of the workpiece W by acquiring the measurement position information of the edges. The control unit 21 functions as a measurement unit. Next, the control unit 21 compares the reference position information registered in step A4 with the measurement position information of the edges acquired in step B3. As a result, the control unit 21 calculates a position correction amount, which is the difference between the reference position indicated by the reference position information and the measurement position indicated by the measurement position information (step B4).

[0053] The difference between the reference position and the measurement position occurs due to a shift in the transport direction (X-axis direction) of the workpiece W in the captured image, a shift in the vehicle height direction (Z-axis direction) of the workpiece W in the captured image, etc. The difference between the reference position and the measurement position may also occur due to enlargement, reduction, or rotation in the XY plane of the workpiece W in the captured image. If the cart D is not placed horizontally on the conveyor 111, or if the workpiece W is not placed horizontally on the cart D, a shift in the transport direction and vehicle height direction of the workpiece W occurs in the captured image. If there is variation in the height (width in the Z-axis direction) of the cart D, a shift in the vehicle height direction of the workpiece W occurs in the captured image. If the detection unit 14 detects that the cart D has reached the inspection start position A and the position of the workpiece W relative to the cart D is deviated from the reference position, a shift in the transport direction of the workpiece W occurs in the captured image.

[0054] FIG. 7 shows an example of the reference position and measurement position of edges in consecutively captured images of the workpiece W. In the example shown in FIG. 7, the difference between the reference position of edge E1 in the image with frame number "1" and the measurement position of edge E11 is (X, Z) = (8 pix, 3 pix). The difference between the reference position of edge E2 in the image with frame number "4" and the measurement position of edge E22 is (X, Z) = (2 pix, 0 pix). In this case, in step B4, the control unit 21 sets the position correction amount in the image with frame number "1" to (X, Z) = (8 pix, 3 pix). The control unit 21 sets the position correction amount in the image with frame number "4" to (X, Z) = (2 pix, 0 pix). Next, the control unit 21 calculates the position correction amount in the images with frame numbers "2," "3," and "5" by linear interpolation based on the position correction amounts in the image with frame number "1" and the image with frame number "4." The control unit 21 may calculate the amount of position correction from the difference between the reference position and the measurement position for all of the consecutively captured images.

[0055] Next, the control unit 21 corrects a first coordinate conversion table that converts image coordinates and workpiece model two-dimensional coordinates based on the position correction amount calculated in step B4 (step B5). In step B5, if the position correction amount is (X, Z) = (ΔX, ΔZ), the control unit 21 corrects the first coordinate conversion table using the following formula (2): (MX, MZ) = f(frame, X - ΔX, Z - ΔZ) Formula (2) For example, since the position correction amount for the image with frame number "1" is (X, Z) = (8 pix, 3 pix), the corrected first coordinate conversion table corresponding to the image with frame number "1" is expressed by the following formula (3). (MX, MZ) = f(1, X - 8pix, Z - 3pix) Equation (3) For example, the position correction amount for the image with frame number "4" is (X, Z) = (2pix, 0pix), so the corrected first coordinate conversion table corresponding to the image with frame number "4" is expressed by the following equation (4): (MX, MZ) = f(4, X - 2pix, Z - 0pix) Equation (4)

[0056] The first coordinate conversion table corrected in step B5 is a second coordinate conversion table that converts image coordinates into workpiece coordinates. In other words, the second coordinate conversion table indicates the correspondence between workpiece coordinates indicating the position of the surface of the workpiece W and image coordinates indicating positions on multiple captured images. As described above, the control unit 21 generates the second coordinate conversion table based on the position of the workpiece W measured in step B3 and the first coordinate conversion table. As a result, the control unit 21 associates the position of the workpiece surface (workpiece coordinates) with positions on multiple captured images (image coordinates). The control unit 21 functions as a conversion unit. The first coordinate conversion table indicates the correspondence between the position of the workpiece model surface based on the three-dimensional data of the workpiece model and the captured model and the positions on each captured image captured under multiple lighting conditions.

[0057] As described above, the control unit 21 associates image coordinates that do not correspond to the workpiece model three-dimensional coordinates with coordinates that are outside the workpiece model three-dimensional coordinates. Therefore, the control unit 21 associates predetermined workpiece coordinates with image coordinates that do not correspond to the workpiece surface in the second coordinate conversion table. The predetermined workpiece coordinates are, for example, (0, 0).

[0058] Next, the control unit 21 extracts pixels whose pixel values ​​are equal to or greater than a predetermined value (e.g., 40) from all captured images of the workpiece W acquired in step B1 (step B6). Pixels in the captured images whose pixel values ​​are equal to or greater than the predetermined value are pixels near the bright zone. Next, the control unit 21 performs coordinate conversion to convert the image coordinates of the pixels extracted in step B6 into workpiece coordinates using a second coordinate conversion table (step B7). In other words, the control unit 21 associates the position of the workpiece surface (workpiece coordinates) with positions on the multiple captured images (image coordinates) only for pixels in the captured images that have pixel values ​​equal to or greater than a predetermined threshold or within a predetermined range.

[0059] The second coordinate conversion table of this embodiment stores correspondences between image coordinates and work coordinates only for a portion of all pixels in the captured image (e.g., 1 / 16 of the total number of pixels). For pixels for which such correspondences are not stored, the control unit 21 performs interpolation using correspondences of other stored pixels. In other words, the control unit 21 associates image coordinates not stored in the second coordinate conversion table (unstored coordinates) with work coordinates interpolated based on the correspondences of other image coordinates stored in the second coordinate conversion table. The other image coordinates include the image coordinates closest to the unstored coordinates and are the image coordinates of four points that form a rectangle surrounding the unstored coordinates.

[0060] Next, the control unit 21 arranges the pixel values ​​at each work coordinate after the coordinate transformation in step B7 in the order of the frame numbers of the captured image, and acquires time-series data of the pixel values ​​at each work coordinate (step B8). In other words, the control unit 21 associates the position on the work surface (work coordinate) with the positions on the multiple captured images (image coordinates), and acquires luminance information under multiple lighting conditions at each position on the work surface (each work coordinate) from the corresponding captured image. The control unit 21 functions as an acquisition unit. The luminance information includes pixel values.

[0061] Next, as shown in FIG. 8 , the control unit 21 normalizes the time-series data of pixel values ​​at each work coordinate acquired in step B8 to 32 pieces of data. The example shown in FIG. 8 illustrates a case where a defect (a convex defect) protruding from the vehicle body surface exists on the vehicle body surface. If the time-series data of pixel values ​​at each work coordinate acquired in step B8 does not have 32 pieces of data, the control unit 21 converts the acquired time-series data of pixel values ​​into 32 pieces of data by interpolating or compressing the data. Next, the control unit 21 divides the 32 pieces of data corresponding to each work coordinate into eight pieces, i.e., the time-series data of pixel values, into four regions, and calculates a representative value of the eight pieces of data included in each region. The representative value of the eight pieces of data is a feature such as the average, median, or mode of the eight pieces of data. The control unit 21 then compresses the time-series data of pixel values ​​at each work coordinate by calculating four representative values ​​corresponding to each of the four regions (step B9). That is, the control unit 21 generates a sequence of luminance information (time-series data) by arranging the luminance information under different lighting conditions in the order of change in the lighting conditions (chronological order), and calculates two or more feature quantities from the sequence. In other words, the control unit 21 calculates the feature quantities in the luminance information. The control unit 21 functions as a calculation unit.

[0062] Next, the control unit 21 inputs the four representative values ​​corresponding to the four regions calculated in step B9 into four channels, R (Red), G (Green), B (Blue), and A (Alpha), one by one. The control unit 21 then generates a composite image, which is a color image of the time-series data of pixel values ​​(step B10). In the example shown in FIG. 8 , the control unit 21 inputs the representative value for region R1 into the A (Alpha) channel. The control unit 21 inputs the representative value for region R2 into the B (Blue) channel. The control unit 21 inputs the representative value for region R3 into the G (Green) channel. The control unit 21 inputs the representative value for region R4 into the R (Red) channel. In step B10, the control unit 21 may generate a composite image of the time-series data of pixel values ​​by inputting the representative values ​​for the divided regions of the time-series data of pixel values ​​into three channels, R, G, and B, one by one. As described above, by displaying time-series data of pixel values ​​as a color image, temporal changes in pixel values ​​can be easily recognized by the human eye, which makes it easier to perform annotation work for machine learning in inspection models (described later).

[0063] Fig. 9 shows an example of a composite image P1 in the case where a convex defect has occurred on the surface of the vehicle body. As shown in Fig. 9, in the composite image P1, a phase shift, which is the timing at which a mirror image of the light source 131 of the illumination unit 13 appears, appears as a difference in color. In the example shown in Fig. 9, a green portion Pg displayed in green in the composite image P1 is a region where no surface defect has occurred. A red portion Pr displayed in red in the composite image P1 is a region where the timing at which the mirror image of the light source 131 appears is earlier than in the green portion Pg due to the convex defect. A blue portion Pb displayed in blue in the composite image P1 is a region where the timing at which the mirror image of the light source 131 appears is later than in the green portion Pg due to the convex defect.

[0064] Next, the control unit 21 divides the composite image generated in step B10 (step B11). Fig. 10 shows an example of a composite image P2 generated by executing step B10 for all work coordinates. In step B11, the control unit 21 divides the entire composite image P2 into tiles T, each having a size of 640 x 640 pixels. The control unit 21 divides the composite image P2 so that there is, for example, a 30% overlap between each tile T.

[0065] Next, the control unit 21 inputs each tile T divided in step B11 into an inspection model and obtains an estimated value representing the likelihood of a defect, which is the output result of the inspection model. Next, if the estimated value is equal to or greater than a preset threshold, the control unit 21 determines that a surface defect has occurred, detects the surface defect (step B12), and terminates the defect inspection process. The inspection model is a machine learning model that has previously been trained on defects on the vehicle body surface, such as a machine learning model using TensorFlow. In other words, the control unit 21 detects defects on the surface of the workpiece W based on the brightness information acquired in step B8. The control unit 21 functions as a defect detection unit. In this embodiment, the control unit 21 generates a multi-channel image (synthetic image) by inputting two or more feature amounts in the brightness information into different channels one by one, and detects the features of defects that appear in the multi-channel image using a machine learning model (inspection model).

[0066] The control unit 21 may perform the following processing on the inspection model. Specifically, the control unit 21 may adjust the defect detection sensitivity by adjusting a threshold for an estimated value representing the likelihood of a defect, which is the output result of the inspection model. The control unit 21 may set the threshold uniformly for all tiles T, or may set the threshold for each work coordinate in the second coordinate conversion table. In other words, when a variable related to defect detection sensitivity is associated with a work coordinate in the second coordinate conversion table, the control unit 21 adjusts the defect detection sensitivity for a specific work coordinate by adjusting the variable. In this case, the control unit 21 may increase the defect detection sensitivity for work coordinates corresponding to a conspicuous location on the vehicle body, or decrease the defect detection sensitivity for work coordinates corresponding to an inconspicuous location or a noisy location on the vehicle body.

[0067] 11A to 11C show an example of a defect on the surface of a vehicle body that the inspection model learns in advance by machine learning. The example shown in FIG. 11A is a convex defect in which a foreign object 330 is mixed into a coating layer 320 formed on a vehicle body material 310, causing the surface of the coating layer 320 to protrude compared to other areas. In the composite image in this case, the green portion Pg is a region where no surface defect occurs. The red portion Pr is a region where the timing at which the mirror image of the light source 131 appears is earlier than in the green portion Pg due to the convex defect. The blue portion Pb is a region where the timing at which the mirror image of the light source 131 appears is later than in the green portion Pg due to the convex defect.

[0068] The example shown in Figure 11B is a concave defect in which, after a coating film layer 320 is applied to a vehicle body material 310, the paint does not adhere uniformly to the vehicle body due to a foreign object 330 or the like, causing a partial depression in the coating film layer 320. In the composite image in this case, the green portion Pg is an area where no surface defect occurs. The red portion Pr is an area where, due to the concave defect, the timing at which the mirror image of the light source 131 appears is delayed compared to the green portion Pg. The blue portion Pb is an area where, due to the concave defect, the timing at which the mirror image of the light source 131 appears is earlier compared to the green portion Pg.

[0069] The example shown in Figure 11C is a nuisance, which is a linear protrusion on the surface of the coating layer 320 formed on the vehicle body material 310 due to the inclusion of a thread-like foreign object 330 in the coating layer 320. In this composite image, the green portion Pg is a region where no surface defects occur. The red portion Pr is a thread-like region where the timing at which the mirror image of the light source 131 appears is earlier than in the green portion Pg due to the nuisance. The blue portion Pb is a thread-like region where the timing at which the mirror image of the light source 131 appears is later than in the green portion Pg due to the nuisance. The defects on the vehicle body surface that the inspection model learns by machine learning in advance include paint defects such as scratches, discoloration, pits, and drips, in addition to convex defects, concave defects, and nuisances.

[0070] <Modifications> Although the present embodiment has been described above, the specific configuration is not limited to the above embodiment and can be modified within the scope of the gist of the invention. Modifications of the present embodiment will be described below. In the modifications, the same components as those in the above embodiment will be assigned the same reference numerals, and their description will be omitted.

[0071] (Variation 1) Fig. 12 is a side view showing the configuration of an inspection unit 10A of Variation 1. The inspection unit 10A of Variation 1 includes an imaging unit 12A, an illumination unit 13A, a robot arm 16A, etc. The imaging unit 12A and the illumination unit 13A are mounted on the tip of the robot arm 16A. The robot arm 16A is an articulated robot. Under the control of the control unit 21, the robot arm 16A can position the imaging unit 12A and the illumination unit 13A at a set predetermined distance from the surface of the workpiece W, which is a curved surface.

[0072] The control unit 21 of the processing device 20 of Modification 1 controls the imaging unit 12A while the illumination unit 13A is turned on to capture an image of the surface of the stationary workpiece W. The control unit 21 controls the robot arm 16A to move the imaging unit 12A and the illumination unit 13A to capture images of multiple locations on the surface of the workpiece W. In this way, the control unit 21 obtains captured images of multiple locations on the surface of the workpiece W.

[0073] (Variation 2) The control unit 21 of the processing device 20 of Variation 2 executes the following process instead of steps B1 to B3 of the above-described defect inspection process. Specifically, the control unit 21 continuously captures images of the workpiece W being transported at predetermined intervals using multiple imaging units 12 to acquire multiple consecutively captured images of the workpiece W. The multiple imaging units 12 include an imaging unit 12 that captures images of the workpiece W from the positive direction of the Z axis, an imaging unit 12 that captures images of the workpiece W from the positive direction of the Y axis, and an imaging unit 12 that captures images of the workpiece W from the negative direction of the Y axis. Next, the control unit 21 performs template matching on the consecutively captured images of the workpiece W corresponding to each imaging unit 12, using the reference edge information saved in step A4 as template data. As a result, the control unit 21 detects edges from the consecutively captured images of the workpiece W corresponding to each imaging unit 12. Next, the control unit 21 acquires edge information of the edges detected from the consecutively captured images corresponding to each imaging unit 12. The edge information of the detected edges includes measurement position information of the detected edges. The control unit 21 measures the position and orientation of the workpiece W by acquiring edge measurement position information from the successively captured images corresponding to each imaging unit 12. This allows the position and orientation of the workpiece W to be measured more accurately.

[0074] The control unit 21 of variant example 2 generates a second coordinate conversion table by correlating the position of the work surface with positions on multiple captured images based on the measured position and posture of the work W and the correspondence (first coordinate conversion table) between the position of the work model surface and the position on each captured image taken under multiple lighting conditions.

[0075] (Variation 3) The system 1 of Variation 3 includes a position measurement unit (not shown) that cooperates with the processing device 20 to execute steps B1 to B5 of the defect inspection process. The configuration of the position measurement unit is similar to that of the inspection unit 10. The position measurement unit is installed upstream of the inspection unit 10 in the transport direction of the workpiece W. In Variation 3, the control unit 21 of the processing device 20 transports the workpiece W to be inspected so that it passes through the position measurement unit, and uses the position measurement unit to execute steps B1 to B5 of the defect inspection process on the workpiece W. As a result, the control unit 21 generates a second coordinate conversion table by correcting the first coordinate conversion table. Next, the control unit 21 transports the workpiece W so that it passes through the inspection unit 10, and executes steps B1 and B6 to B12 of the defect inspection process on the workpiece W. In step B6, the control unit 21 converts image coordinates into workpiece coordinates using the second coordinate conversion table that was previously generated, and executes the subsequent processes. This allows coordinate conversion to be performed using the second coordinate conversion table that has been generated in advance, so that in the defect inspection process, the processes from step B6 onwards can be executed immediately after the workpiece W is imaged.

[0076] 13 in addition to the conveying unit 11, the imaging unit 12, the lighting unit 13, the detection unit 14, and the position detection unit 15. The laser rangefinders 17 are installed on the positive Z-axis side of the workpiece W. The laser rangefinders 17 measure the distance to the surface of the workpiece W near the front end, middle, or rear end of the workpiece W in the X-axis direction, and output the measurement results to the processing device 20.

[0077] The control unit 21 of the processing device 20 of Modification 4 executes the following process instead of the reference position registration process. Specifically, the control unit 21 measures the distance to the surface of the workpiece W using the multiple laser rangefinders 17 and acquires the measurement results. Next, the control unit 21 stores the distance information to the surface of the workpiece W in the memory unit 22, thereby registering the distance information to the surface of the workpiece W as reference position information.

[0078] The control unit 21 of the processing device 20 of Modification 4 executes the following process instead of steps B2 to B4 of the defect inspection process described above. Specifically, the control unit 21 measures the distance to the surface of the workpiece W using multiple laser rangefinders 17 and acquires the measurement results. Next, the control unit 21 compares the reference position information stored in the memory unit 22 with the distance information (measured position information) to the surface of the workpiece W, which is the measurement result. The control unit 21 then calculates a position correction amount, which is the difference between the reference position W0 indicated by the reference position information and the measured position indicated by the distance information to the surface of the workpiece W. For example, if the reference position in the Z-axis direction is 1500 mm and the measured position in the Z-axis direction is 1490 mm, the control unit 21 calculates the position correction amount in the Z-axis direction as -10 mm. This allows the spatial position of the workpiece W to be measured. This allows the difference between the reference position and the measured position due to rotation of the workpiece W in the XY plane to be measured.

[0079] The control unit 21 of the processing device 20 of Modification 4 may execute the following process before the defect inspection process for the workpiece W. Specifically, the control unit 21 continuously captures images of the workpiece W during transport using the imaging unit 12 at predetermined intervals to obtain continuous captured images of the workpiece W. Next, the control unit 21 calculates the length per 1 pix of the surface of the workpiece W in the captured images of the workpiece W. The control unit 21 may calculate the length per 1 pix of the surface of the workpiece W based on the design value of the workpiece W without using the captured images of the workpiece W. In this case, the control unit 21 converts the position correction amount, which is the difference between the reference position W0 indicated by the reference position information and the measurement position indicated by the distance information to the surface of the workpiece W, into pix. For example, if the position correction amount is −10 mm and the length per 1 pix of the surface of the workpiece W is 1 mm, the control unit 21 calculates the position correction amount to be −10 pix.

[0080] (Variation 5) In step B5 of the defect inspection process in the above embodiment, the control unit 21 corrects the coordinate conversion table that converts image coordinates and workpiece coordinates based on the position correction amount calculated in step B4, but this is not limited to this. The control unit 21 of the processing device 20 in Variation 5 may execute the following process in step B5 of the defect inspection process. Specifically, the control unit 21 converts image coordinates into workpiece model two-dimensional coordinates in all captured images of the workpiece W acquired in step B1 using the first coordinate conversion table stored in the storage unit 22. Next, the control unit 21 corrects each coordinate after the coordinate conversion based on the position correction amount calculated in step B4.

[0081] (Modification 6) The control unit 21 of the processing device 20 in Modification 6 generates a third coordinate conversion table for converting the two-dimensional coordinates of the workpiece model into image coordinates based on the three-dimensional data of the workpiece model and the captured model. The third coordinate conversion table for converting the two-dimensional coordinates of the workpiece model into image coordinates is expressed by the following formula (5). Formula (5) (C, frame, X, Z) = f －１ (MX, MY) “C” is identification information that identifies the imaging unit 12 .

[0082] In variant example 6, the control unit 21 inputs the work coordinates (MX, MY) in the composite image generated in step B10 of the defect inspection process into the above equation (5), thereby obtaining the image coordinates (C, frame, X, Z) corresponding to the work coordinates.

[0083] (Variation 7) In Variation 7, the control unit 21 of the processing device 20 may generate a second coordinate conversion table for converting image coordinates and workpiece coordinates by the following process during the execution of the defect inspection process. Specifically, the control unit 21 corrects the three-dimensional data of the workpiece model using the measurement position information acquired in step B3 of the defect inspection process. Next, the control unit 21 generates a second coordinate conversion table based on the three-dimensional data of the workpiece model after correction and the captured model. In other words, the control unit 21 corrects the coordinates of the workpiece model (three-dimensional coordinates of the workpiece model) based on the position of the workpiece W measured in step B3. Next, the control unit 21 associates the position of the workpiece surface with positions on the multiple captured images based on the three-dimensional data of the workpiece model with the corrected coordinates and the captured model. The control unit 21 may acquire light reflection characteristics from light ray information obtained by ray tracing of the illumination light irradiated onto the surface of the workpiece W, and associate the light reflection characteristics with the position of the workpiece model surface (three-dimensional coordinates of the workpiece model).

[0084] (Variation 8) In Variation 8, the control unit 21 of the processing device 20 may not compress the time-series data of pixel values ​​at each coordinate in step B9 of the defect inspection process. In this case, the control unit 21 generates a composite image from the time-series data of 32 pixel values, which are 32-dimensional vectors, in step B10. This increases the amount of information in the composite image compared to when the time-series data of pixel values ​​at each workpiece coordinate is compressed, thereby improving the accuracy of surface defect detection. In this case, the control unit 21 may weight the time-series data of the normalized 32 pixel values. Specifically, the control unit 21 weights the first 8 data points and the last 8 data points, which correspond to the vicinity of the boundary between the bright zone illuminated by the illumination unit 13 and the dark zone not illuminated, more heavily than the 16 data points near the center. This makes it possible to highlight the features of the time-series data of pixel values ​​near the boundary between the bright zone and the dark zone, where the features of surface defects are likely to appear.

[0085] (Modification 9) In Modification 9, the control unit 21 of the processing device 20 may not compress the time-series data of pixel values ​​at each workpiece coordinate in step B9 of the defect inspection process. In this case, the control unit 21 calculates feature quantities for the time-series data of 32 pixel values, which are 32-dimensional vectors at each workpiece coordinate, in step B10. The feature quantities in the time-series data of pixel values ​​in Modification 9 include the phase difference from the time-series data of reference pixel values, the difference between the maximum and minimum pixel values, and the standard deviation of the pixel values. Next, the control unit 21 generates a single-channel grayscale image using the calculated feature quantities in the time-series data of pixel values. The control unit 21 may perform differentiation processing on the generated grayscale image to emphasize the features of defective portions. The control unit 21 may further perform binarization processing on the grayscale image to extract defective portions. The control unit 21 may adjust the threshold value in the binarization processing as a variable to adjust the detection sensitivity of surface defects. In other words, the control unit 21 may perform a predetermined process on the feature quantities in the time-series data of pixel values ​​and adjust the threshold value in the predetermined process to adjust the defect detection sensitivity. Next, the control unit 21 inputs the generated grayscale image to an inspection model to detect surface defects.

[0086] As described above, the control unit 21 of the ninth modification generates a sequence of luminance information (time-series data) by arranging the luminance information under different lighting conditions in the order of change in the lighting conditions (chronological order), and calculates one feature from the sequence. Next, the control unit 21 generates a feature image (grayscale image) based on the feature, and detects the features of defects that appear in the feature image using a machine learning model.

[0087] (Modification 10) The control unit 21 of the processing device 20 of Modification 10 may execute the following process before executing the defect inspection process on the workpiece W. Specifically, after installing multiple imaging units 12, the control unit 21 causes the imaging units 12 to capture images of a reference chart (checkerboard) placed in a specified location, thereby acquiring multiple captured images including the reference chart. Next, the control unit 21 calculates the position and orientation of the imaging units 12 by using a stereo matching method for the multiple captured images including the reference chart. Next, the control unit 21 corrects the imaging model based on the calculated position and orientation of the imaging units 12. In addition to the position and orientation of the imaging units 12, the control unit 21 may measure the focal length and distortion of the lens characteristics of the imaging units 12 and correct the imaging model based on this. This makes it possible to correct errors between the imaging model and the measurement position, variations in the lenses of the imaging units 12, and the like.

[0088] <Effects> As described above, the defect inspection device 1 according to this embodiment is a defect inspection device that detects defects on a workpiece surface based on captured images of the workpiece surface captured while changing the illumination conditions for illuminating the workpiece surface. The defect inspection device 1 includes a measurement unit (control unit 21) that measures the position of the workpiece W. The defect inspection device 1 also includes a conversion unit (control unit 21) that associates the position of the workpiece surface with positions on multiple captured images based on the position of the workpiece W measured by the measurement unit, the position of the workpiece model surface based on the 3D data of the workpiece model and the captured model, and the position on each captured image captured under multiple illumination conditions. The defect inspection device 1 also includes an acquisition unit (control unit 21) that acquires luminance information under multiple illumination conditions at each position on the workpiece surface from the corresponding captured images by associating the position of the workpiece surface with the position on the multiple captured images. The defect inspection device 1 also includes a defect detection unit (control unit 21) that detects defects on the workpiece surface based on the luminance information acquired by the acquisition unit. This enables highly accurate detection of defects on the surface of vehicle bodies transported on a production line. Therefore, highly accurate defect inspection can be performed efficiently. This eliminates the need to expend a great deal of effort setting up multiple cameras when implementing a defect inspection system.

[0089] The defect inspection device 1 according to this embodiment includes a calculation unit (control unit 21) that calculates feature quantities in the luminance information. The defect detection unit (control unit 21) detects defects based on the feature quantities calculated by the calculation unit. This makes it possible to detect defects on the surface of the vehicle body with high accuracy.

[0090] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) performs coordinate conversion between the three-dimensional coordinates of the workpiece model and the image coordinates, which are the coordinates of the captured image, based on the three-dimensional data of the workpiece model and the captured model. As a result, the conversion unit associates the position of the workpiece model surface with a position on the captured image. The conversion unit corrects the correspondence between the position of the workpiece model surface and a position on the captured image based on the position of the workpiece. As a result, the conversion unit associates the position of the workpiece surface with a position on the multiple captured images. This allows the position of the workpiece surface to be accurately associated with a position on the multiple captured images, enabling highly accurate defect inspection of the surface of the vehicle body.

[0091] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) corrects the coordinates of the workpiece model based on the position of the workpiece, and associates the position of the workpiece surface with positions on the multiple captured images based on the 3D data of the workpiece model with the corrected coordinates and the captured model. This allows the position of the workpiece surface to be accurately associated with positions on the multiple captured images, enabling highly accurate defect inspection of the surface of the vehicle body.

[0092] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) corrects the coordinates of the workpiece model based on the position of the workpiece, and associates the position of the workpiece surface with positions on the multiple captured images based on the three-dimensional data of the workpiece model with the corrected coordinates and ray tracing of the illumination light irradiated onto the workpiece surface as an imaged model. This allows the position of the workpiece surface to be accurately associated with positions on the multiple captured images, enabling highly accurate defect inspection of the surface of the vehicle body.

[0093] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) generates a table (second coordinate conversion table) that indicates the correspondence between workpiece coordinates, which indicate the position on the workpiece surface, and image coordinates, which indicate the positions on a plurality of captured images. This makes it possible to easily convert the workpiece coordinates and the image coordinates using the second coordinate conversion table.

[0094] In the defect inspection apparatus 1 according to this embodiment, the table (second coordinate conversion table) stores the correspondence between work coordinates and image coordinates for only some image coordinates. The conversion unit (control unit 21) associates image coordinates not stored in the table with interpolated work coordinates based on the correspondence corresponding to other image coordinates stored in the table, including image coordinates that are closest to the image coordinates not stored in the table. Therefore, the second coordinate conversion table does not need to store the correspondence between work coordinates and image coordinates for all image coordinates in the captured image.

[0095] In the defect inspection device 1 according to this embodiment, the imaging model includes information about the focal length, position, and posture of the imaging optical system of the imaging unit 12 that captures the captured images. This allows the position of the workpiece surface to be accurately associated with positions on the multiple captured images, making it possible to perform highly accurate defect inspection of the surface of the vehicle body.

[0096] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) associates work coordinates indicating the position of the work surface with image coordinates indicating the position on the captured image. The conversion unit associates predetermined work coordinates with image coordinates that do not correspond to the work surface. This allows image coordinates that do not correspond to the work surface to be converted into work coordinates.

[0097] In the defect inspection apparatus 1 according to this embodiment, the calculation unit (control unit 21) generates a sequence of luminance information by arranging luminance information under different lighting conditions in the order of change in the lighting conditions, and calculates two or more feature quantities from the sequence. The defect detection unit (control unit 21) generates a multi-channel image by inputting two or more feature quantities into different channels one by one. The defect detection unit detects the features of defects that appear in the multi-channel image using a machine learning model. This generates a color image, which is a multi-channel image, making it easier for the human eye to recognize changes in pixel values ​​over time. This facilitates annotation work for machine learning in the inspection model.

[0098] In the defect inspection apparatus 1 according to this embodiment, the calculation unit (control unit 21) converts luminance information into a vector with a predetermined number of dimensions by interpolating or compressing it, and calculates feature quantities in the vector. The calculation unit assigns a predetermined weight to the vector. This makes it possible to highlight features in the time-series data of pixel values ​​near the boundary between light and dark bands, where the features of surface defects are likely to appear.

[0099] In the defect inspection apparatus 1 according to this embodiment, the calculation unit (control unit 21) generates a sequence of luminance information by arranging luminance information under different lighting conditions in the order of change in the lighting conditions, and calculates one feature from the sequence. The defect detection unit (control unit 21) generates a feature image based on the feature and detects the features of defects that appear in the feature image using a machine learning model. This increases the amount of information in the composite image compared to when time-series data of pixel values ​​at each workpiece coordinate is compressed, thereby improving the accuracy of surface defect detection.

[0100] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) associates the position of the workpiece surface with the position on the multiple captured images only for pixels in the captured images that have pixel values ​​equal to or greater than a predetermined threshold value or within a predetermined range. This allows the defect inspection to be performed based only on pixels near the bright zone in the captured images, which is the portion necessary for the defect inspection of the surface of the vehicle body.

[0101] In the defect inspection device 1 according to this embodiment, the measurement unit (control unit 21) measures the orientation of the workpiece. The conversion unit (control unit 21) associates the position of the workpiece surface with positions on the multiple captured images based on the position and orientation of the workpiece measured by the measurement unit and the correspondence between the position of the workpiece model surface and positions on each captured image captured under multiple lighting conditions. This allows the position of the workpiece surface to be accurately associated with positions on the multiple captured images, enabling highly accurate defect inspection of the surface of the vehicle body.

[0102] In the defect inspection device 1 according to this embodiment, the defect detection unit (control unit 21) performs a predetermined process on the feature quantities and adjusts the threshold value in the predetermined process to adjust the defect detection sensitivity, thereby enabling an appropriate defect detection sensitivity to be set in the defect inspection of the vehicle body surface.

[0103] In the defect inspection device 1 according to this embodiment, variables related to defect detection sensitivity are associated with workpiece coordinates in a table (second coordinate conversion table). The defect detection unit (control unit 21) adjusts the variables to adjust the defect detection sensitivity for a given workpiece coordinate. This allows appropriate defect detection sensitivity to be set for each workpiece coordinate when inspecting defects on the vehicle body surface.

[0104] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) acquires light ray reflection characteristics from light ray information obtained by ray tracing of illumination light irradiated onto the workpiece surface, and associates the light ray reflection characteristics with positions on the workpiece model surface. This makes it possible to set appropriate defect detection sensitivity according to the light ray reflection characteristics corresponding to positions on the workpiece model surface when inspecting defects on the surface of a car body.

[0105] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) sets an inspection target area, a non-inspection target area, or both, on the three-dimensional data of the workpiece model, thereby enabling defect inspection to be performed on desired areas on the surface of the vehicle body.

[0106] In the defect inspection device 1 according to this embodiment, the machine learning model outputs an estimated value that indicates the likelihood of a defect. The defect detection unit (control unit 21) adjusts the threshold value for the estimated value to adjust the defect detection sensitivity. This allows an appropriate defect detection sensitivity to be set when inspecting the surface of a vehicle body for defects.

[0107] In the defect inspection device 1 according to this embodiment, the conversion unit (control unit 21) corrects the imaging model based on position information and orientation information of the imaging unit 12 that is measured using a reference chart after the imaging unit 12 that captures the captured images is installed. This allows the position of the workpiece surface to be accurately associated with positions on the multiple captured images, enabling highly accurate defect inspection of the surface of the vehicle body.

[0108] The above has been a specific description based on an embodiment of the present invention, but the present invention is not limited to the above embodiment and can be modified within the scope of the gist of the present invention. For example, in the above embodiment, the workpiece W to be inspected is illustrated and described as a car body, but the present invention is not limited to this. By applying the present invention to inspection objects other than car bodies, high-precision defect inspection can be efficiently performed.

[0109] The present invention can be used in a defect inspection device.

[0110] REFERENCE SIGNS LIST 1 Defect inspection device 10, 10A Inspection unit 11 Transport section 111 Conveyor 12, 12A Imaging section 13, 13A Illumination section 131 Light source 14 Detection section 15 Position detection section 16A Robot arm 17 Laser distance meter 20 Processing device 21 Control section 22 Memory section 22a Program 23 Communication section 24 Operation section 25 Display section A Inspection start position D Cart E1, E11, E2, E22 Edge H Imaging range P1, P2 Composite image W Work

Claims

1. A defect inspection device that detects defects on a work surface based on captured images of the work surface while changing the lighting conditions that illuminate the work surface, comprising: a measurement unit that measures the position of the work; a conversion unit that associates the position of the work surface with positions on multiple captured images based on the position of the work measured by the measurement unit, and the correspondence between the position of the work model surface based on the three-dimensional data of the work model and the captured model, and the position on each captured image captured under multiple lighting conditions; an acquisition unit that acquires luminance information under multiple lighting conditions at each position on the work surface from the corresponding captured images by correlating the position of the work surface with the positions on the multiple captured images; and a defect detection unit that detects defects on the work surface based on the luminance information acquired by the acquisition unit.

2. A defect inspection device according to claim 1, further comprising a calculation unit that calculates a feature amount in the luminance information, and wherein the defect detection unit detects the defect based on the feature amount calculated by the calculation unit.

3. The defect inspection device described in claim 1, wherein the conversion unit corresponds the position of the work model surface to a position on the captured image by performing a coordinate conversion between the three-dimensional coordinates of the work model and image coordinates, which are the coordinates of the captured image, based on the three-dimensional data of the work model and the captured model, and corresponds the position of the work surface to a position on multiple of the captured images by correcting the correspondence between the position of the work model surface and the position on the captured image based on the position of the work.

4. The defect inspection device described in claim 1, wherein the conversion unit corrects the coordinates of the work model based on the position of the work, and corresponds the position of the work surface to positions on the multiple captured images based on the three-dimensional data of the work model whose coordinates have been corrected and the captured model.

5. The defect inspection device described in claim 4, wherein the conversion unit corrects the coordinates of the work model based on the position of the work, and corresponds the position of the work surface to positions on the multiple captured images based on the three-dimensional data of the work model with corrected coordinates and ray tracing of illumination light irradiated onto the work surface as the captured model.

6. The defect inspection device according to claim 1, wherein the conversion unit generates a table showing the correspondence between work coordinates indicating the position on the work surface and image coordinates indicating the positions on the multiple captured images.

7. The defect inspection device described in claim 6, wherein the table stores correspondences between the work coordinates and the image coordinates only for some of the image coordinates, and the conversion unit associates the image coordinates not stored in the table with work coordinates interpolated based on the correspondences corresponding to other image coordinates stored in the table and including image coordinates closest to the image coordinates not stored in the table.

8. A defect inspection device according to claim 1, wherein the imaging model includes information on the focal length, position, and orientation of an imaging optical system of an imaging unit that captures the captured image.

9. The defect inspection device of claim 1, wherein the conversion unit associates work coordinates indicating the position of the work surface with image coordinates indicating a position on the captured image, and associates predetermined work coordinates with image coordinates for which no coordinates corresponding to the work surface exist.

10. The defect inspection device described in claim 2, wherein the calculation unit generates a sequence of the brightness information by arranging the brightness information under different lighting conditions in the order in which the lighting conditions change, and calculates two or more feature amounts from the sequence; and the defect detection unit generates a multi-channel image by inputting the two or more feature amounts one by one into a plurality of different channels, and detects features of defects that appear in the multi-channel image using a machine learning model.

11. The defect inspection device according to claim 2, wherein the calculation unit converts the luminance information into a vector with a predetermined number of dimensions by interpolating or compressing it, and calculates the feature amount in the vector.

12. The defect inspection device according to claim 11, wherein the calculation section assigns a predetermined weight to the vector.

13. The defect inspection device described in claim 2, wherein the calculation unit generates a sequence of the brightness information by arranging the brightness information under different lighting conditions in the order of change in the lighting conditions, and calculates one feature from the sequence; and the defect detection unit generates a feature image based on the feature and detects the features of defects that appear in the feature image using a machine learning model.

14. The defect inspection device described in claim 1, wherein the conversion unit associates the position of the work surface with positions on the multiple captured images only for pixels of the captured images that have pixel values ​​above a predetermined threshold or within a predetermined range.

15. The defect inspection device described in claim 1, wherein the measurement unit measures the posture of the workpiece, and the conversion unit associates the position of the workpiece surface with positions on the multiple captured images based on the position and posture of the workpiece measured by the measurement unit and the correspondence between the position of the workpiece model surface and positions on each captured image taken under multiple lighting conditions.

16. The defect inspection device according to claim 2, wherein the defect detection section performs a predetermined process on the feature amount and adjusts the defect detection sensitivity by adjusting a threshold value in the predetermined process.

17. A defect inspection device as described in claim 6, wherein in the table, the work coordinates are associated with variables related to defect detection sensitivity, and the defect detection unit adjusts the variables to adjust the detection sensitivity of the defect corresponding to a given work coordinate.

18. A defect inspection device as described in claim 1, wherein the conversion unit acquires light reflection characteristics from light ray information obtained by ray tracing of illumination light irradiated onto the work surface, and associates the light reflection characteristics that correspond to each position on the work model surface.

19. The defect inspection device according to claim 1, wherein the conversion unit sets an inspection target area, a non-inspection target area, or both, on the three-dimensional data of the workpiece model.

20. A defect inspection device as described in claim 10 or 13, wherein the machine learning model outputs an estimated value representing the likelihood of the defect, and the defect detection unit adjusts the defect detection sensitivity by adjusting a threshold value for the estimated value.

21. The defect inspection device according to claim 1, wherein the conversion unit corrects the imaging model based on position information and orientation information of the imaging unit that captures the captured image, measured using a reference chart after installation of the imaging unit.

Images