Inspection device, inspection method, and program

Abstract

To provide a technology relating to an inspection apparatus using a learning model capable of making the size of each image used for inspection smaller than that of a captured image to facilitate defect detection, reduce the number of images to be inspected, and shorten inspection time.SOLUTION: The inspection apparatus is provided with a captured image acquisition unit that acquires a captured image of an inspection target, an image division unit that divides the acquired captured image into a plurality of first divided images, a first synthetic image generation unit that superimposes the plurality of first divided images and generates a synthetic image and a determination unit that determines whether or not there is a defect in the inspection target using the generated synthetic image and the learning model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an inspection apparatus, an inspection method, and a program. [Background technology]

[0002] 2. Description of the Related Art There is known an analysis device that inputs image data acquired by imaging the surface of a vehicle into a learning model to determine the cause of a paint defect that has occurred on the surface of the vehicle (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-192131 A Summary of the Invention [Problem to be solved by the invention]

[0004] When inspecting a large area such as the side of a vehicle, it is preferable to obtain an image of a larger size in order to reduce the inspection time. However, if the image size is large, the size of the defect to be detected may be small compared to the image size, making it difficult to detect the defect by image judgment using machine learning. Therefore, it is possible to divide the image and inspect using smaller images. However, simply dividing the image to reduce the image size results in a problem that the number of images to be inspected increases, and the inspection time increases. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms. A first embodiment of the present disclosure is an inspection device including an image acquisition unit that acquires an image of an inspection object, an image division unit that divides the acquired image into a plurality of first divided images, a first composite image generation unit that generates a composite image by superimposing the plurality of first divided images, a determination unit that determines whether or not the inspection object has a defect by using the generated composite image and a learning model, and a second composite image generation unit that generates the composite image by further dividing the generated composite image when it is determined that an object is present in at least a part of the periphery of the first divided image or the periphery of the composite image. The present disclosure can also be realized in the following embodiments.

[0006] (1) According to one embodiment of the present disclosure, there is provided an inspection device including: an image acquisition unit that acquires an image of an inspection object; an image division unit that divides the acquired image into a plurality of first divided images; a first composite image generation unit that generates a composite image by superimposing the plurality of first divided images; and a determination unit that determines whether or not the inspection object has a defect by using the generated composite image and a learning model. According to this type of inspection device, the size of each image used for inspection can be made smaller than the captured image, making it easier to detect defects, while also reducing the number of images to be inspected and shortening the inspection time. (2) In the inspection device of the above-described form, a pre-processing unit may be further provided that performs image processing to make areas other than defective areas contained in any of the captured image, the first divided image, and the composite image less noticeable. According to the inspection device of this embodiment, it is possible to reduce or suppress the problem that a defective portion cannot be detected. (3) In the inspection device of the above aspect, the pre-processing unit may perform at least a median filter, a Gaussian filter, and binarization in this order as the image processing. According to the inspection device of this embodiment, it is possible to remove noise from the captured image and to increase the gradation value of defective parts relative to non-defective parts. (4) In the inspection device of the above-described form, a second composite image generation unit may be further provided which, when it is determined that an object is present in at least a portion of the peripheral portion of the first divided image or the peripheral portion of the composite image, generates the composite image using a plurality of second divided images obtained by further dividing the generated composite image. According to the inspection device of this aspect, it is possible to reduce or prevent the problem of an abnormality being overlooked due to the object being divided when the first divided image is generated. (5) In the inspection device of the above-described form, the second composite image generation unit may determine that an object is present in the peripheral portion of the first divided image or the peripheral portion of the composite image when the gradation value of the peripheral portion of the first divided image or the composite image is greater than a predetermined threshold value. According to the inspection device of this aspect, it is possible to detect objects on the periphery of the composite image by a simple method. (6) In the inspection device of the above-described form, the second composite image generation unit may generate the composite image by swapping the positions of second divided images that are positioned opposite each other among the plurality of second divided images and combining them. According to the inspection device of this aspect, it is possible to generate a composite image by a simpler method than when changing the division position and generating the first divided image anew. (7) In the inspection apparatus of the above-described form, a third composite image generation unit may be further provided which extracts provisional defective images from the plurality of first divided images which are predicted to have a high probability of having defects, and overlays the provisional defective images to generate the composite image. According to the inspection device of this embodiment, when generating a composite image, the number of images to be composited is reduced, thereby making it possible to further shorten the inspection time. (8) In the inspection device of the above aspect, the provisional defective image may be at least one of the first divided image having a gradation value greater than a predetermined threshold value and the first divided image extracted by pattern matching. According to the inspection device of this embodiment, it is possible to extract the provisional defective images by a simple method. (9) In the inspection device of the above aspect, the learning model may be a model using R-CNN. According to this type of inspection device, a general-purpose machine learning learning model can be used for defect detection. (10) In the inspection apparatus of the above aspect, the composite image may have a width not greater than 50 times the width of the defect to be detected, and a height not greater than 50 times the height of the defect to be detected. According to the inspection device of this embodiment, it is possible to obtain high detection accuracy for defects to be detected. (11) In the inspection device of the above aspect, the inspection object may be a side surface of a vehicle body. According to the inspection device of this embodiment, the inspection time for inspecting the side of the vehicle body can be significantly reduced. (12) In the inspection device of the above aspect, the defect to be inspected may be a sealer attached to a side surface of the vehicle body. According to the inspection device of this embodiment, it is possible to provide an inspection device suitable for detecting a sealer that is very small compared to the inspection range. The present disclosure can also be realized in various forms other than an inspection device, such as a manufacturing method for an inspection device, a foreign matter detection device, a foreign matter detection method, an inspection method, an inspection system, a control method for an inspection system, a control method for an inspection device, a computer program for implementing these control methods, a non-transitory recording medium on which the computer program is recorded, etc. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram illustrating a schematic overall configuration of an inspection system. [Diagram 2] FIG. 4 is a process diagram showing a manufacturing process including an inspection process performed by an inspection device. [Diagram 3] FIG. 4 is a process diagram showing details of an inspection process performed by an inspection device. [Figure 4] FIG. 4 is an explanatory diagram showing how a camera captures an image of an inspection object. [Diagram 5] FIG. 4 is a process diagram showing a processing routine of pre-processing of a captured image by a pre-processing unit. [Figure 6] FIG. 4 is an explanatory diagram showing an example of a captured image. [Figure 7] FIG. 11 is an explanatory diagram showing an example of a captured image after preprocessing. [Figure 8] FIG. 11 is an explanatory diagram showing an enlarged view of a defective portion in the image after preprocessing. [Figure 9] FIG. 4 is an explanatory diagram showing an example of a first divided image generated by an image dividing unit. [Figure 10] FIG. 4 is an explanatory diagram showing an example of a composite image generated by a first composite image generating unit. [Figure 11] FIG. 4 is a block diagram showing the functional configuration of an inspection apparatus according to a second embodiment of the present disclosure. [Figure 12] 8A to 8C are process diagrams showing details of an inspection method executed by an inspection device according to a second embodiment. [Figure 13] FIG. 4 is an explanatory diagram illustrating a composite image generated by overlapping the first divided images. [Figure 14] FIG. 13 is an explanatory diagram illustrating a composite image generated using the second divided images. [Figure 15] FIG. 11 is a block diagram showing the functional configuration of an inspection device according to a third embodiment of the present disclosure. [Figure 16] 13A to 13C are process diagrams showing details of an inspection method executed by an inspection device as a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: FIG. 1 is an explanatory diagram showing a schematic overall configuration of an inspection system 100. The inspection system 100 includes an inspection device 60 as a first embodiment of the present disclosure and a camera 70. The inspection device 60 performs defect detection using machine learning by using a captured image of an inspection target processed in a manufacturing process. In this embodiment, the inspection target is, for example, a car body CB of an automobile to which a sealer is applied, and the defect is, for example, the sealer as a foreign object attached to the surface of the car body CB. The sealer is a paste material that is applied for the purpose of reducing steps at joints of the car body and improving the airtightness of the car body. Various materials such as polyvinyl chloride, epoxy resin, and synthetic rubber are used for the sealer depending on the purpose of use. The inspection target is not limited to an automobile, and may be various moving bodies such as railway vehicles, aircraft, and ships, and may be various products such as parts that constitute these.

[0009] The camera 70 captures an image of an object to be inspected. The camera 70 is connected to the inspection device 60 via a network. In this embodiment, the captured image acquired by the camera 70 is composed of an RGB input image signal composed of image signal components represented by R (red), G (green), and B (blue). In this embodiment, 8 bits are assigned to each image signal, totaling 24 bits of information. The size of the captured image acquired by the camera 70 may be set arbitrarily according to the resolution of the camera, the defect detection accuracy required for the inspection device 60, the size of the defect to be detected, the image size that can be processed by the learning model, and the like. In this embodiment, the camera 70 acquires a 12 million pixel captured image having a width of 4000 pixels and a height of 3000 pixels. However, the input image signal of the captured image is not limited to the above form, and may be, for example, a YUV image signal composed of Y (luminance signal), U (first color difference signal), and V (second color difference signal), or may be a YCbCr image signal, or a YPbPr image signal. Each image signal may be assigned any amount of information, such as 16 bits, 30 bits, 32 bits, 48 ​​bits, or 64 bits in total. The captured image may be a color image, a monochrome image, or a grayscale image having a luminance value of any color depth, such as 1 bit or 8 bits. The camera 70 is not limited to being separate from the inspection device 60, and may be integrated with the inspection device 60.

[0010] The inspection device 60 includes a central processing unit (CPU) 62, a storage device 64, and a communication unit 66. The CPU 62, the storage device 64, and the communication unit 66 are connected to one another via a bus 61 and are capable of bidirectional communication. The CPU 62 executes various programs stored in the storage device 64 to function as a captured image acquisition unit 620, a preprocessing unit 622, an image division unit 624, a first composite image generation unit 626, and a determination unit 628. The inspection device 60 may include a display unit having a function of displaying data obtained by each process described in detail below and data generated during the process.

[0011] The storage device 64 is, for example, a RAM, a ROM, or a hard disk drive (HDD). Various programs for implementing the functions provided in this embodiment are stored in the HDD or ROM, and the various programs read from the HDD or ROM are expanded on the RAM and executed by the CPU 62. The storage device 64 includes a filter storage unit 642 for storing various filters used in pre-processing of captured images, and a learning model storage unit 644 for storing machine learning models. The storage device 64 temporarily stores various images generated by the pre-processing unit 622, the image division unit 624, and the first composite image generation unit 626, and the defect probability calculated by the determination unit 628. The storage device 64 may be an optical disk, a solid state drive (SSD), a flash memory, or the like.

[0012] The communication unit 66 is an interface that performs communication control to receive captured images to be used for inspection via a network between the camera 70. The captured image acquisition unit 620 acquires the captured images from the camera 70 via the communication unit 66.

[0013] The pre-processing unit 622 performs pre-processing of the captured image. Specifically, the pre-processing unit 622 processes the captured image according to a predetermined procedure using an image processing filter stored in the filter storage unit 642. The pre-processing makes parts other than defective parts such as foreign objects contained in the captured image less noticeable, and reduces gradation values ​​such as RGB values ​​of the background other than the defective parts or luminance values ​​obtained by luminance conversion of RGB values, thereby improving the detection accuracy of defective parts in the captured image. Examples of image processing filters stored in the filter storage unit 642 include various techniques such as an expansion filter, a contraction filter, an averaging filter, a median filter, an edge extraction filter, an edge emphasis filter, a Sobel filter, and a Laplacian filter. In this embodiment, the filter storage unit 642 stores image processing filters that apply histogram equalization, a median filter, a Gaussian filter, binarization, and morphological transformations.

[0014] The image dividing section 624 divides one captured image into a plurality of images. Images generated by dividing the captured image acquired by the captured image acquiring section 620 by the image dividing section 624 are also called "first divided images". The number and size of divisions by the image dividing section 624 can be set arbitrarily in the width direction and height direction of the image. In this embodiment, the image dividing section 624 divides the captured image into images of the same size, and generates 80 first divided images.

[0015] The first composite image generating unit 626 generates a composite image by superimposing a plurality of images. In this embodiment, the first composite image generating unit 626 generates one composite image by accumulating the gradation values ​​of the plurality of first divided images by so-called digital composite. The composite image may be generated by accumulating the R value, the G value, and the B value, or may be generated by accumulating the luminance value. When the plurality of first divided images are different sizes, the first composite image generating unit 626 may composite the images according to the first divided image of the largest size, or may resize the first divided images to an arbitrary size and then composite them. The first composite image generating unit 626 may superimpose the plurality of first divided images after performing a weighted average.

[0016] The determination unit 628 uses machine learning to determine whether or not the inspection target has a defect. Specifically, the determination unit 628 performs inspection on the inspection target by inputting a synthetic image into a learning model stored in the learning model storage unit 644. In this embodiment, the learning model storage unit 644 stores a model using R-CNN (Region Based Convolutional Neural Networks) as a learning model of machine learning. The "model using R-CNN" includes various versions of R-CNN such as Mask R-CNN, Faster R-CNN, Fast R-CNN, YOLO, and Mesh R-CNN. The learning model is not limited to a model using R-CNN, and various models equipped with neural networks may be used, such as a model using a convolutional neural network (CNN) such as SSD (Single Shot MultiBox Detector), a generative adversarial network (GAN), a variational autoencoder (VAE), and an autoregressive generative net.

[0017] The determination unit 628 inputs the image data into the learning model and calculates the defect probability for each predetermined area in the input image data. The defect probability means the probability that the inspection target has a defect. In general, the size of the image data input into the learning model is determined by the performance such as the calculation speed of the CPU 62 and the storage device 64 of the inspection device 60. In this embodiment, the determination unit 628 resizes the image data to a size of about 400 pixels high x 400 pixels wide and then calculates the defect probability. Therefore, for example, if image data larger than a size of about 400 pixels high x 400 pixels wide is input into the learning model, the defect may be compressed to a small size by resizing. As a result, for example, the defective part may be recognized as noise and may not be detected as a defect.

[0018] The determination unit 628 determines whether or not the inspection target is defective using the calculated defect probability. Specifically, the determination unit 628 determines whether or not the inspection target is defective by comparing the calculated defect probability with a threshold value previously stored in the storage device 64. The threshold value can be set in advance using any value. The threshold value can be experimentally obtained, for example, using the correspondence between the presence or absence of defects in the inspection target and the calculated defect probability. In this embodiment, the threshold value is set to 2.0%.

[0019] In this embodiment, the determination unit 628 further transmits, to the production line, a command signal for switching the product transport path in the production line and a command signal for issuing an alarm depending on the inspection result via the communication unit 66. The determination unit 628 may transmit the below-mentioned defect probability, areas with high defect probability, etc. to other inspection devices, a manager's terminal device, a local computer, etc.

[0020] FIG. 2 is a process diagram showing a manufacturing process including an inspection process executed by the inspection device 60. In step S10, a predetermined process in the manufacturing process is performed on the product. The manufacturing process includes various actions of performing specific work on the inspection target, such as product processing, surface treatment, casting, molding, welding, painting, and transportation. In this embodiment, the manufacturing process is a coating process in which a sealer is applied to a car body after electrodeposition coating. In this embodiment, the color of the car body after electrodeposition coating is gray, and the sealer is white. When the sealer adheres as a foreign object to a position other than the planned coating range, it may adhere in a relatively small size of φ1 to 2 mm or less.

[0021] In step S100, an inspection process is performed by the inspection device 60. In this embodiment, the inspection device 60 inspects the surface of the vehicle body CB after the sealer has been applied, and inspects whether or not the sealer is present as a foreign object. In step S180, it is determined whether or not to switch the process to which the product is to be discharged, depending on the inspection result in the inspection process. In this embodiment, if the inspection device 60 determines that the product is defective in the inspection process (S180: YES), the inspection device 60 transmits to the production line a command signal for issuing an alarm and a command signal for switching the conveyance path of the product in the production line. As a result, the product determined to be defective is discharged to a defect removal process in step S182. In the defect removal process, a process for removing defects from the product is performed. In this embodiment, the sealer attached to the surface of the vehicle body CB is removed. As a result, the product is discharged to a subsequent process after defects have been removed. The subsequent process is, for example, a process for performing so-called main painting, including undercoating, base painting, and clear painting. If the inspection device 60 determines that the product is not defective in the inspection process (S180: NO), the inspection device 60 does not output a command signal to the production line, and as a result, the product is sent to the subsequent process and this process ends.

[0022] FIG. 3 is a process diagram showing the details of the inspection process by the inspection device 60. The process shown in FIG. 3 is started, for example, when the inspection device 60 detects a product discharged from the manufacturing process. This process may be started when the inspection device 60 receives a signal from the manufacturing process indicating that processing has been completed. At the time when this process is started, the learning model has already completed learning using images of normal inspection objects and images of inspection objects having defects. The process shown in FIG. 3 shows an example in which the captured image acquisition unit 620 acquires a single captured image, but when the captured image acquisition unit 620 acquires multiple captured images, this process may be repeatedly executed for each of the multiple captured images.

[0023] In step S20, the captured image acquisition unit 620 acquires the captured image of the inspection target captured by the camera 70. The number of imaging locations of the inspection target is not limited to one, and may be multiple, and the number of cameras 70 may be set to one or multiple depending on the number of imaging locations. The acquired captured image is output to the pre-processing unit 622.

[0024] Fig. 4 is an explanatory diagram showing how the camera 70 captures an image of an inspection target. Fig. 4 shows a schematic diagram of a side surface of a vehicle body CB as an inspection target. The side surface of the vehicle body CB tends to have a wider inspection range than other parts of the vehicle body CB, such as the rear surface of the vehicle body CB.

[0025] The camera 70 is fixedly installed on a conveying path for conveying the vehicle body CB along the conveying direction DF in the manufacturing line. In this embodiment, a lighting device (not shown) for irradiating light toward the vehicle body CB from behind the camera 70 is provided on the rear side of the camera 70. This makes it possible to obtain a captured image in which the contrast difference between the vehicle body CB and the sealer is more prominent by utilizing the difference in reflectance between the gray vehicle body CB and the white sealer.

[0026] FIG. 4 shows a schematic image capturing range CR of one captured image. In this embodiment, the camera 70 captures an image of a range of about 400 mm×400 mm per location of the vehicle body CB. The camera 70 repeats capturing images at a predetermined time interval according to the transport speed of the vehicle body CB under irradiation of illumination light from a lighting device. This allows the side of the vehicle body CB to be captured at predetermined intervals in the transport direction DF. In addition, a plurality of cameras 70 are arranged along the vertical direction DZ. This allows the camera 70 to capture images of a plurality of inspection locations along the vertical direction DZ at once.

[0027] As shown in Fig. 3, in step S30, the pre-processing unit 622 performs pre-processing on the captured image. Fig. 5 is a process diagram showing a processing routine of pre-processing on the captured image by the pre-processing unit 622. In step S32, the pre-processing unit 622 performs image processing corresponding to histogram equalization on the captured image. By performing histogram equalization, the distribution of gradation values ​​of the entire captured image is equalized, and the contrast of the captured image can be improved. Note that, if sufficient defect detection accuracy is obtained, histogram equalization may be omitted.

[0028] In step S34, the pre-processing unit 622 performs image processing using a median filter on the captured image. The median filter removes noise components from the captured image. In step S35, the pre-processing unit 622 performs image processing using a Gaussian filter on the captured image. Any smoothing filter other than the Gaussian filter may be used. The median filter and the Gaussian filter make the defective area less noticeable in areas other than the defective area, such as the background. In step S36, the pre-processing unit 622 performs image processing using binarization. The threshold for binarization may be set using any method, such as the mode method, the P-tile method, or the discriminant analysis method. In this embodiment, the threshold for binarization is set to 200 or more.

[0029] In step S38, the pre-processing unit 622 performs image processing using a morphological transformation process on the generated binary image. Specifically, the pre-processing unit 622 performs erosion and dilation processes on the obtained binary image. Erosion is performed by convolving the binary image using a convolution layer. As a result, the part of the binary image that corresponds to white is eroded. Dilation is performed by convolving the binary image using a convolution layer. As a result, the part of the binary image that corresponds to black is eroded. The morphological transformation process may include opening, closing, morphological gradient, top-hat transformation, and black-hat transformation. Opening is performed by eroding an object in an image and then expanding it. Closing is performed by expanding an object in an image and then eroding it. Morphological gradient is a process that calculates the difference between an expanded image and a contracted image. As a result, the outline of the object, that is, the part that may be defective, becomes clear. It should be noted that if sufficient defect detection accuracy can be obtained using a binary image, the morphological transformation may be omitted.

[0030] Fig. 6 is an explanatory diagram showing an example of a captured image. The captured image D2 shown in Fig. 6 is an imaging result of the imaging range CR on the side of the vehicle body CB shown in Fig. 5. In the range AR of the captured image D2, a sealer as a defect attached to the vehicle body CB is imaged. In the captured image D2, the sealer as a defect is imaged as white, and has a higher gradation value than the surface of the vehicle body CB which is imaged as gray.

[0031] FIG. 7 is an explanatory diagram showing an example of a captured image after preprocessing. The captured image after preprocessing is also called a "preprocessed image." The preprocessed image D3 shown in FIG. 7 is an image after the captured image D2 shown in FIG. 6 has been subjected to image processing by the preprocessing unit 622, including histogram equalization, median filtering, Gaussian filtering, binarization, and morphological conversion. As shown in FIG. 7, the median filtering, Gaussian filtering, and binarization are performed in this order, so that noise in the image and the gradation value of the background other than the defective portion can be made to be approximately zero, and the portions other than the defective portion can be made inconspicuous. For example, if the order of the median filtering, Gaussian filtering, and binarization is changed from this order, there is a risk of the detection accuracy being reduced due to the remaining noise in the image and the disappearance or reduction of the defect to be detected.

[0032] Fig. 8 is an explanatory diagram showing an enlarged view of a defective portion in the image after preprocessing. Image D4 shown in Fig. 8 is an image corresponding to the portion of the range AR in Fig. 7. As shown in Fig. 8, in the image D4 after preprocessing, a defective FM as an example of a defect, specifically, a sealer attached to the side of the vehicle body CB, remains as a white image.

[0033] In step S40, the image division unit 624 divides the captured image after preprocessing into a plurality of first divided images. FIG. 9 is an explanatory diagram showing an example of the first divided image generated by the image division unit 624. The plurality of first divided images D5 shown in FIG. 9 are images obtained by dividing the preprocessed image D3 shown in FIG. 7. In this embodiment, the sizes of the first divided images D5 are equal to each other and coincide with the size of the composite image, as described later. The size of the first divided image, that is, the number of divisions, is preferably set based on the size of image data that can be input to the learning model and the size of the defect to be detected. It has been experimentally obtained that a high detection accuracy can be obtained for defects when the width of the image is 50 times or less the width of the defect size to be detected and the height of the image is 50 times or less the height of the defect size. In this embodiment, the size of the defect to be detected is set to 10 pixels or less in width x 10 pixels or less in height. Therefore, in this case, in order to obtain high detection accuracy, it is preferable that the first divided image is image data of a size of 500 pixels or less in width x 500 pixels or less in height. When the size of the defect to be detected is changed, the size of the defect may be adjusted to 10 pixels wide or less by 10 pixels high or less by changing the resolution of the camera or the like.

[0034] In this embodiment, the image dividing unit 624 divides one image into a plurality of first divided images each having a size of 400 pixels wide by 375 pixels high. That is, the image is divided into 10 pieces in the width direction and 8 pieces in the height direction, and a total of 80 first divided images are generated. The size of the first divided images, that is, the composite image, is set to a size close to a size of 400 pixels high by 400 pixels wide, which is the size of the image data to be input to the learning model, thereby reducing the amount of change in the image data due to resizing for input to the learning model. Note that FIG. 9 shows some images of the first divided image D5 divided into 80 pieces. Among them, the first divided image D5a shown in FIG. 9 is an image of a defective FM. The image dividing unit 624 outputs the generated plurality of first divided images D5 to the first composite image generating unit 626.

[0035] In step S50, the first composite image generating unit 626 generates one composite image using a plurality of first divided images. The first composite image generating unit 626 outputs the generated composite image to the determining unit 628. In step S60, the determining unit 628 inputs the acquired composite image into R-CNN as a learning model stored in the learning model storage unit 644 to calculate the defect probability. The defect probability is output for each predetermined area of ​​the inspection range. In step S70, the determining unit 628 compares the defect probability output from the learning model with a threshold value stored in the storage device 64. If the defect probability is less than the threshold value in any area of ​​the inspection range (S70: YES), the determining unit 628 proceeds to step S72, determines that the inspection object is not defective, and ends this process. If there is even one area where the defect probability is equal to or greater than the threshold value (2.0% or more in this embodiment) (S70: NO), the determining unit 628 proceeds to step S74, determines that the inspection object is defective, and ends this process.

[0036] FIG. 10 is an explanatory diagram showing an example of a composite image D6 generated by the first composite image generating unit 626. The composite image D6 shown in FIG. 10 is an image generated by the first composite image generating unit 626 synthesizing the 80 first divided images D5 shown in FIG. 9. The composite image D6 is generated by superimposing images in which the gradation value of the defective FM is high and the gradation value of the area other than the defective FM is approximately zero through preprocessing by the preprocessing unit 622. Therefore, as shown in FIG. 10, the composite image D6 can be generated as an image in which the gradation value of the defective FM is high and the gradation value of the area other than the defective part is approximately zero. The determination unit 628 can detect the defective FM in the composite image D6 by inputting one composite image D6 shown in FIG. 10 into the learning model. That is, the determination unit 628 can determine whether or not there is a defect in the 80 first divided images D5 by inspecting one composite image D6, and can therefore determine whether or not there is a defect within the imaging range CR of the side of the vehicle body CB.

[0037] As described above, the inspection device 60 of this embodiment includes a captured image acquisition unit 620 that acquires a captured image D2 of an inspection target, an image division unit 624 that divides the acquired captured image D2 into a plurality of first divided images D5, a first composite image generation unit 626 that generates a composite image D6 by superimposing the plurality of first divided images D5, and a determination unit 628 that determines whether or not the inspection target has a defect FM by using the generated composite image D6 and a learning model. According to the inspection device 60 of this embodiment, by using one composite image D6 by superimposing the divided plurality of first divided images D5, the size of each image used for inspection can be made smaller than that of the captured image D2. Therefore, even in an inspection using a learning model, defects can be easily detected. In addition, the inspection device 60 can determine the presence or absence of defects in the imaging range CR by one composite image D6. Therefore, compared to the case where each of the plurality of first divided images is inspected, the number of images to be inspected can be reduced while keeping the inspection range the same, and the inspection time can be shortened.

[0038] The inspection device 60 of this embodiment further includes a preprocessing unit 622 that executes preprocessing to make parts other than the defective FM parts in the captured image D2 of the inspection target less noticeable. By making the background other than the defective FM parts less noticeable, even when a plurality of first divided images D5 are superimposed when generating the composite image D6, for example, a problem in which the defective FM parts cannot be detected because the gradation value of the defective FM parts becomes equivalent to the gradation value of the parts other than the defective FM can be reduced or prevented.

[0039] According to the inspection device 60 of this embodiment, the pre-processing unit 622 performs at least a median filter, a Gaussian filter, and binarization in this order as pre-processing on the captured image D2. This makes it possible to remove noise in the captured image, to increase the gradation value of the defective FM portion relative to the portion other than the defective FM, to make the portion other than the defective portion less noticeable, and to emphasize the defective portion.

[0040] According to the inspection device 60 of this embodiment, the learning model is a model using R-CNN. Therefore, a general-purpose machine learning learning model can be used for defect detection.

[0041] According to the inspection device 60 of this embodiment, the composite image is an image whose width is 50 times or less than the width of the defect to be detected and whose height is 50 times or less than the height of the defect to be detected. Therefore, it is possible to obtain high detection accuracy for the defect to be detected.

[0042] According to the inspection device 60 of this embodiment, the inspection target is the side of the vehicle body CB of the automobile. Therefore, by using the inspection device 60 of this embodiment for inspecting the side of the vehicle body CB, which tends to require a wide inspection range, the inspection time for inspecting the side of the vehicle body CB can be significantly reduced.

[0043] According to the inspection device 60 of this embodiment, the defect to be inspected is a sealer attached to the side of the vehicle body CB. It is possible to provide an inspection device 60 suitable for detecting a sealer that is very small compared to the inspection range from the side of the vehicle body CB, which tends to have a wide inspection range.

[0044] B. Second embodiment: 11 is a block diagram showing a functional configuration of an inspection device 60b according to a second embodiment of the present disclosure. The inspection device 60b according to the second embodiment differs from the inspection device 60 according to the first embodiment in that the inspection device 60b according to the second embodiment further includes a CPU 62b including a second composite image generating unit 630 instead of the CPU 62, and the other configurations are the same as those of the inspection device 60 according to the first embodiment. The second composite image generating unit 630 generates a composite image using the second divided image.

[0045] 12 is a process diagram showing details of an inspection method executed by the inspection device 60b of the second embodiment. The inspection method of the second embodiment differs from the inspection method of the first embodiment in that it includes steps S200 to S210 instead of step S60, but other configurations are similar to those of the inspection method of the first embodiment. When the inspection device 60b of this embodiment determines that an object is present on the periphery of the composite image, it divides the composite image to generate second divided images, and performs inspection using the composite image generated by combining the second divided images.

[0046] In step S200, the second composite image generating unit 630 determines whether or not there is an object in the peripheral portion of the composite image. In this disclosure, the "periphery of the image" means pixels arranged on the outermost side of the image. In this embodiment, the second composite image generating unit 630 determines whether or not there is an object in the four corners of the image in the peripheral portion of the image. The presence or absence of an object can be determined, for example, by using the gradation value of a pixel at a predetermined position. The second composite image generating unit 630 compares the gradation values ​​of the pixels at the four corners of the composite image with a predetermined threshold value. The pixels are not limited to the four corners, and may be a group of pixels in a predetermined area including the four corners.

[0047] If at least one of the gradation values ​​of the four corners of the first divided image is smaller than the threshold (S200: YES), the second composite image generating unit 630 proceeds to step S210. Step S210 is similar to step S60 shown in the first embodiment, so a description thereof will be omitted. If all of the gradation values ​​of the pixels at the four corners of the composite image are equal to or greater than the threshold (S200: NO), the second composite image generating unit 630 proceeds to step S204.

[0048] In step S204, the image dividing unit 624 sets a dividing position of the composite image. An image obtained by dividing the composite image is also called a "second divided image." The dividing position for generating the second divided image is set at a position shifted from the edge of the composite image by a predetermined arbitrary number of pixels. The dividing position may be changed in both the width direction and the height direction, or may be changed in only one of them.

[0049] In step S206, the second composite image generating unit 630 generates a composite image using the second divided images. In this embodiment, the composite image is generated by combining the second divided images with each other in a state where the second divided images are swapped based on a predetermined rule, as described later. In step S208, the determining unit 628 inputs the generated composite image into the learning model stored in the learning model storage unit 644 to calculate the defect probability. The steps from step S70 onwards are the same as those in the first embodiment, and therefore will not be described.

[0050] A method for generating the second divided image will be described with reference to Fig. 13 and Fig. 14. Fig. 13 is an explanatory diagram that shows a composite image D7 generated by overlapping the first divided images. The composite image D7 is an example of a composite image determined to have an object in the peripheral portion. Fig. 13 shows second divided images D71 to D74 obtained by dividing the composite image D7. Fig. 14 is an explanatory diagram that shows a composite image D8 generated using the second divided images D71 to D74. In Fig. 13 and Fig. 14, the portions corresponding to foreign objects are hatched to facilitate understanding of the technique.

[0051] As shown in FIG. 13, objects FM21 to FM24 are present at the four corners of the composite image D7. Therefore, the gradation values ​​of the pixels at the four corners of the composite image D7 are higher than those of the non-defective areas such as the background. An example of the case where the objects FM21 to FM24 are present at the four corners of the composite image D6 is a case where the objects are present at the image division position by the image division unit 624. In the example of FIG. 13, one object FM20 shown in FIG. 14 is present at the position where the division position in the height direction and the division position in the width direction intersect, and the object FM20 is divided and exists as objects FM21 to FM24 across four first divided images. As a result, when the first divided images are superimposed by the second composite image generation unit 630, the objects FM21 to FM24 exist at the four corners of one composite image D7. In this case, for example, there is a possibility that the learned learning model cannot recognize the objects FM21 to FM24 as defects.

[0052] Due to pre-processing of the captured image, the objects FM21-FM24 have higher gradation values ​​than the background BK, and all of the gradation values ​​of the four corners of the composite image D7 are equal to or greater than the threshold value. Therefore, the second composite image generating unit 630 determines that there is an object on the periphery of the composite image D7. The image dividing unit 624 divides the composite image D7 based on a predetermined rule to generate second divided images D71-D74.

[0053] In this embodiment, the division positions of the composite image D7 are perpendicular lines to each side of the composite image D7 located at the middle in the width direction and height direction, as shown by dashed lines L1 and L2 in Fig. 13. The image division unit 624 divides the composite image D7 into four at the positions of the dashed lines L1 and L2, and generates four second divided images D71 to D74 of the same size.

[0054] The second composite image generating unit 630 generates a composite image D8 using the generated second divided images D71 to D74. Specifically, the second composite image generating unit 630 generates the composite image D8 by combining the second divided images arranged at diagonal positions with each other in a swapped state. As shown in FIG. 13, the second composite image generating unit 630 swaps the arrangement positions of the second divided images D71 and D74 arranged diagonally with each other, and swaps the arrangement positions of the second divided images D72 and D73 arranged diagonally with each other. The second composite image generating unit 630 generates the composite image D8 shown in FIG. 14 by combining the second divided images D71 to D74 in a swapped state with each other. As a result, as shown in FIG. 14, the composite image D8 displays one object FM20 in which the objects FM21 to FM24 divided in the composite image D7 are combined. As a result, even if the image of the object is split when the first divided images are generated, it can be restored to the shape before split when the composite image D8 is formed. Therefore, the determination unit 628 can correctly determine whether or not there is a defect in the composite image D8 by using the learning model.

[0055] According to the inspection device 60b of this embodiment, when the second composite image generating unit 630 determines that an object is present on the periphery of the composite image D7, it generates a composite image D8 using a plurality of second divided images D71-D74 divided at division positions different from the division positions for generating the first divided image. Even if the object FM20 in the captured image is divided when the first divided image is generated, the shape of the object FM20 can be restored when the composite image D8 is formed. Therefore, it is possible to reduce or prevent a problem in which an abnormality is overlooked due to the object FM20 being divided when the first divided image is generated.

[0056] According to the inspection device 60b of this embodiment, the second composite image generating unit 630 determines that an object is present in the peripheral portion of the composite image D7 when the gradation value of the peripheral portion of the composite image D7 is greater than a predetermined threshold value. Therefore, it is possible to detect an object in the peripheral portion of the composite image D7 by a simple method.

[0057] According to the inspection device 60b of this embodiment, the second composite image generating unit 630 generates a composite image D8 by swapping the positions of the second divided images D71 that are located at positions facing each other among the multiple second divided images D71 and combining them with each other. This makes it possible to generate the composite image D8 by a simpler method than the case where the division positions are changed and the first divided images are generated again.

[0058] C. Third embodiment: 15 is a block diagram showing a functional configuration of an inspection device 60c according to a third embodiment of the present disclosure. The inspection device 60c according to the third embodiment differs from the inspection device 60 according to the first embodiment in that the inspection device 60c according to the third embodiment further includes a CPU 62c including a third composite image generating unit 632 instead of the CPU 62. The other configuration is the same as that of the inspection device 60 according to the first embodiment. The third composite image generating unit 632 extracts a tentative defective image and generates a composite image by superimposing the tentative defective image.

[0059] FIG. 16 is a process diagram showing details of an inspection method executed by an inspection device 60c as a third embodiment of the present disclosure. The inspection method of the third embodiment differs from the inspection method of the first embodiment in that it includes steps S302 and S304 instead of step S50, and other configurations are similar to those of the inspection method of the first embodiment. The inspection device 60c of this embodiment extracts a first divided image that is predicted to have a high possibility of having a defect from among the generated first divided images as a tentative defective image, and performs inspection using the tentative defective image. Note that the tentative defective image is an example of a first divided image.

[0060] In step S302, the third composite image generating unit 632 extracts the first divided images predicted to have a high possibility of having defects as provisional defective images from among the first divided images. In this embodiment, the first divided images predicted to have a high possibility of having defects are extracted by using pattern matching with a pattern image including a defective image stored in advance in the storage device 64. The third composite image generating unit 632 compares each of the generated first divided images with the pattern image by pattern matching, and extracts the first divided images similar to the pattern image as provisional defective images. The provisional defective images are not limited to a single image, and multiple images may be extracted depending on the number of defects detected. The provisional defective images are not limited to pattern matching, and may be extracted, for example, by comparing the gradation value of the first divided image with a predetermined threshold value. In step S304, the third composite image generating unit 632 superimposes the extracted provisional defective images to generate a composite image. The steps from step S60 onwards are the same as those in the first embodiment, so their explanations will be omitted.

[0061] According to the inspection device 60c of this embodiment, the third composite image generating unit 632 extracts a tentative defective image that is predicted to have a high possibility of having a defect from among the multiple first divided images, and generates a composite image using the tentative defective image. Therefore, by reducing the number of images to be composited when generating a composite image, the inspection time can be further shortened.

[0062] According to the inspection device 60c of the present embodiment, the tentative defective image is the first divided image extracted by pattern matching, so that the tentative defective image can be extracted by a simple method.

[0063] D. Other embodiments: (D1) In the above second embodiment, an example was shown in which the second composite image generating unit 630 uses the composite image D7 to detect objects at the four corners of the composite image D7. In contrast, the second composite image generating unit 630 may use the first divided image to determine whether or not an object was divided when the first divided image was generated. In this case, the second composite image generating unit 630 may omit generating a composite image using the first divided image and generate the second divided image.

[0064] (D2) In the above second embodiment, an example was shown in which the second composite image generating unit 630 detected objects at the four corners of the composite image D7, but this is not limited to the four corners of the composite image D7. For example, an object may be detected at least in one of the four corners of the composite image D7. The second composite image generating unit 630 may detect an object present on any two of the four sides of the composite image D7 that face each other. In this case, the second divided image may be divided into two instead of four, and a composite image may be generated by swapping the two divided images.

[0065] (D3) In the above second embodiment, when an object is detected on the periphery of the composite image D7, the determination unit 628 determines the presence or absence of a defect only from the newly generated composite image D8. In contrast, the determination unit 628 may determine the presence or absence of a defect using the composite image D7 in addition to determining the presence or absence of a defect using the composite image D8. In this case, the determination unit 628 inputs the composite image D8 and the composite image D7 individually into a learning model to determine the presence or absence of a defect. The determination unit 628 may further generate a composite image by superimposing the composite image D8 and the composite image D7, and input the generated composite image into the learning model to determine the presence or absence of a defect.

[0066] (D4) In the above third embodiment, the third composite image generating unit 632 extracts a tentative defective image predicted to have a high possibility of having a defect from among a plurality of first divided images, and generates a composite image using the tentative defective image. In contrast, the third composite image generating unit 632 may generate a composite image by combining only the remaining first divided images that were not extracted as the tentative defective images from among the first divided images. In this case, the determining unit 628 may input each of the tentative defective images and the generated composite image into a learning model to determine the presence or absence of a defect. Alternatively, each of the tentative defective images may be input individually into the learning model without combining the tentative defective images to determine the presence or absence of a defect.

[0067] (D5) In the above embodiments, an example was described in which the inspection device 60 detects a white sealant attached to a gray vehicle body CB as a defect. In contrast, the inspection device 60 can also be applied to cases in which defects having colors other than white on a captured image, such as scratches or unevenness on the side of the vehicle body CB, or the attachment of foreign matter other than white, are detected. For example, when detecting a black defect, a captured image may be generated in which the background of the captured image is white and the defective part is black. Also, when detecting a red defect, a captured image with only the R value of the RGB values ​​of the captured image can be used.

[0068] (D6) In the second embodiment, the second divided images are generated by dividing the composite image, and the composite image is generated by combining the second divided images. In contrast, the division position of the image may be changed by changing the division position of the captured image acquired by the captured image acquisition unit 620, thereby generating a new first divided image.

[0069] The control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described in the present disclosure may be realized by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer.

[0070] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0071] 60, 60b, 60c... inspection device, 61... bus, 62, 62b, 62c... CPU, 64... storage device, 66... ​​communication unit, 70... camera, 100... inspection system, 620... captured image acquisition unit, 622... pre-processing unit, 624... image division unit, 626... composite image generation unit, 628... judgment unit, 630... second composite image generation unit, 632... third composite image generation unit, 642... filter memory unit, 644... learning model memory unit, BK... background, CB... vehicle body, CR... imaging range, D2... captured image, D3... pre-processed image, D4... image, D5, D5a... first divided image, D6, D7... composite image, D71 to D74... second divided image, D8... composite image, DF... conveying direction, DZ... vertical direction, FM... defect, FM21 to 24... object

Claims

1. An inspection device comprising: an image acquisition unit for acquiring an image of an object to be inspected; an image division unit that divides the acquired captured image into a plurality of first divided images; a first composite image generating unit that generates a composite image by superimposing the first divided images; a determination unit that determines whether or not the inspection object has a defect by using the generated synthetic image and a learning model; and a second composite image generating unit that generates the composite image using a plurality of second divided images obtained by further dividing the generated composite image when it is determined that an object is present in at least a part of the peripheral portion of the first divided image or the peripheral portion of the composite image. Inspection equipment.

2. 2. The inspection device according to claim 1, Further, a pre-processing unit is provided that performs image processing to make parts other than the defective parts included in any one of the captured image, the first divided image, and the composite image less noticeable. Inspection equipment.

3. 3. The inspection device according to claim 2, The pre-processing unit performs at least a median filter, a Gaussian filter, and binarization in this order as the image processing. Inspection equipment.

4. The inspection device according to any one of claims 1 to 3, the second composite image generating unit determines that an object is present in the peripheral portion of the first divided image or the peripheral portion of the composite image when a gradation value of the peripheral portion of the first divided image or the composite image is greater than a predetermined threshold value; Inspection equipment.

5. The inspection device according to any one of claims 1 to 4, the second composite image generating unit generates the composite image by swapping positions of the second divided images that are positioned opposite to each other among the plurality of second divided images and combining them with each other. Inspection equipment.

6. The inspection device according to any one of claims 1 to 5, Further, a third composite image generating unit is provided which extracts a tentative defective image predicted to have a high possibility of having a defect from among the plurality of first divided images, and generates the composite image by superimposing the tentative defective image. Inspection equipment.

7. 7. The inspection device according to claim 6, The provisional defective image is at least one of the first divided image having a gradation value greater than a predetermined threshold value and the first divided image extracted by pattern matching. Inspection equipment.

8. The inspection device according to any one of claims 1 to 7, The learning model is a model using R-CNN. Inspection equipment.

9. The inspection device according to any one of claims 1 to 8, The composite image has a width of 50 times or less than the width of the defect to be detected, and a height of 50 times or less than the height of the defect to be detected. Inspection equipment.

10. The inspection device according to any one of claims 1 to 9, The inspection object is a side of a vehicle body. Inspection equipment.

11. The inspection device according to claim 10, The defect to be inspected is a sealer attached to the side of the vehicle body. Inspection equipment.

12. 1. A testing method comprising: Acquire an image of the inspection object; Dividing the acquired captured image into a plurality of first divided images; generating a composite image by superimposing the plurality of first divided images; Using the generated synthetic image and a learning model, it is determined whether or not the inspection object has a defect; When it is determined that an object is present in at least a part of the peripheral portion of the first divided image or the peripheral portion of the composite image, the generated composite image is further divided into a plurality of second divided images to generate the composite image. Testing method.

13. A program executed by a computer provided in an inspection device, A function of acquiring an image of an object to be inspected; A function of dividing the acquired captured image into a plurality of first divided images; A function of generating a composite image by superimposing the plurality of first divided images; A function of determining whether or not the inspection object has a defect using the generated synthetic image and a learning model; and and generating the composite image using a plurality of second divided images obtained by further dividing the generated composite image when it is determined that an object is present in at least a part of the peripheral portion of the first divided image or the peripheral portion of the composite image. program.

Images