Determination device

Abstract

To suppress overdetection and improve determination accuracy.SOLUTION: A determination device of the embodiment has a first determination unit that outputs a first pass / fail determination result for an input captured image based on features extracted from a captured image of a determination object that belongs to a pass determination, an image processing unit that performs preprocessing by applying predetermined image processing to the captured image and outputs a preprocessed image, a second determination unit that outputs a second pass / fail determination result for the input preprocessed image based on features extracted from a preprocessed image of the determination object that belongs to a pass determination, and a determination control unit that determines the pass / fail of the determination object using the first and second pass / fail determination results. The determination control unit determines a result as pass if an abnormal area detected in one pass / fail determination result is different from an abnormal area detected in the other pass / fail determination result, and determines a result as fail if an abnormal area detected in one pass / fail determination result is the same as an abnormal area detected in the other pass / fail determination result.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to a pass / fail determination technique for a determination target.

Background Art

[0002] As a pass / fail determination technique for a determination target, for example, there is a technique of photographing an article and determining the pass / fail of the article shown in the photographed image.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide a determination device that can suppress over-detection and improve determination accuracy.

Means for Solving the Problems

[0005] The determination device according to the embodiment determines pass / fail using a photographed image of a determination target. The determination device includes: a first determination unit that outputs a first pass / fail determination result for the input photographed image based on a feature amount extracted from the photographed image of the determination target belonging to a pass determination; an image processing unit that performs preprocessing for performing predetermined image processing on the photographed image and outputs a preprocessed image; a second determination unit that outputs a second pass / fail determination result for the input preprocessed image based on a feature amount extracted from the preprocessed image of the determination target belonging to a pass determination; and a determination control unit that determines the pass / fail of the determination target using the first pass / fail determination result and the second pass / fail determination result. When an abnormal region detected in one pass / fail determination result is different from an abnormal region detected in the other pass / fail determination result, the determination control unit determines it as pass. When an abnormal region detected in one pass / fail determination result is the same as an abnormal region detected in the other pass / fail determination result, the determination control unit determines it as fail.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Best Mode for Carrying Out the Invention

[0007] Hereinafter, embodiments will be described with reference to the drawings.

[0008] (First Embodiment) FIGS. 1 to 5 are diagrams for explaining the determination device according to the first embodiment.

[0009] FIG. 1 is a functional block diagram of the determination device 100 according to the present embodiment. The determination device 100 performs a pass / fail determination on a determination target using the pass / fail determination model generated by the determination model generation device 200.

[0010] The determination device 100 according to the present embodiment can be used, for example, for the appearance inspection of an article. In this case, the monitoring device K is an imaging device that captures an article, and the monitoring data is a captured image. Hereinafter, the determination device 100 according to the present embodiment will be described by taking the appearance inspection of an article as an example.

[0011] The imaging device K captures an article A to be inspected (judgment target) and outputs a captured image (image data) to the judgment device 100. The judgment device 100 stores the acquired captured image in the storage device 150. The imaging device K can acquire a still image or a moving image of the article A and output it to the judgment device 100. In the case of a moving image, a still image cut out from the moving image is used as the captured image of the article A. The orientation for capturing the article A is appropriately set so as to obtain a captured image suitable for inspection. For example, a plurality of articles A are sequentially conveyed by a conveying device H. The imaging device K is fixed and sequentially captures a plurality of conveyed articles A, and the judgment device 100 sequentially receives the captured images of the plurality of articles.

[0012] The judgment device 100 can sequentially determine the pass / fail of a plurality of different articles conveyed continuously in time series. The pass / fail judgment result of the judgment device 100 is output to the display device D. Although an example of a mode of continuously judging a plurality of different articles has been described, the present invention is not limited to this.

[0013] The judgment device 100 includes a judgment control unit 110, a first judgment unit 120, a second judgment unit 130, an image processing unit 140, and a storage unit 150. The judgment model generation device 200 includes a model generation unit 210, an image processing unit 220, and a storage unit 230.

[0014] FIG. 2 is an explanatory diagram of a first pass / fail judgment model and a second pass / fail judgment model used in the judgment device 100. The first pass / fail judgment model and the second pass / fail judgment model can be configured using an AI model and are generated by the model generation unit 210. The model generation in the present embodiment uses monitoring data (captured images, pre-processed images) of judgment targets belonging to pass judgments and does not use monitoring data of judgment targets belonging to fail judgments (negative judgments).

[0015] The first pass / fail determination model and the second pass / fail determination model are individual pass / fail determination functional units of each other, and each outputs a pass / fail determination result individually for one determination target. The first pass / fail determination model performs determination processing using a captured image, and the second pass / fail determination model performs determination processing using a pre-processed image obtained by subjecting the captured image to predetermined image processing.

[0016] The first determination unit 120 is a first pass / fail determination model generated by learning processing using captured images of determination targets belonging to pre-collected good determinations as learning data. The first determination unit 120 extracts feature amounts from the input captured images of the determination targets and determines the presence or absence of abnormalities.

[0017] That is, the first determination unit 120 is a processing unit that determines the pass / fail of article A based on the feature amounts extracted from a group of captured images of determination targets belonging to pre-collected good determinations, and outputs a first pass / fail determination result for the input captured image. The feature amounts extracted from the group of captured images of determination targets belonging to pre-collected good determinations are good determination feature information used by the first determination unit 120 and are generated by the learning processing of the AI model.

[0018] The second determination unit 130 is a second pass / fail determination model generated by learning processing using pre-processed images of determination targets belonging to pre-collected good determinations as learning data. The second determination unit 130 extracts feature amounts from the input pre-processed images of the determination targets and determines the presence or absence of abnormalities.

[0019] The second determination unit 130 is a processing unit that determines the quality of article A based on the feature amounts extracted from a group of preprocessed images of the determination target belonging to the good determination collected in advance, and outputs a second pass / fail determination result for the input preprocessed image. The feature amounts extracted from the group of preprocessed images of the determination target belonging to the good determination collected in advance are the good determination feature information used by the second determination unit 130, and are generated by the learning process of the AI model. The preprocessed image is an image (preprocessed image) obtained by performing predetermined image processing on the captured image captured by the imaging device K, and is generated by the image processing unit 220. That is, the image processing unit 220 performs predetermined image processing on the captured images belonging to the good determination used in the learning process of the first pass / fail determination model to generate a preprocessed image. Note that the learning data of the second pass / fail determination model may be a preprocessed image obtained by performing predetermined image processing on the captured images belonging to the good determination, and does not have to be a preprocessed image generated from the captured images belonging to the good determination used in the learning process of the first pass / fail determination model.

[0020] Note that, as an example of the first determination unit 120 and the second determination unit 130, an embodiment using an AI model has been described, but the present invention is not limited thereto. As described above, the first determination unit 120 and the second determination unit 130 may be configured as processing units that determine the quality of the determination target (article) using the feature amounts (good determination feature information) extracted from a group of monitoring data (captured images, preprocessed images) of the determination target belonging to the good determination collected in advance, and output a pass / fail determination result.

[0021] Here, the determination process of the first determination unit 120 will be described. First, the pixel value at each pixel position of the captured image corresponds to a feature amount (feature information). Then, the good determination feature information used by the first determination unit 120 is configured as information including the average pixel value at the same pixel position in each captured image of a plurality of articles belonging to the good determination.

[0022] A width that can be determined as a good determination is set for this average pixel value. For example, at each pixel position, a range of the average pixel value + a predetermined value (first threshold value) x1 and the average pixel value - a predetermined value (second threshold value) x2 is set. That is, the good determination feature information is information in which an upper limit value and a lower limit value based on the average pixel value are set for each pixel position. Note that the predetermined values x1 and x2 may be different values at each pixel position, or may be the same value at a plurality of each pixel position.

[0023] The predetermined values x1 and x2 can be set based on the distribution range (variance or deviation) of the pixel values of each photographed image of the article belonging to the good determination. The determination process determines whether the pixel value (feature amount) at each pixel position in the photographed image of the article A to be determined belongs to the good determination range (is below the upper limit value or above the lower limit value), and outputs a pass / fail determination result.

[0024] Note that, as described above, in the determination process of the present embodiment, when the feature amount to be determined does not exceed the range defined in the good determination feature information, the determination result is determined to be "good", but it can be a more detailed determination process. For example, the determination process can be configured to be divided into two stages: whether there is an abnormality in the object to be determined, and if there is, whether the abnormality exceeds a predetermined allowable range.

[0025] In this case, the good determination feature information can include first good determination feature amount information for determining the presence or absence of an abnormality in the object to be determined, and second good determination feature information for determining whether the abnormality is within a predetermined allowable range.

[0026] Taking the appearance inspection of the above article as an example, first, the pixel value at each pixel position of the photographed image is checked to determine whether there is a pixel exceeding the threshold value (first determination process). The good determination feature information used at this time corresponds to the first good determination feature information, and is the average pixel value + a predetermined value x1 and the average pixel value - a predetermined value x2 at each pixel position composed of the photographed image group belonging to the good determination described above.

[0027] Next, when it is determined that there are pixels exceeding the threshold value, in other words, when it is determined that there is an abnormality in the object to be determined, it is determined whether or not the abnormality is within a predetermined allowable range (second determination process). The good determination feature information used at this time corresponds to the second good determination feature information. For example, in the first determination unit 120, in the first determination process, a plurality of pixel values in the image data are respectively compared with the first good determination feature information, and abnormal pixels deviating from the threshold value can be extracted. Therefore, the first determination unit 120 can grasp the extracted abnormal pixel group, that is, the abnormal region.

[0028] Then, using the features of the abnormal region, for example, the area of the abnormal region, the shape of the abnormal region, and the distribution of pixel values in the abnormal region, etc., a second determination process is performed to determine whether or not the abnormal region grasped through the first determination process is within the range of good determination. The information regarding the features of the abnormal region at this time becomes the second good determination feature information. For example, when the area is used as the feature of the abnormal region, a threshold value for the area is set in advance. The area is represented by the number of pixels included in the abnormal region. The first determination unit 120 determines that the article A is "good" if the area of the abnormal region does not exceed the threshold value, and determines that the article A is "defective (no)" if it exceeds the threshold value.

[0029] For the shape of the abnormal region, for example, the aspect ratio can be used. The aspect ratio is the ratio of the length in the long axis direction to the length in the short axis direction. The long axis direction is the direction parallel to the longest line segment among the line segments obtained by connecting any two points on the outer edge of the particle. The short axis direction is perpendicular to the long axis direction. When the shape is used as the feature of the abnormal region, a threshold value for the aspect ratio is set in advance. The magnitude relationship between the aspect ratio and the threshold value is set according to the article. For example, when the quality of the article to be inspected is poor and the aspect ratio of the abnormal region tends to be large, the first determination unit 120 determines that the article is "defective (no)" when the aspect ratio exceeds the threshold value, and determines that the article is "good" if the aspect ratio does not exceed the threshold value.

[0030] The distribution of the abnormal region can use, for example, the ratio of the number of pixels below the lower threshold value to the number of pixels in the abnormal region, the ratio of the number of pixels above the upper threshold value to the number of pixels in the abnormal region, and the like. When the distribution is used as a feature of the abnormal region, thresholds are set for each of these ratios. The determination device 100 determines, for example, that the article is "good" if each of these ratios does not exceed the threshold, and determines that the article is "defective (no)" when any one of these ratios exceeds the corresponding threshold.

[0031] When the first determination unit 120 determines that there is no abnormality in the first determination process, and when it satisfies the second good determination feature information in the second determination process even if it is determined that there is an abnormality in the first determination process, it makes a "good" determination.

[0032] As described above, the first determination unit 120 of the present embodiment performs a first determination process for determining the presence or absence of an abnormality in the determination target based on the feature amount extracted from the captured image of the determination target belonging to the good determination, and when it is determined that there is an abnormality in the first determination process, a second determination process for determining whether or not the detected abnormal region exceeds a predetermined allowable range. Then, when it is determined that there is an abnormality in the first determination process and that the detected abnormal region exceeds the predetermined allowable range in the second determination process, the first determination unit 120 outputs a first pass / fail determination result indicating that an abnormal region has been detected in the input captured image. On the other hand, when it is determined that there is an abnormality in the first determination process and that the detected abnormal region does not exceed the predetermined allowable range in the second determination process, the first determination unit 120 outputs a first pass / fail determination result indicating that no abnormal region has been detected in the input captured image.

[0033] Next, the determination process of the second determination unit 130 will be described. As described above, the second determination unit 130 performs a determination process using a pre-processed image generated by subjecting the captured image used by the first determination unit 120 to predetermined image processing.

[0034] Therefore, the good judgment feature information used by the second determination unit 130 is configured as information including the average pixel value at the same pixel position in a group of pre-processed images obtained by image processing of captured images of a plurality of articles belonging to good judgment.

[0035] Then, a width within which a good judgment can be made is set for this average pixel value. For example, at each pixel position, a range of average pixel value + predetermined value (third threshold value) x3 and average pixel value - predetermined value (fourth threshold value) x4 is set. The good judgment feature information used by the second determination unit 130 is information in which an upper limit value and a lower limit value based on the average pixel value are set for each pixel position. Note that the predetermined values x3 and x4 may be different values at each pixel position or the same value at a plurality of each pixel position.

[0036] The predetermined values x3 and x4 can be set based on the distribution range (variance or deviation) of the pixel values of each pre-processed image of the articles belonging to good judgment. The determination process determines whether the pixel value (feature amount) at each pixel position in the pre-processed image of the article A to be determined belongs to the good judgment range (is below the upper limit value or above the lower limit value), and outputs a pass / fail determination result.

[0037] Note that the second determination unit 130 can also apply the subdivided determination process of the first determination unit 120 described above. That is, it is possible to perform the determination process in two stages: whether there is an abnormality in the determination target, and if so, whether the abnormality exceeds a predetermined allowable range.

[0038] In this case, the good judgment feature information used by the second determination unit 130 can include third good judgment feature amount information for determining the presence or absence of an abnormality in the determination target and fourth good judgment feature information for determining whether the abnormality is within a predetermined allowable range.

[0039] Taking the appearance inspection of the above-mentioned article as an example, first, the pixel value at each pixel position of the pre-processed image is checked to determine whether there is a pixel exceeding the threshold value (third determination process). The good determination feature information used at this time corresponds to the third good determination feature information, and is the average pixel value + a predetermined value x3 and the average pixel value - a predetermined value x4 at each pixel position composed of the pre-processed image group belonging to the good determination as described above.

[0040] Next, when it is determined that there is a pixel exceeding the threshold value, in other words, when it is determined that there is an abnormality in the determination target, it is determined whether or not the abnormality is within a predetermined allowable range (fourth determination process). The good determination feature information used at this time corresponds to the fourth good determination feature information. For example, in the third determination process, the second determination unit 130 compares a plurality of pixel values in the image data with the third good determination feature information respectively, and extracts abnormal pixels deviating from the threshold value. Therefore, the second determination unit 130 can grasp the extracted abnormal pixel group, that is, the abnormal region.

[0041] Then, as described above, using the features of the abnormal region, for example, the area of the abnormal region, the shape of the abnormal region, and the distribution of pixel values in the abnormal region, etc., the second determination unit 130 performs a fourth determination process to determine whether the abnormal region grasped through the third determination process is within the range of good determination. The information regarding the features of the abnormal region at this time becomes the fourth good determination feature information. For example, when the area is used as the feature of the abnormal region, a threshold value for the area is set in advance. The area is represented by the number of pixels included in the abnormal region. The second determination unit 130 determines that article A is "good" if the area of the abnormal region does not exceed the threshold value, and determines that article A is "defective (no)" if it exceeds. Regarding the shape of the abnormal region and the distribution of pixel values in the abnormal region, it can be applied in the same manner as the first determination unit 120 described above, so the description is omitted.

[0042] The second determination unit 130 makes a "good" determination when it is determined that there is no abnormality in the third determination process, and when it satisfies the fourth good determination feature information in the fourth determination process even if it is determined that there is an abnormality in the third determination process.

[0043] As described above, the second determination unit 130 of the present embodiment performs a third determination process for determining the presence or absence of an abnormality in the determination target based on the feature amount extracted from the pre-processed image belonging to the good determination, and when it is determined that there is an abnormality in the third determination process, a fourth determination process for determining whether or not the detected abnormal region exceeds a predetermined allowable range. Then, when it is determined that there is an abnormality in the third determination process and it is determined that the detected abnormal region exceeds the predetermined allowable range in the fourth determination process, the second pass / fail determination result indicating that an abnormal region has been detected in the input pre-processed image is output. On the other hand, when it is determined that there is an abnormality in the third determination process and it is determined that the detected abnormal region does not exceed the predetermined allowable range in the fourth determination process, the second pass / fail determination result indicating that no abnormal region has been detected in the input pre-processed image is output.

[0044] Here, the image processing for generating the pre-processed image will be described. The image processing is performed by the image processing unit 140 of the determination device 100 and the image processing unit 220 of the determination model generation device 200, and both functional units can perform the same image processing (same pre-processing).

[0045] The image processing unit 140 can perform a first pre-processing and a second pre-processing. The first pre-processing performs edge extraction processing on the captured image to generate a first image, performs averaging processing on the captured image to generate a second image, and generates a difference image between the generated first image and the second image.

[0046] The second pre-processing performs contrast conversion processing on the difference image generated in the first pre-processing, and further performs reduction processing and dilation processing on the difference image subjected to the contrast conversion processing to generate a pre-processed image.

[0047] Edge extraction processing is, for example, an image processing for removing noise in the X or Y direction from a captured image, and emphasizes the boundary between bright and dark areas. Specifically, it is a process of detecting locations where the luminance in the captured image changes significantly. Edge extraction means detecting the brightness or color gradient of pixel values between adjacent pixels on the left and right (X) or above and below (Y). Edge extraction Y is the result of detecting the gradient of pixels adjacent above and below, and is an image with noise in the X direction removed. On the other hand, edge extraction X is the result of detecting the gradient of pixels adjacent left and right, and is an image with noise in the Y direction removed. It is arbitrary which direction of noise in the X or Y direction to remove.

[0048] Averaging processing is a known image processing for removing noise, similar to edge extraction processing. For example, the center of a 3×3 pixel is replaced with the average luminance value of 9 pixels. This can blur the image and reduce the influence of noise components.

[0049] A difference image is image data having pixel values obtained by subtracting a second image subjected to averaging processing from a first image subjected to edge extraction processing at each pixel position.

[0050] Next, the second preprocessing performs contrast conversion processing and dilation / erosion processing on the difference image generated in the first preprocessing. The contrast conversion processing is a process for increasing the contrast between the surroundings and the black dots. The stronger the contrast, the brighter the bright parts become and the darker the dark parts become. That is, the higher the contrast, the brighter the bright parts and the darker the dark parts, and the difference between light and dark becomes clearer. Conversely, the lower the contrast, the smaller the difference between the bright and dark parts, and it becomes blurry.

[0051] This embodiment can perform two-stage contrast conversion processing. For example, a contrast conversion process of minus 30 is performed on the difference image generated in the first preprocessing. This can lower the contrast, reduce the difference between bright and dark parts, and overall remove dark components. Further, as the second stage, a contrast conversion process of plus 120 is performed on the image after the contrast conversion process of minus 30. This increases the contrast and enlarges the difference between bright and dark parts. That is, the contrast conversion process of plus 120 converts the overall brightness to an intermediate density and is a process for making it easier for people to see. This is because the abnormal area may become too dark after the difference image processing and may be difficult to distinguish with the human eye.

[0052] Note that, although a case where the contrast conversion process is performed in two stages has been described as an example, it is not limited to this. For example, the contrast conversion process may be performed only once, or may be performed three or more times. Considering the material and shape of the article to be judged, the illuminance and color tone at the time of shooting, etc., it can be adjusted to an arbitrary contrast degree at an arbitrary number of times.

[0053] The dilation and erosion processing is an image processing performed on the image data after the contrast conversion processing. This dilation and erosion processing is a process that combines dilation processing and erosion processing. First, it is eroded and minimized, and then dilated and maximized.

[0054] The erosion processing is, for example, an image processing that replaces the center of a 3×3 pixel with the darkest luminance value among the nine pixels, emphasizing black pixels. The dilation processing is, for example, an image processing that replaces the center of a 3×3 pixel with the brightest luminance value among the nine pixels, removing black noise components.

[0055] The preprocessed image of this embodiment is generated by performing image processing including first preprocessing and second preprocessing on the captured image acquired by the imaging device K, and is used for the determination process of the second determination unit 130 and the learning process of the second pass / fail determination model of the model generation unit 210.

[0056] For example, the defective part to be determined is photographed as a lump of dark part of a certain size. However, on the surface of the object to be determined, there may be noise components called hair lines (fine lines formed in the vertical (Y) direction or the horizontal (X) direction). Therefore, in order to "suppress the noise components" and "emphasize the defective parts", image processing including a first preprocessing and a second preprocessing can be performed. Specifically, when there are noise components called hair lines in the vertical (Y) direction on the surface of the object to be determined, edge extraction can be performed to suppress the density change in the X direction (noise components as hair lines) and emphasize the density change in the Y direction (actual defective parts). Then, in order to further emphasize the defective parts, difference processing with the averaged image can be performed, or contrast conversion processing, reduction processing, and dilation processing can be further performed on the difference image to generate a preprocessed image.

[0057] Note that the preprocessing including the above-described image processes such as edge extraction, averaging process, difference process, contrast conversion process, and reduction process and dilation process is described separately as the first preprocessing and the second preprocessing for convenience of explanation, but it is not limited to this. For example, the difference process may be configured as the second preprocessing, or the whole may be configured as one preprocessing without distinguishing between the first preprocessing and the second preprocessing.

[0058] FIG. 3 is an explanatory diagram of the determination process of the present embodiment. The determination control unit 110 determines the pass / fail of the object to be determined using the first pass / fail determination result of the first determination unit 120 and the second pass / fail determination result of the second determination unit 130. The determination control unit 110 outputs the final pass / fail determination result of the object to be determined.

[0059] As shown in FIG. 3, one captured image is acquired for one object to be determined. The acquired captured image is output to the first determination unit 120 and also to the image processing unit 140. The image processing unit 140 performs preprocessing including the above-described first preprocessing and second preprocessing, and outputs the preprocessed image to the second determination unit 130.

[0060] The first determination unit 120 and the second determination unit 130 each perform a determination process individually, and individual pass / fail determination results (the first pass / fail determination result, the second pass / fail determination result) are output to the determination control unit 110 for one determination target.

[0061] When the abnormal area detected in one of the pass / fail determination results of the determination control unit 110 is different from the abnormal area detected in the other pass / fail determination result, it is determined as "pass". Also, when the abnormal area detected in one of the pass / fail determination results is the same as the abnormal area detected in the other pass / fail determination result, it is determined as "fail (no)".

[0062] Conventionally, suppression of over-detection is important in the technical field of pass / fail determination. Over-detection means being determined as a defective product when it is actually a non-defective product. By suppressing over-detection, the determination accuracy can be improved.

[0063] On the other hand, there is a limit to suppressing over-detection with only one determination unit (one pass / fail determination model). That is, over-detection and mis-detection are two sides of the same coin. Suppressing over-detection, where a non-defective product is determined as a defective product, increases mis-detection, where a defective product is determined as a non-defective product, and the determination accuracy decreases.

[0064] Therefore, the determination device 100 of the present embodiment combines the "pass / fail determination result based on the captured image" and the "pass / fail determination result based on the pre-processed image obtained by pre-processing the captured image" to suppress over-detection and improve the determination accuracy.

[0065] Specifically, it utilizes the fact that in the captured image and the preprocessed image of the same determination target, abnormal regions (defective parts) are often detected at the same position, and over-detected parts often occur at different positions. As shown in FIG. 3, the AND part (overlapping part) of the first pass / fail determination result of the first pass / fail determination model and the second pass / fail determination result of the second pass / fail determination model is taken. If an abnormal region is detected at the same position in the pass / fail determination results of both, a "defective (fail)" determination is made, and if the position of the detected abnormal region is detected at different positions in the pass / fail determination results of both, a "good" determination is made.

[0066] In this way, the determination device 100 of the present embodiment determines that the abnormal region detected under the AND condition of the first pass / fail determination result and the second pass / fail determination result is a defective part, so the occurrence frequency of over-detected parts can be reduced, and the detection accuracy of defective parts can be improved. That is, if the abnormal regions detected in the pass / fail determination results of both are at the same position, it goes in the direction of denying the possibility of over-detection of the abnormal region and in the direction of affirming the possibility of being a defective part. Conversely, if the abnormal regions detected in the pass / fail determination results of both are at different positions, it goes in the direction of affirming the possibility of over-detection of the abnormal region and in the direction of denying the possibility of being a defective part. Therefore, the determination device 100 of the present embodiment can absorb the variation of the good product parts (the allowable range determined as good) to suppress over-detection and improve the determination accuracy.

[0067] FIG. 4 is a flowchart of the model generation process of the present embodiment.

[0068] As shown in FIG. 4, in the storage unit 230 of the determination model generation device 200, a group of captured images belonging to the good determination among a plurality of different captured images of determination targets collected in advance is stored (S201). Then, the model generation unit 210 performs a learning process using the group of captured images belonging to the good determination (S202), extracts feature amounts (for example, luminance values at each pixel position) necessary for determining the pass / fail of the determination target from each captured image, and generates a first pass / fail determination model based on the good determination feature information (S203). The model generation unit 210 stores the generated first pass / fail determination model in the storage unit 230 (S204).

[0069] Similarly, in the storage unit 230 of the determination model generation device 200, a group of captured images belonging to the positive determination among a plurality of different captured images of determination targets collected in advance is stored (S211). The image processing unit 220 performs image processing (pre-processing) on each of the groups of captured images belonging to the positive determination to generate a group of pre-processed images belonging to the positive determination (S212). The generated group of pre-processed images is stored in the storage unit 230, and the model generation unit 210 performs a learning process using the group of pre-processed images belonging to the positive determination (S213), extracts feature amounts (for example, luminance values at each pixel position) necessary for determining the pass / fail of the determination target from each pre-processed image, and generates a second pass / fail determination model based on the positive determination feature information (S214). The model generation unit 210 stores the generated second pass / fail determination model in the storage unit 230 (S215).

[0070] FIG. 5 is a flowchart of the determination process of the present embodiment. When starting the determination process, the determination device 100 acquires the first pass / fail determination model and the second pass / fail determination model from the determination model generation device 200 and applies them to the first determination unit 120 and the second determination unit 130.

[0071] As shown in FIG. 5, the determination device 100 receives an input of a captured image (S101), and performs a determination process triggered by the input of the captured image.

[0072] The first determination unit 120 of the determination device 100 extracts feature amounts from the input captured image, compares the positive determination feature information with the feature amounts of the determination target, and performs a pass / fail determination. The first determination unit 120 outputs the first pass / fail determination result to the determination control unit 110 (S102). The determination device 100 performs predetermined image processing on the captured image input to the first determination unit 120 to generate a pre-processed image. The generated pre-processed image is output to the second determination unit 130. The second determination unit 130 of the determination device 100 extracts feature amounts from the input pre-processed image, compares the positive determination feature information with the feature amounts of the determination target, and performs a pass / fail determination. The second determination unit 130 outputs the second pass / fail determination result to the determination control unit 110 (S104).

[0073] Note that the determination process of the first determination unit 120 and the determination process of the second determination unit 130 are processes that are independently performed on the same determination target, and both determination processes can be performed in parallel, or the other determination process can be started after one determination process is completed.

[0074] The determination control unit 110 determines whether or not the abnormal area detected in one of the first pass / fail determination result and the second pass / fail determination result is the same location as the abnormal area detected in the other pass / fail determination result (S105).

[0075] When the determination control unit 110 determines that the abnormal area detected in one pass / fail determination result is the same location as the abnormal area detected in the other pass / fail determination result, it makes a "defective (no)" determination (S106). When it is determined that the abnormal area detected in one pass / fail determination result is a different location from the abnormal area detected in the other pass / fail determination result, a "good" determination is made (S107).

[0076] The determination device 100 outputs the pass / fail determination result to the display device D (S108). Also, the pass / fail determination result is stored in the storage unit 150 as a log. Note that the first pass / fail determination result and the second pass / fail determination result can also be configured to be stored in association with the pass / fail determination result output by the determination control unit 110.

[0077] The determination device 100 determines whether or not a captured image of the next determination target has been input in time series (S109). When the next captured image has been input (YES in S109), it returns to step S101 and performs the determination process based on the first pass / fail determination result and the second pass / fail determination result. When it is determined in step S109 that there is no input of the next monitoring data (NO in S109), the determination device 100 checks for the presence of an end instruction to end the determination process. When there is an end instruction (YSE in S110), the determination process is ended. When there is no end instruction (NO in S110), it waits until the next captured image is input.

[0078] In addition, whether the abnormal regions of both sides are in the same location can be determined by considering, for example, the degree of overlap between the abnormal region detected in one pass / fail determination result and the abnormal region detected in the other pass / fail determination result, in addition to the case where the pixel positions of the abnormal regions of both sides are the same. That is, it is possible to detect the degree of overlap of the pixel groups indicating abnormalities, and when an overlap of a predetermined value or more is recognized, it can be determined that the abnormal regions of both sides are in the same location. As another example, for instance, if at least a part overlaps, it can also be determined that the abnormal regions are detected at the same location. As yet another example, even if the abnormal regions do not overlap with each other, if the distance between the abnormal regions is within a predetermined range, it may be determined that the abnormal regions are detected at the same location.

[0079] In addition, the size of the abnormal region may be considered. For example, even if the abnormal region of one pass / fail determination result overlaps with the abnormal region of the other pass / fail determination result, if the size of one abnormal region is extremely smaller than the size of the other abnormal region, the smaller abnormal region may be over-detected. Therefore, even if abnormal regions are detected at the same location, if the ratio of the sizes of the abnormal regions of each other is larger than a predetermined value, a "good" determination may be made, and if it is smaller than the predetermined value (when there is not much difference in their sizes), it may be configured to make a "bad" determination.

[0080] (Second Embodiment) FIGS. 6 to 8 are diagrams for explaining the determination device of the second embodiment. FIG. 6 is a functional block diagram of the determination device 100 of the present embodiment. The determination device 100 of the present embodiment has a configuration in which a preprocessing setting unit 141 and a preprocessing setting unit 221 are added to the above-described first embodiment. The same components as those in the above-described first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

[0081] The determination device 100 has previously set information regarding preprocessing, such as what image processing to perform, in what order to perform each image processing, and what processing parameters of the image processing to set, in order to obtain the preprocessed image to be input to the second determination unit 130. This is linked to the design of the good determination feature information (design of the second pass / fail determination model) of the second determination unit 130.

[0082] On the other hand, the "features of defective portions" and the "features of noise (such as surface conditions)" differ depending on the determination target, and thus what preprocessing to perform also differs. As described above, when a noise component called a hairline existing on the surface of the determination target appears in the vertical (Y) direction in the captured image, it is necessary to perform edge processing Y. Conversely, when a noise component called a hairline existing on the surface of the determination target appears in the horizontal (X) direction in the captured image, it is necessary to perform edge processing X that suppresses the density change in the vertical (Y) direction and emphasizes the density change in the horizontal (X) direction.

[0083] The same applies to contrast conversion processing, reduction processing, and dilation processing. Settings are made such as at what strength to perform them or whether to apply each of the contrast conversion processing, reduction processing, and dilation processing itself, according to the situations of the "features of defective portions" and the "features of noise (such as surface conditions)".

[0084] Such preprocessing settings are determined through trial and error according to the "features of defective portions" and the "features of noise (such as surface conditions)" of the discrimination target when designing the good determination feature information (second pass / fail determination model) of the second determination unit 130.

[0085] Therefore, the determination device 100 of the present embodiment includes a preprocessing setting unit 141 that controllably selects preprocessing according to the "features of defective portions" and the "features of noise (such as surface conditions)" of the determination target. Similarly, the determination model generation device 200 includes a preprocessing setting unit 221.

[0086] The determination device 100 of the present embodiment includes a preprocessing setting unit 141, and based on information related to preprocessing (preprocessing setting information) for generating a preprocessed image used in the learning process of the second pass / fail determination model, it can controllably select the image processing that constitutes the preprocessing performed by the image processing unit 140 and can controllably set the processing parameters of the selected image processing. Therefore, the preprocessing can be appropriately selected (designed), and it can flexibly respond according to the "characteristics of defective parts" and "characteristics of noise (such as surface conditions)" of the determination target so that "defective parts are emphasized" and "noise (pseudo-defects) is suppressed".

[0087] FIG. 7 is a flowchart of the second pass / fail determination model generation process of the present embodiment. It corresponds to the right figure of FIG. 4 in the first embodiment.

[0088] As shown in FIG. 7, in the storage unit 230 of the determination model generation device 200, a group of captured images belonging to the pass determination among a plurality of different captured images of determination targets collected in advance is stored (S211). The preprocessing setting unit 221 can controllably select one or more image processes that constitute the preprocessing. Also, it can controllably set various parameters of the selected image processing. The preprocessing setting unit 221 generates preprocessing setting information including the selected image processing and the parameters of each image processing, and stores it in the storage unit 230 (S212a).

[0089] The image processing unit 220 performs image processing (preprocessing) on each of the group of captured images belonging to the pass determination based on the preprocessing setting information, and generates a group of preprocessed images belonging to the pass determination (S212). The generated group of preprocessed images is stored in the storage unit 230, and the model generation unit 210 performs a learning process using the group of preprocessed images belonging to the pass determination (S213), extracts feature amounts (for example, luminance values at each pixel position) necessary for determining the pass / fail of the determination target from each preprocessed image, and generates a second pass / fail determination model based on the pass determination feature information (S214). The model generation unit 210 stores the generated second pass / fail determination model in the storage unit 230 (S215).

[0090] Further, the determination model generation device 200 may be configured to have a function of providing preprocessing setting information to the determination device 100 that uses the generated second pass / fail determination model. That is, the image processing unit 140 of the determination device 100 may be configured to operate in conjunction with the image processing unit 220, and the preprocessing setting information may be associated with the generated second pass / fail determination model and provided to the determination device 100 (S215a).

[0091] FIG. 8 is a flowchart of the determination process of the present embodiment, corresponding to FIG. 5 of the first embodiment above.

[0092] When starting the determination process, the determination device 100 acquires the first pass / fail determination model and the second pass / fail determination model from the determination model generation device 200 and applies them to the first determination unit 120 and the second determination unit 130. At this time, the preprocessing setting unit 141 acquires the preprocessing setting information associated with the second pass / fail determination model and sets (generates) the preprocessing setting information used by the preprocessing setting unit 141 (S1001).

[0093] Specifically, the preprocessing setting unit 141 takes in the preprocessing setting information provided from the determination model generation device 200, controllably selects one or more image processes that constitute the preprocessing of the image processing unit 140, and further controllably sets various parameters of the selected image processes. The preprocessing setting unit 141 can automatically generate the preprocessing setting information by taking it in, or receive the selection input of one or more image processes and the setting input of various parameters of the image processes, and generate the preprocessing setting information used by the image processing unit 140.

[0094] The determination device 100 receives the input of a captured image (S101), and performs a determination process triggered by the input of the captured image. In step S103, the image processing unit 140 performs predetermined image processing based on the preprocessing setting information on the captured image input to the first determination unit 120 to generate a preprocessed image. For other processing steps, they are the same as those in the first embodiment above, and the description is omitted by attaching the same reference numerals.

[0095] In the above embodiments, the quality determination using the captured image of an article as monitoring data was described by taking the appearance inspection of the article as an example. However, for example, it is also possible to configure a determination device 100 that applies 3D image data (stereoscopic image data) such as 3D ultrasonic image data as monitoring data. In this case, the monitoring device K can apply a 3D ultrasonic image acquisition device (3D echo (3D ultrasonic) inspection device).

[0096] Also, although the mode of sequentially performing quality determination on a plurality of different determination targets in time series has been described, for example, articles that are continuously manufactured while flowing (rolled steel materials subjected to rolling, steel materials such as H-beams and I-beams, long members such as bar-shaped reinforcing bars) can be used as determination targets. In this case, the determination target is divided in the longitudinal direction and continuously photographed. Then, each of the continuously photographed images obtained by the division is input as monitoring data, and it is also possible to configure the system to determine the quality of the long article for each part in the longitudinal direction.

[0097] The determination device 100 and the determination model generation device 200 may be configured as one device. Also, the determination device 100 can be constructed in a cloud-based service provision form. That is, the monitoring data (for example, the captured image) output from the monitoring device K is configured to be transmitted to the determination device 100 through the IP network. Thereby, the quality determination process can be performed on the cloud side, and the quality determination result can be transmitted to the display device D, the monitoring terminal, the monitoring equipment, etc. through the IP network. Both the determination device 100 and the determination model generation device 200 may be constructed in a cloud-based service provision form.

[0098] Also, the determination device 100 and the determination model generation device 200 are computer devices equipped with computing functions, storage functions, communication functions, etc. such as server devices. Also, as a hardware configuration, it can be equipped with a memory (main storage device), operation input means such as a mouse, keyboard, touch panel, output means such as a printer, auxiliary storage device (hard disk, etc.).

[0099] In addition, each function of the present invention can be realized by a program. A computer program prepared in advance to realize each function is stored in an auxiliary storage device, and a control unit such as a CPU reads the program stored in the auxiliary storage device into a main storage device and executes the program read into the main storage device, enabling the computer to operate the functions of each part of the present invention. On the other hand, each function of the apparatus 100 can also be configured by individual apparatuses, or a computer system can be configured by connecting a plurality of apparatuses directly or via a network.

[0100] Also, the above program can be provided to a computer in a state recorded on a computer-readable recording medium. Examples of computer-readable recording media include optical discs such as CD-ROM, Blu-ray (registered trademark) Disc Rewritable, phase change optical discs such as DVD-ROM, magneto-optical discs such as MO (Magneto Optical), magnetic discs such as floppy (registered trademark) discs and hard discs, memory cards such as SD memory cards and USB flash drives. In addition, hardware devices such as integrated circuits (IC chips such as ROM and RAM) specially designed and configured for the purpose of the present invention are also included as recording media. Furthermore, the present invention including the above program is not limited to being executed on the architecture of a von Neumann-type computer, and may be executed on the architecture of a so-called non-von Neumann-type computer such as a neurocomputer based on the mechanism of a cerebral nerve circuit or a quantum computer applying quantum mechanics to information processing.

[0101] Although the embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0102] 100 Determination device 110 Determination control unit 120 First determination unit 130 Second determination unit 140 Image processing unit 141 Pre - processing setting unit 150 Memory unit 200 Determination model generation device 210 Model generation unit 220 Image processing unit 221 Pre - processing setting unit 230 Memory unit A Object to be determined D Display device H Conveying device K Monitoring device

Claims

1. A determination device that determines pass / fail using a captured image to be determined, comprising: a first determination unit that outputs a first pass / fail determination result for the input captured image based on feature amounts extracted from captured images to be determined belonging to a pass determination; an image processing unit that performs preprocessing for performing predetermined image processing on the captured image and outputs a preprocessed image; a second determination unit that outputs a second pass / fail determination result for the input preprocessed image based on feature amounts extracted from preprocessed images to be determined belonging to a pass determination; a determination control unit that determines the pass / fail of the determination target using the first pass / fail determination result and the second pass / fail determination result; wherein the determination control unit: determines as pass when an abnormal region detected in one pass / fail determination result is a different location from the abnormal region detected in the other pass / fail determination result; determines as fail when an abnormal region detected in one pass / fail determination result is the same location as the abnormal region detected in the other pass / fail determination result. A determination device characterized by the above.

2. The image processing unit: performs first preprocessing of generating a first image by performing edge extraction processing on the captured image, generating a second image by performing averaging processing on the captured image, and generating a difference image between the first image and the second image; performs second preprocessing of performing contrast conversion processing on the difference image generated in the first preprocessing, and performing reduction processing and dilation processing on the difference image subjected to the contrast conversion processing to generate the preprocessed image. The determination device according to claim 1, characterized by performing the above.

3. The first determination unit is a first pass / fail determination model generated by a learning process using captured images to be determined belonging to a pass determination collected in advance, and extracts feature amounts from the input captured images to be determined to determine the presence or absence of abnormalities; The second determination unit is a second pass / fail determination model generated by a learning process using preprocessed images to be determined belonging to a pass determination collected in advance, and extracts feature amounts from the input preprocessed images to be determined to determine the presence or absence of abnormalities. The determination device according to claim 1 or 2, characterized by the above.

4. The first determination unit performs a first determination process of determining the presence or absence of abnormalities in the determination target based on feature amounts extracted from captured images to be determined belonging to a pass determination, and when it is determined that there are abnormalities in the first determination process, performs a second determination process of determining whether the detected abnormal region exceeds a predetermined allowable range. When the first determination unit determines that there is an abnormality in the first determination process and determines that it exceeds a predetermined allowable range in the second determination process, it outputs the first pass / fail determination result indicating that an abnormal region has been detected in the input captured image. When the first determination unit determines that there is an abnormality in the first determination process and determines that it does not exceed a predetermined allowable range in the second determination process, it outputs the first pass / fail determination result indicating that no abnormal region has been detected in the input captured image. The determination device according to claim 1 or 2, characterized in that.

5. The second determination unit performs a third determination process for determining the presence or absence of an abnormality in the determination target based on the feature amount extracted from the preprocessed image of the determination target belonging to the good determination, and when it is determined that there is an abnormality in the third determination process, a fourth determination process for determining whether the detected abnormal region exceeds a predetermined allowable range. When the second determination unit determines that there is an abnormality in the third determination process and determines that it exceeds a predetermined allowable range in the fourth determination process, it outputs the second pass / fail determination result indicating that an abnormal region has been detected in the input preprocessed image. When the second determination unit determines that there is an abnormality in the third determination process and determines that it does not exceed a predetermined allowable range in the fourth determination process, it outputs the second pass / fail determination result indicating that no abnormal region has been detected in the input preprocessed image. The determination device according to claim 4, characterized in that.

6. The second determination unit is a second pass / fail determination model generated by a learning process using preprocessed images of determination targets belonging to good determinations collected in advance, and extracts a feature amount from the input preprocessed image of the determination target to determine the presence or absence of an abnormality. Based on information regarding preprocessing for generating the preprocessed image used in the learning process of the second good determination model, it is provided with a preprocessing setting unit that can selectively control the image processing constituting the preprocessing performed by the image processing unit and can controllably set the processing parameters of the selected image processing. The determination device according to claim 1, characterized in that.

7. A program executed by a computer device that determines pass / fail using a captured image of a determination target, the computer device having a first function that outputs a first pass / fail determination result for the input captured image based on the feature amount extracted from the captured image of the determination target belonging to the good determination. Perform preprocessing to apply predetermined image processing to the captured image and output a preprocessed image; A third function that outputs a second pass / fail determination result for the input preprocessed image based on the feature amount extracted from the preprocessed image of the determination target belonging to the pass determination; A fourth function that determines the pass / fail of the determination target using the first pass / fail determination result and the second pass / fail determination result is realized; The fourth function is as follows: When the abnormal area detected in one pass / fail determination result is different from the abnormal area detected in the other pass / fail determination result, it is determined as pass; When the abnormal area detected in one pass / fail determination result is the same as the abnormal area detected in the other pass / fail determination result, it is determined as fail; A program characterized by the above.

Images