Image processing method and image processing device

Abstract

To provide an image processing method which is applied with machine learning and achieves both of detection accuracy and a processing speed and an image processing device.SOLUTION: An image processing method according to the present disclosure comprises the steps of: acquiring a plurality of images by imaging different portions of a prescribed range of an object (2); performing an individual evaluation on whether each of the plurality of images includes a feature of the detection object by applying a prescribed image processing method acquired by using machine learning (S3); and performing a total evaluation. The total evaluation evaluates that the object includes a singular part of the detection object (S5) when a ratio X1 / Y1 of the number Y1 of the plurality of images constituting a prescribed range or a partial range being a portion of the prescribed range to the number X1 of the images evaluated to include the features in the individual evaluation among the Y1 images is equal to or greater than a first prescribed value (S4).SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing method and an image processing device. [Background technology]

[0002] There are known defect inspection systems, anomaly inspection systems, and anomalous part detection systems that inspect for the presence or absence of an abnormal state or a part having characteristics different from others. In recent years, a method has been proposed for evaluating the presence or absence of a part having characteristics different from most other regions (hereinafter referred to as a "unique part"), including defects, by using machine learning.

[0003] Patent Document 1 describes a unique part detection system that extracts a unique part image containing a unique part with any characteristic from an image of an object captured by an imaging unit, and identifies the type of unique part through machine learning using the unique part image as input. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 003813 Summary of the Invention [Problem to be solved by the invention]

[0005] The unique portion to be detected may have various shapes. When the unique portion has a large shape or is formed in an elongated shape, the image of the unique portion extracted from the captured image will also be large.

[0006] However, if such a singular part is detected by machine learning, the learning time for generating a trained model becomes long. In addition, the calculation time for detecting the singular part using the trained model increases. Although such a problem can be solved by using an information processing device with high calculation power, it will lead to an increase in cost, which will be a hindrance when implementing it in a manufacturing line for applications such as in-line inspection.

[0007] Furthermore, if the size of the unique part image is larger than the image size set as the image size to which machine learning is applied, the unique part image must be extracted without including part of the unique part, resulting in a decrease in the accuracy of detection of the unique part.

[0008] Therefore, in order to solve the above problems, an object of the present invention is to provide an image processing method and an image processing device that achieve both detection accuracy and processing speed in an image processing method to which machine learning is applied. [Means for solving the problem]

[0009] The image processing method according to the present disclosure includes: acquiring a plurality of images of an object, each of which captures a different portion of a predetermined range of the object; applying a predetermined image processing method acquired using machine learning to each of the plurality of images to perform an individual evaluation of whether or not each of the plurality of images contains a feature of the object to be detected; and performing a comprehensive evaluation of whether or not the object contains a unique portion of the object to be detected based on whether or not the feature is contained in two or more of the plurality of images. Here, the comprehensive evaluation evaluates that the object contains the unique portion of the object to be detected when a ratio X1 / Y1 of the number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to the number X1 of the images evaluated in the individual evaluation as containing the feature is equal to or greater than a first predetermined value.

[0010] According to this image processing method, a predetermined image processing method acquired using machine learning is applied to perform individual evaluation of whether each of the multiple images contains the feature of the detection target, and a comprehensive evaluation of whether the object contains the unique part of the detection target is performed based on whether the feature is included in two or more images of the multiple images. Specifically, the presence or absence of the unique part is evaluated based on the number Y1 of at least one of the multiple images corresponding to a predetermined range or the multiple images constituting a partial range that is a part of the predetermined range, and the number X1 of images evaluated as containing the feature. In other words, whether the object contains the unique part of the detection target is evaluated based on the number of images that have an estimated score indicating that the feature is included in the individual evaluation that is equal to or greater than a predetermined value among the multiple images constituting the predetermined range or the partial range of the object. Therefore, even if the size of the unique part image is larger than the image size set as the image size to which machine learning is applied, individual evaluation is performed at that image size, and a comprehensive evaluation is performed based on the number of multiple images containing the feature in the predetermined range or the partial range, thereby making it possible to shorten the learning time and calculation time while detecting a unique part larger than the image size.

[0011] In this disclosure, "evaluation" includes determining the presence or absence of a unique part, determining the presence or absence of a unique part together with probability information, and outputting probability information (estimated score) regarding the presence or absence of a unique part. In addition, in the present disclosure, "evaluation" includes, when there are multiple types of unique parts, outputting information regarding the presence or absence of the unique part and / or information regarding the type of the unique part.

[0012] In addition, different or the same evaluations may be performed before or after the individual evaluation and the comprehensive evaluation described above. For example, after performing a comprehensive evaluation for a predetermined range, a comprehensive evaluation may be further performed once or multiple times for a partial range that is a part of the predetermined range.

[0013] Moreover, one aspect of the present disclosure further includes dividing an original image obtained by capturing an image of a predetermined range of the object, and generating the multiple images each having the same image size.

[0014] Also, in one aspect of the present disclosure, when the comprehensive evaluation is such that the ratio X1 / Y1 is less than the first predetermined value, and is equal to or greater than a second predetermined value smaller than the first predetermined value, and the ratio X2 / Y2 of the number X2 of images, which is less than Y1, among the plurality of images constituting a second partial range that is a part of the predetermined range or the partial range composed of the Y1 images and in which the feature is evaluated to be included in the individual evaluation of the Y2 images is equal to or greater than a third predetermined value, it is evaluated that the object includes a specific part to be detected.

[0015] Also, in one aspect of the present disclosure, in the comprehensive evaluation, when there is an image A evaluated to include the feature in the individual evaluation, and among the plurality of images located around the image A in the individual evaluation, the feature is evaluated to be included in two or more images, it is also possible to evaluate that the object includes a specific part to be detected.

[0016] Also, in one aspect of the present disclosure, in the comprehensive evaluation, when there is an image A evaluated to include the feature in the individual evaluation, there is an image B located around the image A and evaluated to include the feature in the individual evaluation, there is an image C located around the image B but not around the image A and evaluated not to include the feature in the individual evaluation, there is an image D located around the image C but not around the image A and the image B and evaluated to include the feature in the individual evaluation, and there is an image E located around the image D but not around the image A, the image B, and the image C and evaluated to include the feature in the individual evaluation, it is also possible to evaluate that the object includes a specific part to be detected.

[0017] The present disclosure discloses an image processing device, which includes an image acquisition means for acquiring a plurality of images capturing different parts of a predetermined range of an object, an individual evaluation means for performing an individual evaluation of whether each of the plurality of images contains a feature of the detection target by applying a predetermined image processing method acquired using machine learning to each of the plurality of images, and a comprehensive evaluation means for performing a comprehensive evaluation of whether the object contains a peculiar part of the detection target based on whether the feature is contained in two or more images of the plurality of images, and the comprehensive evaluation means is configured to evaluate the object as containing the peculiar part of the detection target when a ratio X1 / Y1 between the number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range and the number X1 of the images evaluated by the individual evaluation means as containing the feature is equal to or greater than a first predetermined value.

[0018] According to such an image processing device, it is possible to perform an individual evaluation of whether each of a plurality of images contains the feature of the detection target by applying a predetermined image processing method acquired using machine learning, and to perform a comprehensive evaluation of whether the target contains the unique part of the detection target based on whether the feature is included in two or more images out of the plurality of images, specifically, it is possible to evaluate the presence or absence of a unique part based on the number Y1 of a plurality of images constituting a predetermined range or a partial range that is a part of the predetermined range, and the number X1 of images evaluated as containing the feature. Therefore, even if the size of the unique part image is larger than the image size set as the image size to which machine learning is applied, it is possible to shorten the learning time and calculation time while detecting a unique part larger than the image size by performing an individual evaluation at that image size and performing a comprehensive evaluation based on the number of images containing the feature within the predetermined range or partial range.

[0019] The image processing device according to the present disclosure may further include means for dividing an original image obtained by capturing an image of a predetermined range of the object, and generating the plurality of images each having the same image size.

[0020] Also, in one aspect of the present disclosure, when the ratio X1 / Y1 is less than the first predetermined value, and is equal to or greater than a second predetermined value smaller than the first predetermined value, and the ratio X2 / Y2 of the number Y2 (<Y1) of a plurality of images constituting a second partial range that is a part of the predetermined range or the partial range composed of the Y1 images, and the number X2 of images among the Y2 images evaluated by the individual evaluation means to include the feature is equal to or greater than a third predetermined value, the comprehensive evaluation means is configured to evaluate that the object includes a specific part to be detected.

[0021] Also, in one aspect of the present disclosure, the comprehensive evaluation means may be configured to evaluate that the object includes a specific part to be detected when there is an image A evaluated by the individual evaluation means to include the feature, and when the feature is evaluated to be included in two or more of the plurality of images located around the image A by the individual evaluation means.

[0022] Also, in one aspect of the present disclosure, the comprehensive evaluation means is configured such that there is an image A evaluated by the individual evaluation means to include the feature, there is an image B located around the image A and evaluated by the individual evaluation means to include the feature, there is an image C located around the image B but not around the image A and evaluated by the individual evaluation means not to include the feature, there is an image D located around the image C but not around the image A, the image B, and evaluated by the individual evaluation means to include the feature, there is an image E located around the image D but not around the image A, the image B, and the image C and evaluated by the individual evaluation means to include the feature, and in this case, the comprehensive evaluation means is configured to evaluate that the object includes a specific part to be detected.

[0023] Note that an image being "located around" another image refers to a positional relationship in which no other image exists between the one image and the other image, typically meaning that the vertices or edges of the one image and the other image face each other or are shared, and that an image not being "located around" the other image refers to a positional relationship in which there is yet another image between the one image and the other image.

[0024] In addition, in one aspect of the present disclosure, the unique portion is a defect of the object.

[0025] A computer program according to the present disclosure includes instructions to cause a computer to execute the following: acquiring a plurality of images of an object, each of which is an image of a different portion of a predetermined range of the object; applying a predetermined image processing method acquired using machine learning to each of the plurality of images to perform an individual evaluation of whether or not each of the plurality of images includes a feature of the object to be detected; and performing a comprehensive evaluation of whether or not the object includes a unique portion of the object to be detected based on whether or not the feature is included in two or more of the plurality of images. Here, the comprehensive evaluation evaluates that the object includes the unique portion of the object to be detected when a ratio X1 / Y1 of the number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to the number X1 of the images evaluated in the individual evaluation as including the feature is equal to or greater than a first predetermined value.

[0026] The computer program may be stored in a non-transitory storage medium. The non-transitory storage medium includes a non-volatile semiconductor storage element.

[0027] By causing a computer to execute such a computer program, it is possible to achieve the same effects as the image processing method described above.

[0028] The image processing system according to the present disclosure includes the image processing device described above and a server. The server stores a first image data group including different types of features and a second image data group including no features as training data for machine learning. The first image data may be assigned the same label. The second image data group is assigned a label different from the label assigned to the first image data. The image processing system according to the present disclosure includes an aspect in which the execution instructions included in the computer program according to the present disclosure described above are executed in cooperation with a plurality of arithmetic devices including a processor and a memory and connected to each other wirelessly or by wire so as to be able to communicate with each other.

[0029] According to such an image processing system, it is possible to achieve the same effect as the above-mentioned image processing device. It becomes possible to achieve results. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a functional block diagram of an image processing system according to this embodiment. [Diagram 2] FIG. 2 is a block diagram showing the hardware configuration of the image processing system according to this embodiment. [Diagram 3] FIG. 3 is a flowchart of the image processing method according to this embodiment. [Figure 4] FIG. 4 is a diagram showing a schematic diagram of the results of individual evaluations of divided images of partial ranges. [Diagram 5] FIG. 5 shows an example of a pattern of positional relationships between a plurality of divided images used in the overall evaluation of the image processing method according to the modified example of this embodiment. [Figure 6] FIG. 6 shows an example of a pattern of positional relationships between a plurality of divided images used in the overall evaluation of the image processing method according to the modified example of this embodiment. [Figure 7] FIG. 7 is a diagram showing the relationship between the size of a unique portion and a divided image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to the embodiments.

[0032] 1 is a functional block diagram of an image processing system 10 according to this embodiment. This image processing system 10 is an image processing system for determining whether or not a foreign object is present inside a plastic bottle (an example of an "object") that contains liquid. However, as will be described later, the present invention can be applied to various objects and various unique parts.

[0033] This image processing system 10 includes an image processing device 20 having an imaging unit 12, an image division unit 14, an individual evaluation unit 16, and a comprehensive evaluation unit 18. Furthermore, the image processing system 10 further includes a server device connected to the image processing device 20 via a network N. The server device includes a database DB1 for storing images of normal samples and a database DB2 for storing images of abnormal samples. In this embodiment, a normal sample is a sample that does not include a unique part, in other words, a sample in which the object has no abnormality and corresponds to a non-defective product. An abnormal sample is a sample that includes a unique part, in other words, a sample in which the object has an abnormality and corresponds to a defective product.

[0034] The imaging unit 12 captures an image of a part or the whole of an object and acquires an image (an example of an "original image"). The imaging unit 12 can be configured, for example, from an image sensor such as a line sensor or an area sensor. For example, when the object has a shape that is rotationally symmetrical with respect to a central axis (such as a container having an internal space for accommodating contents), an image corresponding to the entire side surface of the object (an example of a "predetermined range") can be acquired by capturing an image of the side surface of the object with a line sensor while rotating the object around the central axis. Note that the object is not limited to a cylindrical shape. In the case of this embodiment, the imaging unit 12 is configured from a line sensor, and acquires grayscale data having a luminance value of 8 bits (256 gradations) for each pixel of 5000 pixels x 5000 pixels as the image of the object. Note that the imaging unit 12 may acquire color data such as RGB data as the image of the object. Also, instead of capturing an image of the object with the imaging unit 12 and acquiring an image, an image of the object may be acquired from a database or the like that stores images of the object.

[0035] The image division unit 14 (an example of an "image acquisition means") divides the image acquired by the imaging unit 12 into a plurality of images of the same size, and acquires a plurality of images capturing different parts of a predetermined range of the object (hereinafter, each divided image may be referred to as a divided image). The divided images are set to have a size suitable for an image processing method using machine learning. Here, the size of the divided images does not need to be set to include the size that the peculiar part to be detected may have, and may be set to be smaller than the size that the peculiar part may have. For example, when the size of the predetermined range is 5000 pixels x 5000 pixels and the size of the divided images is 75 pixels x 75 pixels, the image division unit 14 acquires approximately 4400 divided images. By setting the size of the divided images small so as to be suitable for machine learning in this way, it is possible to shorten the calculation time of machine learning compared to the case where the image size is set to include the peculiar part. Furthermore, it is possible to reduce the number of images required for learning, which has the great advantage of shortening the learning time and facilitating the collection of images of abnormal samples.

[0036] The individual evaluation unit 16 applies a predetermined image processing method acquired using machine learning to each of the multiple divided images, and performs an individual evaluation of whether each of the multiple divided images includes the features to be detected.

[0037] Here, the unique portion refers to a portion that has characteristics different from the surrounding area in terms of at least one of shape, brightness, and color, and the characteristic is the unique portion itself or a part of the unique portion. Therefore, the characteristic refers to a portion that has characteristics different from the surrounding area in terms of at least one of shape, brightness, and color.

[0038] The unique parts and features may be defects, or may be artificially set marks or patterns. The unique parts and features related to shape refer to parts that have shape characteristics (e.g., volume, area, height, particle size, aspect ratio, circularity, contour shape, etc.) that differ from those of the surrounding areas. Similarly, the unique parts and features related to luminance refer to parts that have characteristics that differ from those of the surrounding areas in luminance characteristics such as brightness and histogram, and the unique parts and features related to color refer to parts that have characteristics that differ from those of the surrounding areas in color characteristics such as spectrum, central wavelength, maximum wavelength, etc.

[0039] The unique parts and features in this embodiment are dust, human hair, air bubbles, and dirt on the container that may be contained in the liquid inside the plastic bottle.

[0040] The individual evaluation unit 16 of this embodiment includes a trained model composed of a convolutional neural network. The convolutional neural network includes an input layer for inputting data of a divided image, an output layer for outputting the presence or absence of a feature, and an intermediate layer for connecting the two layers. The intermediate layer includes a combination of a convolutional layer that performs a convolution operation to calculate the sum of products of overlapping parts as features by sliding a filter to generate a map having features, and a pooling layer that extracts the maximum value of the two-dimensional array data output from the convolutional layer, and a combination layer that combines the internal parameters as weighting coefficients to provide an evaluation result to the output layer. In addition to a convolutional neural network, the neural network can be configured from a recurrent neural network (RNN), an Elman network, a Jordan network, an echo state network (ESN), a long short term memory network (LSTM), a bi-directional RNN (BRNN), and the like. The output layer outputs probability information (sometimes called "classification probability") indicating the possibility that the target divided image has a feature. However, the output from the output layer may be a binary result of whether or not a feature is present, or may be an output for each type of feature.

[0041] The server device includes a database DB1 that stores sample images of normal objects and a database DB2 that stores sample images of abnormal objects. The database DB1 stores data of divided images that do not include a unique part. In this embodiment, the database DB1 stores data of divided images that does not include a unique part or features, which is 75 pixels x 75 pixels, acquired by capturing an image of the side of a plastic bottle, which is an object.

[0042] The database DB2 stores a second image data group including a unique portion or a feature. In this embodiment, the database DB2 stores data of a divided image including at least one of a unique portion or a feature of 75 pixels x 75 pixels acquired by imaging the side of a plastic bottle, which is an object. In this embodiment, the unique portion and the feature are, for example, human hair, dust, stains on a container, or parts thereof. Therefore, the database DB2 stores data of a plurality of divided images including human hair, data of a plurality of divided images including dust, data of a plurality of divided images including stains on a container, and the like.

[0043] The unique parts and features differ depending on the detection target. For example, when air bubbles are the detection target, data of the divided image including the air bubbles is stored in database DB2 as second image data including the unique parts and features. However, when a plastic bottle containing a carbonated drink is the detection target, air bubbles are not the detection target, so data of the divided image including the air bubbles is stored in database DB1 as first image data including the unique parts or features.

[0044] The individual evaluation unit 16 acquires a trained model including internal parameters for evaluating the presence or absence of features by learning the data of the segmented images stored in the databases DB1 and DB2 as training data. The internal parameters are, for example, bias values ​​in the convolution layer, weighting coefficients in the connection layer, etc. The internal parameters vary by adding training data.

[0045] The trained model in this embodiment is a trained model configured to output information regarding the presence or absence of a feature included in an input image of a predetermined size. The trained model in this embodiment is generated by machine learning using training data in which each of the data of a plurality of segmented images is labeled with the presence or absence of a feature. The trained model may output information regarding the presence or absence of a feature included in the input image and / or information regarding the type of feature. Such a trained model is generated by machine learning using training data in which each of the data of a plurality of segmented images is labeled with the presence or absence of a feature and / or the type of feature.

[0046] The individual evaluation unit 16 applies the trained model obtained using machine learning as described above to each of the divided images, and individually evaluates whether or not each of the 1,000 or more divided images contains features, and obtains probability information indicating the possibility of whether or not a feature exists.

[0047] The overall evaluation unit 18 evaluates the presence or absence of a unique portion based on two or more divided images that have been evaluated by the individual evaluation unit 16 as possibly including a feature among the multiple divided images. More specifically, the overall evaluation unit 18 acquires the number Y1 of divided images within a predetermined range and the number X1 of divided images that have been evaluated by the individual evaluation unit 16 as possibly including a feature, and evaluates that the object includes a unique portion if the ratio X1 / Y1 is equal to or greater than a predetermined threshold value.

[0048] FIG. 2 is a block diagram showing a hardware configuration for realizing the image processing device 20. As described above, the imaging unit 12 captures an image of an object to obtain an image of a predetermined range of the object, and is composed of, for example, a line camera. The processor 22 executes a computer program (including a trained model for evaluating the presence or absence of a singular part) stored in the storage unit 24 to perform each arithmetic processing shown in this embodiment. Therefore, the processor 22 and the storage unit 24 cooperate to function as the image division unit 14, the individual evaluation unit 16, and the overall evaluation unit 18. The processor 22 is composed of, for example, an ASIC (Application Specific Integrated Circuit) having multiple arithmetic cores, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a quantum computer, or the like. The storage unit 24 stores each instruction including a computer program (including a trained model for evaluating the presence or absence of a feature) for executing each arithmetic processing shown in this embodiment and other information. The storage unit 24 is composed of a non-volatile semiconductor storage element (non-transient storage element) such as a NAND flash memory, FeRAM, or MRAM that can electrically record and read information, or a magnetic storage element such as a HDD (Hard Disk Drive). The RAM 26 is composed of a volatile semiconductor storage element such as an SRAM (Static Random Access Memory) and a DRAM (Dynamic Random Access Memory) for temporarily storing data and other information used for each arithmetic processing and the like shown in this embodiment. The display unit 28 has a display for displaying the results of the arithmetic operations performed by the processor 22. The arithmetic operations include the evaluation results performed by the individual evaluation unit 16 and the overall evaluation unit 18. The communication I / F unit 30 is connected to the databases DB1 and DB2 via the network N, and can receive information from the databases DB1 and DB2. The input unit 32 has an input device such as a keyboard for an inspector to input information to the image processing system 10. These components are connected to each other via a bus so that they can transmit and receive data.However, some of the components may be provided at a remote location via the network N, or may be integrated with other components. Some of the functions executed by the processor 22 may be executed by other components, such as the imaging unit 12. The server device may also have the hardware configuration shown in FIG. 2. The databases DB1 and DB2 may be configured as information processing devices integrated with the image processing device 20. The information processing devices that execute the image segmentation unit 14, the individual evaluation unit 16, and the overall evaluation unit 18 of the image processing device 20 may be provided separately from the image processing device 20, and these may be electrically connected to each other via the network N to configure the image processing system 10.

[0049] A description will now be given of an image processing method using the above-described image processing system 10. Fig. 3 is a flowchart of the image processing method shown in this embodiment.

[0050] First, the imaging unit 12 captures an image of an object to obtain a base image (step S1). Specifically, while rotating the object, which is a plastic bottle, around its central axis, the imaging unit 12 captures an image of the side of the plastic bottle using a line camera to capture a base image corresponding to a predetermined range. In this embodiment, the imaging unit 12 captures an image of 5000 pixels x 5000 pixels as a base image corresponding to a predetermined range. Also, in this embodiment, the imaging unit 12 captures grayscale data having a luminance value of 8 bits (256 gradations) for each pixel of the 5000 pixels x 5000 pixels as the base image. Here, the plastic bottle container is translucent. Therefore, the imaging unit 12 captures an image of the plastic bottle container and the liquid contained therein.

[0051] Next, the image dividing unit 14 divides the base image corresponding to the side surface of the plastic bottle acquired by the imaging unit 12 into divided images of 75 pixels×75 pixels, and obtains 1000 or more divided images (step S2).

[0052] Then, the individual evaluation unit 16 individually evaluates each divided image based on a trained model generated using machine learning, and evaluates whether or not each divided image includes a feature (step S3).

[0053] FIG. 4 is a diagram showing a schematic result of individual evaluation of divided images of a partial range that is a part of a predetermined range. The partial range in the figure includes 80 divided images arranged in 8 rows in the vertical direction of the paper and 10 columns in the horizontal direction of the paper. The numerical value shown in each divided image indicates probability information obtained by the individual evaluation unit 16, which indicates the possibility that each divided image has a feature. In this embodiment, the closer the numerical value is to 0, the higher the possibility that the divided image has a feature, and the closer the numerical value is to 1, the lower the possibility that the divided image has a feature. For example, the probability information of the divided image DI1 is 0.041, so the probability that the divided image DI1 does not have a feature is 4.1% (the probability that the divided image has a feature is 95.9%). The probability information of the divided image DI2 is 0.444, so the probability that the divided image DI2 does not have a feature is 44.4% (the probability that the divided image has a feature is 55.6%).

[0054] The image processing device 20 may be configured to display probability information of each divided image on the display unit 28. For example, the probability information may be classified by three thresholds, and each divided image may be configured to be displayed in a different color, thereby making it easy to see how divided images evaluated to have a feature are distributed within a predetermined range or a partial range. The threshold value that is the standard for evaluating whether an image has a feature can be set appropriately depending on the detection target. For example, the individual evaluation unit 16 may be configured to evaluate an image as not having a feature when the probability of not having a feature is 90% or more.

[0055] There are various reasons why an image is evaluated as having a feature. For example, an image may be evaluated as having a feature due to noise. Also, an image may be evaluated as having a feature due to dust, human hair, or the like. For example, if a human hair H (FIG. 4) is mixed in, an image may be evaluated as having a feature across multiple adjacent divided images, such as divided images DI3 to DI7.

[0056] Then, the overall evaluation unit 18 obtains the number Y1 of divided images within the specified range and the number X1 of divided images evaluated by the individual evaluation unit 16 as possibly containing features, and determines whether the ratio X1 / Y1 is greater than or equal to a specified threshold value (an example of a "first specified value") (step S4).

[0057] For example, if Y1 is 5000 and X1 is 10, X1 / Y1 is 0.002. If this value is equal to or greater than a predetermined threshold (YES), the object to be detected is evaluated to contain a unique portion (step S5). If it is less than the threshold (NO), the object to be detected is evaluated to not contain a unique portion (step S6).

[0058] If the detection object is evaluated to include a unique part, the display unit 28 outputs information to that effect, and prompts the inspector to perform a visual inspection (step S7). At this time, the display unit 28 is configured to display the position within a predetermined range of the divided image evaluated to include a unique part or feature. With this configuration, the inspector can easily identify the area evaluated to include a unique part or feature. This makes it possible to reduce the visual inspection time. Note that, if the detection object is evaluated to include a unique part, the image processing device 20 may output information to that effect to a controller (e.g., a programmable logic controller (PLC) or the like) that controls the controlled object of the manufacturing line, and the controller that receives the information may control the manufacturing line so as to discharge the object evaluated to include a unique part as a defective product or a candidate for a defective product outside the manufacturing line system.

[0059] Even if the detection object is evaluated as not containing a unique portion, a visual inspection may be further performed.

[0060] Furthermore, when the detection object is large, the image processing method according to the present embodiment including steps S1 to S7 may be repeatedly executed for different ranges. On the other hand, when the singular portion is a defect and the locations where the defect may occur are concentrated in a certain region, even if the detection object is large, the image processing method according to the present embodiment including steps S1 to S7 may be executed for that region, and the detection object may be inspected for another region by a different method (for example, visual inspection).

[0061] According to the image processing system 10 and the image processing method as described above, it is possible to set the size of the divided image small so as to be suitable for machine learning, and therefore it is possible to shorten the learning time and the calculation time of the machine learning compared to the case where the image size is set large so as to include the unique part. Furthermore, the inventors of the present application have focused on the fact that noise and the like that should not be evaluated as a unique part or feature rarely spans multiple divided images, while dust, human hair, and the like that should be evaluated as a unique part or feature are relatively often spread across multiple divided images, and have devised an image processing system 10 and an image processing method that evaluates a unique part based on the fact that multiple divided images have characteristics, rather than evaluating that a single divided image has a characteristic. As a result, it is possible to reduce the possibility of overdetection due to a mistaken recognition that a unique part exists due to noise, etc., and to suppress overlooking dust, human hair, and the like that should be evaluated as a unique part or feature, thereby improving the detection accuracy.

[0062] In other words, by setting the size of the divided images so that the unique portion spans multiple divided images, it can be said that an image processing system 10 and an image processing method that achieve both high detection accuracy and high processing speed are provided.

[0063] In the case of objects that are mass-produced, even the presence of a unique part in just one of them can cause a major problem. However, by using the image processing system 10 and image processing method of this embodiment and generating divided images that are intentionally smaller than the unique part, it is possible to reduce the rate at which the unique part is overlooked.

[0064] Furthermore, the image processing method of this embodiment employs a method that uses a model obtained by machine learning that can autonomously generate estimation rules for input data based on teacher data (training data), and by using a divided image with a small number of pixels as the teacher data, it is possible to shorten the learning time.

[0065] In particular, assuming that the evaluation method of the present embodiment is implemented in a manufacturing line and used for product inspection, it is preferable that the time required to generate a trained model is short in order to flexibly respond to changes in the lot or model of the product to be manufactured. As described above, by suppressing an increase in the learning time, an evaluation method suitable for implementation in a manufacturing line can be realized.

[0066] In order to further enhance the above-mentioned effects, it is preferable to use, as the first method, a rule-based image processing method which has few elements that need to be changed in response to changes in the lot or model of the product to be manufactured and which is also applicable to image processing of large amounts of data.

[0067] In addition, multiple objects may be imaged simultaneously by moving the objects so as to pass through the area imaged by the imaging unit 12. This allows multiple objects to be efficiently evaluated. In this case, the image processing system 10 may be configured to be able to evaluate the entire side by moving the objects while rotating them around the central axis. In addition, while imaging a first object using the imaging unit 12, an evaluation of the presence or absence of a feature or a unique part may be performed on a second object using the individual evaluation unit 16 or the overall evaluation unit 18 at the same time.

[0068] [First Modification] The following describes a first modified example of the first embodiment. Note that, in the modified example described below, explanations of points that a person skilled in the art would reasonably understand to have the same or similar configuration as the first embodiment or other modified examples will be omitted or simplified, and the description will focus on the differences.

[0069] The image processing device and image processing method according to the first embodiment performed an overall evaluation for a specified range based on the ratio of the number Y1 of pixels constituting the specified range to the number X1 of images that were evaluated as containing features in the individual evaluation of the Y1 images.

[0070] In the image processing method of the first variant, a comprehensive evaluation is performed not on a specified range, but on a subrange that is a part of the specified range, based on the ratio of the number Y1 of pixels constituting the subrange to the number X1 of images that are evaluated as containing features in the individual evaluation among the Y1 images.

[0071] The size of the partial range can be set appropriately. For example, the partial range is set so as to include the size of the unique part assumed to be the detection target. For example, when the size of a typical unique part is often included in 16 divided images (a partial range consisting of 16 divided images of 4 rows and 4 columns), the partial range may be set as a part consisting of the same 16 divided images of 4 rows and 4 columns, and when the number of pixels evaluated as including a feature in the individual evaluation is 4 or more, that is, when Y1 is 16, X1 is 4, and X1 / Y1 is 0.25 or more, the comprehensive evaluation unit 18 may be configured to evaluate that the unique part is included.

[0072] [Second modified example] A second modified example of the first embodiment will be described below. The image processing method according to the second modification performs an evaluation by combining the comprehensive evaluation described in the first embodiment and the comprehensive evaluation described in the first modification.

[0073] Specifically, in step S4, even when X1 / Y1 is less than the threshold value (NO), if it is equal to or greater than a second threshold value (an example of the "second predetermined value") that is smaller than this threshold value, the ratio X2 / Y2 of the number Y2 (<Y1) of divided images that constitute a second partial range, which is a part of a predetermined range composed of Y1 divided images, and the number X2 of images evaluated as including features in the individual evaluation among the Y2 images is equal to or greater than a third threshold value (an example of the "third predetermined value"), the comprehensive evaluation unit 18 may be configured to evaluate that the specific part is included.

[0074] By adopting such a configuration, even when X1 within the predetermined range is small, it becomes possible to capture the case where the divided images evaluated as including features in some regions are concentrated, so that it becomes possible to suppress the overlooking rate of the specific part.

[0075] [Third Modification Example] Hereinafter, a third modification example of the first embodiment will be described. The image processing method according to the third modification example performs a comprehensive evaluation based on the pattern of the positional relationship of a plurality of divided images evaluated as including features in the individual evaluation.

[0076] FIG. 5 shows an example of the pattern of the positional relationship of a plurality of divided images. In the figure, the black cells indicate the divided images evaluated as including features in the individual evaluation (step S3) of the divided images, and the white cells indicate the divided images evaluated as not including features.

[0077] As shown in the figure, Fig. 5(A) and Fig. 5(B) show a pattern in which, among the eight divided images located around a central divided image (an example of "Image A") among the divided images evaluated to include features in a partial range consisting of nine divided images in three rows and three columns, two divided images evaluated to include features are present. Fig. 5(A) and Fig. 5(B) both show patterns in which there is one divided image evaluated to include features in each column (vertical direction) and two divided images adjacent in a diagonal direction, Fig. 5(A) shows a pattern in which three divided images are connected diagonally, and Fig. 5(B) shows a pattern in which one divided image is connected diagonally with respect to the central divided image, and another divided image is connected in the vertical or horizontal direction.

[0078] The pattern shown in the figure may be rotated by 90 degrees, or may be symmetrical up and down or left and right on the page.

[0079] If it is known in advance that there is a high possibility that the unique portion extends diagonally for the divided images arranged in a matrix, as shown in the same figure, the overall evaluation unit 18 may be configured to evaluate the divided image as including a unique portion if, in the individual evaluation, two or more of the eight divided images positioned around a central divided image are evaluated as including a feature, and at least one divided image is positioned diagonally relative to the divided image evaluated as including a feature.

[0080] In addition, when the other six divided images are individually evaluated as not containing features, the overall evaluation unit 18 may be configured to evaluate them as including a unique portion, and even when the other six divided images are individually evaluated as containing features, the overall evaluation unit 18 may be configured to evaluate them as including a unique portion.

[0081] When the unique portion is linear and has a width narrower than the size of the divided image, it is preferable to perform the comprehensive evaluation as in the former. On the other hand, even if the unique portion is linear, when the width is wider than the size of the divided image, it is also possible to perform the comprehensive evaluation as in the latter. In other words, it is preferable to set a pattern of positional relationships of the multiple divided images that are evaluated as including features in the individual evaluations according to the expected shape of the unique portion.

[0082] Such a comprehensive evaluation based on the pattern of the positional relationships of multiple divided images that have been evaluated as containing features in the individual evaluations may be performed in step S4 of the first embodiment, and the detected object may be evaluated as including a singular part when X1 / Y1 is equal to or greater than a predetermined threshold value and a pattern such as that shown in FIG. 5(A) or FIG. 5(B) is detected.

[0083] Alternatively, instead of performing step S4 of checking whether X1 / Y1 is equal to or greater than a predetermined threshold, a comprehensive evaluation may be performed after the individual evaluation, in which the detection object is evaluated as including a unique part based on the positional relationship pattern of the multiple divided images. An image processing method in such a case will be described later.

[0084] [Fourth Variation] Hereinafter, a fourth modified example of the first embodiment will be described. The image processing method according to the fourth modification is common to the third modification in that a comprehensive evaluation is performed based on the pattern of positional relationships of a plurality of segmented images that have been evaluated as including features in the individual evaluations.

[0085] The inventors of the present application have noted that even in cases other than the case described in the third modified example (when, among the partial range consisting of nine divided images in three rows and three columns, a central divided image (an example of "image A") among the divided images evaluated to include a feature, two divided images evaluated to include a feature are present among the eight divided images located around the central divided image (an example of "image A") among the divided images evaluated to include a feature), a singular part may exist. Such a case is a pattern in which two divided images evaluated to include a feature in the individual evaluation (an example of "image A" and "image B") are adjacent to each other, two other divided images evaluated to include a feature in the individual evaluation (an example of "image D" and "image E") are adjacent to each other, and images A and B and images C and D are not adjacent to each other, and are separated by one divided image evaluated to include no feature in the individual evaluation (an example of "image C").

[0086] In other words, when there is image A that is evaluated to include a feature in the individual evaluation, and is located around image A, there is image B that is evaluated to include the feature in the individual evaluation, and image C that is not located around image A but around image B, there is image C that is evaluated to not include the feature in the individual evaluation, there is image D that is not located around images A and B but around image C, and there is image D that is evaluated to include the feature in the individual evaluation, and there is image E that is not located around images A, B, and C but is located around image D, and there is image E that is evaluated to include the feature in the individual evaluation, the overall evaluation unit 18 is configured to evaluate that the object includes the unique portion to be detected.

[0087] Such a pattern is shown in Fig. 6. Note that a pattern obtained by rotating the pattern shown in the figure by 90 degrees and a pattern obtained by making this pattern symmetrical up and down or left and right on the page are omitted. In these figures, black cells indicate segmented images that are evaluated as including features in the individual evaluation of the segmented images (step S3), and white cells indicate segmented images that are evaluated as not including features.

[0088] 6A shows a pattern that partially includes a pattern in which images B, C, and D are arranged in a straight line in the row or column direction. Image A may be any one of images A1 to A3. Image E may be any one of images E1 to E3.

[0089] 6B shows a pattern including a pattern in which, of images B, C, and D, images B and C are arranged in a straight line in the row or column direction, and image D is arranged diagonally relative to image C. Image A may be any one of images A1 to A3. Image E may be any one of images E1 to E5.

[0090] 6C shows a pattern including a pattern in which, among images B, C, and D, images B and D are arranged at the same position in the row or column direction, and image C is arranged diagonally with respect to images B and D. Image A may be any one of images A1 to A3. Image E may be any one of images E1 to E4.

[0091] 6D shows a pattern including a pattern in which images B, C, and D are arranged in a straight line in a diagonal direction. Image A may be any one of images A1 to A5. Image E may be any one of images E1 to E5.

[0092] Note that the pattern shown in FIG. 6(E) corresponds to a pattern that is obtained by rotating FIG. 6(B) by 90 degrees and then inverting it symmetrically, and therefore is substantially identical to the pattern in FIG. 6(B), and therefore a description thereof will be omitted.

[0093] Such an overall evaluation based on the pattern of positional relationships of multiple divided images that have been evaluated as containing features in the individual evaluations may be performed together with or independently of the evaluation that X1 / Y1 is greater than or equal to a predetermined threshold, which is performed in step S4 in the first embodiment.

[0094] According to the image processing method and image processing device described above, even if a feature that should be contained in the divided pixel corresponding to image C is overlooked, it is possible to detect a singular part based on the surrounding divided pixels, thereby preventing the singular part from being overlooked and improving the detection accuracy.

[0095] [How to generate training data] A method for generating the training data to be stored in the databases DB1 and DB2 will be described below.

[0096] Conventionally, when performing sensory testing on objects with complex structures that generate noise factors, images were captured while masking areas that generate noise factors, such as boundaries between parts, in order to prevent overdetection.

[0097] However, when the unique portion is divided by the mask, the detection performance of the unique portion is reduced. Therefore, in this embodiment, when an object composed of multiple parts is imaged to obtain a base image in order to generate teacher data, a predetermined range of the object is imaged so as to include the multiple parts and their boundaries without using a mask. Here, the multiple parts include a part composed of a flat surface and a part composed of a curved surface connected to the flat surface, or a part composed of a certain material and another part connected to the part and composed of a different material.

[0098] Then, the base image is divided into a plurality of divided images to generate teacher data. The divided images that do not include a singular part are stored in the database DB1. Here, these divided images constitute a predetermined range including a plurality of parts and their boundaries, and the divided images that become teacher data are generated without performing processing for removing noise such as filtering. For this reason, these divided images may contain noise.

[0099] Thereafter, when imaging the object to obtain a base image in step S1, the imaging unit 12 images a predetermined range of the object so as to include a plurality of parts and their boundaries, and a divided image is obtained in step S2 based on this.

[0100] Then, in step S3, each divided image is individually evaluated based on a trained model generated using machine learning to evaluate whether or not each divided image contains features. Since the trained model uses trained data of divided images constituting a specified range including multiple parts and their boundaries, it is possible to accurately individually evaluate whether or not each divided image constituting this specified range contains features.

[0101] The database DB2 stores divided images including unique parts or features. These divided images also constitute a predetermined range including a plurality of parts and their boundaries. FIG. 7 shows an example of unique parts H2 to H4 and an example of a divided image generated based on the unique parts. As shown in this figure, the unique part may be larger than the size of the divided image (in other words, the size of the divided image is set so as to be smaller than some of the unique parts). Even if the unique part is smaller than the size of the divided image, the unique part may span multiple divided images. The overall evaluation unit 18 shown in this embodiment performs an overall evaluation based on two or more divided images, making it possible to improve the detection accuracy of such unique parts.

[0102] [Scope of application] The image processing system and image processing device according to the present disclosure can be applied to evaluate various objects. For example, the image processing system and image processing device can be used to detect foreign matter, scratches, wrinkles, etc. that may be contained in a transparent film as a specific part in a manufacturing process of a transparent film made of polyvinylidene chloride (PVDC).

[0103] Moreover, the present invention can be used to detect foreign matter, scratches, and the like that may be contained in a polymer film as a specific portion in a manufacturing process of the polymer film for use in a flat panel display or the like.

[0104] Furthermore, the present invention can be used to detect cracks, scratches, and the like that may be present in glass as unique parts during the manufacturing process of glass products, etc. In this case, the glass product is illuminated and an image is taken, and the unique parts can be detected by utilizing the fact that the brightness of the unique parts is higher than that of other parts.

[0105] In addition, the present invention can be used to detect deformation or bubbles on the membrane surface as unique parts in the manufacturing process of separation membranes for chromatography. In this case, the separation membrane is illuminated and imaged, and the unique parts can be detected by utilizing the fact that the luminance of the unique parts is lower or higher than that of other parts.

[0106] In addition, the present disclosure can be modified in various ways without departing from the spirit of the present disclosure. For example, some components in one embodiment can be added to other embodiments within the scope of ordinary creativity of a person skilled in the art. Also, some components in one embodiment can be replaced with corresponding components in other embodiments.

[0107] For example, the image processing method according to the present disclosure includes: Obtaining a plurality of images of an object, each image capturing a different portion of a predetermined range of the object; applying a predetermined image processing method acquired using machine learning to each of the plurality of images to individually evaluate whether each of the plurality of images includes a feature to be detected; performing a comprehensive evaluation of whether the object includes a specific portion to be detected based on whether the feature is included in two or more images of the plurality of images; Including, The comprehensive evaluation evaluates whether or not the object includes a singular portion to be detected based on a positional relationship of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range in the individual evaluation. It may be an image processing method.

[0108] For example, the image processing device according to the present disclosure includes: A means for acquiring a plurality of images of an object, each of which captures a different portion of a predetermined range of the object; a means for applying a predetermined image processing method acquired by machine learning to each of the plurality of images to individually evaluate whether each of the plurality of images includes a feature to be detected; a means for performing a comprehensive evaluation as to whether or not the object includes a specific portion to be detected based on whether or not the feature is included in two or more images of the plurality of images; Including, The means for performing the comprehensive evaluation is configured to evaluate, in the individual evaluation, whether or not the object includes a unique portion to be detected based on a positional relationship of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range. It may be an image processing device.

[0109] In such an image processing method and image processing device, Each of the plurality of images is an image of a rectangular area having a long side and a short side, The overall evaluation may be configured to evaluate the object as including a peculiar portion to be detected if at least two of the images evaluated as including the feature in the individual evaluation are adjacent in a diagonal direction. [Explanation of symbols]

[0110] 10 Image Processing System 12 Imaging unit 14 Image division section 16 Individual Evaluation Section 18. Overall Evaluation Department 18. Overall Evaluation Department 20 Image processing device 20 Image processing device 22 processors 24 Memory section 28 Display section 32 Input section DB1 database DB2 database

Claims

1. acquiring a plurality of images of an object, each of the images including a different portion of a predetermined range of the object; applying a predetermined image processing method acquired using machine learning to each of the plurality of images to individually evaluate whether each of the plurality of images includes a feature to be detected; performing a comprehensive evaluation as to whether the object includes a specific portion to be detected based on whether the feature is included in two or more images of the plurality of images; Including, the comprehensive evaluation is performed by evaluating that the object includes a unique portion to be detected when a ratio X1 / Y1 of a number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to a number X1 of the Y1 images that are evaluated in the individual evaluation as including the feature is equal to or greater than a first predetermined value; Furthermore, the comprehensive evaluation evaluates that the object includes a unique portion to be detected when the ratio X1 / Y1 is less than the first predetermined value and is equal to or greater than a second predetermined value smaller than the first predetermined value, and when a ratio X2 / Y2 of a number Y2 (<Y1) of a plurality of images constituting the predetermined range or a second partial range that is a part of the partial range constituted by the Y1 images and a number X2 of images evaluated as including the feature in the individual evaluation among the Y2 images is equal to or greater than a third predetermined value. Image processing methods.

2. Dividing an original image obtained by capturing an image of a predetermined range of the object, and generating the plurality of images each having the same image size; The image processing method of claim 1 further comprising:

3. The overall evaluation is as follows: There is an image A that is evaluated as including the feature in the individual evaluation, and In the individual evaluation, when it is evaluated that two or more of the plurality of images located around the image A include the feature, Evaluating that the object includes a specific portion to be detected; 3. The image processing method according to claim 1.

4. Obtaining a plurality of images of an object, each image including a different portion of a predetermined range of the object; applying a predetermined image processing method acquired using machine learning to each of the plurality of images to individually evaluate whether each of the plurality of images includes a feature to be detected; performing a comprehensive evaluation as to whether the object includes a specific portion to be detected based on whether the feature is included in two or more images of the plurality of images; Including, The comprehensive evaluation is performed in the following case: an image A evaluated to include the feature in the individual evaluation exists, and is located around the image A; an image B evaluated to include the feature in the individual evaluation exists, and is not located around the image A but is located around the image B; an image C evaluated to not include the feature in the individual evaluation exists, and is not located around the images A and B but is located around the image C; an image D evaluated to include the feature in the individual evaluation exists, and an image E evaluated to include the feature in the individual evaluation exists, and is not located around the images A, B, and C but is located around the image D; Image processing methods.

5. The image processing method according to claim 1 , wherein the unique portion is a defect of the object.

6. an image acquisition means for acquiring a plurality of images of different parts of a predetermined range of an object; an individual evaluation means for applying a predetermined image processing method acquired by machine learning to each of the plurality of images to perform an individual evaluation of whether each of the plurality of images includes a feature to be detected; a comprehensive evaluation means for performing a comprehensive evaluation as to whether or not the object includes a specific portion to be detected, based on whether or not the feature is included in two or more images among the plurality of images; Equipped with the comprehensive evaluation means is configured to evaluate that the object includes a unique portion to be detected when a ratio X1 / Y1 of a number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to a number X1 of the Y1 images evaluated by the individual evaluation means as including the feature is equal to or greater than a first predetermined value; Furthermore, the comprehensive evaluation means is configured to evaluate that the object includes a unique portion to be detected when the ratio X1 / Y1 is less than the first predetermined value and is equal to or greater than a second predetermined value smaller than the first predetermined value, and when a ratio X2 / Y2 between a number Y2 (<Y1) of a plurality of images constituting the predetermined range or a second partial range that is a part of the partial range constituted by the Y1 images and a number X2 of images evaluated by the individual evaluation means to include the feature among the Y2 images is equal to or greater than a third predetermined value. Image processing device.

7. a means for dividing an original image obtained by capturing an image of a predetermined range of the object, and generating the plurality of images each having the same image size; The image processing apparatus according to claim 6 , further comprising:

8. The comprehensive evaluation means includes: There is an image A that has been evaluated by the individual evaluation means as including the feature, and When the individual evaluation means evaluates that two or more of the plurality of images located around the image A include the feature, The object is configured to evaluate that it includes a specific portion to be detected; 8. The image processing device according to claim 6 or 7.

9. An image acquisition means for acquiring a plurality of images capturing different parts of a predetermined range of an object; an individual evaluation means for applying a predetermined image processing method acquired by machine learning to each of the plurality of images to perform an individual evaluation of whether each of the plurality of images includes a feature to be detected; a comprehensive evaluation means for performing a comprehensive evaluation as to whether or not the object includes a specific portion to be detected, based on whether or not the feature is included in two or more images among the plurality of images; Equipped with The comprehensive evaluation means is configured to evaluate that the object includes a unique portion to be detected when an image A evaluated by the individual evaluation means to include the feature exists, an image B evaluated by the individual evaluation means to include the feature exists, an image C not located around the image A but located around the image B, an image D evaluated by the individual evaluation means to include the feature exists, an image D not located around the images A and B but located around the image C, an image D evaluated by the individual evaluation means to include the feature exists, and an image E evaluated by the individual evaluation means to include the feature exists, an image E not located around the images A, B, and C but located around the image D, Image processing device.

10. The image processing device according to claim 6 , wherein the unique portion is a defect of the object.

Images