Support device and method

Abstract

To provide a support device and a support method, capable of stably determining various defect types generated in a sheet product by using AI processing.SOLUTION: In an inspection support system 1, an inspection support device 10 includes a feature amount classification processing unit 20 that calculates a feature amount of an abnormality generated in an object 2 to be inspected from a captured image, classifies the calculated feature amount into a plurality of classes, and selects one or more classes to be processed to determine a type of the abnormality from among the plurality of classes, by using a learning determination unit, and a single class AI processing unit 30 including for each type of the abnormality a learning discriminator that has learned a determination reference to determine whether the abnormality to be determined is a type as a right answer or a type other than the type as the right answer. The single class AI processing unit 30 determines the type of the abnormality for the class to be processed, selected by the feature amount classification processing unit 20, on the basis of individual determination results of single class AI processors 30a to 30n that have learned the determination reference for each type of the abnormality.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for detecting an abnormal portion of a sheet-like object to be inspected. [Background technology]

[0002] In production lines for manufacturing or processing sheet-like articles (hereinafter referred to as "sheet products" or simply "sheets"), inspection devices are used to detect abnormalities on the sheet (such as foreign matter, stains, wrinkles, etc.) by irradiating the sheet with visible light or ultraviolet light and capturing the transmitted or reflected light with a camera (see, for example, Patent Document 1).

[0003] Conventional inspection devices can detect abnormalities on a sheet, but cannot precisely determine the type of abnormality that is detected. For this reason, conventionally, sheets on which abnormalities are detected must be discarded, downgraded, or sent for detailed visual inspection. However, in reality, there are various types of abnormalities that can occur on sheets (hereinafter, abnormalities that occur on sheets are also referred to as "defects"), and some do not need to be deemed defective depending on the type of product, application, material, etc.

[0004] In recent years, with the advancement of information processing technology, attempts have been made to classify (multi-class classification) the types of defects detected on sheets using AI (e.g., machine learning, deep learning, etc.) and replace visual inspections that were previously performed manually to determine the shipping of sheet products. In AI that performs multi-class classification (hereinafter also referred to as "multi-class classification AI"), for example, a wide variety of defect types such as air bubbles, holes, insects, foreign bodies, wrinkles, and fibers are identified in images of defect locations based on previously learned defect images and correct defect types. However, since multi-class classification AI requires a set of data of "defect image + correct defect type" for learning the AI ​​for each defect type, it may not be able to identify defect types that are detected infrequently. For this reason, for example, an image of an insect or the like that has different characteristics from the learned image of an insect may be identified as a foreign body or a hole. Furthermore, there are technical challenges in the process of performing multi-class classification, such as the need for tuning to optimize the network structure and parameters in the algorithm. In addition, when the multi-class classification AI is used to identify defect types, the output results are unstable when the AI ​​is retrained using new learning data, so currently visual inspection is used to make the final shipping decision. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-8174 A (Patent No. 4950951 A) Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and has an object to provide a technique that can stably distinguish various defect types that occur in sheet products using AI processing. [Means for solving the problem]

[0007] One aspect of the disclosed technique for solving the above problem is to 1. A support device for supporting a judgment inspection that detects an abnormality occurring in an object to be inspected from a captured image of the object to be inspected and judges a type of the detected abnormality, comprising: A feature amount of an abnormality occurring in the inspection object is calculated from the captured image, and the calculated feature amount is classified into a plurality of classes. a classification processing unit for selecting one or more classes to be processed, the classes being classified according to the type of the object; a learning discrimination unit for each type of abnormality, the learning discriminator having a discrimination criterion for discriminating whether an abnormality to be discriminated is a type that is a correct answer or a type other than the type that is a correct answer based on set data combining an element image that induces the abnormality and a type of abnormality that is a correct answer for the element image, the learning and discriminating unit discriminates the type of abnormality for the processing target class selected by the classification processing unit based on an individual discrimination result of a learning discriminator in which discrimination criteria for each of the types of abnormality have been learned. It is characterized by:

[0008] As a result, the inspection support device 10, which is a support device, can select one or more feature amount patterns to which the single-class AI processing is applied as a pre-processing before the defect type is determined by the single-class AI processing. Since the feature amount related to the determination of the defect type of the abnormal portion detected in the sheet product can be reduced in advance, the processing load related to the determination using the single-class AI processing can be relatively reduced. In addition, the single-class AI processing unit 30, which is a learning and determining unit, can be provided with a plurality of single-class AI processors (30a to 30n) which are learning classifiers. In the single-class AI processor, it is only necessary to prepare a data set ("defect image + correct defect type") for determining a single defect type as learning data, and it is not necessary to accumulate a large amount of set data of "defect image + correct defect type" for all defect types in a database, as in an AI processor that performs multi-class classification. In addition, in the single-class AI processing, for example, it is sufficient to determine a single defect type from others, so there is no need to build a complex network structure as in the multi-class AI processing, and no need to perform advanced tuning of parameters. For this reason, the operator is not required to have advanced skills. Furthermore, since only a single defect type is identified, learning new set data does not affect the identification of other defect types, and stable identification results can be obtained even after re-learning. The inspection support device 10 can provide a technology that can stably identify various defect types that occur in sheet products using AI processing.

[0009] In one embodiment of the disclosed technology, the learning and discriminator may have a first learning discriminator having learned a discrimination criterion for discriminating a first abnormality and a second learning discriminator having learned a discrimination criterion for discriminating a second abnormality, and may process one of the plurality of classes, and when the discrimination result of the first learning discriminator discriminates the first abnormality and the discrimination result of the second learning discriminator discriminates a type other than the second abnormality, discriminate the abnormality type of the class as the first abnormality. By combining the discrimination results of the single-class AI processors, it is possible to improve discrimination accuracy for the defect types trained in each single-class AI.

[0010] In one embodiment of the disclosed technology, one of the multiple classes may be processed, and when the discrimination result of the first learning discriminator discriminates a type other than the first abnormality and the discrimination result of the second learning discriminator discriminates the second abnormality, the type of abnormality of the class may be discriminated as the second abnormality. Even in this embodiment, it is possible to combine the discrimination results of the single-class AI processors to improve discrimination accuracy for the defect types learned by each single-class AI.

[0011] In one embodiment of the disclosed technology, one of the multiple classes may be treated as a processing target, and when the discrimination result of the first learning discriminator discriminates a first abnormality and the discrimination result of the second learning discriminator discriminates a second abnormality, or when the discrimination result of the first learning discriminator discriminates a type other than the first abnormality and the discrimination result of the second learning discriminator discriminates a type other than the second abnormality, the discrimination of the abnormality type of the class may be handed over to a designated process. This makes it possible to combine the discrimination results of the single-class AI processors and exclude the feature amount of the defect location from the defect type discrimination targets. Discrimination by single-class AI processing The ability to combine and arbitrate results reduces the processing burden associated with defect type discrimination and enables online processing time to be shortened. By combining the discrimination results of single-class AI processors, it becomes possible to improve discrimination accuracy for the defect types trained in each single-class AI.

[0012] In one embodiment of the disclosed technology, the classification processing unit may further classify the feature amount of the class in which the type of abnormality is determined into a plurality of subclasses. This allows the feature amount of the defect type determined using the single-class AI processing to be further classified into subclasses, making it possible to rank the product type of the sheet product in which the defect type is determined by the single-class AI processing.

[0013] In addition, another aspect of the disclosed technology is A method executed by a computer of a support device for supporting a judgment inspection for detecting an abnormality occurring in an object to be inspected from a captured image of the object and determining a type of the detected abnormality, comprising: calculating a feature amount of an abnormality occurring in the inspection object from the captured image, classifying the calculated feature amount into a plurality of classes, and selecting a predetermined class from the plurality of classes; learning a discrimination criterion for discriminating whether an abnormality to be discriminated is the type that is determined as a correct answer or is other than the type that is determined as a correct answer, based on set data that combines an element image that induces the abnormality and a type of abnormality for which the element image is determined as a correct answer, for each type of abnormality; Discriminating the type of anomaly for the predetermined class based on an individual discrimination result discriminated according to the learned discrimination criterion for each type of anomaly; The present invention is characterized by comprising:

[0014] Even in this form, the inspection support device 10, which is a support device, can include a plurality of single-class AI processors (30a to 30n) that are learning classifiers in the single-class AI processing unit 30, which is a learning classifier. In the single-class AI processor, it is sufficient to prepare a data set ("defect image + correct defect type") for discriminating a single defect type as learning data, and it is not necessary to accumulate a large amount of set data of "defect image + correct defect type" for all defect types in a database as in an AI processor that performs multi-class classification. In addition, in the single-class AI processing, for example, it is sufficient to discriminate between a single defect type and others, so there is no need to build a complex network structure as in the multi-class AI processing, and advanced tuning of parameters is also not required. Therefore, an operator is not required to have advanced skills. Furthermore, since the defect type to be discriminated is single, learning new set data does not affect the discrimination of other defect types, and stable discrimination results can be obtained even after re-learning. The inspection support device 10 can provide a technology that can stably discriminate various defect types occurring in a sheet product using AI processing. Effect of the Invention

[0015] According to the present invention, a technique can be provided that can stably distinguish various types of defects occurring in a sheet product using AI processing. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a schematic configuration of an examination support system according to a first embodiment of the present invention. [Diagram 2] 1 is a diagram illustrating an example of a hardware configuration of an examination support device according to a first embodiment of the present invention. [Diagram 3] 11A to 11C are diagrams illustrating a defect type discrimination process using an AI that performs multi-class classification in a comparative example. [Figure 4] FIG. 2 is a diagram for explaining the discrimination process of a single-class AI processor according to the first embodiment of the present invention. [Diagram 5] 5 is a flowchart showing an example of a defect type determination process of the inspection support apparatus according to the first embodiment of the present invention. [Figure 6] FIG. 4 is a diagram for explaining arbitration of the determination results in the first embodiment of the present invention. [Figure 7] 10 is a flowchart showing an example of a defect type determination process of the inspection support apparatus according to the first modified example of the present invention. [Figure 8] 11 is a flowchart showing an example of a defect type determination process of the inspection support apparatus according to Modification 2 of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] [Application example] Hereinafter, application examples of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing a schematic configuration of an inspection support system 1 according to an application example of the present invention. The inspection support system 1 in this application example is a system that supports determining the type of abnormality of an abnormal portion detected from a captured image of a sheet-like object (hereinafter also referred to as a "sheet product" or simply as a "sheet"). The inspection support system 1 individually determines the type of cause (e.g., foreign matter contamination, stains, wrinkles, etc.) that may cause an abnormality (hereinafter also referred to as a "defect") that may occur in the sheet product using single-class AI processors (30a to 30n).

[0018] The inspection support system 1 according to the application example of the present invention includes an inspection support device 10 connected to a processing device 5 via a communication network N. The inspection support device 10 is a computer that supports the function of the processing device 5 connected to the communication network N in determining the type of anomaly (hereinafter, an anomaly detected from a sheet is also referred to as a "defect"). The inspection support device 10 includes a feature amount classification processing unit 20 and a single-class AI processing unit 30 in a control unit 11. The inspection support device 10 also includes a detection data DB (database) 12, a feature amount classification DB 13, and a learning data DB 14. The detection data DB 12 stores captured images that are transmitted from the processing device 5 via the communication network N and that are determined to have an anomaly detected, for offline processing. The feature amount classification DB stores classification criteria for feature amount classification. The learning data DB 14 stores set data that includes a set of "defect image + correct defect type" for learning each single-class AI processor.

[0019] In the inspection support device 10 according to an application example of the present invention, the feature amount classification processing unit 20 classifies the feature amounts of detected defect locations into a plurality of classes based on classification criteria stored in the feature amount classification DB 13. The single-class AI processing unit 30 includes a plurality of single-class AI processors (30a to 30n) that discriminate a single defect type for each class to which the feature amount classified by the feature amount classification processing unit 20 belongs. For some classes to which the feature amount classified by the feature amount classification processing unit 20 belongs, the single-class AI processing unit 30 uses the single-class AI processing units (30a to 30n) to identify the defect type classified into the class.

[0020] As shown in FIG. 4, in the discrimination of the single-class AI processor according to this application example, the defect type contained in the image to be classified is classified into the defect type learned from the set data and the non-classified type when the defect type is determined to be other than the defect type. Here, in the multi-class classification AI, as shown in FIG. 3, the defect type contained in the image to be classified is classified into each of the multiple defect types learned from the set data. For this reason, as shown in circle 1 in FIG. 3, it is necessary to prepare set data of "defect image + correct defect type" for AI learning for each defect type, so a large amount of learning data for classifying the six types of defect types is accumulated in the database. In addition, as shown in circle 2, it is necessary to construct a complex network structure that extracts features for classifying the six types of defect types from the image of the cut-out defect location and models the type discrimination, and this requires advanced tuning of parameters such as weighting between the nodes connecting the network. Technology with such skills In cases where a specialist is not available, it can be difficult to apply multi-class classification AI. Furthermore, the criteria for the extraction of the above-mentioned features and modeling leading to the defect classification change with each learning. For example, when a new "defect image + correct defect type" for the defect type "wrinkle" is learned, it affects the criteria for determining other types of defect types, and there are cases where defect images that were stably classified as "wrinkles" before learning are more frequently classified as "fibers" or "foreign objects," causing instability.

[0021] In the single-class AI processor according to this application example, a data set for discriminating a single defect type ("defect image + correct defect type") is prepared as learning data, and it is not necessary to accumulate a large amount of set data of "defect image + correct defect type" for all defect types in a database as in an AI processor that performs multi-class classification. In addition, in the single-class AI processing, for example, it is sufficient to discriminate between a single defect type and others, so there is no need to build a complex network structure as in the multi-class AI processing, and advanced tuning of parameters is also not required. Therefore, the operator is not required to have advanced skills. Furthermore, since the defect type to be discriminated is single, learning new set data does not affect the discrimination of other defect types, and stable discrimination results can be obtained even after re-learning. According to the inspection support device 10 according to this application example, a technology that can stably discriminate various defect types occurring in a sheet product using AI processing can be provided.

[0022] Example 1 Hereinafter, specific embodiments of the present invention will be described in more detail with reference to the drawings.

[0023] <System configuration> FIG. 1 is a block diagram showing a schematic configuration of an inspection support system 1 according to an embodiment of the present invention. The inspection support system 1 in this embodiment is a system that supports the determination of the type of abnormality of an abnormal portion detected from a captured image of a sheet-like object (hereinafter, also referred to as a "sheet product" or simply as a "sheet"). The inspection support system 1 uses single-class AI processors (30a to 30n) to individually determine the type of cause (e.g., foreign matter contamination, stains, wrinkles, etc.) that may cause an abnormality (hereinafter, also referred to as a "defect") that may occur in the sheet product. The inspection support system 1 according to this embodiment includes an inspection support device 10 that is connected to a processing device 5 constituting a sheet inspection device via a communication network N. The communication network N is a wired or wireless network. The communication network N can adopt an appropriate configuration depending on the connection method with the connected device. A plurality of sheet inspection devices and a plurality of inspection support devices 10 can be connected to the communication network N.

[0024] As shown in FIG. 1, the sheet inspection apparatus has, as an illumination system, a visible light source 3a that irradiates the top surface (first surface) of the inspection object 2 with visible light, and a visible light source for transmitted light 3b that irradiates the bottom surface (second surface) of the inspection object 2 with visible light. The sheet inspection apparatus also has an imaging device 4 as a measurement system. The visible light source 3a and the visible light source for transmitted light 3b are arranged so as to irradiate light at the same position of the inspection object 2 (however, the surfaces on which the light hits are different), and the imaging device 4 is arranged so as to be able to image the inspection object 2 with light irradiated from the visible light source 3a and reflected by the top surface of the inspection object 2, and light irradiated from the visible light source for transmitted light 3b and transmitted straight through the inspection object 2. The illumination system and the measurement system may be configured to include a plurality of imaging devices 4 in the width direction of the inspection object 2 in accordance with the width of the inspection object 2 so that the entire width of the inspection object 2 can be imaged by the imaging device 4. The sheet inspection apparatus has a processing device 5 that detects abnormal parts contained in the inspection object 2 and determines the type of abnormality based on the output signal of the imaging device 4. The sheet inspection device has a function of detecting abnormal locations contained in the inspection object 2 based on the image obtained by the imaging device 4, determining the type of the detected abnormality, and outputting the result.

[0025] The object to be inspected 2 is formed, for example, in a sheet shape, and is conveyed in the direction of the arrow in FIG. Examples of the object 2 to be inspected include paper, film, resin, cellulose, etc. Also, the object 2 to be inspected may be a separator used in a secondary battery, an optical sheet used in a liquid crystal, etc. In the embodiment shown in FIG. 1, the illumination system and the measurement system are fixed and the object 2 to be inspected is moved, but instead, the object 2 to be inspected may be fixed and the illumination system and the measurement system may be moved. The illumination system may be an LED or the like with a limited wavelength range, or a wavelength filter may be used with a limited wavelength range. Also, the measurement system may be an imaging device (camera) equipped with a CCD image sensor in which light receiving elements are arranged in series, for example. In each light receiving element, light is converted into an electric charge according to the amount of light received. The electric charge output from each light receiving element is input to the processing device 5 as an output signal (imaging data).

[0026] The processing device 5 includes functional elements including a signal processing unit 51, a detection threshold storage unit 52, an abnormality detection unit 53, a judgment threshold storage unit 54, a judgment unit 55, and an output unit 56. In the signal processing unit 51, a white shading process is performed on the signal of the imaging data output from the imaging device 4, and the variation in the output level of each light receiving element constituting the imaging device 4 is corrected. For example, a process is performed in which a correction coefficient determined for each pixel position of one line of 4096 pixels is multiplied by the corresponding pixel to smooth out the variation in the output level caused by the aberration of an optical lens or the like. In addition, the signal processing unit 51 may obtain the luminance value of the captured image using a luminance ratio obtained by dividing the output luminance value after the white shading by the output luminance value (normal value) when the object 2 to be inspected has no abnormality.

[0027] The abnormality detection unit 53 detects an abnormality in the object 2 using a threshold value for detecting an abnormality stored in the detection threshold storage unit 52. For example, the detection threshold storage unit 52 stores a threshold value indicating the degree of change in pixel value to be determined as abnormal, and the abnormality detection unit 53 determines an abnormality when the degree of change in pixel value of the image output from the imaging device 4 exceeds the threshold value. The threshold value stored in the detection threshold storage unit is set by a user or the like according to an inspection standard corresponding to the type, use, material, etc. of the object 2 to be inspected. The determination unit 55 determines the type of abnormality when an abnormality is detected based on a plurality of threshold values ​​used in a process for determining the type of abnormality stored in the determination threshold storage unit 54. The output unit 58 is a function for outputting information about the abnormality. The destination of the information is typically a display device such as an LCD, but it is also possible to output information to a printer, to notify a message or an alarm from a speaker, or to output information to an external information processing device. In this embodiment, the output unit 58 transmits information about the abnormality to the inspection support device 10 via a communication network N connected to the processing device 5.

[0028] The inspection support device 10 according to the present embodiment is a computer that supports the function of the processing device 5 connected to the communication network N to determine the type of abnormality (hereinafter, an abnormality detected from a sheet is also referred to as a "defect"). The inspection support device 10 includes a feature amount classification processing unit 20 and a single-class AI processing unit 30 in a control unit 11. The inspection support device 10 also includes a detection data DB (database) 12, a feature amount classification DB 13, and a learning data DB 14. The detection data DB 12 stores captured images that are determined to have an abnormality detected and are transmitted from the processing device 5 via the communication network N for offline processing. The feature amount classification DB stores classification criteria related to feature amount classification. The learning data DB 14 stores set data that includes a set of "defect image + correct defect type" for learning each single-class AI processor. As shown in a thin dashed rectangular frame 90, the processing device 5 may include the function of the inspection support device in its configuration.

[0029] As will be described in detail later, the feature amount classification processing unit 20 classifies the feature amounts of the detected defective portions into a plurality of classes based on classification criteria stored in the feature amount classification DB 13. The single-class AI processing unit 30 includes a plurality of single-class AI processors (30a to 30n) that discriminate a single defect type for each class to which the feature amount classified by the feature amount classification processing unit 20 belongs. The single-class AI processing unit 30 uses single-class AI processing units (30a to 30n) to identify defect types classified into some classes to which the features classified by the feature classification processing unit 20 belong. The inspection support device 10 according to this embodiment provides the above-mentioned discrimination function to the processing device 5 offline or online, based on information on abnormal locations transmitted from the output unit 58.

[0030] <Device configuration> Fig. 2 is a diagram showing an example of a hardware configuration of the test support device 10 according to the present embodiment. As shown in Fig. 2, the test support device 10 is a computer including components such as a processor 101, a main memory device 102, an auxiliary memory device 103, a communication IF 104, and an input / output IF 105, which are connected to each other by a connection bus 106. The main memory device 102 and the auxiliary memory device 103 are recording media that can be read by the test support device 10. The main memory device 102 and the auxiliary memory device 103 constitute the memory of the test support device 10. Each of the above components may be provided in multiple units, or some of the components may not be provided.

[0031] The processor 101 is a central processing unit that controls the entire test support device 10. The processor 101 is, for example, a central processing unit (CPU) or a digital signal processor (DSP). The processor 101 receives, for example, The program is executablely deployed in the working area of ​​the main memory device 102, and peripheral devices are controlled through the execution of the program to provide functions that meet a predetermined purpose. In this embodiment, the functions of the feature classification processing unit 20, single-class AI processing unit 30 (30a to 30n), detection data DB 12, feature classification DB 13, and learning data DB 14 are provided through the execution of the program. Each DB is, for example, a relational database constructed by a database management system (DBMS) program executed by the processor 101 of the inspection support device 10 managing data stored in the auxiliary memory device 103 or the like. Note that some or all of the functions may be provided by an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or the like. In a similar manner, Some or all of the functions are implemented using FPGA (Field-Programmable Gate Array), numerical calculation processor, etc. The examination support device 10 may be realized by a dedicated LSI (large scale integration) such as a processor, a vector processor, an image processor, or other hardware circuits. It may be realized in a single physical configuration, or may be realized by a configuration of multiple computers working in conjunction with each other.

[0032] The main memory device 102 stores the programs executed by the processor 101, the data processed by the processor, etc. The main memory device 102 includes a flash memory, a RAM (Random Access Memory), and a ROM (Read Only Memory). The auxiliary memory device 103 stores various programs and various data in a readable and writable recording medium. The auxiliary memory device 103 is also called an external memory device. For example, an OS (Operating System), various programs, various tables, etc. are stored in the auxiliary memory device 103. The OS includes, for example, a communication interface program that transfers data to and from an external device, etc. connected via the communication IF 104. The external device, etc. are, for example, a processing device 5 connected to the communication network N, a computer such as an information processing terminal, an external memory device, etc.

[0033] The auxiliary storage device 103 is used as a storage area that supports the main storage device 102, and stores programs executed by the processor 101, data processed by the processor 101, etc. The auxiliary storage device 103 is a non-volatile semiconductor memory (flash memory, EPROM (Erasable Programmable Read Only Memory) Silicon disks, solid state drive devices, hard disks, Hard disk drives (HDDs) and other storage devices. Examples of the removable recording medium 03 include a drive device for a removable recording medium such as a CD drive device, a DVD drive device, and a BD drive device. Examples include a USB (Universal Serial Bus) memory and a SD (Secure Digital) memory card.

[0034] The communication IF 104 is an interface for connecting the test support device 10 to the communication network N. The communication IF 104 can adopt an appropriate configuration according to the connection method with the communication network N. The input / output IF 105 is an interface for inputting and outputting data between the test support device 10 and a device connected to the test support device 10. For example, input devices such as a keyboard, a pointing device such as a touch panel or a mouse, and a microphone are connected to the input / output IF 105. The test support device 10 accepts operation instructions from an operator who operates the input device via the input / output IF 105. In addition, for example, display devices such as an LCD, an EL panel, and an organic EL panel, and output devices such as a printer and a speaker are connected to the input / output IF 105. The test support device 10 outputs data and information processed by the processor 101, and data and information stored in the main storage device 102 and the auxiliary storage device 103 via the input / output IF 105.

[0035] <Processing configuration> First, as a comparative example, a defect type discrimination process using an AI that performs multi-class classification will be described with reference to FIG. 3. As shown in FIG. 3, when an abnormality is detected in a sheet product and a multi-class classification AI is adopted to discriminate the type of the abnormality, an image is captured from an imaging device that is a camera (A1), and the position of the abnormality (defective part) in the image is identified (A2). The position of the defective part in the image can be expressed as, for example, two-dimensional coordinates (X, Y) from the upper left corner of the image as the origin to the lower right corner. For example, a part where the degree of change in pixel value of the image exceeds a threshold is identified using a method similar to that of the abnormality detection unit 53 already described, and an image of a partial area that includes the part (such as a rectangular area inscribed by the part) is cut out (A3). Then, an AI process that performs multi-class classification is applied to the cut-out image of the defective part (A4).

[0036] The multi-class classification AI is pre-trained for all defect types using set data of "defect images + correct defect types" stored in a database (A5). Using the pre-trained defect images and correct defect types, the multi-class classification AI automatically performs a process of extracting features for each type and modeling to distinguish the type from the extracted features (A6), and outputs the classification results from this modeling (A7). In the example of Figure 3, six types of classification results (air bubbles, holes, insects, foreign objects, wrinkles, and fibers) that indicate the type of abnormality the defect is are output. Using the multi-class classification AI makes it possible to distinguish six types of abnormality (multi-class discrimination) that indicate the type of abnormality that a defect occurring in a sheet product is.

[0037] However, in the case of multi-class classification AI, as shown in circle 1, it is necessary to prepare a set of data of "defect image + correct defect type" for AI training for each defect type, so a large amount of training data for classifying the six defect types is accumulated in the database. Note that for defect types with low occurrence frequency, the above data set cannot be prepared sufficiently, and there are cases where the learning for classification judgment has to be performed in an incomplete state. In addition, as shown in circle 2, it is necessary to construct a complex network structure from the image of the cut-out defect part to extract features for classifying the six defect types and modeling the type discrimination, and this requires advanced tuning of parameters such as weighting between nodes connecting the network. If engineers with such skills are not available, it may be difficult to apply multi-class classification AI. Furthermore, the criteria for the extraction of the above features and modeling leading to the defect classification change each time learning is performed. For example, when a new "defect image + correct defect type" for the defect type "wrinkle" was trained, it affected the criteria for determining other types of defects, and defect images that had been consistently determined to be "wrinkles" before training were more frequently determined to be "fibers" or "foreign objects," resulting in instability.

[0038] The inspection support device 10 according to this embodiment includes, as a processing function, a single-class AI processing unit 30. Each single-class AI processor constituting the single-class AI processing unit 30 is an AI for discriminating an individual defect type. Set data for discriminating a single defect type is used as learning data for each single-class AI processor, and the correct defect type specified by the set data is learned.

[0039] FIG. 4 is a diagram for explaining the discrimination process of the single-class AI processor. FIG. 4 illustrates a discrimination example using the single-class AI processor. If the single defect type discriminated by the single-class AI processor A9 is "A", the single-class AI processor learns set data limited to "defect image (A) + correct defect type (A)" as learning data (A10). The single-class AI processor discriminating the single defect type "A" learns the process of extracting feature amounts from the set data limited to "defect image (A) + correct defect type (A)" and modeling for determining the type from the extracted feature amounts. Then, the single-class AI processor discriminating the single defect type "A" automatically performs the process of extracting the learned feature amounts and modeling for determining the type from the extracted feature amounts, and discriminates whether the image of the defect location (A8) is "A" or "Not A". For example, if "A" is "wrinkle," the discrimination result of the single-class AI processor is classified into two values: "wrinkle" and "not classified" which is classified as something other than wrinkles. In this way, in the discrimination of the single-class AI processor according to this embodiment, the defect type contained in the image to be classified is classified into the defect type learned from the set data and "not classified" which is classified as something other than the defect type.

[0040] In this embodiment, a plurality of single-class AI processors are used to discriminate the defect type for the image of the defect portion. As described in FIG. 4, the single-class AI processor only needs to prepare a data set ("defect image + correct defect type") for discriminating a single defect type as learning data, and does not need to accumulate a large amount of set data of "defect image + correct defect type" for all defect types in a database as in the case of an AI processor that performs multi-class classification. Furthermore, in single-class AI processing, for example, it is sufficient to discriminate between a single defect type and others, so there is no need to build a complex network structure as in multi-class AI processing, and no need to perform advanced tuning of parameters. Therefore, the operator is not required to have advanced skills. Furthermore, since the defect type to be discriminated is single, even if new set data is learned, it does not affect the discrimination of other defect types, and stable discrimination results can be obtained even after re-learning.

[0041] FIG. 5 is a flowchart showing an example of a defect type determination process provided by the inspection support device 10 according to the present embodiment. In the flow of the present embodiment, as a pre-processing before determining the defect type by single-class AI processing, feature analysis of the abnormal part occurring in the sheet product is performed based on the degree of change in luminance value of the abnormal part and the size of the abnormal part. As a result of the feature analysis, the feature of the abnormal part is classified into a plurality of patterns, and a pattern for determining the defect type using single-class AI processing is selected from the classified plurality of patterns. Then, in this flow, single-class AI discrimination processing is performed on the selected feature pattern, and individual defect types associated with each single-class AI discrimination are determined. The processing of this flow shown in FIG. 5 is provided by cooperation between the feature classification processing unit 20 and the single-class AI processing unit 30 composed of a plurality of single-class AI processors (30a to 30n).

[0042] After the process starts, an image is captured by the imaging device 4, which is a camera (step S101), and feature amounts for abnormal areas (defective areas) in the image are calculated (step S102). In calculating the feature amounts, for example, the degree and magnitude of change in luminance value is found from an image of an abnormal area that has occurred in the sheet product. For example, a method similar to that of the abnormality detection unit 53 is used to identify an area where the degree of change in pixel values ​​of the image exceeds a threshold, and an inscribed region that includes the area is calculated. The size is obtained by calculating the area of ​​the rectangular region. In addition, the degree of change in the luminance value of the defective portion is obtained by calculating the average luminance value of the inscribed rectangle. In step S103, the calculated feature values ​​are classified (feature value classification). For example, the calculated average luminance value is classified into "bright" that is equal to or greater than a preset threshold value, and "dark" that is less than the threshold value. Similarly, the area of ​​the calculated inscribed rectangular region is used to classify the relative sizes into "large", "medium", and "small". As a result of the classification, the feature values ​​of the defective portion are classified into six patterns, "bright small", "bright medium", "bright large", "dark small", "dark medium", and "dark large", which are combinations of the average luminance value and the size (step S104). Note that the feature values ​​of the defective portion may be classified using an index other than the average luminance value and the size, or may be classified by combining the average luminance value and the size with another index. For example, the defective portion may be classified using optical characteristics such as the color and polarization of the defective portion. It is sufficient to classify into a pattern to which a single-class AI can be applied to single-classify the defect type.

[0043] In step S105, one or more patterns for determining the defect type using the single-class AI processing are selected from the classified patterns. For example, as shown in FIG. 5, the patterns of "dark small" and "dark medium" are selected as patterns to which the defect type by the single-class AI processing is applied. Such a selection criterion can be set in advance by a user who inspects the sheet product. In the example of FIG. 5, it is determined by empirical knowledge that the defect type of the pattern classified as "light small" and "light medium" as the feature amount of the defect portion is "air bubble," the defect type of the pattern classified as "light large" is "hole," and the defect type of the pattern classified as "dark large" is "fiber." Therefore, in step S105, the patterns classified as "dark small" and "dark medium" are selected to determine the defect type by the single-class AI processing. In the selected pattern, when the defect portion is cut out, the processing proceeds to step S106. Note that the processing from step S106 to step S108 is provided by the single-class AI processing unit 30.

[0044] In step S106, the defect type is determined using a plurality of single-class AI processes. For example, when the defect types to be determined are "insects," "foreign matter," and "wrinkles," the single-class AI processors 30a, 30b, and 30c that determine the respective defect types perform determination processing on the images of the defect parts extracted. The single-class AI processor 30a performs learning using set data of "defect image + correct defect type (insect)," and determines the defect type of the image of the defect part as "insects" and defect types other than "insects." The single-class AI processor 30b performs learning using set data of "defect image + correct defect type (foreign matter)," and determines the defect type of the image of the defect part as "foreign matter" and defect types other than "foreign matter." The single-class AI processor 30c performs learning using set data of "defect image + correct defect type (wrinkles)," and determines the defect type of the image of the defect part as "wrinkles" and defect types other than "wrinkles." In the following description, it is assumed that a single-class AI processor 30a that discriminates the defect type "insect" and a single-class AI processor 30b that discriminates the defect type "foreign matter" are applied to an image of a defect area extracted from the "dark small" feature classification pattern, and a single-class AI processor 30c that discriminates the defect type "wrinkle" is applied to an image of a defect area extracted from the "dark small" feature classification pattern.

[0045] In step S107, when discrimination by multiple single-class AI processors is applied to the same defect image, arbitration of discrimination results is performed. FIG. 6 is a diagram for explaining arbitration of discrimination results in step S107. In this flow, a single-class AI processor 30a that discriminates the defect type "insect" and a single-class AI processor 30b that discriminates the defect type "foreign matter" are applied to an image of a defect portion extracted from the feature amount classification pattern "dark small". The discrimination results of each processor for the same defect image are four combinations, as shown in FIG. 6(a). That is, the combinations obtained are "insect" and "foreign matter", "insect" and "non-foreign matter", "foreign matter" and "non-insect", and "non-foreign matter" and "non-insect".

[0046] Here, the single-class AI processor 30a that distinguishes the defect type "insect" distinguishes "insect" and When the single-class AI processor 30b that discriminates the defect type "foreign matter" judges it to be "other than a foreign matter," the defect type of the defect image can be identified as "insect." Similarly, when the single-class AI processor 30a that discriminates the defect type "insect" judges it to be "other than an insect" and the single-class AI processor 30b that discriminates the defect type "foreign matter" judges it to be "foreign matter," the defect type of the defect image can be identified as "foreign matter." This is because the defect type of the defect image is the defect type that each single-class AI processor has learned from the set data that it has been made to learn.

[0047] However, when the single-class AI processor 30a discriminating the defect type "insect" discriminates "insect" and the single-class AI processor 30b discriminating the defect type "foreign matter" judges it as "foreign matter", it is not possible to specify whether the defect type of the defect image is "insect" or "foreign matter". Such a case may occur, for example, when there is insufficient learning data. In step S107, a criterion for arbitrating such a discrimination result is set in advance. In this flow, when the single-class AI processor discriminating between "A" and "other than A" and the single-class AI processor discriminating between "B" and "other than B" discriminate the defect types that they have each learned, they each judge that visual confirmation is required, as shown in FIG. 6(a). Such a criterion is set in advance by a user who inspects the sheet product. In addition, when the single-class AI processor 30a, which discriminates the defect type "insects," discriminates "other than insects," and the single-class AI processor 30b, which discriminates the defect type "foreign matter," judges it to be "other than foreign matter," it is judged to be other than a classification that excludes the features of the defect location from the defect type discrimination targets.

[0048] Also, as shown in FIG. 6(b), the single-class AI processor 30c that discriminates the defect type "wrinkles" is applied to the image of the defect portion extracted from the feature classification pattern "in the dark", and "wrinkles" and "other than wrinkles" are discriminated according to the discrimination criteria in the AI ​​processing learned from the wrinkle learning data. In step S107, if the single-class AI processor 30c discriminates the defect type as "other than wrinkles", it discriminates the defect type as "out of classification" by excluding the feature of the defect portion from the defect type discrimination targets according to the criteria previously set by the user. In this way, by providing an arbitration function for the discrimination result, the processing load related to the defect type discrimination can be reduced, and the processing time during online processing can be shortened.

[0049] In step S108, the defect type determined using the single-class AI processors 30a, 30b, and 30c is output. That is, "insects," "foreign matter," and "wrinkles," determined from an image of a defect portion cut out from the feature classification pattern "dark small," are output. The determination results may be output to the processing device 5 connected online, or may be displayed on a display panel or the like provided in the inspection support device 10 offline. After the processing of step S108, this routine is temporarily terminated.

[0050] As described above, in the inspection support device 10 according to this embodiment, as preprocessing before determining the defect type by single-class AI processing, feature analysis of the abnormal part occurring in the sheet product is performed based on the degree of change in luminance value and the size of the abnormal part, and a feature pattern to which the single-class AI processing is applied can be selected. Therefore, the feature amounts related to determining the defect type of the abnormal part detected in the sheet product can be reduced in advance, and the processing load related to the determination using single-class AI processing can be relatively reduced.

[0051] In the inspection support device 10 according to this embodiment, single-class AI processing can be used to distinguish defect types. As explained in FIG. 4, in a single-class AI processor, it is sufficient to prepare a data set for distinguishing a single defect type ("defect image + correct defect type") as learning data, and there is no need to accumulate a large amount of set data of "defect image + correct defect type" for all defect types in a database, as in an AI processor that performs multi-class classification. Also, in single-class AI processing, for example, if it is possible to distinguish between a single defect type and others, Since only a single defect type is detected, there is no need to build a complex network structure as in multi-class AI processing, nor is there a need for advanced parameter tuning. This means that operators are not required to have advanced skills. Furthermore, since only a single defect type is detected, learning a new set of data does not affect the discrimination of other defect types, and stable discrimination results can be obtained even after re-learning.

[0052] Moreover, in the inspection support device 10 according to this embodiment, discrimination by multiple single-class AI processes can be applied to the same defect image. Therefore, by combining the discrimination results of the single-class AI processors, it is possible to improve the discrimination accuracy for the defect types that each single-class AI has been trained to learn. Furthermore, by combining the discrimination results of the single-class AI processors, it is also possible to exclude the feature amount of the defect location from the defect type discrimination targets. In this way, by combining and arbitrating the discrimination results of the single-class AI processes, it is possible to reduce the processing load related to the discrimination of the defect type, and it is possible to shorten the processing time when online. According to this embodiment, it is possible to provide a technology that can stably discriminate various types of defects occurring in a sheet product using AI processing.

[0053] [Variation 1] Fig. 7 is a flowchart showing an example of a defect type determination process provided by the inspection support device 10 according to the first modified example. The flow shown in Fig. 7 illustrates an example in which feature amount classification is further applied to the defect type determined using the single-class AI process. The process of this flow enables ranking to identify the product type of the sheet product whose defect type has been determined by the single-class AI process. In the flow shown in Fig. 7, the processes from step S101 to step S108 are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0054] In step S109, feature classification is performed for the purpose of ranking sheet products having an abnormality determined to have the defect type "foreign matter", and the result of the classification is output (step S110). In step S109, the abnormality determined to have the defect type "foreign matter" is classified more precisely than the feature classification in step S103. Specifically, the defect area is calculated in an image cut out from the defect area. Then, based on this defect area, a ranking index for products having an abnormality determined to have the defect type "foreign matter" is classified (step S110).

[0055] For example, when the number exceeds a first threshold (e.g., 10,000), it is classified as a "large foreign object," and when the number is equal to or less than the first threshold and equal to or more than a second threshold (e.g., 100), it is classified as a "medium foreign object." Similarly, when the number is less than the second threshold, it is classified as a "small foreign object." Such thresholds are set in advance by a user who inspects the sheet product. In step S110, an example of three-level classification is shown, but two levels or five levels may be used as long as the classification contributes to ranking sheet products having abnormalities. Furthermore, in step S109, feature classification is performed as a rule-based classification, but it goes without saying that AI processing may be used for this processing. After the processing of step S110, this routine is temporarily terminated.

[0056] [Modification 2] In the first embodiment and the first modification, pattern classification (step S103) is performed on the feature values ​​calculated from the abnormal portion (defect portion), but the calculated feature values ​​may be used to determine whether the defect type can be determined by AI processing and to select a single-class AI processor to be applied to the defect type determination. The processing of the second modification can optimize the determination process for the defect type.

[0057] FIG. 8 is a flowchart showing an example of a defect type determination process provided by the inspection support device 10 according to the second modification. In the flow of FIG. 8, the processes from step S101 to step S102 are the same as those in the first embodiment, and therefore the description thereof will be omitted. In step S111, a calculation Depending on the feature quantity, it is determined whether or not to apply AI processing to determine the defect type, and if so, which single-class AI processor to use is selected. Specifically, the defect area, which is the average density of the abnormal area, which is the calculated feature quantity, is used to determine whether or not the defect is acceptable and to select the single-class AI processor.

[0058] In step S111, for example, if the defect area exceeds the first threshold, it is determined that AI processing is not to be applied to the defect type discrimination, and the process proceeds to step S112. If the defect area is equal to or smaller than the first threshold and equal to or larger than the second threshold, it is determined that AI processing is to be applied to the defect type discrimination, and a single-class AI processor 30a that discriminates the defect type "insect" and a single-class AI processor 30b that discriminates the defect type "foreign matter" are selected, and the process proceeds to step S113. Similarly, if the defect area is less than the second threshold, it is determined that AI processing is to be applied to the defect type discrimination, and a single-class AI processor 30c that discriminates the defect type "wrinkle" is selected, and the process proceeds to step S114. Here, the first threshold and the second threshold are the same thresholds as those in the first modification.

[0059] The process of step S111 can reduce the processing load related to the determination of the defect type, and can optimize the determination process for the defect type.

[0060] In step S112, feature classification processing similar to steps S103 to S104 in the first embodiment is performed, and the defect types "bubble", "hole", and "fiber" are identified. In step S113, the single-class AI processor 30a for discriminating the defect type "bubble" and the single-class AI processor 30b for discriminating the defect type "foreign matter" perform the processing of steps S105 to S108 in the first embodiment, and the defect types "bug" and "foreign matter" are discriminated. Similarly, in step S114, the single-class AI processor 30c for discriminating the defect type "wrinkle" performs the processing of steps S105 to S108 in the first embodiment, and the defect type "wrinkle" is discriminated. After the processing of step S114, this routine is temporarily terminated.

[0061] As described above, in the second modified example as well, the processing load associated with discrimination using single-class AI processing can be relatively reduced by using feature amounts calculated from an abnormal location that has occurred in a sheet product.

[0062] (others) The above embodiment is merely an example, and the disclosure of the present embodiment may be modified as appropriate without departing from the spirit of the present disclosure. The processes and means described in the present disclosure may be freely combined and implemented as long as no technical contradiction occurs.

[0063] Moreover, the processing described as being performed by one device may be shared and executed by a plurality of devices. Alternatively, the processing described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration for implementing each function can be flexibly changed. For example, as shown in a thin dashed rectangular frame 90, the processing device 5 of the first embodiment may include the configuration of the inspection support device 10. Moreover, the processing device 5 may include a feature classification processing unit 20, a single-class AI processing unit 30, a feature classification DB 13, and a learning data DB 14, and perform the functions of the inspection support device 10 in this embodiment.

[0064] <Computer-readable recording medium> A program for causing an information processing device or other machine or device (hereinafter, computer, etc.) to realize any of the above functions can be recorded on a recording medium readable by the computer, etc. Then, the computer, etc. can provide the function by reading and executing the program from the recording medium.

[0065] Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer, etc. Among such recording media, those that can be removed from a computer, etc. include, for example, flexible disks, magneto-optical disks, CD-ROMs, CD-R / Ws, DVDs, Blu-ray disks, DATs, 8mm tapes, memory cards such as flash memory, etc. Furthermore, examples of recording media that are fixed to a computer, etc. include hard disks and ROMs.

[0066] In the following, the components of the present invention will be described with reference to the reference numerals in the drawings in order to make it possible to compare the components of the present invention with the configurations of the embodiments. <Invention 1> A support device (10) that supports a judgment inspection for detecting an abnormality occurring in an object (2) to be inspected from a captured image of the object (2) and determining a type of the detected abnormality, a classification processing unit (20) that calculates a feature amount of an abnormality occurring in the inspection object (2) from the captured image, classifies the calculated feature amount into a plurality of classes, and selects one or more classes to be processed, from the plurality of classes, in which a type of abnormality is to be discriminated using a learning discrimination unit (30); a learning discrimination unit (30) including, for each type of abnormality, learning discriminators (30a to 30n) that have learned discrimination criteria for discriminating whether an abnormality to be discriminated is a type that is a correct answer or a type other than the type that is a correct answer based on set data that combines an element image that induces the abnormality and a type of abnormality that is a correct answer for the element image, the learning and discriminating unit (30) discriminates the type of abnormality for the processing target class selected by the classification processing unit (20) based on individual discrimination results of the learning discriminators (30a to 30n) in which discrimination criteria for each type of abnormality have been learned; An assistance device (10). [Explanation of symbols]

[0067] 1 inspection support system, 2 inspection object (sheet, sheet product), 3a visible light source, 3b transmitted light visible light source, 4 imaging device, 5 processing device, 10 inspection support device, 11 control unit, 12 detection data DB, 13 feature amount classification DB, 14 learning data DB, 20 feature amount classification processing unit, 30 single class AI processing unit, 30a-30n Single-class AI processor, 101 processor, 102 main memory device, 13 auxiliary memory device, 14 communication IF, 15 input / output IF, 106 connection bus

Claims

1. 1. A support device for supporting a judgment inspection that detects an abnormality occurring in an object to be inspected from a captured image of the object to be inspected and judges a type of the detected abnormality, comprising: a classification processing unit that calculates feature amounts of an abnormality occurring in the inspection object from the captured image, classifies the calculated feature amounts into a plurality of classes based on a preset classification criterion, and selects one or more classes from the plurality of classes as processing targets for which a type of abnormality is to be discriminated using a learning discrimination unit; a learning discrimination unit for each type of abnormality, the learning discriminator having a discrimination criterion for discriminating whether an abnormality to be discriminated is a type that is a correct answer or a type other than the type that is a correct answer based on set data combining an element image that induces the abnormality and a type of abnormality that is a correct answer for the element image, the learning and discriminating unit discriminates the type of abnormality for the processing target class selected by the classification processing unit based on an individual discrimination result of a learning discriminator in which discrimination criteria for each of the types of abnormality have been learned.

13. An assistance device comprising:

2. the learning and discriminator includes a first learning discriminator that has learned a discrimination criterion for discriminating a first abnormality, and a second learning discriminator that has learned a discrimination criterion for discriminating a second abnormality, 2. The support device according to claim 1, wherein, when one of the plurality of classes is processed and the discrimination result of the first learning discriminator discriminates a first abnormality and the discrimination result of the second learning discriminator discriminates a type other than the second abnormality, the type of abnormality of the class is discriminated as the first abnormality.

3. 3. The support device according to claim 2, characterized in that, when one of the plurality of classes is processed and the discrimination result of the first learning discriminator discriminates a type other than a first abnormality and the discrimination result of the second learning discriminator discriminates a second abnormality, the type of abnormality of the class is discriminated to be the second abnormality.

4. When one of the plurality of classes is a processing target, and the discrimination result of the first learning discriminator discriminates a first abnormality and the discrimination result of the second learning discriminator discriminates a second abnormality, or when the discrimination result of the first learning discriminator discriminates a type other than the first abnormality and 4. The support device according to claim 2, wherein when the discrimination result of the second learning discriminator discriminates a type other than the second abnormality, discrimination of the type of abnormality of said class is handed over to a designated process.

5. 5. The support device according to claim 1, wherein the classification processing unit further classifies the feature amount of the class in which the type of the abnormality has been determined into a plurality of subclasses.

6. A method executed by a computer of a support device for supporting a judgment inspection for detecting an abnormality occurring in an object to be inspected from a captured image of the object and determining a type of the detected abnormality, comprising: calculating feature amounts of an abnormality occurring in the inspection object from the captured image, and classifying the calculated feature amounts into a plurality of classes based on a preset classification criterion, and selecting a predetermined class from the plurality of classes; learning a discrimination criterion for discriminating whether an abnormality to be discriminated is the type that is the correct answer or is other than the type that is the correct answer, based on set data that combines an element image that induces the abnormality and a type of abnormality for which the element image is a correct answer, for each type of abnormality; Discriminating the type of anomaly for the predetermined class based on an individual discrimination result discriminated according to the learned discrimination criterion for each type of anomaly; The method according to claim 1, further comprising:

Images