Information processing apparatus and information processing method

Abstract

To effectively determine a defect occurring in an object to be inspected using a learning model.SOLUTION: Acquiring a first image group to an N-th image group extracted from a plurality of captured images obtained by imaging the same inspection target under a plurality of different imaging conditions, where N is an integer of 2 or more; I is a variable that is an integer of 1 or more and N or less, M is an integer of 1 or more, for each of i=1 to N, i-th preprocessing is performed based on the captured images included in the i-th image group to generate M preprocessed images, and for each of i=1 to N, the M preprocessed images generated in the preprocessing are input to a learned learning model included in an i-th learning model group of machine learning, acquiring an output from the learning model, the first image group to the N-th image group each including one or more of the captured images and being different image groups, and the i-th learning model group including one or more of the learning models.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] Defects may occur in products during or after their manufacture. When controlling the quality of products, it is necessary to determine the types of defects. There are various objects (inspection targets) that can be inspected in this way, such as a glass substrate.

[0003] The defect inspection device described in Patent Document 1 classifies defects (faults) occurring on a substrate based on an image of the substrate that is the object of inspection. The defect inspection device estimates the type of defect using deep learning (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-124591 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, there is still room for improvement in determining the type of defect occurring in an object to be inspected.

[0006] The present disclosure has been made in consideration of these circumstances, and aims to provide an information processing device and an information processing method that can effectively determine defects that occur in an object to be inspected using a machine learning learning model. [Means for solving the problem]

[0007] One aspect of the present disclosure is an information processing device, where N is an integer greater than or equal to 2, that acquires a first image group to an Nth image group extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions, i is a variable greater than or equal to 1 and N is an integer greater than or equal to 1, and M is an integer greater than or equal to 1, and for each of i=1 to N, an i-th preprocessing is performed based on the captured images included in the i-th image group to generate M preprocessed images, and for each of i=1 to N, the M preprocessed images generated by the preprocessing are input into a trained learning model included in an i-th learning model group of machine learning to obtain output from the learning model, and the first image group to the N-th image group each include one or more of the captured images and are mutually different image groups, and the i-th learning model group each includes one or more of the learning models.

[0008] One aspect of the present disclosure is an information processing method in which an information processing device performs the following processes: acquiring a first image group to an Nth image group, where N is an integer greater than or equal to 2, extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions; i is a variable between 1 and N inclusive, and M is an integer greater than or equal to 1; for each of i=1 to N, performing an i-th preprocessing based on the captured images included in the i-th image group to generate M preprocessed images; and for each of i=1 to N, inputting the M preprocessed images generated by the preprocessing into a trained learning model included in an i-th learning model group of machine learning to acquire output from the learning model; wherein the first image group to the Nth image group each include one or more of the captured images and are mutually different image groups, and the i-th learning model group each includes one or more of the learning models.

[0009] One aspect of the present disclosure is an information processing device, where N is an integer greater than or equal to 2, that acquires a first image group to an Nth image group extracted from a plurality of captured images of the same object under a plurality of different imaging conditions, i is a variable greater than or equal to 1 and N is an integer greater than or equal to 1, and M is an integer greater than or equal to 1, and for each of i=1 to N, an i-th preprocessing is performed based on the captured images included in the i-th image group to generate M preprocessed images, and for each of i=1 to N, the M preprocessed images generated by the preprocessing are input into an untrained learning model included in an i-th learning model group of machine learning to obtain an output from the learning model, and learning of the learning model is performed based on the output, and the first image group to the Nth image group each include one or more of the captured images and are mutually different image groups, and the i-th learning model group each includes one or more of the learning models. [Effects of the Invention]

[0010] According to the information processing device, information processing device, and information processing method disclosed herein, it is possible to effectively determine defects occurring in an object to be inspected using a machine learning learning model. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram illustrating an example of the configuration of functional blocks of the information processing device according to the embodiment. [Figure 2A] FIG. 10 is a diagram illustrating an example of input and output of a learning model during learning according to an embodiment. [Figure 2B] FIG. 10 is a diagram illustrating an example of input and output of a trained learning model according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of a learning device and a determination device according to a modified example of the embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing unit according to the embodiment. [Figure 5] 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 6A] FIG. 10 is a diagram showing an example of an inspection using bright-field and reflected light imaging conditions. [Figure 6B]FIG. 10 is a diagram showing an example of an inspection using dark-field and reflected light imaging conditions. [Figure 6C] FIG. 10 is a diagram showing an example of an inspection using bright-field and transmission imaging conditions. [Figure 6D] FIG. 10 is a diagram showing an example of an inspection using dark-field and transmission imaging conditions. [Figure 7A] FIG. 10 is a diagram illustrating an example of a procedure of a process at the time of determination performed by the information processing apparatus according to the embodiment. [Figure 7B] FIG. 2 is a diagram showing an example of a procedure of an i-th process performed by the information processing device according to the embodiment. [Figure 8A] FIG. 4 is a diagram schematically illustrating an example of a first process according to the embodiment. [Figure 8B] FIG. 10 is a diagram schematically illustrating an example of a second process according to the embodiment. [Figure 8C] FIG. 10 is a diagram schematically illustrating an example of a third process according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] [Information processing device] FIG. 1 is a diagram showing an example of the functional block configuration of an information processing device 11 according to the embodiment. The information processing device 11 is configured using, for example, a computer.

[0014] The information processing device 11 includes an input unit 131 , an output unit 132 , a communication unit 133 , a storage unit 134 , and a control unit 135 . The input unit 131 includes an operation unit 151 . The output unit 132 includes a display unit 152 . The control unit 135 includes a learning control unit 171 and a determination control unit 172 .

[0015] The input unit 131 receives information from the outside. The input unit 131 has, for example, an operation unit 151 that accepts an operation performed by a user, and inputs information according to the operation accepted by the operation unit 151. The operation unit 151 may have a function to accept operations from a touch panel, or may have a function to accept operations from physical keys. The operation unit 151 may also have a function to accept operations by voice (for example, the user's voice). Furthermore, the input unit 131 may be connected to an external device and receive information output from the external device. The external device may be, for example, a portable recording medium. The recording medium may be called, for example, a storage medium.

[0016] The output unit 132 outputs the information. The output unit 132 has, for example, a display unit 152, and displays (outputs) information on the screen of the display unit 152. The screen may have a touch panel function. The output unit 132 may also be connected to an external device and output information to the external device. The external device may be, for example, a portable recording medium. The output unit 132 may output information in a form other than a display, such as audio output.

[0017] In this embodiment, the input unit 131 and the output unit 132 are shown as separate functional units, but the input unit 131 and the output unit 132 may be configured as a common functional unit (input / output unit). The input / output unit may be configured as a touch panel.

[0018] The communication unit 133 has a function of communicating information with an external device. This communication may be wired or wireless. Here, in this embodiment, the communication unit 133 is shown as a functional unit separate from the input unit 131 and the output unit 132, but the receiving function of the communication unit 133 may be considered to be included in the function of the input unit 131, and the transmitting function of the communication unit 133 may be considered to be included in the function of the output unit 132.

[0019] The storage unit 134 stores information. The storage unit 134 may store any information. In the example of FIG. 1, a learning model A1 and training data A2 stored in the storage unit 134 are shown. As the learning model A1, various learning models may be used, for example, a deep learning model may be used. While machine learning is being performed, the learning model A1 becomes a learning model under training, and after machine learning is completed, the learning model A1 becomes a trained learning model. Furthermore, the training data A2 may be used for machine learning. However, if the training data A2 is not used, the training data A2 does not need to be stored in the storage unit 134.

[0020] The control unit 135 performs various controls and processes. In this embodiment, the control unit 135 includes a processor such as a CPU (Central Processing Unit), and executes a predetermined program (control program) by the processor to perform control and processing as defined in the program. The program may be stored in the storage unit 134, for example.

[0021] In the example of FIG. 1, the functions of the control unit 135 are shown as a learning control unit 171 and a determination control unit 172. The learning control unit 171 controls the machine learning process. The determination control unit 172 controls the determination process using the results of machine learning.

[0022] <Overview of machine learning training> FIG. 2A is a diagram illustrating an example of input and output of a learning model A1a during learning according to the embodiment. Here, learning model A1a represents the state in which learning model A1 shown in FIG. 1 is undergoing machine learning. In general, the learning control unit 171 updates the parameters of the learning model A1a based on the output data when predetermined input data is input to the learning model A1a. At this time, the learning control unit 171 may perform supervised learning using training data A2. As another example, unsupervised learning may be performed without using the training data A2. It should be noted that the schematic diagram shown in FIG. 2A is a diagram for the purpose of general explanation and is not necessarily precise. The learning model under training may be referred to as, for example, a model under training.

[0023] <Outline of judgment using machine learning results> FIG. 2B is a diagram illustrating an example of input and output of a trained learning model A1b according to the embodiment. Here, learning model A1b represents the state in which learning model A1 shown in FIG. 1 has undergone machine learning. In general, the determination control unit 172 obtains a predetermined determination result based on output data when predetermined input data is input to the learning model A1b. The determination result may be, for example, the output data from the learning model A1b itself, or may be other data obtained based on the output data from the learning model A1b. It should be noted that the schematic diagram shown in FIG. 2B is a diagram for the purpose of general explanation and is not necessarily precise. A trained learning model may be referred to as, for example, a trained model.

[0024] Hereinafter, for the sake of convenience, the learning model A1a and the learning model A1b will not be distinguished from each other, and will be referred to as the learning model A1. Hereinafter, unless it is explicitly stated that the learning model is "currently learning," the learning model A1 refers to a learned learning model (learning model A1b).

[0025] In this embodiment, the output from the trained learning model A1 (learning model A1b in the example of FIG. 2B) will be described as an inference result that represents the result of inference by the learning model A1. Inference results from a machine learning learning model include, for example, classification results or regression results. In this embodiment, the case where the inference result is a classification result will be exemplified. Inference may also be referred to as, for example, prediction, estimation, or conjecture. Furthermore, in this embodiment, a case is shown in which a predetermined judgment is made based on an inference result, but for example, the inference result itself may be used as the result of the judgment, in which case the inference result essentially becomes the result of the judgment, and the judgment processing is omitted.

[0026] In this embodiment, the output (inference result) from the learning model A1 is the dependent variable, and other items that are thought to affect the dependent variable are the explanatory variables. As a specific example, if the type of defect that has occurred on a glass substrate is the objective variable, the explanatory variables may be one or more of the shape of the defect, the size of the defect, the color of the defect, the position of the defect on the glass substrate, the orientation of the defect on the glass substrate, etc. The objective variable and the explanatory variables are included in the feature quantities of the defect. It should be noted that this specific example is an example for the purpose of explanation and is not necessarily limited to this.

[0027] <Example in which the learning function and the judgment function are provided in separate devices> Here, the example of FIG. 1 illustrates a case where the information processing device 11 has both a learning function for performing machine learning learning and a determination function for making a determination based on the machine learning result. As another example, the learning function and the judgment function may be provided in separate devices.

[0028] <Learning device> FIG. 3 is a diagram illustrating an example of a learning device 211 and a determination device 212 according to a modified example of the embodiment. In the example of FIG. 3, the learning device 211 and the determination device 212 are configured as separate devices. The learning device 211 has a configuration similar to that of the information processing device 11 shown in FIG. 1 except that the determination control unit 172 is not included. Furthermore, if the learning device 211 does not use the teacher data A2, the learning device 211 does not need to store the teacher data A2.

[0029] The learning device 211 performs machine learning learning and provides the learning results (for example, a learned learning model) to the determination device 212. In addition, the transfer of learning results (e.g., a learned learning model) from the learning device 211 to the determination device 212 may be performed, for example, by communication between the learning device 211 and the determination device 212, or may be performed using a portable recording medium, etc. The communication between the learning device 211 and the determination device 212 may be performed via, for example, a server device or a relay device.

[0030] <Judgment device> The determination device 212 has a configuration similar to that of the information processing device 11 shown in FIG. 1 except that the learning control unit 171 is not included. Furthermore, if the determination device 212 does not use the teacher data A2, the determination device 212 does not need to store the teacher data A2.

[0031] The determination device 212 acquires a predetermined determination result based on the learning result (for example, a learned learning model) provided by the learning device 211.

[0032] <Example of hardware configuration for each device> FIG. 4 is a diagram illustrating an example of the hardware configuration of the information processing unit 1001 according to the embodiment. The configuration shown in Fig. 4 may be applied to the information processing device 11 shown in Fig. 1, or to each of the learning device 211 and the determination device 212 shown in Fig. 3. In this case, the functions of each device are realized by the functions of the information processing unit 1001 shown in Fig. 4. For example, the configuration of the information processing unit 1001 or a part of the configuration of the information processing unit 1001 may be used in any part of the information processing device 11, the learning device 211, or the determination device 212.

[0033] In the example of Figure 4, the information processing unit 1001 includes a processor 1011, an operation device 1012, a display device 1013, a storage device 1014, a memory 1015, an input / output interface 1016, a network interface 1017, and a bus 1021 connecting these. When the information processing unit 1001 is applied to any device, it is not necessarily required that all of the processing units shown in FIG. 4 are provided, and any processing unit not shown in FIG. 4 may be added.

[0034] The processor 1011 is composed of a CPU and the like, and executes a program to perform control and processing defined in the program. The operation device 1012 includes one or more input devices such as a keyboard and a mouse, and receives operations performed by a person (user) or the like. The display device 1013 has a screen and displays and outputs information on the screen.

[0035] The storage device 1014 is a non-volatile storage unit, and is configured, for example, by a hard disk, and stores information. The memory 1015 is a volatile storage unit, and is configured with RAM (Random Access Memory) or the like, and temporarily stores information. As the RAM, for example, a DRAM (Dynamic Random Access Memory) may be used. The storage device 1014 or memory 1015 may store information for a program executed by the processor 1011, for example.

[0036] The input / output interface 1016 is an interface for connecting to an external recording medium or the like. The network interface 1017 is an interface for connecting to an external network.

[0037] Here, the information processing unit 1001 may include one processor or two or more processors as the processor 1011. As an example, the information processing unit 1001 may include multiple CPUs, each of which executes its own control and processing, and these multiple CPUs may work together to realize overall control and processing.

[0038] [Example of judgment process] An example of the determination process according to this embodiment will be described below. In this embodiment, an example is described in which an image of a defect that occurs on a glass substrate during the manufacturing process is captured, and a predetermined judgment result is obtained using the image data and a trained learning model A1. Defects occurring on the glass substrate may be caused by, for example, scratches on the glass substrate or dirt on the glass substrate. The glass substrate is manufactured by, for example, a float method or a fusion method. The defects may be called, for example, defects or faults.

[0039] Here, the types of defects may be, for example, linear defects, point defects, ball-shaped defects, etc., in terms of shape. In addition, the types of defects can be classified into small defects (micro defects), medium-sized defects, large defects, etc., from the viewpoint of size. The size can be, for example, length, Or it could be area, etc.

[0040] <Example of multiple inspection machine layout> FIG. 5 is a diagram showing an example of the configuration of an information processing system 301 according to the embodiment. In FIG. 5, the glass substrate manufacturing process 2011 is shown schematically by arrows. FIG. 5 also shows the locations where five inspection machines, a first inspection machine B1 to a fifth inspection machine B5, are arranged in the manufacturing process 2011. FIG. 5 also shows an information processing device 11.

[0041] In this embodiment, the direction toward the start point of the manufacturing process 2011 is referred to as upstream, and the direction toward the end point of the manufacturing process 2011 is referred to as downstream. The first inspection machine B1 and the second inspection machine B2 are arranged at the same position or at a nearby position on the upstream side of the manufacturing process 2011. The first inspection machine B1 and the second inspection machine B2 may be configured using a common device, or may be configured using separate devices. The third inspection machine B3 is disposed at a position downstream of the first inspection machine B1 and the second inspection machine B2. The fourth inspection machine B4 is disposed at a position downstream of the third inspection machine B3. The fifth inspection machine B5 is disposed at a position downstream of the fourth inspection machine B4.

[0042] In this embodiment, the first inspection machine B1 to the fifth inspection machine B5 each have an imaging unit, and acquire images captured by the imaging unit as inspection results. As the imaging unit, for example, a visible light camera may be used, or other devices may be used.

[0043] In this embodiment, the first inspector B1 and the second inspector B2 acquire images of the entire glass substrate to be manufactured. Furthermore, the third inspection machine B3 to the fifth inspection machine B5 may each acquire an image capturing an area focusing on the position of a defect included in an image acquired by, for example, the first inspection machine B1 or the second inspection machine B2. In this case, for example, the first inspection machine B1 or the second inspection machine B2 detects the position of the defect based on the captured image and notifies the third inspection machine B3 to the fifth inspection machine B5 of the detection result. It should be noted that this type of series of inspections is an example for the purpose of explanation, and any other type of inspection may be used.

[0044] Here, in this embodiment, a case is shown in which five inspection machines are used in the manufacturing process 2011, but the number of inspection machines may be any number, for example, two or more. Furthermore, the arrangement of the multiple inspection machines relative to the manufacturing process is not limited to this example, and various other arrangements may be used.

[0045] <Examples of imaging conditions for inspection> 6A to 6D, examples of imaging conditions for an image for inspection are shown. In this embodiment, the object to be inspected is a glass substrate in the manufacturing process 2011 . The object to be inspected may be called, for example, an object to be inspected.

[0046] FIG. 6A is a diagram showing an example of an inspection using bright field and reflected light imaging conditions. FIG. 6A shows an example of the arrangement of the image capturing unit 411, the illumination 421, and the object to be inspected C1 when bright field and reflected light imaging conditions are used. In the example of FIG. 6A, light from the illumination 421 is reflected by the inspection object C1, and the reflected light is incident on the imaging section 411. In FIG. 6A, light is schematically shown using arrows.

[0047] FIG. 6B is a diagram showing an example of an inspection using dark-field and reflected light imaging conditions. FIG. 6B shows an example of the arrangement of the image capturing unit 412, the illumination 422, and the object to be inspected C2 when dark-field and reflective imaging conditions are used. In the example of FIG. 6B, light from the illumination 422 is reflected by the object C2 to be inspected, and the scattered light at the time of the reflection is incident on the imaging section 412. In FIG. 6B, the light is schematically shown by arrows.

[0048] FIG. 6C is a diagram showing an example of an inspection using bright-field and transmission imaging conditions. FIG. 6C shows an example of the arrangement of the image capturing unit 413, the illumination 423, and the object C3 to be inspected when bright field and transmission imaging conditions are used. In the example of FIG. 6C, light from the illumination 423 passes through the object C3 to be inspected, and the transmitted light is incident on the imaging section 413. In FIG. 6C, the light is schematically shown by arrows.

[0049] FIG. 6D is a diagram showing an example of an inspection using dark-field and transmission imaging conditions. FIG. 6D shows an example of the arrangement of the imaging unit 414, the illumination 424, and the object C4 to be inspected when dark-field and transmission imaging conditions are used. In the example of FIG. 6D, light from the illumination 424 passes through the object C4 to be inspected, and the scattered light from the transmission enters the imaging section 414. In FIG. 6D, the light is schematically shown by arrows.

[0050] Here, the four imaging conditions are divided into two categories based on the combination of brightness and imaging direction, and the illumination method is divided into two categories: reflection and transmission. The combinations of brightness and imaging direction include bright field and dark field. A bright field is an imaging condition in which a light beam emitted by an illumination source is irradiated onto an object to be inspected, and an imaging unit (a camera in this embodiment) is installed at a position and in a direction such that the incident direction of the reflected light from the object to be inspected or the transmitted light that has passed through the object to be inspected is included in the field of view. In a bright field, the reflected light or transmitted light from the object to be inspected is directly incident on the imaging unit, so a bright image is captured. A dark field is an imaging condition in which a light beam emitted by an illumination source is irradiated onto an object to be inspected, and the imaging unit (a camera in this embodiment) is installed at a position and orientation such that the object to be inspected is included in the field of view, but the direction of incidence of the reflected light from the object to be inspected or the transmitted light that has passed through the object to be inspected is not included in the field of view. In a dark field, the reflected light or transmitted light from the object to be inspected does not directly enter the imaging unit, and scattered light generated on the surface of the object to be inspected enters the imaging unit, resulting in an image that is darker than that captured in a bright field.

[0051] <Outline of the judgment process> An outline of the process when the information processing device 11 performs a determination process using the trained learning model A1 will be described with reference to FIGS. 7A and 7B. In this embodiment, N or more determination processes are performed, where N represents an integer of 2 or more, and a comprehensive determination process is performed based on the results of these N determination processes.

[0052] FIG. 7A is a diagram showing an example of a procedure of a process at the time of determination performed by the information processing device 11 according to the embodiment. For convenience of explanation, the variable i represents an integer between 1 and N, inclusive.

[0053] (Step S1-i) In the information processing device 11, the i-th process, which is the i-th determination process, is performed by the determination control unit 172. Then, in the information processing device 11, the process proceeds to step S2. Here, (Step S1-i) includes (Step S1-1) to (Step S1-N). These processes may be performed in parallel in time, or may be performed at different times in a time-division manner.

[0054] (Step S2) In the information processing device 11, the determination control unit 172 performs a comprehensive determination process based on the results of the N determination processes, and acquires a comprehensive determination result, which is the determination result. Then, in the information processing device 11, the process of this flow ends.

[0055] Here, various methods may be used as a method for obtaining a comprehensive judgment result based on a plurality of judgment results, and for example, a method may be used in which one judgment result having the highest possibility (likelihood) among the plurality of judgment results is obtained as the comprehensive judgment result, or a method may be used in which one judgment result having the highest priority among the plurality of judgment results is obtained as the comprehensive judgment result. Note that the priority may be set in advance, for example. As another example, a method for obtaining an overall judgment result based on multiple judgment results may be used in which the result obtained by combining two or more of these multiple judgment results is obtained as the overall judgment result.

[0056] In this embodiment, a case where a comprehensive judgment result is obtained based on N judgment results is described. However, as another example, a configuration in which a comprehensive judgment is not performed by the information processing device 11 may be used, in which case the processing of step S2 shown in FIG. 7A may not be provided.

[0057] FIG. 7B is a diagram showing an example of the procedure of the i-th process performed by the information processing device 11 according to the embodiment. In this embodiment, the processes in (step S1-1) to (step S1-N) are generally similar to each other, and these will be collectively described as the i-th process. However, for each of the processes (step S1-1) to (step S1-N), for example, one or more of the input data, pre-processing, inference processing, and judgment processing may be different.

[0058] (Step S11) In the information processing device 11, the input data is acquired by the determination control unit 172. Then, in the information processing device 11, the process proceeds to step S12.

[0059] Here, in this embodiment, the input data may be data of an image captured by an imaging unit of an inspection machine, or other data obtained based on the image. The input data may also include, for example, data on images captured by imaging units of two or more inspection machines, or other data obtained based on the images. That is, the input data includes one or both of data on images captured by imaging units of one or more inspection machines, or other data obtained based on the images.

[0060] (Step S12) In the information processing device 11, predetermined pre-processing is performed on the input data by the determination control unit 172. Then, in the information processing device 11, the process proceeds to step S13.

[0061] Here, for example, the content of the pre-processing may be different in one or more of the first process to the Nth process. As an example, the contents of the pre-processing of each of the first process to the Nth process may all be different.

[0062] Note that various types of processing may be used as the preprocessing. For example, the preprocessing may involve one or more of the following: cutting out a portion of an image; enlarging a portion of an image; combining two or more images; or a combination of any two or more of these processes. The process of cutting out a part of an image may involve, for example, cutting out a point of interest contained in the image (in this embodiment, a point of a defect that should be noted) as the center of the cut-out image. The process of enlarging a part of an image may involve, for example, a process of generating an enlarged image so as to enlarge a notable portion included in the image (in this embodiment, a notable defect portion).

[0063] As an example of the process of combining two or more images, a process of generating an image by averaging pixel values ​​(e.g., luminance) of two or more images that fit into the same frame may be used. In the averaging, for example, weighting may be performed using weights set for each of the two or more images to be combined. Another example of the process for combining two or more images may be a process for arranging these two or more images in different areas within a frame to generate one image. In the process of combining two or more images, for example, images that are captured under some or all of the same imaging conditions may be used as the two or more images to be combined. As a specific example, two or more images captured under dark-field imaging conditions may be combined, or two or more images captured under bright-field imaging conditions may be combined.

[0064] (Step S13) In the information processing device 11, the judgment control unit 172 inputs the preprocessed data into a predetermined trained learning model A1, and obtains output data from the learning model A1 as inference result data. Then, the information processing device 11 proceeds to the processing of step S14.

[0065] Here, in the i-th process, for example, one learning model may be used as the trained learning model A1, and one inference result may be obtained using that learning model, or two or more different learning models may be used, and two or more inference results may be obtained using these learning models.

[0066] Here, for example, the content of the inference process (the learning model used for the inference) may be different in one or more of the first process to the Nth process. As an example, the contents of the inference processes (learning models used for inference) of the first process to the Nth process may all be different.

[0067] (Step S14) In the information processing device 11, the determination control unit 172 makes a predetermined determination based on the acquired data of the inference result and acquires the determination result. Then, in the information processing device 11, the processing of this flow ends. When the inference result itself is used as the determination result, for example, the process of step S14 may be common to the process of step S13.

[0068] Here, when two or more learning models are used in the i-th process, for example, each learning model may be suitable for application to different types of defects. Each learning model may output, as an inference result, for example, a result of classifying the presence or absence of a defect of the type corresponding to each learning model into two levels, or a result of classifying the possibility of a defect of the type into three or more levels. As another example, each learning model may output, as an inference result, the result of classification regarding other matters.

[0069] For convenience of explanation, it is assumed that the variable j represents an integer between 1 and N, and is a different number from the variable i. In the jth process, for example, a learning model different from the learning model used in the ith process may be used, and for example, each learning model may be suitable for applying different types of defects.

[0070] <Specific example of judgment process> A specific example of the determination process will be shown schematically with reference to FIGS. 8A, 8B, and 8C. In this example, the case where N=3 is shown, that is, the case where the comprehensive judgment process is performed after the first to third processes are performed. It should be noted that the images and defects shown in FIGS. 8A, 8B, and 8C are schematic for the purpose of explanation and are not necessarily strict.

[0071] First, the first image G1 to the fifth image G5 will be described. FIG. 8A shows data of a first image G1 to a fourth image G4 as an example of input data. FIG. 8B shows data of a fifth image G5 as an example of input data. FIG. 8C shows data of a first image G1 to a fifth image G5 as an example of input data. Here, the first image G1 to the fourth image G4 shown in FIG. 8C are the same as those shown in FIG. 8A, and the fifth image G5 shown in FIG. 8C is the same as that shown in FIG. 8B.

[0072] The first image G1 is an image detected by the first inspection machine B1. The second image G2 is an image detected by the second inspection machine B2. The third image G3 is an image detected by the third inspection machine B3. The fourth image G4 is an image detected by the fourth inspection machine B4. The fifth image G5 is an image detected by the fifth inspection machine B5.

[0073] In this embodiment, as shown in FIG. 5, in the manufacturing process 2011, the first inspection machine B1 and the second inspection machine B2 are arranged at the most upstream side, and from there, the third inspection machine B3, the fourth inspection machine B4, and the fifth inspection machine B5 are arranged in that order downstream. In terms of resolution, the fifth inspection machine B5 has the highest resolution, followed by the third inspection machine B3 and the fourth inspection machine B4, then the second inspection machine B2, and finally the first inspection machine B1 has the lowest resolution. The resolution of the fifth inspection machine B5 is significantly higher than that of the third inspection machine B3 and the fourth inspection machine B4. The resolution of the third inspection machine B3 and the fourth inspection machine B4 is the same (or approximately the same).

[0074] In the first inspection machine B1 and the third inspection machine B3, dark field imaging is performed by the imaging unit. In the second inspection machine B2, the fourth inspection machine B4, and the fifth inspection machine B5, the imaging units capture bright field images. In the first inspection machine B1, the second inspection machine B2, the third inspection machine B3, and the fourth inspection machine B4, imaging is performed by a reflective optical system, not a vertical optical system. In the fifth inspection machine B5, imaging is performed by a vertical optical system.

[0075] In the first inspection machine B1, the second inspection machine B2, the third inspection machine B3, and the fourth inspection machine B4, the reflected image of the front surface and the reflected image of the back surface of the object to be inspected (a glass substrate in this embodiment) appear in the image, so it is possible to distinguish between the front and back surfaces. Specifically, if a defect exists on the front surface of the object to be inspected, a real image (image reflected from the front surface) and a virtual image (image reflected from the back surface) of the defect appear in the image, but if a defect exists on the back surface of the object to be inspected, only the real image (image reflected from the back surface) of the defect appears in the image. On the other hand, the fifth inspector B5 usually clearly shows the features (for example, defects on the surface) of only one side of the object to be inspected (a glass substrate in this embodiment), and therefore cannot distinguish between the front and back sides. Here, the surface of the substrate-shaped object to be inspected on the side where the imaging unit of the inspection machine is placed (one side of the object to be inspected) is called the front surface, and the opposite side (the other side of the object to be inspected) is called the back surface.

[0076] In the four-image plate process shown in FIG. 8A, the types of defects suitable for judgment include evaporative defects and backside defects. For example, evaporation defects appear in images taken by the first inspection machine B1 and the second inspection machine B2 located upstream in the manufacturing process 2011, but do not appear in images taken by the third inspection machine B3, the fourth inspection machine B4, or the fifth inspection machine B5 located further downstream. In other words, in this example, the evaporation defects have already evaporated and disappeared at the position of the third inspection machine B3 and positions further downstream. For example, since defects on the back side are not clearly visible in the image captured by the fifth inspection machine B5, it is more difficult (or impossible) to determine defects on the back side in the example of Figure 8B or the example of Figure 8C than in the example of Figure 8A.

[0077] In the processing of the single image plate shown in FIG. 8B, the type of defect suitable for judgment is a minute linear defect system. For example, in this example, such tiny defects can be identified in images taken by the fifth inspection machine B5, which has the highest resolution, but are difficult (or impossible) to identify in images taken by the first inspection machine B1 to the fourth inspection machine B4. Furthermore, in the example of Figure 8B, only images captured by the fifth inspection machine B5 are used, so there is less disturbance (e.g., no disturbance) when images captured by other inspection machines (first inspection machine B1 to fourth inspection machine B4) are used.

[0078] In the five-image plate processing shown in FIG. 8C, the type of defect suitable for judgment is a minute defect type. For example, in this example, such tiny defects can be identified in images taken by the fifth inspection machine B5, which has the highest resolution, but are difficult (or impossible) to identify in images taken by the first inspection machine B1 to the fourth inspection machine B4.

[0079] In this manner, in this embodiment, images of the same defect occurring on the glass substrate to be inspected are captured by each of the multiple inspectors (first inspector B1 to fifth inspector B5) under different imaging conditions. In this embodiment, the images captured by each of the plurality of inspection machines (first inspection machine B1 to fifth inspection machine B5) are black and white images. Then, a predetermined number of images are selected from the multiple images captured by the multiple inspection machines (first inspection machine B1 to fifth inspection machine B5), and pre-processing and judgment processing are performed as shown in the examples of Figures 8A, 8B, and 8C.

[0080] FIG. 8A is a diagram schematically illustrating an example of a first process according to the embodiment. First, as an input data acquisition process, the information processing device 11 causes the determination control unit 172 to acquire data of the first image G1, data of the second image G2, data of the third image G3, and data of the fourth image G4.

[0081] Next, as preprocessing, the information processing device 11 generates data for a first preprocessed image H1 based on data for the first image G1, generates data for a second preprocessed image H2 based on data for the second image G2, and generates data for a third preprocessed image H3 based on data for the third image G3 and data for the fourth image G4 using the judgment control unit 172.

[0082] Here, the first preprocessed image H1 is an image obtained by cutting out a partial area of ​​the first image G1, centered on the defect portion of the first image G1. The cut-out area may be, for example, an area similar in shape to the area of ​​the first image G1, or may be an area of ​​another shape. The second preprocessed image H2 is an image obtained by cutting out a part of the second image G2, centered on the defect portion of the second image G2. The cut-out part may be, for example, a region similar in shape to the region of the second image G2, or may be a region of another shape.

[0083] The third preprocessed image H3 is a composite image of an image obtained by cutting out a portion of the third image G3 centered on the defect portion shown in the third image G3, and an image obtained by cutting out a portion of the fourth image G4 centered on the defect portion shown in the fourth image G4. Here, the region cut out from the third image G3 may be, for example, a region having a similar shape to the region of the third image G3, or may be a region having another shape. Furthermore, the region cut out from the fourth image G4 may be, for example, a region similar in shape to the region of the fourth image G4, or may be a region of another shape. Furthermore, various methods may be used to combine two images, and for example, weights may be set for each of the images to be combined.

[0084] In this example, a first preprocessed image H1, a second preprocessed image H2, and a third preprocessed image H3 are used as the first inference-use image I1, the second inference-use image I2, and the third inference-use image I3, respectively. As another example, the results of performing a predetermined processing on each of the first preprocessed image H1, the second preprocessed image H2, and the third preprocessed image H3 may be used as the first inference image I1, the second inference image I2, and the third inference image I3, respectively.

[0085] Next, in the information processing device 11, the determination control unit 172 inputs the first inference-use image I1, the second inference-use image I2, and the third inference-use image I3 into the predetermined learned learning model A1. Then, in the information processing device 11, the judgment control unit 172 acquires the output data from the learning model A1 as inference result data, makes a predetermined judgment based on the inference result, and acquires the judgment result (for convenience of explanation, referred to as the first judgment result).

[0086] Here, in this embodiment, a case is shown in which the learning model A1 inputs three images (a first inference-use image I1, a second inference-use image I2, and a third inference-use image I3) as one set. In the learning model A1, these three image data may be treated as image data corresponding to R (Red), image data corresponding to G (Green), and image data corresponding to B (Blue), respectively. In this embodiment, the three images (first inference image I1, second inference image I2, and third inference image I3) are all black and white images, but are input into the learning model A1 as pseudo-RGB image data. The learning model A1 does not necessarily have to be one that corresponds to RGB, that is, a learning model other than a learning model for color images may be used.

[0087] FIG. 8B is a diagram schematically illustrating an example of the second process according to the embodiment. First, as an input data acquisition process, the information processing device 11 causes the determination control unit 172 to acquire data of the fifth image G5.

[0088] Next, as preprocessing, the information processing device 11 generates an eleventh inference-use image I11, a twelfth inference-use image I12, and a thirteenth inference-use image I13 based on the data of the fifth image G5 by the determination control unit 172. The eleventh inference image I11, the twelfth inference image I12, and the thirteenth inference image I13 may also be called the eleventh preprocessed image, the twelfth preprocessed image, and the thirteenth preprocessed image, respectively.

[0089] Here, the eleventh inference image I11 is an image obtained by cutting out a part of the fifth image G5 around the defect part in the fifth image G5. The cut-out part may be, for example, a part similar to the part of the fifth image G5, or may be a part of another shape. The twelfth inference-use image I12 is the same image as the eleventh inference-use image I11. For example, the twelfth inference-use image I12 may be an image obtained by copying the eleventh inference-use image I11. The thirteenth inference-use image I13 is the same image as the eleventh inference-use image I11. For example, the thirteenth inference-use image I13 may be an image obtained by copying the eleventh inference-use image I11.

[0090] In this example, the eleventh inference-use image I11, the twelfth inference-use image I12, and the thirteenth inference-use image I13 are images obtained by performing preprocessing on the fifth image G5. As another example, the results of performing a predetermined processing on the image resulting from preprocessing of the fifth image G5 may be used as the eleventh inference image I11, the twelfth inference image I12, and the thirteenth inference image I13, respectively.

[0091] Next, in the information processing device 11, the determination control unit 172 inputs the eleventh inference-use image I11, the twelfth inference-use image I12, and the thirteenth inference-use image I13 into the predetermined trained learning model A1. Then, in the information processing device 11, the judgment control unit 172 acquires the output data from the learning model A1 as inference result data, makes a predetermined judgment based on the inference result, and acquires the judgment result (for convenience of explanation, referred to as the second judgment result).

[0092] Here, in this embodiment, the learning model A1 is shown to input three images (an eleventh inference-use image I11, a twelfth inference-use image I12, and a thirteenth inference-use image I13) as one set. In the learning model A1, these three image data may be treated as image data corresponding to R (Red), image data corresponding to G (Green), and image data corresponding to B (Blue), respectively. In this embodiment, the three images (the eleventh inference image I11, the twelfth inference image I12, and the thirteenth inference image I13) are all black and white images, but are input into the learning model A1 as pseudo-RGB image data. The learning model A1 does not necessarily have to be one that corresponds to RGB, that is, a learning model other than a learning model for color images may be used.

[0093] FIG. 8C is a diagram schematically illustrating an example of a third process according to the embodiment. First, as an input data acquisition process, the information processing device 11 acquires data of the first image G1, data of the second image G2, data of the third image G3, data of the fourth image G4, and data of the fifth image G5 by the judgment control unit 172.

[0094] Next, as preprocessing, the information processing device 11 generates data for a 21st preprocessed image H21 based on the data of the first image G1 and the data of the third image G3, generates data for a 22nd preprocessed image H22 based on the data of the second image G2 and the data of the fourth image G4, and generates data for a 23rd preprocessed image H23 based on the data of the fifth image G5 by the judgment control unit 172.

[0095] Here, the 21st preprocessed image H21 is configured by combining four images: a 21-1 image J1, a 21-2 image J2, a 21-3 image J3, and a 21-4 image J4. In the example of FIG. 8C, the 21-1 image J1, the 21-2 image J2, the 21-3 image J3, and the 21-4 image J4 are arranged at the top left, top right, bottom right, and bottom left, respectively. In this way, the combination of the 21-1 image J1, the 21-2 image J2, the 21-3 image J3, and the 21-4 image J4 combines images captured under dark field conditions (in this example, the first image G1 and the third image G3).

[0096] The 21-1 image J1 is an image obtained by cutting out a portion of the first image G1, centered on the defect portion of the first image G1. The cut-out portion may be, for example, a region similar in shape to the region of the first image G1, or may be a region of another shape. The 21-1 image J1 may be, for example, the same image as the first preprocessed image H1 shown in FIG. 8A. The 21-2 image J2 is an image obtained by cutting out a part of the area of ​​the 21-1 image J1, centered on the defect part shown in the 21-1 image J1, and is an enlarged image. The 21-3rd image J3 is an image obtained by cutting out a part of the third image G3, centered on the defect part of the third image G3. The cut-out part may be, for example, a similar area to the area of ​​the third image G3, or may be a different shape. The 21-4th image J4 is an image obtained by cutting out a part of the area of ​​the 21-3th image J3, centered on the defect part that appears in the 21-3th image J3, and is an enlarged image.

[0097] The 22nd preprocessed image H22 is formed by combining four images: the 22-1 image K1, the 22-2 image K2, the 22-3 image K3, and the 22-4 image K4. In the example of FIG. 8C, the 22-1 image K2, the 22-2 image K2, the 22-3 image K3, and the 22-4 image K4 are arranged at the upper left, upper right, lower right, and lower left, respectively. In this way, the combination of the 22-1 image K1, the 22-2 image K2, the 22-3 image K3, and the 22-4 image K4 combines images captured under bright field conditions (in this example, the second image G2 and the fourth image G4).

[0098] The 22-1 image K1 is an image obtained by cutting out a portion of the second image G2, centered on the defect portion of the second image G2. The cut-out portion may be, for example, a region similar in shape to the region of the second image G2, or may be a region of another shape. The 22-1 image K1 may be, for example, the same image as the second preprocessed image H2 shown in FIG. 8A. The 22-2 image K2 is an image obtained by cutting out a part of the area of ​​the 22-1 image K1, centered on the defect part that appears in the 22-1 image K1, and is an enlarged image. The 22-3rd image K3 is an image obtained by cutting out a portion of the fourth image G4, centered on the defect portion of the fourth image G4. The cut-out portion may be, for example, a region similar in shape to the region of the fourth image G4, or may be a region of another shape. The 22-4th image K4 is an image obtained by cutting out a part of the area of ​​the 22-3th image K3, centered on the defect part that appears in the 22-3th image K3, and is an enlarged image.

[0099] In addition, the 23rd preprocessed image H23 is the same image as the 5th image G5 in this example. As another example, the 23rd preprocessed image H23 may be an image obtained by cutting out a portion of the fifth image G5 centered on the defect portion of the fifth image G5. The cut-out portion may be, for example, a region similar in shape to the region of the fifth image G5, or may be a region of another shape. The 23rd preprocessed image H23 may be, for example, the same image as the 11th inference image I11 shown in FIG. 8B.

[0100] In this example, the 21st preprocessed image H21, the 22nd preprocessed image H22, and the 23rd preprocessed image H23 are used as the 21st inference-use image I21, the 22nd inference-use image I22, and the 23rd inference-use image I23, respectively. As another example, the results of performing a predetermined processing on each of the 21st preprocessed image H21, the 22nd preprocessed image H22, and the 23rd preprocessed image H23 may be used as the 21st inference image I21, the 22nd inference image I22, and the 23rd inference image I23, respectively.

[0101] Next, in the information processing device 11, the determination control unit 172 inputs the 21st inference-use image I21, the 22nd inference-use image I22, and the 23rd inference-use image I23 into the predetermined trained learning model A1. Then, in the information processing device 11, the judgment control unit 172 acquires the output data from the learning model A1 as inference result data, makes a predetermined judgment based on the inference result, and acquires the judgment result (for convenience of explanation, referred to as the third judgment result).

[0102] In this embodiment, the learning model A1 receives three images (the 21st inference-use image I21, the 22nd inference-use image I22, and the 23rd inference-use image I23) as one set. In the learning model A1, these three image data may be treated as image data corresponding to R (Red), image data corresponding to G (Green), and image data corresponding to B (Blue), respectively. In this embodiment, the three images (the 21st inference image I21, the 22nd inference image I22, and the 23rd inference image I23) are all black and white images, but are input into the learning model A1 as pseudo-RGB image data. The learning model A1 does not necessarily have to be one that corresponds to RGB, that is, a learning model other than a learning model for color images may be used.

[0103] <Example of comprehensive judgment process> Here, the learning model A1 used in the first process may be one learning model, or may be two or more different learning models. Furthermore, the learning model A1 used in the second process may be one learning model, or may be two or more different learning models. Furthermore, the learning model A1 used in the third process may be one learning model, or may be two or more different learning models.

[0104] Furthermore, the one or more learning models used in the first process and the one or more learning models used in the second process may, for example, all be different, or all be the same, or some may be the same and other parts may be different. Furthermore, the one or more learning models used in the first process and the one or more learning models used in the third process may, for example, all be different, or all be the same, or some may be the same and other parts may be different. Furthermore, the one or more learning models used in the second process and the one or more learning models used in the third process may, for example, all be different, or all be the same, or some may be the same and other parts may be different.

[0105] As a specific example, in the comprehensive judgment process, the information processing device 11 may have the judgment control unit 172 rank the output values ​​(values ​​representing the inference results) of all learning models used in the first to third processes, and adopt a predetermined number of the highest ranked inference results. The predetermined number may be, for example, one, two, or three. In this case, the information processing device 11 may also take into consideration, for example, the priority set in advance for each learning model for ranking.

[0106] For example, the pre-processing and learning model used in the first process may be those suitable for determining a first type of defect, which is a specific type. Furthermore, the pre-processing and learning model used in the second processing may be those suitable for determining defects of a specific type, that is, a second type. Furthermore, the pre-processing and learning model used in the third process may be those suitable for determining a specific type of defect, a third type. Here, the first type, the second type, and the third type are different types from each other.

[0107] As a specific example, in the comprehensive judgment process, the information processing device 11 may have the judgment control unit 172 obtain a comprehensive judgment result based on the first judgment result for the first type, the second judgment result for the second type, and the third judgment result for the third type. As the overall judgment result, for example, the judgment result that is judged to be the most likely of the first to third judgment results may be adopted. In this configuration, if the first judgment result is adopted, the overall judgment result obtained is that the defect is likely to be of the first type, if the second judgment result is adopted, the overall judgment result obtained is that the defect is likely to be of the second type, and if the third judgment result is adopted, the overall judgment result obtained is that the defect is likely to be of the third type.

[0108] As another example, the comprehensive judgment result may be a result in which all of the first to third judgment results are comprehensively considered, rather than being an alternative. The method for obtaining the comprehensively considered result is not particularly limited, and may be set in advance, for example.

[0109] Here, for the sake of simplicity, the first process, the second process, and the third process are shown as being suitable for the first type, the second type, and the third type, respectively, but for example, two or more types of learning models may be used in one or more of the first process to the third process. Furthermore, in two or more of the first to third processes, a learning model suitable for the same type (type of defect) may be used. As an example, each of the first to third processes may be configured to use a learning model that is suitable for a type (type of defect) that is not used in the other processes.

[0110] Furthermore, although the first to third processes have been described above as an example, the same applies to the first to Nth processes being performed in general. That is, in each of the first process to the Nth process, for example, a determination process using one learning model suitable for one type (type of defect) may be performed, or a determination process using two or more learning models suitable for each of two or more types (types of defect) may be performed. In this case, in each of the first process to the Nth process, for example, a determination process using two or more different learning models suitable for one type (type of defect) may be performed. Furthermore, in two or more of the first to Nth processes, a learning model suitable for the same type (type of defect) may be used.

[0111] As an example, a configuration may be adopted in which a learning model suited to a type (type of defect) that is not used in the other processes is always used in each of the first process to the Nth process. As another example, a configuration may be adopted in which a learning model suited to a different type (type of defect) is used in each of the first process to the Nth process.

[0112] It should be noted that there may be multiple patterns of learning models suitable for the same type (type of defect). Furthermore, there may be multiple patterns of pre-treatment suitable for the same type (type of defect).

[0113] [Other configuration examples] In this embodiment, the case of determining defects in a glass substrate is described, but as another example, the present invention may be applied to an object to be inspected other than a glass substrate, such as a transparent body other than a glass substrate, or an object other than a transparent body. In addition, in this embodiment, a case where a judgment is made based on an image taken during the manufacturing process of the object to be inspected is shown, but as another example, it may also be applied to a case where a judgment is made based on an image taken of the object to be inspected, which is a finished product.

[0114] In the example of FIG. 1, one information processing device 11 has various functions, but these functions may be distributed among a plurality of devices (for example, information processing devices). In the example of FIG. 3, one learning device 211 has various functions for learning, but these functions may be distributed among a plurality of devices (for example, information processing devices). In the example of FIG. 3, one determination device 212 has various functions for determination, but these functions may be distributed among a plurality of devices (for example, information processing devices). Here, when various functions are distributed among a plurality of devices, these devices may include, for example, a terminal device and a cloud device (for example, a server device).

[0115] [Regarding the above embodiment] As described above, the information processing device 11 according to this embodiment can effectively determine defects occurring in an object to be inspected using a machine learning learning model.

[0116] As an example of configuration, the information processing device 11 has the following configuration. The information processing device 11 acquires a first image group to an N-th image group extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions, where N is an integer of 2 or more. The information processing device 11 performs the ith preprocessing based on the captured images included in the ith image group, for each of i=1 to N, to generate M preprocessed images, where i is a variable that is an integer between 1 and N, and M is an integer that is 1 or greater. For each of i=1 to N, the information processing device 11 inputs M preprocessed images generated by preprocessing into a trained learning model included in the i-th learning model group of machine learning, and obtains output from the learning model. Here, the first to Nth image groups each include one or more captured images and are different from one another. Each of the i-th learning model groups includes one or more learning models. In this embodiment, for convenience of explanation, it is assumed that the image group includes only one captured image, and that the learning model group includes only one learning model. In addition, examples of image groups being different from each other include a situation in which one or more images included in a certain image group do not completely match one or more images included in another image group, and some images may overlap between the certain image group and the other image group.

[0117] Therefore, the information processing device 11 performs preprocessing on each of the different image groups (first image group to Nth image group) of the same object to be inspected, and inputs M preprocessed images generated from each image group into a trained learning model to obtain the output, thereby effectively determining defects occurring in the object to be inspected using a machine learning learning model based on the N image groups in which the characteristics of the defects are captured in different ways.

[0118] As one configuration example, in the information processing device 11, in at least one of i=1 to N, the ith image group includes two or more captured images. Therefore, in the information processing device 11, by including two or more captured images in one image group, the accuracy of the determination can be improved.

[0119] As one configuration example, in the information processing device 11, the i-th preprocessing includes one or more of cutting out an image portion, enlarging an image portion, combining two or more images, and copying an image. Therefore, the information processing device 11 can improve the accuracy of the determination by performing preprocessing suited to the determination content (for example, identification of the type of defect).

[0120] As one configuration example, in at least one of the information processing devices 11, where i=1 to N, the ith learning model group includes two or more learning models. Therefore, in the information processing device 11, by using two or more learning models for one image group, many determination processes can be performed, and the accuracy of the determination can be improved.

[0121] As an example configuration, in the information processing device 11, for each of i=1 to N, the ith learning model group includes one or more learning models that perform the type of inference suitable for the ith preprocessing. Therefore, in the information processing device 11, the accuracy of judgment can be improved by matching the preprocessing of the image group with the type of inference (for example, type of defect) performed in the learning model used for the image group.

[0122] As one configuration example, in the information processing device 11, in at least one of i=1 to N, the i-th pre-processing performs processing to combine captured images captured under the same imaging conditions. Therefore, in the information processing device 11, by synthesizing captured images captured under the same imaging conditions in a predetermined pre-processing, it becomes easier to judge defects and the accuracy of the judgment can be improved.

[0123] As one configuration example, in the information processing device 11, the object to be inspected is a glass plate in the manufacturing process. The plurality of captured images are images captured by imaging units of inspection machines arranged at different positions in the manufacturing process. Therefore, the information processing device 11 can make highly accurate judgments regarding defects that occur in glass plates during the manufacturing process.

[0124] As one configuration example (examples of FIGS. 8A, 8B, and 8C), the information processing device 11 has the following configuration. N=3. In the manufacturing process, downstream of the first and second inspection machines are the third, fourth, and fifth inspection machines. The first image group includes a first image captured by a first inspection machine under dark-field reflective system imaging conditions, a second image captured by a second inspection machine under bright-field reflective system imaging conditions, a third image captured by a third inspection machine under dark-field reflective system imaging conditions, and a fourth image captured by a fourth inspection machine under dark-field reflective system imaging conditions. The second image group includes a fifth image captured by a fifth inspection machine under imaging conditions of a bright-field vertical optical system. The third image group includes a first image, a second image, a third image, a fourth image, and a fifth image. The first learning model group includes one or more learning models that perform inference suitable for evaporative defects and one or more learning models that perform inference suitable for back surface defects. The second learning model group includes one or more learning models that perform inference suitable for a microdefect system. The third learning model group includes one or more learning models that perform inference suitable for a small linear defect system. Therefore, the information processing device 11 can make highly accurate judgments regarding defects that occur in glass plates during the manufacturing process.

[0125] As one configuration example (examples of FIGS. 8A, 8B, and 8C), the information processing device 11 has the following configuration. M=3. As the learning model, a learning model corresponding to RGB input is used. Therefore, even when a black and white captured image is acquired, the information processing device 11 can use a learning model corresponding to RGB input to make highly accurate judgments regarding defects that occur in glass plates during the manufacturing process.

[0126] In this embodiment, it is also possible to provide a method of processing performed in the information processing device 11. As one configuration example, in an information processing method, an information processing device performs the following processes: acquiring a first image group to an Nth image group extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions; performing an i-th preprocessing process based on the captured image included in the i-th image group for each of i = 1 to N to generate M preprocessed images; and inputting the M preprocessed images generated by the preprocessing process for each of i = 1 to N into a trained learning model included in the i-th learning model group of machine learning to obtain output from the learning model. Here, N is an integer equal to or greater than 2. i is a variable that is an integer equal to or greater than 1 and equal to or less than N, and M is an integer equal to or greater than 1. The first to Nth image groups each include one or more captured images and are different from one another. Each of the i-th learning model groups includes one or more learning models.

[0127] In this embodiment, it is also possible to provide a device (information processing device) that performs learning (machine learning) to generate the above-described trained learning model. As an example of configuration, the information processing device has the following configuration. The information processing device acquires a first image group to an N-th image group extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions, where N is an integer of 2 or more. The information processing device performs the ith preprocessing based on the captured image included in the ith image group, for each of i=1 to N, to generate M preprocessed images, where i is a variable that is an integer between 1 and N, and M is an integer that is 1 or greater. The information processing device inputs, for each of i = 1 to N, M preprocessed images generated by preprocessing into an untrained learning model included in the i-th learning model group of machine learning, obtains output from the learning model, and trains the learning model based on the output. The first to Nth image groups each include one or more captured images and are different from one another. Each of the i-th learning model groups includes one or more learning models.

[0128] In addition, this embodiment can also provide a processing method performed in such an information processing device. The above-described learning mode of the learning model is an example, and is not necessarily limited to the above example. Furthermore, when learning the learning model, for example, training data may be used, or training data may not be used.

[0129] A program for implementing the functions of any of the components of any of the above-described devices may be recorded on a computer-readable recording medium and loaded into a computer system for execution. The term "computer system" as used herein includes hardware such as an operating system or peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and compact discs (CDs) or read-only memories (ROMs), as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory within a computer system that acts as a server or client when a program is transmitted over a network such as the Internet or a communication line such as a telephone line. Such volatile memory may be, for example, random access memory (RAM). The recording medium may also be, for example, a non-transitory recording medium.

[0130] The above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, the above program may be a so-called differential file that can realize the above-mentioned functions in combination with a program already recorded in a computer system. A differential file may also be called a differential program.

[0131] Furthermore, the functions of any of the components in any of the above-described devices may be implemented by a processor. For example, each process in the embodiments may be implemented by a processor operating based on information such as a program and a computer-readable recording medium storing information such as the program. Here, the functions of each unit of the processor may be implemented by, for example, individual hardware, or may be implemented by integrated hardware. For example, the processor may include hardware, and the hardware may include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the processor may be configured using one or more circuit devices mounted on a circuit board, or one or both of one or more circuit elements. An integrated circuit (IC) or the like may be used as the circuit device, and a resistor or a capacitor may be used as the circuit element.

[0132] Here, the processor may be, for example, a CPU. However, the processor is not limited to a CPU, and various types of processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) may be used. The processor may also be, for example, a hardware circuit such as an ASIC (Application Specific Integrated Circuit). The processor may also be, for example, composed of multiple CPUs, or may be, for example, composed of a hardware circuit such as a multiple ASIC. The processor may also be, for example, composed of a combination of multiple CPUs and a hardware circuit such as a multiple ASIC. The processor may also include, for example, one or more of an amplifier circuit or a filter circuit that processes analog signals.

[0133] The embodiments of this disclosure have been described in detail above with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs within the scope that do not deviate from the gist of this disclosure.

[0134] [Note] (Configuration example 1) to (Configuration example 11) are shown.

[0135] (Configuration example 1) N is an integer equal to or greater than 2, and a first image group to an Nth image group are extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions; i is a variable that is an integer between 1 and N, and M is an integer that is 1 or greater, and for each of i=1 to N, an i-th preprocessing is performed based on the captured image included in the i-th image group to generate M preprocessed images; For each of i = 1 to N, M preprocessed images generated by the preprocessing are input to a trained learning model included in an i-th learning model group of machine learning, and an output from the learning model is obtained; the first image group to the Nth image group each include one or more of the captured images and are different from one another; Each of the i learning model groups includes one or more of the learning models. Information processing device.

[0136] (Configuration example 2) In at least one of i=1 to N, the i image group includes two or more of the captured images. The information processing device described in (Configuration Example 1).

[0137] (Configuration example 3) The i preprocessing includes one or more of cutting out an image portion, enlarging an image portion, combining two or more images, and copying an image. The information processing device according to (Configuration Example 1) or (Configuration Example 2).

[0138] (Configuration Example 4) In at least one of i=1 to N, the i learning model group includes two or more of the learning models. The information processing device according to any one of (Configuration Example 1) to (Configuration Example 3).

[0139] (Configuration Example 5) For each of i=1 to N, the i learning model group includes one or more learning models that perform inference of a type suitable for the i preprocessing. The information processing device according to (Configuration Example 4).

[0140] (Configuration Example 6) In at least one of i=1 to N, the i-th pre-processing includes processing for synthesizing the captured images captured under the same imaging conditions. The information processing device according to any one of (Configuration Example 1) to (Configuration Example 5).

[0141] (Configuration Example 7) the object to be inspected is a glass plate in a manufacturing process, the plurality of captured images are images captured by imaging units of inspection machines disposed at different positions in the manufacturing process, The information processing device described in (Configuration Example 1).

[0142] (Configuration Example 8) N=3, In the manufacturing process, a third inspection machine, a fourth inspection machine, and a fifth inspection machine are located downstream of the first inspection machine and the second inspection machine, the first image group includes a first image captured by the first inspection machine under imaging conditions of a dark-field reflective system, a second image captured by the second inspection machine under imaging conditions of a bright-field reflective system, a third image captured by the third inspection machine under imaging conditions of a dark-field reflective system, and a fourth image captured by the fourth inspection machine under imaging conditions of a dark-field reflective system, the second image group includes a fifth image captured by the fifth inspection machine under imaging conditions of a bright-field vertical optical system; the third image group includes the first image, the second image, the third image, the fourth image, and the fifth image; the first learning model group includes one or more learning models that perform inference suitable for evaporative defects and one or more learning models that perform inference suitable for back surface defects; the second learning model group includes one or more of the learning models that perform inference suitable for a small defect system; The third learning model group includes one or more of the learning models that perform inference suitable for a small linear defect system. The information processing device according to (Configuration Example 7).

[0143] (Configuration Example 9) M=3, As the learning model, a learning model corresponding to RGB input is used. The information processing device according to (Configuration Example 8).

[0144] (Configuration Example 10) The information processing device A process of acquiring a first image group to an N-th image group, where N is an integer equal to or greater than 2, extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions; i is a variable that is an integer between 1 and N, and M is an integer that is 1 or more, and for each of i=1 to N, a process of performing an i-th pre-processing based on the captured image included in the i-th image group to generate M pre-processed images; For each of i=1 to N, a process of inputting the M preprocessed images generated by the preprocessing into a trained learning model included in an i-th learning model group of machine learning, and acquiring an output from the learning model; and the first image group to the Nth image group each include one or more of the captured images and are different from one another; Each of the i learning model groups includes one or more of the learning models. Information processing methods.

[0145] (Configuration Example 11) N is an integer equal to or greater than 2, and a first image group to an Nth image group are extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions; i is a variable that is an integer between 1 and N, and M is an integer that is 1 or greater, and for each of i=1 to N, an i-th preprocessing is performed based on the captured image included in the i-th image group to generate M preprocessed images; For each of i=1 to N, M preprocessed images generated by the preprocessing are input to an untrained learning model included in an i-th learning model group of machine learning, an output from the learning model is obtained, and learning of the learning model is performed based on the output; the first image group to the Nth image group each include one or more of the captured images and are different from one another; Each of the i learning model groups includes one or more of the learning models. Information processing device. [Explanation of symbols]

[0146] 11...information processing device, 131...input unit, 132...output unit, 133...communication unit, 134...storage unit, 135...control unit, 151...operation unit, 152...display unit, 171...learning control unit, 172...judgment control unit, 211...learning device, 212...judgment device, 301...information processing system, 411-414...imaging unit, 421-424...lighting, 1001...information processing unit, 1011...processor, 1012... Operation device, 1013...display device, 1014...storage device, 1015...memory, 1016...input / output interface, 1017...network interface, 1021...bus, 2011...manufacturing process, A1...learning model, A1a...learning model under training, A1b...trained learning model, A2...teaching data, B1...first inspection machine, B2...second inspection machine, B3...third inspection machine, B4...third 4 inspection machines, B5...5th inspection machine, C1 to C4...object to be inspected, G1...1st image, G2...2nd image, G3...3rd image, G4...4th image, G5...5th image, H1...1st preprocessed image, H2...2nd preprocessed image, H3...3rd preprocessed image, H21...21st preprocessed image, H22...22nd preprocessed image, H23...23rd preprocessed image, I1...1st inference image, I2...2nd inference image, I3...3rd inference image Image, I11...11th image for inference, I12...12th image for inference, I13...13th image for inference, I21...21st image for inference, I22...22nd image for inference, I23...23rd image for inference, J1...21-1st image, J2...21-2nd image, J3...21-3rd image, J4...21-4th image, K1...22-1st image, K2...22-2nd image, K3...22-3rd image, K4...22-4th image

Claims

1. N is an integer equal to or greater than 2, and a first image group to an Nth image group are extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions; i is a variable that is an integer between 1 and N, and M is an integer that is 1 or greater, and for each of i=1 to N, an i-th preprocessing is performed based on the captured image included in the i-th image group to generate M preprocessed images; For each of i = 1 to N, the M preprocessed images generated by the preprocessing are input to a trained learning model included in an i-th learning model group of machine learning, and an output from the learning model is obtained; the first image group to the Nth image group each include one or more of the captured images and are different from one another; Each of the i learning model groups includes one or more of the learning models. Information processing device.

2. In at least one of i=1 to N, the i-th image group includes two or more of the captured images. The information processing device according to claim 1 .

3. The i-th preprocessing includes one or more of cutting out an image portion, enlarging an image portion, combining two or more images, and copying an image.

3. The information processing device according to claim 1.

4. For at least one of i = 1 to N, the i learning model group includes two or more of the learning models.

3. The information processing device according to claim 1.

5. For each of i = 1 to N, the i learning model group includes one or more learning models that perform inference of a type suitable for the i preprocessing. The information processing device according to claim 4 .

6. In at least one of i=1 to N, the i-th pre-processing performs a process of synthesizing the captured images captured under the same imaging conditions.

3. The information processing device according to claim 1.

7. the object to be inspected is a glass plate in a manufacturing process, the plurality of captured images are images captured by imaging units of inspection machines disposed at different positions in the manufacturing process, The information processing device according to claim 1 .

8. N=3, In the manufacturing process, a third inspection machine, a fourth inspection machine, and a fifth inspection machine are located downstream of the first inspection machine and the second inspection machine, the first image group includes a first image captured by the first inspection machine under imaging conditions of a dark-field reflective system, a second image captured by the second inspection machine under imaging conditions of a bright-field reflective system, a third image captured by the third inspection machine under imaging conditions of a dark-field reflective system, and a fourth image captured by the fourth inspection machine under imaging conditions of a dark-field reflective system, the second image group includes a fifth image captured by the fifth inspection machine under imaging conditions of a bright-field vertical optical system; the third image group includes the first image, the second image, the third image, the fourth image, and the fifth image; the first learning model group includes one or more learning models that perform inference suitable for evaporative defects and one or more learning models that perform inference suitable for back surface defects; the second learning model group includes one or more of the learning models that perform inference suitable for a small defect system; the third learning model group includes one or more of the learning models that perform inference suitable for a small linear defect system; The information processing device according to claim 7 .

9. M=3, As the learning model, a learning model corresponding to RGB input is used. The information processing device according to claim 8 .

10. The information processing device a process of acquiring a first image group to an N-th image group extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions, where N is an integer equal to or greater than 2; i is a variable that is an integer between 1 and N, and M is an integer that is 1 or greater, and for each of i=1 to N, a process of performing an i-th preprocessing based on the captured image included in the i-th image group to generate M preprocessed images; For each of i = 1 to N, a process of inputting the M preprocessed images generated by the preprocessing into a trained learning model included in an i-th learning model group of machine learning, and acquiring an output from the learning model; and the first image group to the Nth image group each include one or more of the captured images and are different from one another; Each of the i learning model groups includes one or more of the learning models. Information processing methods.

11. N is an integer equal to or greater than 2, and a first image group to an Nth image group are extracted from a plurality of captured images of the same object to be inspected under a plurality of different imaging conditions; i is a variable that is an integer between 1 and N, and M is an integer that is 1 or greater, and for each of i=1 to N, an i-th preprocessing is performed based on the captured image included in the i-th image group to generate M preprocessed images; For each of i = 1 to N, M preprocessed images generated by the preprocessing are input to an untrained learning model included in an i-th learning model group of machine learning, an output from the learning model is obtained, and learning of the learning model is performed based on the output; the first image group to the Nth image group each include one or more of the captured images and are different from one another; Each of the i learning model groups includes one or more of the learning models. Information processing device.

Images