Information processing system, information processor and machine learning method

Abstract

To provide an information processing system, an information processor and a machine learning method which can secondarily generate a teacher image having high expression accuracy.SOLUTION: An information processing system includes: division means for dividing a captured image; correction formula creation means for creating a correction formula on the basis of a feature amount expressing the images of division regions of the divided captured images, for each of the division regions; analogical image creation means for creating an analogical image expressed by the feature amount, for each of the division regions, according to the correction formulae of each of the division regions; and learning means for executing machine learning on a learning model using the analogical image as teacher data and the captured image as input data.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing apparatus, and a machine learning method, and more particularly to a machine learning technique in an information processing system that performs inspection based on images.

Background Art

[0002] In a system for inspecting defects or the like of an inspection target based on an image, for example, in machine learning of image recognition, a classified image is learned using the classified image as teacher data. Further, the classification is performed based on features extracted from the image. Then, a learned inspection system is generated by repeating learning so as to bring the error from the classified teacher image closer to 0, thereby improving the classification accuracy in inspection. Patent Document 1 describes that, for this teacher image, points corresponding to points within a predetermined distance from the points occupied by the teacher image obtained from an actual defect image in a feature space composed of a plurality of types of feature amounts are created as pseudo (secondary) teacher images. That is, a defect image as a teacher image is created secondarily, and the number of teacher data in machine learning is increased.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, depending on the actually obtained image, for example, the luminance and contrast may differ for each region in the image. In this case, in Patent Document 1, the obtained feature amounts may not accurately reflect the luminance and contrast for each region. As a result, the secondarily generated teacher image may have low expression accuracy.

[0005] Therefore, an object of the present disclosure is to provide an information processing system, an information processing apparatus, and a machine learning method that enable secondary generation of a teacher image with high expression accuracy. **Means for Solving the Problems**

[0006] The present disclosure includes a dividing means for dividing a captured image, a correction formula creating means for creating a correction formula for each divided region of the captured image based on a feature amount representing the image of the divided region, an analog image creating means for creating an analog image represented by a feature amount for each divided region according to the correction formula for each divided region, and a learning means for performing machine learning on a learning model using the analog image as teacher data and the captured image as input data. **Advantages of the Invention**

[0007] According to the present disclosure, it is possible to secondarily generate a teacher image with high expression accuracy. **Brief Description of the Drawings**

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail. Note that the components described in this embodiment are examples of the form of the present disclosure, and the scope of this disclosure is not limited only to them.

[0010] <First Embodiment> (Configuration of the Processing System) FIG. 1 is a diagram showing the configuration of a processing system 100 according to an embodiment of the present disclosure. The present processing system 100 is composed of a cloud server 200, an edge server 300, and a device 400 connected by a local area network 102 and the Internet 104. As will be described later with reference to FIG. 7 and the like, the cloud server 200 has a configuration related to machine learning, and the edge server 300 also has a configuration related to inference using a learned model, as will be described later with reference to FIG. 7 and the like. That is, the processing system 100 of the present embodiment functions as an information processing system that executes machine learning related to substrate inspection based on an image obtained by imaging a substrate, and performs inference related to the inspection using a learned model obtained by machine learning. Further, as described above, the cloud server 200 functions as an information processing device that generates teacher data (teacher images) and executes learning using the generated teacher data.

[0011] Device 400 includes various devices capable of network connection. For example, it includes smartphone 500, printer 600, client terminals 401 such as personal computers and workstations, digital camera 402, and the like. Device 400 is not limited to these types, and may be, for example, home appliances such as refrigerators, televisions, and air conditioners. These devices 400 are interconnected by local area network 102 and can be connected to the Internet 104 via router 103 installed in local area network 102. Here, router 103 is illustrated as a device connecting local area network 102 and the Internet 104, but it may have a wireless LAN access point function that constitutes local area network 102. In this case, in addition to connecting to router 103 via a wired LAN, each device 400 can be configured to connect via a wireless LAN and access local area network 102. For example, printer 600 and client terminal 401 may be configured to connect via a wired LAN, and smartphone 500 and digital camera 402 may be configured to connect via a wireless LAN. Each device 400 and edge server 300 can communicate with cloud server 200 via Internet 104 connected via router 103.

[0012] Edge server 300 and each device 400 can communicate with each other via local area network 102. Also, each device 400 can communicate with each other via local area network 102. Further, smartphone 500 and printer 600 can communicate via short-range wireless communication 101. As short-range wireless communication 101, wireless communication conforming to the Bluetooth (registered trademark) standard or NFC standard may be used. Also, smartphone 500 is connected to mobile phone line network 105, and can also communicate with cloud server 200 via this mobile phone line network 105.

[0013] Note that this system configuration is an example of the present disclosure and may have a different configuration. For example, although an example where the router 103 has an access point function has been shown, the access point may be configured with a device different from the router 103. Also, the connection between the edge server 300 and each device 400 may use connection means other than the local area network 102. For example, it may use wireless communication such as LPWA other than wireless LAN, ZigBee, Bluetooth (registered trademark), short-range wireless communication, etc., or wired connection such as USB, or infrared communication.

[0014] (Server) FIG. 2 is a block diagram showing the configurations of the cloud server 200 and the edge server 300. Here, it will be described that the hardware configurations of the cloud server 200 and the edge server 300 use common ones. The servers 200 and 300 are composed of a main board 210 that controls the entire device, a network connection unit 201, and a hard disk unit 202.

[0015] On the main board 210, a CPU 211, a program memory 213, a data memory 214, a hard disk control circuit 216, a GPU 217, and a network control circuit are arranged. The CPU 211 in the form of a microprocessor operates according to a control program stored in the program memory 213 connected via an internal bus 212 and the content of the data memory 214. The CPU 211 controls the network connection unit 201 via the network control circuit 215 to connect to networks such as the Internet 101 and the local area network 102 and communicate with other devices. The CPU 211 reads and writes data to the hard disk unit 202 connected via the hard disk control circuit 216.

[0016] The hard disk unit 202 stores an operating system to be loaded into the program memory 213 and used, control software for the servers 200 and 300, as well as various types of data.

[0017] A GPU 217 is connected to the main board 210, and it is possible to execute various arithmetic processes in place of the CPU 211. Since the GPU 217 can perform efficient arithmetic operations by processing more data in parallel, it is effective to use the GPU 217 for processing when performing learning multiple times using a learning model such as deep learning. Therefore, in the present embodiment, the GPU 217 is used in addition to the CPU 211 for the processing by the learning unit 251 described later. Specifically, when executing a learning program including a learning model, learning is performed by the CPU 211 and the GPU 217 cooperating to perform arithmetic operations. Note that the processing of the learning unit 251 may be performed by only the CPU 211 or the GPU 217. Also, the inference unit 351 described later may also use the GPU 217 in the same manner as the learning unit 251.

[0018] Also, in the present embodiment, the cloud server 200 and the edge server 300 have been described as using a common configuration, but the present disclosure is not limited to this configuration. For example, the cloud server 200 may be configured to have a GPU 217 but the edge server 300 may not, or they may be configured to use GPUs 217 with different performances.

[0019] (Smartphone) FIG. 3 is an external view of a smartphone 500 which is an example of the device 400. A smartphone is a multifunctional mobile phone equipped with a camera, a web browser, an email function, etc. in addition to the functions of a mobile phone. The short-range wireless communication unit 501 is a unit that performs short-range wireless communication and communicates with the short-range wireless communication unit of a communication partner within a predetermined distance. The wireless LAN unit 502 is a unit for connecting to and communicating with the local area network 102 (see FIG. 1) using wireless LAN and is disposed inside the device. The line connection unit 503 is a unit for connecting to a mobile phone line and communicating and is disposed inside the device. The touch panel display 504 has both an LCD display mechanism and a touch panel operation mechanism and is provided on the front surface of the smartphone 500. A typical operation method is to display button-shaped operation parts on the touch panel display 504 and issue an event of button pressing when the user performs a touch operation on the touch panel display 504. The power button 505 is used to turn on and off the power of the smartphone.

[0020] (Printer) FIG. 4 is a diagram showing a printer 600 which is also an example of the device 400. In the present embodiment, a multifunction printer (MFP) having a scanner and other functions is taken as an example of the printer. FIG. 4(a) is a perspective view schematically showing the overall external appearance of the printer 600. The document table 601 is a glass-like transparent table and is used when placing a document and reading it with a scanner. The document table platen 602 is a cover for pressing the document against the document table so that the document does not float when reading with the scanner and preventing external light from entering the scanner unit. The printing paper insertion slot 603 is an insertion slot for setting papers of various sizes. The papers set in the printing paper insertion slot 603 are conveyed one by one to the printing unit, printed as desired, and discharged from the printing paper discharge port 604.

[0021] FIG. 4(b) is a diagram schematically showing the appearance of the upper surface of the printer 600. An operation panel 605 and a short-range wireless communication unit 606 are provided above the platen 602. The short-range wireless communication unit 606 is a unit for performing short-range wireless communication, and communicates with the short-range wireless communication unit of a communication partner within a predetermined distance. The wireless LAN antenna 607 has an antenna embedded therein for connecting to and communicating with the local area network 102 using the wireless LAN.

[0022] (Processing Configuration of the Smartphone) FIG. 5 is a block diagram showing the configuration of the smartphone 500 shown in FIG. 3. The smartphone 500 includes a main board 510 that controls the entire device, a short-range wireless communication unit 501, a wireless LAN unit 502, and a line connection unit 503.

[0023] On the main board 510, a CPU 511, a program memory 513, a data memory 514, a wireless LAN control circuit 515, a short-range wireless communication control circuit 516, and a line control circuit 517 are arranged. Also, on the main board 510, an operation unit control circuit 518, a camera 519, and a non-volatile memory 521 are arranged. The CPU 511 in the form of a microprocessor operates according to a control program stored in the program memory 513 in the form of a ROM connected via an internal bus 512 and the content of the data memory 514 in the form of a RAM.

[0024] The CPU 511 controls the wireless LAN unit 502 via the wireless LAN control circuit 515 to perform wireless LAN communication with other communication terminal devices. The CPU 511 can detect a connection with other short-range wireless communication terminals or perform data transmission and reception with other short-range wireless communication terminals by controlling the short-range wireless communication unit 501 via the short-range wireless communication control circuit 516. Also, the CPU 511 can connect to the mobile phone line network 105 and perform calls and data transmission and reception by controlling the line connection unit 503 via the line control circuit 517. The CPU 511 can perform a desired display on the touch panel display 504 or accept operations from the user by controlling the operation unit control circuit 518.

[0025] The CPU 511 controls the camera 519 to take an image and stores the taken image in the image memory 520 in the data memory 514. Also, in addition to the taken image, it is possible to store an image acquired from the outside through the mobile phone line, the local area network 102, or the short-range wireless communication 101 in the image memory 520, or conversely, to transmit it to the outside.

[0026] The non-volatile memory 521 is composed of a flash memory or the like and stores data that needs to be saved even after the power is turned off. For example, in addition to phone book data, various communication connection information, and device information of devices connected in the past, image data that needs to be saved, or application software that realizes various functions in the smartphone 500 is stored.

[0027] (Processing Configuration of Printer) FIG. 6 is a block diagram showing the configuration of the printer 600 shown in FIG. 4. The printer 600 includes a main board 610 that controls the entire apparatus, a wireless LAN unit 608, and a short-range wireless communication unit 606. The main board 610 is provided with a CUP 611, a program memory 613, a data memory 614, a scanner 615, a printing unit 617, a wireless LAN control circuit 618, a short-range wireless communication control circuit 619, and an operation unit control circuit 620. The CPU 611 in the form of a microprocessor operates based on a control program stored in the program memory 613 in the form of a ROM connected via an internal bus 612 and data in the data memory 614 in the form of a RAM.

[0028] The CPU 611 controls the scanner unit 615 to read a document and stores it in the image memory 616 in the data memory 614. Further, the CPU 611 can control the printing unit 617 to print the image in the image memory 616 in the data memory 614 on a recording medium. The CPU 611 controls the wireless LAN unit 608 through the wireless LAN communication control unit 618 to perform wireless LAN communication with other communication terminal devices.

[0029] Also, the CPU 611 can detect a connection with other short-range wireless communication terminals or perform data transmission and reception with other short-range wireless communication terminals by controlling the short-range wireless communication unit 606 via the short-range wireless communication control circuit 619.

[0030] The CPU 611 can control the operation unit control circuit 620 to display the state of the printer 600 or a function selection menu on the operation panel 605 and receive operations from the user. The operation panel 605 is provided with a backlight, and the CPU 611 can control the lighting and extinguishing of the backlight via the operation unit control circuit 621. When the backlight of the operation panel 605 is extinguished, the display on the operation panel 605 becomes difficult to see, but the power consumption of the printer 600 can be suppressed.

[0031] (Software Configuration) FIG. 7 is a block diagram showing the software configuration of the processing system 100 described above in FIG. 1. In this figure, only the software components related to the learning and inference processes in this embodiment are shown, and other software modules are not shown. For example, the operating system operating on each device or server, various middleware, applications for maintenance, etc. are omitted from the illustration.

[0032] The cloud server 200 includes a learning data generation unit 250, a learning unit 251, and a learning model 252. The learning data generation unit 250 is a module that generates learning data processable by the learning unit 251 from the data received from the outside. The learning data is a pair of the input data X of the learning unit 251 and the teacher data T indicating the correct answer of the learning result. The learning unit 251 is a program module that executes learning on the learning model 252 using the learning data received from the learning data generation unit 250. The learning model 252 accumulates the learning results obtained by the learning unit 251. Here, an example of realizing the learning model 252 as a neural network will be described. By optimizing the weighting parameters between the nodes of the neural network, it is possible to classify the input data or determine the evaluation value. The accumulated learning model 252 is distributed to the edge server 300 as a learned model and used for the inference process in the edge server 300.

[0033] The edge server 300 includes a data collection and provision unit 350, an inference unit 351, and a learned model 352. The data collection and provision unit 350 is a module that transmits the data received from the device 400 and the data collected by the edge server 300 itself to the cloud server 200 as a data group for learning. The inference unit 351 is a program module that performs inference using the learned model 352 based on the data sent from the device 400 and returns the result to the device 400. The input data of the inference unit 351 is the data sent from the device 400. The learned model 352 is used for the inference performed by the edge server 300. Assume that the learned model 352 is also realized as a neural network similar to the learning model 252. The learned model 352 may be the same as the learning model 252 as described later, or may extract and use a part of the learning model 252. The learned model 352 is obtained by transmitting the learning model 252 stored in the cloud server 200 and storing the transmitted learning model 252. The learned model 352 may be the one in which all of the learning model 252 is transmitted, or may be the one in which only a part necessary for the inference at the edge server 300 is extracted and transmitted from the learning model 252.

[0034] The device 400 includes an application unit 450 and a data transmission / reception unit 451. The application unit 450 is a module that realizes various functions executed by the device 400 and is a module that utilizes the mechanism of learning and inference by machine learning. The data transmission / reception unit 451 is a module that requests learning or inference from the edge server 300. At the time of learning, the data used for learning is transmitted to the data collection and provision unit 350 of the edge server 300 according to the instruction of the application unit 450. Also, at the time of inference, the data used for inference is transmitted to the edge server 300 according to the instruction of the application unit 450, and the result is received and returned to the application unit 450.

[0035] In addition, in this embodiment, the learning model 252 learned by the cloud server 200 is transmitted to the edge server 300 as the learned model 352 and used for inference. However, the embodiment is not limited to this form. Whether learning and inference are executed on any of the cloud server 200, the edge server 300, and the device 400 can be determined according to the distribution of hardware resources, the amount of calculation, and the amount of data communication. Alternatively, it may be configured to dynamically change according to the increase or decrease in the distribution of these resources, the amount of calculation, and the amount of data communication. When the entities performing learning and inference are different, the inference side can reduce the logic used only for inference and the capacity of the learned model 352, or configure it to be executed at a higher speed.

[0036] (Learning model) FIG. 8 is a conceptual diagram showing the input / output structure when using the learning model 252 and the learned model 352.

[0037] FIG. 8(a) shows the relationship between the learning model 252 and its input / output data during learning.

[0038] The input data X (801) is data input to the input layer of the learning model 252. Details of the input data X in this embodiment will be described later with reference to FIG. 9 and the like. As a result of recognizing the input data X (801) using the learning model 252 which is a machine learning model, the output data Y (803) is output. During learning, teacher data T (802), which will also be described later with reference to FIG. 9 and the like, is given as correct data for the recognition result of the input data X. Then, by giving the output data Y (803) and the teacher data T (802) to the loss function 804, the deviation amount L (805) from the correct answer of the recognition result is obtained. In this way, for a large number of learning data consisting of input data and teacher data, the coupling weight coefficients between the nodes of the neural network in the learning model 252 are updated so that the deviation amount L becomes smaller. In this embodiment, the error backpropagation method is used to adjust the coupling weight coefficients between the nodes of each neural network so that the above error becomes smaller.

[0039] The specific algorithm of this machine learning is not limited to the error backpropagation method. For example, the nearest neighbor method, naive Bayes method, decision tree, support vector machine, etc. can also be used. In addition, deep learning (deep neural network) that uses a neural network to generate its own feature quantities and connection weight coefficients for learning can also be used.

[0040] FIG. 8(b) shows the relationship between the learned model 352 and its input / output data during inference. The input data X(801) is the data of the input layer of the learned model 352. The details of the input data X(801) in the present embodiment will be described later with reference to FIG. 15 and the like. As a result of recognizing the input data X(801) using the learning model 252 which is a machine learning model, the output data Y(803) is output. During inference, this output data Y(803) is used as the inference result. Note that although the learned model 352 during inference has been described as having a neural network equivalent to the learning model 252 during learning, it is also possible to prepare, as the learned model 352, a model obtained by extracting only the necessary parts for inference. This makes it possible to reduce the data amount of the learned model 352 and shorten the neural network processing time during inference.

[0041] An embodiment of the present disclosure based on the configuration described above will be described.

[0042] <First Embodiment> (Learning Process) FIG. 9 is a flowchart showing the learning process according to the first embodiment of the present disclosure. This process is executed by the learning unit 251 of the cloud server 200. The process of FIG. 9 is started when the user inputs an instruction to start substrate inspection to the information processing apparatus. In FIG. 9, steps S901 to S904 show a process of generating an analog image as teacher data based on the captured image, and step S905 shows a learning phase using the generated analog image as teacher data. Note that the symbol "S" in the description of each process means a step in the flowchart diagram (the same applies to the flowchart in this specification hereinafter).

[0043] First, in S901, an imaging image 901 is acquired. The imaging image 901 is an image obtained by the digital camera 402 in the device 400 capturing the inspection area of the substrate to be inspected with infrared light. In this embodiment, the imaging image uses the digital camera 402, but an infrared camera or the like may also be used. The imaging image is obtained as binary data for each pixel constituting the imaging image. The acquisition of this imaging image is performed by five times of imaging with different luminances in order to obtain five levels of luminance (see FIG. 12(b)) as described later. Specifically, it is performed by imaging the inspection area with different luminances by varying the illuminance on the inspection area of the substrate (the following formula (1)).

[0044] Here, in this embodiment, the substrate to be inspected constitutes the recording head of the printer. That is, the processing system 100 of this embodiment executes inspection of the substrate in the manufacturing process of the recording head. In this inspection area, for example, wirings, adhesive parts, etc. exist on the substrate, and the luminance and contrast in the imaging image are often not uniform throughout the inspection area depending on the sites where they exist. Therefore, as will be described later with reference to FIG. 10 and the like, the embodiment of the present disclosure divides the inspection area into a plurality of areas, and acquires the luminance and contrast for each divided area. Then, by correcting the luminance and contrast for each divided area of the imaging image, a secondary image (hereinafter also referred to as an analog image) with respect to the original imaging image is created, and this is used as a teacher image for machine learning.

[0045] Note that the elements representing the imaging image and the feature amounts to be corrected are not limited to luminance and contrast, of course. For example, lightness, chroma, etc. can also be used. Also, in this embodiment, an image is acquired using infrared light, but imaging may be performed with visible light.

[0046] FIG. 10 is a diagram showing the captured image obtained in S901. The captured image 901 obtained by imaging the inspection area on the substrate includes, as an example, an area 902, an area 903, and an area 904. And for each of these areas, depending on the presence of wirings and adhesive parts, the thickness, refractive index, surface roughness, etc. are different, and thus they may have different brightness and contrast. For example, the contrast and brightness represented by the area 902 may be lower than the contrast and brightness represented by the area 903.

[0047] Next, in S902, the image is divided for each area. FIG. 11 shows this divided captured image, and the captured image 901 is divided into an area 902, an area 903, and an area 904. This division can be determined in advance, for example, according to the differences in substrate component patterns such as circuit parts and wiring parts in the inspection area of the substrate. Here, if the captured image is not divided, when generating a teacher image by changing (correcting) the brightness and contrast based on the captured image, the brightness and contrast will be changed (corrected) uniformly between areas, and there is a possibility of generating a teacher image with a biased tendency. Therefore, in this embodiment, the captured image is divided so that the tendency of brightness and the like can be easily reproduced for each divided area, and a teacher image is generated by performing correction for each divided area. Note that the division of the area may be performed in accordance with the pattern boundary as described above, or the area to be divided may be determined from the color tone around the boundary.

[0048] Next, in S903, a correction formula is created based on each of the images divided for each area. That is, based on the captured images of the divided area 902, the divided area 903, and the divided area 904, a correction formula is created for each divided area. And in S904, an analog image is created for each divided area according to the created correction formula for each divided area, and this is used as the teacher image. Hereinafter, for the sake of simplicity of explanation, the creation of the correction formula and the correction of the brightness and contrast based thereon for the divided area 902 shown in FIG. 11 will be described.

[0049] <Creation of Brightness Correction Formula> First, the creation of the luminance correction formula will be described. The luminance L of an image can be expressed by the following formula (1) using the illuminance W and the reflectance R.

[0050] L = R / πW...(1) Here, the reflectance R of a single layer can be expressed by the following formula (2) using the refractive index N.

[0051] R = ((1 - N) / (1 + N))^2...(2) Based on the luminance L of the divided region 902 obtained according to the above formulas (1) and (2), the correction formula C shown in FIG. 12(a) is created. Specifically, it is as follows.

[0052] Regarding the divided region 902 of the captured image, the luminance L of each pixel constituting the divided region 902 is obtained, and its histogram is obtained. FIG. 12(b) shows this histogram, which is represented as histogram 16. That is, histogram 16 shows that the luminance L obtained from the divided region 902 is distributed in five levels of values, and the number of pixels of each luminance is one of the three levels of the number of pixels as shown in the figure.

[0053] Next, among the five levels of luminance L values obtained in this way, for the three luminances L indicated by (11), (12), and (13) in FIG. 12(b), the reflectance R obtained in advance according to formula (2) based on the refractive index N of the region is corresponded. This correspondence is represented as the points of black circles 11, 12, and 13 in FIG. 12(a). That is, the points of black circles 11, 12, and 13 have the luminances (11), (12), and (13) in FIG. 12(b), respectively. By obtaining the first-order correlation of these three black circle points, the correction formula C indicated by the broken line is obtained (S903). In this embodiment, the correlation formula is obtained using three luminances, but the number is not limited to this. To increase the correction accuracy more, the correlation formula may be obtained using more than three luminances. In this case, the correlation formula may be a second-order or higher-order correlation. Also, since the reflectance may be affected by the surface roughness of the substrate, when considering the surface roughness, it is desirable to make the correlation formula second-order or higher.

[0054] (Brightness correction) Next, the brightness is corrected (changed) using the correction formula C obtained as described above, and an analog image is created (S904). Specifically, in FIG. 12(a), an analog image having the brightness of white circles 14 and 15 is created so as to lie on the straight line indicating the correction formula C. The analog image created in this way is used as a teacher image in the machine learning in the next S905. Note that although two analog images are shown in the example of FIG. 12(a), this is for simplification of illustration and explanation, and the number of analog images created is of course not limited to this.

[0055] In this case, it is preferable to create an analog image in which the brightness is determined in consideration of the assumed variation in the reflectance R. For example, in FIG. 12(a), it is preferable to create a plurality of analog images in which the brightness is determined corresponding to the maximum reflectance and the minimum reflectance. Further, the reflectance R can also be created by fixing two of the thickness, refractive index, and surface roughness and varying one main item. For example, when the surface roughness is a main factor in variations such as brightness, it is preferable to obtain a correction formula from the captured image in which the surface roughness is varied. In the present embodiment, one item is varied, but various variations of images can also be created by creating analog images with two or more items variable. Furthermore, the brightness may be determined at a reflectance exceeding the assumed range. Thereby, an analog image corresponding to a wider variation in brightness can be created.

[0056] FIG. 13 is a diagram for explaining the analog image created as described above. In FIG. 13, the captured image 906 of the divided region 902 becomes an analog image 907 (white circle 14 in FIG. 12(a)) with increased brightness and an analog image 908 (white circle 15 in FIG. 12(a)) with decreased brightness by the above-described brightness correction (change) process.

[0057] (Contrast correction) The contrast correction (modification) is performed on the luminance histogram 16 shown in FIG. 12(b). The correction (modification) methods include distribution expansion or flattening. In this embodiment, a flattening process is performed. Specifically, the flattening is performed by narrowing or widening the luminance range while maintaining the area (total number of pixels) of the histogram 16. The luminance histograms 17 and 18 show the correction results by this flattening. The histogram 17 of the correction result shown in the figure shows that as the luminance range narrows, the number of pixels for each luminance in the narrowed luminance range increases, that is, the contrast increases. On the other hand, the histogram 18 of the correction result shows that as the luminance range expands, the number of pixels for each luminance in the widened luminance range decreases, that is, the contrast decreases.

[0058] FIG. 14 is a diagram showing an example of the contrast correction (modification) result for the divided region 902. As shown in the figure, the original image 906 (histogram 16 in FIG. 12(b)) of the divided region 902 becomes an analog image 909 with increased contrast (histogram 17 in FIG. 12(b)) or an analog image 910 with decreased contrast (histogram 18 in FIG. 12(b)) by the correction.

[0059] Note that shading correction may be performed to suppress luminance unevenness in the captured image. This shading correction is preferably performed before creating the correction formula in S903.

[0060] The above description of creating the correction formula and the correction using it was made for the divided region 902 shown in FIG. 11, but it is obvious from the above description that the same applies to the other divided regions 903 and 904. Therefore, the description of the other divided regions is omitted.

[0061] The analog images for each divided region described above are stored in a predetermined memory as data of one teacher image based on luminance and contrast (S904). Thereby, more analog images (teacher images) can be obtained based on a relatively small number of captured images (original images).

[0062] Referring again to FIG. 9, when creating an analogy image as described above, in the following S905, machine learning is performed using a plurality of analogy images for each divided region as teacher images. Here, as shown in FIG. 8(a), the captured image of the inspection region is used as the input data X (801). This captured image of the inspection region is also divided into three regions 902, 903, and 904 in the same manner as when creating the analogy image described above, and for each of these divided regions, the learning described above is performed with reference to FIG. 8(a). That is, learning is performed for each corresponding region of the region-divided captured image (input data X: 801) and the analogy image (teacher data T: 802), and the learning model 252 is generated.

[0063] In this embodiment, an analogy image is created by correcting binary data. However, in the case of a captured image using visible light, an analogy image may be created by correcting the RGB values. In this case, the color tone range of each pattern is determined from the process range to create an analogy image.

[0064] (Inference process) FIGS. 15(a) and (b) are diagrams for explaining the inference process of the first embodiment of the present disclosure, and are processes executed by the inference unit 351 and the learned model 352 of the edge server 300. FIG. 15(a) shows a flowchart of the inference process, and FIG. 15(b) shows the concept of the process.

[0065] First, a determination image 911 is obtained in the same manner as when obtaining the captured image described above, and the determination image 911 is divided into regions (S1501). As a result, the determination image 911 of the inspection region on the substrate is divided into three divided regions 902, 903, and 904. Next, inference (351: Inference A, Inference B, Inference C) is performed by the learned model 352 for each region (learned models A, B, and C) (S1502).

[0066] Thereafter, a final determination is made based on the number of votes of the classification classes by the learned model 352 for each region (S1503). Specifically, it is as follows.

[0067] In this embodiment, two-class classification is performed using the learned models 352 (learned models A, B, and C) of the three divided regions. That is, in the inspection, the determination that the defect of the substrate is unacceptable is tentatively defined as the E classification determination, and the determination that is acceptable including the case where there is no defect is defined as the F classification determination. Then, among the respective inference units 351 (Inference A, Inference B, Inference C) related to the three divided regions, if there is even one E classification determination, it is determined as the E classification determination, that is, a defective determination indicating that there is a defect in the substrate inspection. When all of the respective inference units 351 (Inference A, Inference B, Inference C) related to the three divided regions are F classification determinations, it is determined as a good determination.

[0068] As described above, according to the first embodiment of the present disclosure, it is possible to secondarily create more teacher images with a relatively small number of captured images and to increase the expression accuracy of the teacher images.

[0069] <Second Embodiment> The second embodiment of the present disclosure will be described. Note that mainly the parts different from the first embodiment will be described.

[0070] (Learning Process) FIG. 16 is a flowchart showing the learning process according to the second embodiment. In the first embodiment, the learned model 252 is created for each region from the region-divided captured image 901 and the analog image. However, in the second embodiment, after creating the region-divided analog images, the divided images are combined to create the analog image 905 (S1605).

[0071] Specifically, first, the captured image 901 (see FIG. 10) is acquired (S1601), and it is region-divided (S1602). By this division, the captured image 901 is divided into the region 902, the region 903, and the region 904. Further, a correction formula is created (S1603), and according to the correction formula, the analog image of each divided region is created (S1604).

[0072] Thereafter, analogous images for each divided region are combined to generate an analogous image (S1605). FIG. 17 is a diagram for explaining this combination. In this figure, the analogous images connected by thick solid lines or broken lines indicate combinations of analogous images for each divided region. As described above, in the present embodiment, analogous images for each divided region are mutually combined, and an analogous image obtained as this combination (including a combination of only one of brightness or contrast) is generated. In FIG. 17, the combination connected by the thick solid line is a combination for generating the analogous image 905 shown in this figure.

[0073] According to the second embodiment described above, a larger number of analogous images (teacher images) can be generated as compared with the first embodiment.

[0074] (Inference processing) FIG. 18 is a flowchart showing the inference processing according to the second embodiment of the present disclosure. In the first embodiment, the determination image 911 is inferred by the learned model for each divided region, but in the second embodiment, the determination image 911 is inferred without region division.

[0075] Specifically, first, one determination image is inferred by the learned model (S1801). Then, a final determination is made based on the number of votes of the classification classes of the learned model (S1802).

[0076] With such a configuration, the learning model can be reduced, and the efficiency of learning and inference can be improved.

[0077] <Third Embodiment> The third embodiment of the present disclosure will be described. Note that mainly the parts different from the first embodiment will be described.

[0078] (Image generation method) FIG. 19 is a diagram showing region division according to the third embodiment of the present disclosure. In the first embodiment, the region was divided according to each pattern. However, as shown in the figure, it may be divided into an image 912 of only the pattern and an image 913 of only the background. As shown in FIG. 20, after creating an analog image that is region-divided in the same manner as in the second embodiment from the divided images, they are combined again to create an analog image. Then, a learning model is created using the captured image and the analog image (teacher image). Furthermore, a learning model may be created while keeping the region division as in the first embodiment.

[0079] Note that a learning model may be created by combining the first to third embodiments. For example, an analog image 905 can also be created from images that are region-divided by the background and each pattern of the region.

[0080] By adopting such a configuration, the learning model can be reduced, and an information processing system having high determination accuracy can be created.

[0081] <Other Embodiments> The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and having a computer of the system or apparatus read and execute the program. The computer may have one or more processors or circuits, and may include a network of a plurality of separate computers or a plurality of separate processors or circuits in order to read and execute computer-executable instructions.

[0082] The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The processor or circuit may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0083] The present disclosure is suitable for semiconductor processes, particularly for recording element substrates for liquid ejection. Since multiple films are laminated in a state where multiple substrates overlap in the process of the recording element substrate, the behavior of the luminance and contrast of each region and pattern is intricately intertwined, and it is expected that the learning data for creating a learning model will become enormous. Therefore, by applying the above-described embodiments, it is possible to create an information processing system having high determination accuracy with a small number of images.

[0084] The present disclosure includes a configuration represented by the following information processing system.

[0085] <Configuration 1> Division means for dividing a captured image, Correction formula creation means for creating a correction formula for each divided region of the divided captured image based on a feature amount representing the image of the divided region, Analogy image creation means for creating an analogy image represented by the feature amount for each divided region according to the correction formula for each divided region, Learning means for performing machine learning on a learning model using the analogy image as teacher data and the captured image as input data, An information processing system characterized by comprising:

[0086] <Configuration 2> Inference means for inferring about an inspection image obtained by imaging using the learning model after the machine learning, Determination means for determining the inspection image based on the inference by the inference means, The information processing system according to Configuration 1, further comprising:

[0087] <Configuration 3> The learning model outputs output data as a result of recognizing the input data by machine learning. The information processing system according to Configuration 1 or 2, characterized by:

[0088] <Configuration 4> The information processing system according to any one of Configurations 1 to 3, wherein the captured image is captured using infrared light.

[0089] <Configuration 5> The information processing system according to any one of Configurations 1 to 4, wherein the correction formula is created from three or more captured images with variable thickness, surface roughness, and refractive index of the subject.

[0090] <Configuration 6> The information processing system according to any one of Configurations 1 to 5, wherein the captured image is captured using visible light.

[0091] <Configuration 7> The information processing system according to any one of Configurations 1 to 6, wherein the correction formula is created from the image data obtained by RGB correction.

[0092] <Configuration 8> The information processing system according to any one of Configurations 1 to 7, wherein the correction formula is created from the image data obtained by binarization correction.

[0093] <Configuration 9> The information processing system according to Configuration 7 or Configuration 8, wherein the correction formula is calculated by first-order correction.

[0094] <Configuration 10> The information processing system according to Configuration 7 or Configuration 8, wherein the correction formula is calculated by polynomial approximation correction.

[0095] <Configuration 11> The information processing system according to any one of Configurations 1 to 10, wherein the dividing means divides by each pattern of the subject.

[0096] <Configuration 12> The information processing system according to any one of Configurations 1 to 11, wherein the dividing means divides the subject into a pattern and a background.

[0097] <Configuration 13> The information processing system according to any one of Configurations 1 to 12, wherein the captured image is an image for inspecting a semiconductor process or an image for inspecting a recording element substrate capable of liquid ejection.

[0098] <Configuration 14> Input means for receiving an input of a captured image, Division means for dividing the captured image, Correction formula creation means for creating a correction formula based on a feature amount representing the image of the divided region for each divided region of the divided captured image, Analog image creation means for creating an analog image represented by the feature amount for each divided region according to the correction formula for each divided region, Learning means for performing machine learning on a learning model using the analog image as teacher data and the captured image input by the input means as input data. An information processing apparatus characterized by comprising the above.

[0099] <Configuration 15> A division step of dividing a captured image, A correction formula creation step of creating a correction formula based on a feature amount representing the image of the divided region for each divided region of the divided captured image, An analog image creation step of creating an analog image represented by the feature amount for each divided region according to the correction formula for each divided region, A learning step of performing machine learning on a learning model using the analog image as teacher data and the captured image as input data. A machine learning method characterized by comprising the above.

Explanation of Signs

[0100] 100 Processing system 252 Learning model 901 Captured image 902 Region 903 Region 904 Region 905 Analog image 911 determination image C correction formula T teacher data

Claims

1. a dividing means for dividing a captured image; a correction formula creating means for creating a correction formula for each divided area of the divided captured image based on a feature amount representing the image of the divided area; an analog image creating means for creating an analog image represented by the feature amount for each divided area according to the correction formula for each divided area; a learning means for performing machine learning on a learning model using the analog image as teacher data and the captured image as input data; An information processing system comprising the above.

2. an inference means for inferring about an inspection image obtained by imaging using the learning model after the machine learning; a determination means for determining the inspection image based on the inference by the inference means; The information processing system according to claim 1, further comprising the above.

3. The information processing system according to claim 1, wherein the learning model outputs output data as a result of recognizing the input data by machine learning.

4. The information processing system according to claim 1, wherein the captured image is captured using infrared light.

5. The information processing system according to claim 1, wherein the correction formula is created from three or more captured images with variable thickness, surface roughness, and refractive index of the subject.

6. The information processing system according to claim 1, wherein the captured image is captured using visible light.

7. The information processing system according to claim 1, wherein the correction formula is created from image data obtained by RGB correction.

8. The information processing system according to claim 1, wherein the correction formula is created from image data obtained by binarization correction.

9. The information processing system according to claim 7, wherein the correction formula is calculated by first-order correction.

10. The information processing system according to claim 7, wherein the correction formula is calculated by polynomial approximation correction.

11. The information processing system according to claim 1, wherein the dividing means divides for each pattern of the subject.

12. The information processing system according to claim 1, wherein the dividing means divides the subject pattern and the background.

13. The information processing system according to claim 1, wherein the captured image is an image for inspecting a semiconductor process or an image for inspecting a recording element substrate capable of liquid ejection.

14. Input receiving means for receiving an input of a captured image; Segmenting means for segmenting the captured image; Correction formula creating means for creating a correction formula based on a feature amount representing an image of a segmented area for each segmented area of the captured image; Analog image creating means for creating an analog image represented by the feature amount for each segmented area according to the correction formula for each segmented area; Learning means for performing machine learning on a learning model using the analog image as teacher data and the captured image input by the input receiving means as input data; An information processing apparatus characterized by comprising the above.

15. A segmentation step of segmenting a captured image; A correction formula creating step of creating a correction formula based on a feature amount representing an image of a segmented area for each segmented area of the captured image; An analog image creating step of creating an analog image represented by the feature amount for each segmented area according to the correction formula for each segmented area; A learning step of performing machine learning on a learning model using the analog image as teacher data and a captured image as input data; A machine learning method characterized by comprising the above.

Images