Printed image defect detection device and detection method

Abstract

To allow for discriminating defects that may exist in a printed image, including unknown defects.SOLUTION: A defect discrimination method for printed images comprises: preparing a learning model that has undergone machine learning using teacher images containing defects that may occur during printing and being associated with defect kinds, so as to be able to output similarity for each of defect kinds; then, acquiring an image of printed matter, which is printed, and preparing a target image being an inspection target; using, with respect to the target image, the learning model to acquire similarities to known defect kinds, of a defect present in the target image, and discriminating the defect present in the target image as at least one of the known defect kinds, using the similarities; and when updating the learning model based on the discrimination result, subjecting the learning model to machine learning for defect kinds different from the discriminated defect kinds or defect kinds associated with unknown defects.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present disclosure relates to techniques for determining defects in printed images. [Background technology]

[0002] Patent Document 1 describes a device that uses machine learning to identify defects in printed matter in advance, detects defects in actual printed matter, and displays necessary maintenance information for the printing device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-101540 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology of Patent Document 1 can only distinguish defects in printed matter that have been learned in advance, and cannot handle unknown defects. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms or application examples. (1) One embodiment of the present disclosure is a defect discrimination device for printed images. The defect discrimination device for printed images includes: an image containing defects that may occur during printing, and a learning model that has been trained in advance through machine learning using teacher images associated with defect types and is capable of outputting a similarity for each of the defect types; a target image acquisition unit that acquires an image of a printed matter and prepares a target image to be inspected; a classifier that uses the learning model for the target image to acquire a similarity of a defect present in the target image to the defect type and classifies the defect present in the target image as at least one of known defect types; and a learning unit that, when it is determined based on the result of the discrimination by the classifier that learning of the learning model is necessary, causes the learning model to perform machine learning on a defect type different from the determined defect type or a defect type associated with an unknown defect.

[0006] Another embodiment of the present disclosure relates to a method for determining defects in a printed image. The method for determining defects in a printed image includes: performing machine learning in advance using a teacher image associated with a defect type for an image containing defects that may occur during printing, preparing a learning model that has been trained to be capable of outputting a similarity for each of the defect types; acquiring an image of a printed matter to prepare a target image to be inspected; using the learning model for the target image to determine a similarity of a defect present in the target image to the defect type; using the similarity, determining that the defect present in the target image is at least one defect type of a known defect; and, if it is determined based on the determination result that learning of the learning model is necessary, performing machine learning on a defect type different from the determined defect type or a defect type associated with an unknown defect. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of a classification system according to an embodiment. [Figure 2] FIG. 1 is a block diagram of an information processing device. [Figure 3]FIG. 1 is an explanatory diagram showing the configuration of a machine learning model. [Figure 4] Flowchart showing the process of preparing a machine learning model. [Figure 5] FIG. [Figure 6] FIG. 10 is an explanatory diagram showing how known feature information is created using training data. [Figure 7] FIG. 3 is an explanatory diagram showing the configuration of known feature information. [Figure 8] 10 is a flowchart showing a processing procedure for determining a type of defect in a printed image. [Figure 9] FIG. 10 is an explanatory diagram showing how class-specific similarities regarding data to be discriminated are calculated. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of a method for calculating similarity by class. [Figure 11] FIG. 10 is an explanatory diagram showing an example of a method for determining a discrimination class using a plurality of specific layers. [Figure 12] FIG. 10 is an explanatory diagram showing an example of a display of a defect type discrimination result. [Figure 13] FIG. 4 is an explanatory diagram illustrating a maintenance process. [Figure 14] An explanatory diagram showing a schematic diagram of incremental learning and reinforcement learning techniques. [Figure 15] FIG. 10 is an explanatory diagram schematically showing how each learning model performs discrimination. [Figure 16] 10 is a flowchart showing a process for generating a difference image used during relearning. [Figure 17] FIG. 10 is an explanatory diagram showing a first technique for generating explanatory information. [Figure 18] FIG. 10 is an explanatory diagram showing a second technique for generating explanatory information. [Figure 19] FIG. 10 is an explanatory diagram illustrating a configuration in which a learning model is prepared for each label. [Figure 20] FIG. 10 is a schematic diagram showing how defect types are distinguished when a learning model is constructed for each label. DETAILED DESCRIPTION OF THE INVENTION

[0008] A. First embodiment: (1) System configuration: FIG. 1 is a block diagram showing the schematic configuration of an inspection image defect detection system 100 according to one embodiment. This defect detection system 100 is a printing system including a printer 10, an information processing device 20, and a scanner 30 as an imaging unit. Upon receiving image data to be printed from the information processing device 20, the printer 10 transports a print medium P stored in a stack SK and prints the image on the print medium P. The method for forming the image on the print medium P is not critical. For example, printing may be performed using an inkjet method in which ink droplets are ejected onto the print medium P, a xerography method in which a latent image is formed on a photosensitive drum and the toner is transferred to the print medium P, or a lithographic printing method in which ink is transferred onto a lithographic plate. The image output and printed by the printer 10 corresponds to the reference image, and the image printed on the print medium P and scanned by the scanner 30 corresponds to the inspection image. However, as will be described later, the reference image and the inspection image are subjected to a process of subtracting them during inspection, so the formats of both images must be identical. For example, if the original image org to be printed is in RGB format and has a resolution of r1, the print image prt that will actually be printed on the print medium P is in CMYK format and has a resolution of r2, and the captured image scg read by the scanner 30 is in RGB format and has a resolution of r3, then the captured image scg read by the scanner 30 can be used as the inspection image, and the original image org can be converted to a resolution of r3 and used as the reference image. Alternatively, the reference image can be prepared by using the scanner 30 to read an image printed by the printer 10 in a state that has been confirmed to be free of defects.

[0009] The scanner 30 is installed inside the printer 10 or at a location where the printed print medium P is ejected. It can scan the printed surface of the print medium P to capture an inspection image. In the present disclosure, the inspection image obtained by the scanner 30 is used to generate a difference image by subtracting it from a reference image. The difference image is input into a machine learning model (described below). The information processing device 20 uses the machine learning model to perform a defect type classification process and determine which of multiple classes the defect type present in the image formed on the print medium P belongs to. The "defect type class" refers to the type of defect present in the printed image. The information processing device 20 controls the printer 10 to print under appropriate printing conditions depending on the type of print medium P. The inspection image defect classification system according to the present disclosure can also be configured without the printer 10. Alternatively, a camera or other device can be used instead of the scanner 30. Since inspection is possible without generating a difference image, the image to be subjected to defect type classification may simply be referred to as the "target image."

[0010] 2 is a block diagram showing the functions of the information processing device 20. The information processing device 20 has a processor 110, a storage device 120, an interface circuit 130, and an input device 22 and a display unit 21 connected to the interface circuit 130. A scanner 30 and a printer 10 are also connected to the interface circuit 130. For example, but not limited to, the processor 110 not only has the function of executing the processes described in detail below, but also has the function of displaying on the display unit 21 data obtained by the processes, data generated during the processes, and even maintenance information for the printer 10 as a result of the processes.

[0011] The processor 110 functions as a print processing unit 112, a class discrimination processing unit 114, a learning / relearning unit 116, a difference image generation unit 118, etc. The print processing unit 112 realizes the function of executing a process of printing an original image input from the input device 22 using the printer 10. The class discrimination processing unit 114 also executes a process of discriminating defect types that may be present in an image printed on the print medium P. The class discrimination processing unit 114 includes a similarity calculation unit 310 and a class determination unit 320. The learning / relearning unit 116 causes the learning model 200 stored in the storage device 120 to perform re-learning. Re-learning includes at least one of various learning methods that correct a trained learning model, such as reinforcement learning, incremental learning, and transfer learning. The difference image generation unit 118 generates a difference image by extracting the difference between the reference image and the inspection image described above. The differential image generating unit 118 generates a differential image between the reference image and the defect image or the inspection image, but if the differential image is not used, the inspection image can be used as the target image, and the differential image generating unit 118 does not need to be used.

[0012] Here, a "reference image" refers to an image to be printed that does not contain any defect types. A "defect type," as will be described in detail later, refers to a partial phenomenon that differs from the reference image and that impairs the quality of the printed image, resulting from printing by the printer 10. Defect types include known defect types learned by the machine learning model 200 as well as unknown defect types that have not been learned but are recognized as defects by the person inspecting the image. A "defect image" refers to an image that contains such known or unknown defect types. An "inspection image" refers to an image to be inspected, regardless of whether it contains defect types—here, an image scanned by the scanner 30 from an image printed based on an original image. Therefore, a difference image obtained by subtracting a reference image from a defect image for learning purposes will always contain defect types. However, a difference image between a reference image and an inspection image does not necessarily contain defect types and may not contain defects. To reiterate, if the inspection image is a predetermined specific image, defect types can be identified without using a difference image. In this case, the inspection image can be used directly as the target image. The "reference image" is not limited to one that can be seen with the naked eye, but also includes data that can be processed by the information processing device 20 or the like when taking the difference from the inspection image.

[0013] These units 112 to 118 are realized by the processor 110 executing a computer program stored in the storage device 120. However, these units 112 to 118 may also be realized by hardware circuits. The term "processor" used in this specification also includes such hardware circuits. Furthermore, the processor that executes the class discrimination process may be a processor included in a remote computer connected to the information processing device 20 via a network.

[0014] The storage device 120 stores a machine learning model 200, training data TD, known feature information KSp, and a maintenance information table MIT. The machine learning model 200 is used for processing by the class discrimination processing unit 114. An example configuration and operation of the machine learning model 200 will be described later. The training data TD is a collection of labeled data used to train the machine learning model 200. In this embodiment, the training data TD is a collection of difference images corresponding to defect types. The known feature information KSp is a collection of feature spectra obtained when the training data TD is input into the trained machine learning model 200. The feature spectra will be described later. The maintenance information table MIT is a table that registers information indicating the content of maintenance that should be performed on the printer 10 when a defect is detected in a printed image. The information indicating the content of maintenance is displayed on the display unit 21 and presented to the user. This information may include not only text but also images, videos, audio, and the like.

[0015] (2) Machine learning model configuration: FIG. 3 is an explanatory diagram showing the configuration of a machine learning model 200. In this machine learning model 200, input data IM is two-dimensional array data. This machine learning model 200 includes, in order from the input data IM side, a convolutional layer 210, a primary vector neuron layer 220, a first convolutional vector neuron layer 230, a second convolutional vector neuron layer 240, and a classification vector neuron layer 250. Of these five layers 210 to 250, the convolutional layer 210 is the lowest layer, and the classification vector neuron layer 250 is the highest layer. In the following description, the layers 210 to 250 are also referred to as the "Conv layer 210," the "PrimeVN layer 220," the "ConvVN1 layer 230," the "ConvVN2 layer 240," and the "ClassVN layer 250," respectively.

[0016] The Conv layer 210 is a layer made up of scalar neurons. The other four layers 220 to 250 are layers made up of vector neurons. A vector neuron is a neuron that uses vectors as input and output. In the following, the term "node" is used as a broader concept of scalar neurons and vector neurons.

[0017] In this embodiment, the input data IM is image data, and therefore is a two-dimensional array of data. For example, the input data IM is image data of 32 pixels x 32 pixels. Each pixel may have only a luminance value, or may have RGB format data.

[0018] In the example of Figure 3, two convolution vector neuron layers 230 and 240 are used, but the number of convolution vector neuron layers is arbitrary, and the convolution vector neuron layer may be omitted. However, it is preferable to use one or more convolution vector neuron layers. The configuration of each layer 210 to 250 in Figure 3 can be described as follows. <Description of each layer configuration> ·Conv layer 210: Conv[32,5,2] ·PrimeVN layer 220: PrimeVN[16,1,1] ·ConvVN1 layer 230:ConvVN1[12,3,1] ·ConvVN2 layer 240:ConvVN2[6,7,2] ·ClassVN layer 250:ClassVN[n1,3,1] Vector dimension VD: VD=16

[0019] In the description of each of these layers 210-250, the character string before the parentheses is the layer name, and the numbers in the parentheses are, in order, the number of channels, the kernel surface size, and the stride. For example, the layer name of the Conv layer 210 is "Conv," the number of channels is 32, the kernel surface size is 5x5, and the stride is 2. In FIG. 3, these descriptions are shown below each layer. The number of channels is the number of neuron layers in each layer. Since the Conv layer 210 is composed of scalar neurons, it is written as "Kernel" in FIG. 3. Since the other layers 220-250 are composed of vector neurons, the number of channels is written as "VN." The hatched rectangles drawn in each layer represent the surface size of the kernel used to calculate the output vector of the adjacent higher layer. In this embodiment, since the input data IM is a two-dimensional array of image data, the kernel surface size is also two-dimensional. In the above description, the dimension of the output vector of each vector neuron is constant at 16. The parameter values ​​used in the description of each of the layers 210 to 250 are merely examples and can be changed as desired.

[0020] FIG. 3 shows the Conv layer 210, with a first axis x and a second axis y defining the planar coordinates of the node array, and a third axis z representing depth. It also shows that the Conv layer 210 has dimensions of 14, 14, and 32 in the x, y, and z directions. The dimensions in the x and y directions are referred to as "resolution." In this embodiment, the resolution in the x and y directions is the same for each layer. In FIG. 3, the resolution is shown above each layer. Specifically, the resolution is 14x14 for the Conv layer 210, 14x14 for the PrimeVN layer 220, 12x12 for the ConvNV1 layer 230, 3x3 for the ConvNV2 layer 240, and 1x1 for the ClassVN layer 250. The size in the z direction is the number of channels. These three axes x, y, and z are also used as coordinate axes indicating the position of each node in other layers. However, in FIG. 3, the axes x, y, and z are not shown in layers other than the Conv layer 210.

[0021] As is well known, the resolution W1 in the x and y directions after convolution is given by the following equation: W1 = Ceil{(W0 - Wk + 1) / S} …(1) Here, W0 is the target resolution for convolution processing, Wk is the surface size of the kernel, S is the stride, and Ceil{X} is a function that performs the calculation of rounding up X. Taking the Conv layer 210 as an example, W0=32, Wk=5, and S=2, so W1=Ceil{(32-5+1) / 2}=14 The resolution of each layer shown in Fig. 3 is an example in which the resolution in the y direction of the input data IM is set to 32, and the actual resolution of each layer is changed appropriately depending on the size of the input data IM.

[0022] The ClassVN layer 250 has n1 channels. In the example of FIG. 3, n1=3. Generally, n1 is an integer equal to or greater than 2 and is the number of known classes that can be distinguished using the machine learning model 200. The three channels of the ClassVN layer 250 output judgment values ​​Class1 to Class8 for eight known classes (defect types in this embodiment). Typically, the class having the largest value among these judgment values ​​Class1 to Class8 is used as the class discrimination result for the input data IM. Furthermore, if the largest value among the judgment values ​​Class1 to Class8 is less than a predetermined threshold, the class of the input data IM may be determined to be unknown.

[0023] In the present disclosure, as will be described later, instead of using the judgment values ​​Class1 to Class8 of the ClassVN layer 250, which is the output layer, a class-specific similarity calculated from the output of a specific vector neuron layer is used to determine the discrimination class, in this embodiment, the type of defect (defect type).

[0024] FIG. 3 also illustrates subregions Rn in each layer 210, 220, 230, 240, and 250. The subscript "n" in subregion Rn refers to the reference number of the layer. For example, subregion R210 indicates a subregion in the Conv layer 210. A "subregion Rn" is a region in each layer that is identified by a planar position (x, y) defined by the position of the first axis x and the position of the second axis y, and that includes multiple channels along the third axis z. The subregion Rn has dimensions of "Width" × "Height" × "Depth," corresponding to the first axis x, the second axis y, and the third axis z. In this embodiment, the number of nodes included in one "subregion Rn" is "1 × 1 × depth count," i.e., "1 × 1 × number of channels."

[0025] 3, a feature spectrum Sp_ConvVN1 (described later) is calculated from the output of the ConvVN1 layer 230 and input to the similarity calculation unit 310. Similarly, feature spectra Sp_ConvVN2 and Sp_ClassVN are calculated from the outputs of the ConvVN2 layer 240 and the ClassVN layer 250, respectively, and input to the similarity calculation unit 310. The similarity calculation unit 310 calculates class-specific similarities Sclass_ConvVN1, Sclass_ConvVN2, and Sclass_ClassVN (described later) using these feature spectra Sp_ConvVN1, Sp_ConvVN, and Sp_ClassVN and previously generated known feature information KSp. The class determination unit 320 generates a discrimination result RD using at least a portion of these class-specific similarities Sclass_ConvVN1, Sclass_ConvVN2, and Sclass_ClassVN. The discrimination result RD includes a discrimination class D_class and a similarity value S_value corresponding to the discrimination class D_class.

[0026] In this disclosure, the vector neuron layer used to calculate the similarity is also referred to as the "specific layer." Any number of vector neuron layers, one or more, can be used as the specific layer. The configuration of the feature spectrum, the method for calculating the similarity using the feature spectrum, and the method for determining the discriminant class will be described later.

[0027] 4 is a flowchart showing the processing steps of the machine learning model preparation process. This preparation process is performed, for example, by the manufacturer of the printer 10. When this process starts, a reference image is first prepared (step S110). This process is not performed internally by the information processing device 20, but rather a reference image required for machine learning is selected.

[0028] Next, a process is performed to acquire a defect image corresponding to this reference image (step S120). The reference image is the original image printed by printer 10, and the defect image is an image that includes defects that may occur when printing this reference image with printer 10. In this embodiment, in order to learn about various defects that may occur when printing with printer 10, the reference image and the corresponding defect image are required when preparing a learning model. In this embodiment, in step S120, the defect image is acquired by reading an image in which a defect has occurred with scanner 30.

[0029] Possible defects that may occur in the printer 10 include, for example, the following. <1> Banding <2> Droplets <3> Nozzle missing <4> Pinhole <5> Foreign matter 6. Distortion <7> Cock ring <8> Color Fluctuation

[0030] The defect images may be acquired by preparing a reference image and a corresponding defect image using the same type of printer 10 and scanner 30 that produce each defect. The reference image may be obtained by printing RGB format original image data using the printer 10 and reading this with the scanner 30, or by generating an image equivalent to the printed image from the RGB format original image data through simulation and using this image.

[0031] After the reference image and the defect image are acquired, preprocessing is performed on the reference image acquired in step S110 and the defect image acquired in step S120 (step S130). Here, preprocessing refers to at least one of the following processes, but in this embodiment, all of the processes are performed. (A) Level correction: This is a correction to match the levels of brightness, hue, saturation, etc. between the reference image and the defective image. Originally, both images are the same overall, but since they are read using the scanner 30, it may be necessary to match the levels. (B) Geometric correction: This is a correction that makes the reference image and the defective image geometrically identical in shape. Corrections are performed by adjusting the magnification of both images and correcting distortions. (C) Alignment: The positions of the reference image and the defective image may be misaligned due to factors such as bending of the paper feed direction when printing with the printer 10 or the reading angle of the scanner 30. To correct this, the positions of both images are aligned by performing translation correction, rotation correction, etc.

[0032] After performing this preprocessing (step S130), a difference image between the reference image and the defect image is generated by the difference image generation unit 118 and prepared as training data (step S140). The processes of steps S110 to S140 may be performed separately to prepare a difference image corresponding to the defect type in advance. This difference image is training data TD used in the training performed by the class discrimination processing unit 114.

[0033] The class discrimination processing unit 114 performs machine learning using the thus prepared plurality of training data TD to generate and update the machine learning model 200 (step S150). Each training data TD is assigned a label in advance. In this embodiment, it is assumed that labels corresponding to the above-mentioned defects <1> to <8> are assigned to each training data TD. These labels correspond to the eight classes Class1 to Class8 of the machine learning model 200. In this disclosure, "label" and "class" mean the same thing.

[0034] When learning using the plurality of pieces of training data TD is completed, the trained machine learning model 200 is stored in the storage device 120. Once learning of the machine learning model 200 is completed, next, in step S160 of FIG. 4, the plurality of pieces of training data TD are input again into the trained machine learning model 200 to generate known feature information KSp. The generated known feature information KSp is stored in the storage device 120. The known feature information KSp is a set of feature spectra, which will be described below.

[0035] (3) Generation of known feature information: FIG. 5 is an explanatory diagram showing a feature spectrum Sp obtained by inputting arbitrary input data to the trained machine learning model 200. Here, the feature spectrum Sp obtained from the output of the ConvVN1 layer 230 will be described. The horizontal axis of FIG. 5 represents the position of a vector element in the output vector of multiple nodes included in one subregion R230 of the ConvVN1 layer 230. The position of this vector element is represented by a combination of the element number ND of the output vector of each node and the channel number NC. In this embodiment, since the vector dimension is 16, the element numbers ND of the output vector are 16, ranging from 0 to 15. Furthermore, since the ConvVN1 layer 230 has 12 channels, the channel numbers NC are 12, ranging from 0 to 11. In other words, this feature spectrum Sp is obtained by arranging multiple element values ​​of the output vector of each vector neuron included in one subregion R230 across multiple channels along the third axis z.

[0036] The vertical axis of Fig. 5 represents the feature value C at each spectral position. V In this example, the feature value C V is the value of each element of the output vector V ND Note that the feature value C V As the value of each element of the output vector V ND Alternatively, the normalization coefficient may be used as it is. In the latter case, the feature value C included in the feature spectrum Sp may be used as it is. VThe number of is equal to the number of channels, which is 12. The normalization coefficient is a value corresponding to the vector length of the output vector of the node.

[0037] The number of feature spectra Sp obtained from the output of the ConvVN1 layer 230 for one piece of input data is 12 × 12 (144) because it is equal to the number of planar positions (x, y) of the ConvVN1 layer 230, i.e., the number of subregions R230. Similarly, for one piece of input data, 3 × 3 (9) feature spectra Sp are obtained from the output of the ConvVN2 layer 240, and one feature spectrum Sp is obtained from the output of the ClassVN layer 250.

[0038] When the training data TD is input again to the trained machine learning model 200, the similarity calculation unit 310 calculates the feature spectrum in the same manner as the feature spectrum Sp shown in Fig. 5. This feature spectrum differs from the feature spectrum obtained when actually detecting and discriminating defect types and is prepared in advance, so in order to distinguish between the two, it will be referred to as known feature information hereinafter.

[0039] 6 is an explanatory diagram showing how known feature information KSp is created using training data TD. In this example, training data TD with labels 1 to 8 is input to a trained machine learning model 200, and known feature information KSp associated with each label or class is extracted from the outputs of three vector neuron layers, namely, the ConvVN1 layer 230, the ConvVN2 layer 240, and the ClassVN layer 250. information KSp_ConvVN1, KSp_ConvVN2, and KSp_ClassVN are obtained. These pieces of known feature information KSp_ConvVN1, KSp_ConvVN2, and KSp_ClassVN are stored in the storage device 120 as known feature information KSp.

[0040] Fig. 7 is an explanatory diagram showing the configuration of known feature information KSp. In this example, known feature information KSp_ConvVN1 obtained from the output of the ConvVN1 layer 230 is shown. Known feature information KSp_ConvVN2 obtained from the output of the ConvVN2 layer 240 and known feature information KSp_ClassVN obtained from the output of the ClassVN layer 250 also have similar configurations, but are not shown in Fig. 7. Note that it is sufficient that the known feature information KSp is obtained from the output of at least one vector neuron layer.

[0041] Each record of the known feature information KSp_ConvVN1 includes a parameter i indicating the order of the label or class, a parameter j indicating the order of the specific layer, a parameter k indicating the order of the subregion Rn, a parameter q indicating a data number, and a known feature information component. The known feature information component is a component corresponding to the data number q for each subregion k of the known feature information KSp_ConvVN1 corresponding to each label or class.

[0042] The class parameter i takes the same value as the label, 1 to 8. The specific layer parameter j takes a value from 1 to 3 indicating which of the three specific layers 230, 240, and 250 it is. The subregion Rn parameter k takes a value indicating which of the multiple subregions Rn included in each specific layer it is, i.e., which planar position (x, y) it is. For the ConvVN1 layer 230, there are 144 subregions R230, so k = 1 to 144. The data number parameter q indicates the number of training data with the same label, and takes a value from 1 to max1 for class 1, 1 to max2 for class 2, and so on, to 1 to max8 for class 8.

[0043] The plurality of teacher data TD used in step S120 does not need to be the same as the plurality of teacher data TD used in step S110. However, if some or all of the plurality of teacher data TD used in step S110 is used in step S120 as well, there is an advantage in that there is no need to prepare new teacher data.

[0044] (4) Defect type discrimination process: After the preparations described above have been made, a process is performed to identify defect types that may occur in an image printed on the print medium P. FIG. 8 is a flowchart showing a print image defect type identification process routine using a trained machine learning model. This process routine is shown primarily as being performed by the processor 110 of the information processing device 20, but some of the process can also be performed by the user of the printer 10. When the process can be performed by the user, this will be explained each time.

[0045] 8 begins when a desired image is printed by printer 10. When this process begins, the printed image is first read as a test image using scanner 30 (step S410). This reading process may be performed every time an image is printed on print medium P, or may be performed by the user at a specific timing.

[0046] Next, a process for acquiring a reference image is performed (step S420). As already explained, the reference image may be one obtained by printing the RGB format original image data to be printed using a serviced printer, i.e., a printer guaranteed not to produce defective images, and then scanning the printed image with scanner 30. Alternatively, the reference image may be one obtained by generating an image equivalent to the printed image from the RGB format original image data through simulation. This process is performed in advance before printing, and the image is prepared as a reference image.

[0047] Next, preprocessing is performed on the inspection image and the reference image (step S430). This process is similar to the process in step S130 of the preparation process shown in FIG. 4. Through the preprocessing, the inspection image and the reference image are matched in terms of position, size, rotation angle, image brightness, and other levels, and then a difference image is generated (step S435). The difference image thus obtained becomes discrimination data for discriminating defect types, as described below. The preprocessing may be performed automatically or individually by the user.

[0048] Once the differential image, i.e., the discriminated data, has been generated, the class discrimination processing unit 114 then inputs the discriminated data into the trained machine learning model 200 to calculate the feature spectrum Sp (step S440). Using the obtained feature spectrum Sp and the known feature information KSp generated and saved in step S160 of Fig. 4, the similarity calculation unit 310 calculates class-specific similarities (step S450).

[0049] A method for calculating class-specific similarities will now be described. Fig. 9 is an explanatory diagram showing how class-specific similarities for data to be discriminated are calculated. When data to be discriminated is input to the machine learning model 200, the class discrimination processing unit 114 calculates feature spectra Sp_ConvVN1, Sp_ConvVN2, and Sp_ClassVN from the outputs of the ConvVN1 layer 230, the ConvVN2 layer 240, and the ClassVN layer 250. Upon receiving these feature spectra, the similarity calculation unit 310 calculates the class-specific similarities for each specific layer. Specifically, a class-specific similarity Sclass_ConvVN1 is calculated using the feature spectrum Sp_ConvVN1 obtained from the output of the ConvVN1 layer 230 and known feature information KSp_ConvVN1, a class-specific similarity Sclass_ConvVN2 is calculated using the feature spectrum Sp_ConvVN2 obtained from the output of the ConvVN2 layer 240 and known feature information KSp_ConvVN2, and a class-specific similarity Sclass_SclassVN is calculated using the feature spectrum Sp_ClassVN obtained from the output of the ClassVN layer 250 and known feature information KSp_ClassVN. In the figure, known feature information is depicted only for the ConvVN1 layer 230, but class-specific similarities are calculated in a similar manner for the ConvVN2 layer 240 and the ClassVN layer 250.

[0050] Although it is not necessary to generate all of the class-specific similarities Sclass_ConvVN1, Sclass_ConvVN2, and Sclass_ClassVN using each of the three vector neuron layers 230, 240, and 250, it is preferable to calculate the class-specific similarities using one or more of these vector neuron layers. As described above, in this disclosure, the vector neuron layer used to calculate the similarities is referred to as a "specific layer." As described below, in this embodiment, the class-specific similarities Sclass_ConvVN1, Sclass_ConvVN2, and Sclass_ClassVN are all generated using outputs from the three vector neuron layers 230, 240, and 250, and these multiple class-specific similarities are used to determine the class. Alternatively, one or two of the outputs from the three vector neuron layers 230, 240, and 250 may be used to calculate one or two of the class-specific similarities Sclass_ConvVN1, Sclass_ConvVN2, and Sclass_ClassVN to determine the defect type.

[0051] An example of a method for calculating the class-specific similarity will be described below. In this embodiment, the class-specific similarity is calculated without taking the partial region Rn into consideration at all, as will be described below. A calculation method for calculating class-specific similarity without considering the correspondence between the feature spectrum Sp and the subregion Rn in the known feature information KSp, A calculation method for calculating class-specific similarity between the feature spectrum Sp and the corresponding subregion Rn of the known feature information KSp, It is also known that:

[0052] The calculation method for class-specific similarity is as follows: (1) Calculating, for each class, the class-specific similarity Sclass(i,j) between all feature spectra Sp obtained from the output of a specific layer j according to the data to be discriminated and all known feature information KSsp associated with that specific layer j and each class i; (2) For each class i, the maximum value of the multiple class-specific similarities Sclass(i,j) is calculated as the similarity value S_value between the feature spectrum Sp and the known feature information KSp; (3) The class associated with the maximum similarity value S_value across multiple classes i is determined as the discriminant class D_class.

[0053] 10 is an explanatory diagram showing a method for calculating class similarity in this embodiment. In this calculation method, the class similarity Sclass(i,j) is calculated from the outputs of specific layers, ConvVN1 layer 230, ConvVN2 layer 240, and ClassVN layer 250, without taking local similarity into consideration.

[0054] The class-specific similarity Sclass(i,j) obtained by this calculation method is calculated using the following formula. Sclass(i,j)=max[G{Sp(j,k=all), KSp(i,j,k=all,q=all)}] …(2) where: Sp(j,k=all) is the feature spectrum obtained from the output of all subregions k of a specific layer j according to the data to be discriminated. The function max[] is a function that finds the maximum value of the elements in [].

[0055] In the example of FIG. 10, the final discrimination result RD_ConvVN1 is further determined from the class-specific similarity Sclass(i,j). The discrimination result RD_ConvVN1 can be expressed in a format including a discrimination class D_class and a similarity value S_value corresponding to the discrimination class D_class. The similarity value S_value is obtained by taking the maximum value of the similarity values ​​for the eight classes 1 to 8 in the class-specific similarity Sclass(i,j). The discrimination class D_class is the class with the maximum similarity value in the class-specific similarity Sclass(i,j). As shown in the figure, for j=1, that is, in the vector neuron layer 230, the class-specific similarity Sclass(3,1) of class 3 is 0.95, which is the maximum value among all classes, so the final discrimination result RD_ConvVN1 is RD_ConvVN1(D_class,S_value)=(3,0.95) Similarly, for the ConvVN2 layer 240 and the ClassVN layer 250, the class similarities Sclass_ConvVN2 and Sclass_ClassVN are calculated, and the final discrimination results RD_ConvVN2 and RD_ClassVN are determined for each layer.

[0056] In this way, the class similarities Sclass_ConvVN1, Sclass_ConvVN2, Sclass_ClassVN, and ultimately the final discrimination results RD_ConvVN1, RD_ConvVN2, RD_ClassVN are calculated using the outputs from the three vector neuron layers 230, 240, 250. Then, Figure 11 As shown in Fig. 1, the most frequent class is determined from the final discrimination result based on the outputs from these three vector neuron layers 230, 240, and 250. In this example, of the three vector neuron layers 230 to 250, the outputs of the vector neuron layers 230 and 240 are class 3, so the final discrimination result is class 3.

[0057] In step S460, the class determination unit 320 determines the class of the difference image obtained from the inspection image and the reference image, i.e., the type of defect present in the printed image, based on the class-specific similarity obtained in step S450. <1> Banding <2> Droplets <3> Nozzle missing <4> Pinhole <5> Foreign matter 6. Distortion <7> Cock ring <8> Color Fluctuation Eight defect types are learned. Here, each number corresponds to a class to be distinguished. In addition to identifying these defect types, it is also possible to provide an output indicating that no defect type has been identified. In other words, if the defect type distinction result does not identify any defect type, and does not identify the existence of an unknown defect, this is classified as a class of "no defect type identified." Therefore, the number of classes related to defect type distinction is ultimately nine in total, <0> No defect type identified It should be noted that this class of "no defect type identified" may be prepared as an output of the machine learning model 200, or may be prepared as a class that is output when none of the outputs (eight in this example) corresponding to the defect types have a predetermined similarity or higher.

[0058] After determining the defect type in this way, it is determined whether or not there is a defect in the printed image (step S470). If it is determined that there is no defect (step S470: "NO"), in this embodiment, the process goes directly to "END" and ends this processing routine, but it is also possible to accept a user's decision as to whether the determination that there is no defect is correct, and if the determination is incorrect, to perform additional learning, which will be described later.

[0059] If any defect type is found (step S470: "YES"), a process is performed to display the defect type found in the printed inspection image on the display unit 21 (step S480). FIG. 12 shows an example of such a defect type display. The figure shows an example of a defect type determined to be a nozzle missing. In this case, the defect type is displayed using the corresponding difference image 211. That is, the display unit 21 displays the difference image 211 generated in step S435 and the defect type determination result, in this case, the text 212 "nozzle missing." The display unit 21 also displays an approval button 215 indicating that the display result is accepted, an additional learning button 216 which instructs the user to perform additional learning of a new defect type since the defect type in the displayed image is not a known one, a field 216a for inputting a defect name to be used as a label for the new defect type during additional learning, a reinforcement learning button 217 which instructs the user to perform reinforcement learning to strengthen the discrimination of known defect types based on the display result, and a field 217a for selecting an existing label in a pop-up format. Furthermore, if a defect is found but the defect type cannot be identified, the approval button 215 may be set to an invalid state in which it cannot be operated, since the defect type to be accepted is unknown.

[0060] The system determines which of these selection buttons was operated (step S490). If the user views the differential image 211 and the discrimination result 212 displayed on the display unit 21, determines that the defect type is correct, and presses the approve button 215, the system executes the maintenance process (step S500). The maintenance process retrieves the maintenance method corresponding to the identified defect type from the maintenance information table MIT stored in the storage device 120 and displays it on the display unit 21. An example of the maintenance process display is shown in FIG. 13. In this example, the maintenance method for nozzle clogging is displayed using text 330 and an image 335. The details of the maintenance process may be presented to the user as necessary using audio guidance, video, or AR display on a head-mounted display. In this example, the user understands the details of the maintenance process, operates the cleaning button 337 to instruct the printer 10 to perform cleaning, and, upon viewing the results, determines that the defect has been resolved. If the user operates the end button 338 displayed on the display unit 21, the display of the maintenance process ends, the system exits to "END," and the processing routine ends.

[0061] On the other hand, if the user operates the additional learning button 216 as determined in step S490, additional learning is performed (step S510), and if the user operates the reinforcement learning button 217, reinforcement learning is performed (step S520). If the additional learning button 216 is operated, that is, if the defect type discrimination result by the machine learning model 200 is an unknown defect and the user determines that the detected defect is different from a known defect type, the user inputs a new label (defect name) in field 216a, and additional learning is performed. Also, if the defect type was discriminated as a known defect type but the user determines that the detected defect is different from the known defect type, additional learning can also be selected. On the other hand, if the reinforcement learning button 217 is operated, the detected defect type is known, but the user determines that the defect type discrimination result by the machine learning model 200 is incorrect, so a correct label is assigned to the difference image 211 and reinforcement learning is performed. In this case, the correct label (defect name to be learned) is selected from existing labels displayed in a pop-up format in field 217a.

[0062] The methods of incremental learning and reinforcement learning are shown schematically in FIG. 14, along with a method for constructing a machine learning model. Column (A) in the figure shows a schematic representation of the machine learning method shown in FIG. 4. As already explained, in supervised machine learning, a difference image to which a label indicating the defect type is attached is prepared, and this is trained by the learning model 200 shown in FIG. 3, thereby obtaining a trained learning model A. In the figure, the learning model drawn with a dashed line indicates the model before training, and the learning model drawn with a solid line indicates the trained model A.

[0063] The classifier 114, which is equipped with a similarity calculation unit 310 and a class determination unit 320, can use this learning model A to classify the class (defect type) of the difference image. The manner of classification in this case is shown in column (A) of FIG. 15. The mechanism for classifying the defect type using the trained learning model A is called classifier 1. The classifier 1 uses learning model A to determine the class (defect type) of a new difference image obtained from the printed printing medium P.

[0064] Column (B) of Figure 14 shows a schematic diagram of how additional learning is performed. First, as shown in the upper part of column (B), an image printed on a print medium P is read, and the defect type present in the inspection image is identified using learning model A for the difference image between the read inspection image and the reference image. If the user determines that the defect type is unknown as a result of the identification, supervised machine learning is performed again, as shown in the lower part of column (B) of the same figure, using a new label and the difference image for additional learning. The model used in this case is the model before learning. As a result of the additional learning, a new learning model B is obtained.

[0065] The difference images used for such additional learning are generated by performing a predetermined operation on the original difference image to generate multiple difference images, which are then used. The method for generating multiple difference images will be explained in detail later. New labels for discrimination are attached to the generated multiple difference images, and supervised machine learning is performed again to obtain a trained learning model B. The manner in which discrimination is performed using this learning model B is shown in column (B) of Figure 15. Since the trained learning model B discriminates new defects that have been additionally learned, the discriminator 2 that performs discrimination including this uses the initially trained learning model A and the additionally trained learning model B to determine the class (defect type) of the difference image obtained from the printed printing medium P.

[0066] Column (C) of FIG. 14 shows a schematic diagram of how reinforcement learning is performed. First, as shown in the upper part of Column (C), an image printed on a print medium P is read, and the defect type present in the inspection image is identified using a learning model A based on a difference image between the read inspection image and a reference image. If the user determines that the defect type identification is incorrect and presses the reinforcement learning button 217 shown in FIG. 12, supervised machine learning is performed again as shown in the lower part of Column (C) of FIG. 14 using the label designated by the user as the corresponding defect type and the difference image for reinforcement learning. At this time, machine learning is performed based on the trained learning model A using the difference image during training and the difference image for reinforcement learning. As a result of reinforcement learning, learning model A becomes learning model A1, whose ability to identify images with that label has been enhanced. Note that reinforcement learning may be performed if the correct defect type is displayed as a result of the initial identification using learning model A. The difference image used for reinforcement learning is obtained by performing one or more operations on the difference image for inspection. The operation of generating multiple difference images is the same as the difference images for additional learning, and will be described later. To be carried out .

[0067] The multiple difference images obtained in this way are labeled for discrimination, and supervised machine learning is performed to obtain a reinforcement-learned learning model A1. The manner in which discrimination is performed using this learning model A1 is shown in column (C) of Figure 15. The reinforcement-learned learning model A1 is an enhanced version of the trained learning model A's function for discriminating known defects, so the discriminator 3, which performs discrimination, determines the class (defect type) of the difference image obtained from the printed printing medium P using only the trained model A1.

[0068] A method for generating a plurality of difference images used in additional learning and reinforcement learning will be described. FIG. 16 is a flowchart showing a processing routine for generating a difference image for re-learning. This processing routine is started when the user presses the additional learning button 216 or the reinforcement learning button 217 in FIG. 12, and is executed by the information processing device 20. When the processing starts, first, a difference image to be used for re-learning is identified (step S610). In this case, a difference image when the defect type could not be correctly identified is identified. Teaching When performing supervised machine learning, it is desirable to prepare multiple labeled difference images, but at this point, the only difference images that can be prepared are those between the test image used for inspection and the reference image.

[0069] Next, explanatory information is generated (step S620). The explanatory information is information indicating what kind of judgment was made for which part of the image based on the feature spectrum output by each vector neuron layer 230, 240, and 250 of the learning model 200 for the difference information between a given test image and a reference image. In this embodiment, the final class determination for the difference image is not obtained from the output of the ClassVN layer 250, but is made based on the feature spectrum output by the vector neuron layers 230 to 250. Therefore, it is possible to know what kind of judgment was made for which part of the difference image for each defect type. This is the explanatory information.

[0070] Column (A) of Figure 17 shows one of the processes for obtaining learning information to be used when generating a difference image using such explanatory information. The degree of similarity between the difference image DIG and various defect types when defect types are identified using learning model A can be known from the feature spectrum of each vector neuron layer 230-250 of learning model A. Column (B) of the same figure shows examples of which areas of the difference image have the degree of similarity for the defect types "missing nozzle", "cockling", and "foreign matter" in each of the images NRG, KKG, and EXG. In each figure, the areas where the inspection image and the reference image do not match are shown as highlighted areas. In the image NRG corresponding to the nozzle missing defect, the degree of similarity between the nozzle missing defect type and the reference image is shown as highlighted areas. of Similarity Corresponding to Highlights show characteristic spectra (low concentration) On the other hand, in the image KKG corresponding to the cock ring defect, the part showing the characteristic spectrum with high similarity to the cock ring defect type is shown with low highlight, and the similarity is high. In the image EXG corresponding to the foreign object defect, the part showing the characteristic spectrum with high similarity to the foreign object defect type is shown with low highlight, and the similarity is high. of Similarity Corresponding to The area showing the characteristic spectrum is uneven, with some areas shown with low highlights, and the similarity in those areas is high.

[0071] Therefore, as shown in (C) of the figure, a combination of these is generated as explanatory information for learning as image MSG. Since explanatory information is not the image itself, image MSG is a schematic representation of explanatory information being generated as having a predetermined value (density) for each region of the differential image.

[0072] A second method for generating explanatory information for learning is shown in FIG. 18. In this example, the user operates the cursor HND to directly draw explanatory information DGG while viewing the difference image DIG in which the defect type has been identified and displayed on the display unit 21. If the display unit 21 is equipped with a touch panel, the range of explanatory information may be indicated by directly tracing the screen instead of using the cursor. The density may be changed by changing the drawing speed; for example, a portion drawn slowly may be displayed darker, thereby emphasizing that portion of the difference image.

[0073] Once the explanatory information for learning is generated by this method (step S620), it is used to perform a process of emphasizing the difference image to be used for re-learning (step S630). Specifically, the corresponding area of ​​the difference image is emphasized according to the density of the highlight part of the image MSG. The emphasis is performed by increasing the brightness of the difference image, emphasis This is done by adding a premium depending on the concentration of sexual information.

[0074] Next, a correction operation is performed on the enhanced difference image to generate multiple images (step S640). This correction operation is performed on the difference image used for inspection, and multiple operations are performed to generate multiple difference images for re-learning. Examples of such operations include Operation 1: Rotating the difference image by a predetermined angle, for example, 90 degrees, to obtain each rotated image. Operation 2: translating the difference image by a predetermined distance, for example, 1 / 4 of the image width, to obtain each translation image; Operation 3: Flip the difference image to obtain the inverted image. It is not necessary to perform all of these operations, but by performing operations 1 to 3 in order, the original difference image, three types of rotated images, images of these images moved horizontally (4 x 3 images), images of these images moved vertically (16 x 3 images), and their inverted images (64 images) can be obtained. Using this method, multiple additional difference images for learning are generated. Note that the selection and combination of multiple operations, the order of operations, etc. are arbitrary. Increasing the number of difference images for learning through correction processing generally improves the accuracy of the discrimination results obtained through learning.

[0075] The multiple difference images thus obtained are provided as difference images for re-learning (step S650), and the process exits to "END" to terminate this processing routine. Upon receiving multiple pieces of difference information, additional learning or reinforcement learning is performed as shown in columns (B) and (C) of Figure 14, and a new learning model B or a learning model A1 that is an enhanced version of the existing learning model A is obtained.

[0076] According to the discrimination system 100 of the first embodiment described above, a learning model for discriminating defect types contained in images printed by the printer 10 is constructed by learning from multiple difference images between an image containing a defect type and a reference image that does not contain the defect type as training data, thereby making it possible to accurately discriminate defect types that may be contained in the printed image. Moreover, if the discrimination system 100 is unable to correctly discriminate a defect type present in the printed image as a known defect type, the discrimination accuracy can be improved by performing machine learning again, and by having the discrimination system learn unknown defect types, the number of types of defect types that can be discriminated can be gradually increased.

[0077] Furthermore, in this embodiment, similarity is determined using the feature spectrum output by the specific layers 230 to 250, which are vector neuron layers, and known feature information previously stored in the storage device 120 for class discrimination. This allows for accurate class discrimination based on similarity, and in addition, if an unknown defect type whose class cannot be discriminated is found or if a defect type is erroneously discriminated, during re-learning, the explanatory information obtained from each of the specific layers 230 to 250 can be used to emphasize the difference information used for learning, thereby improving the efficiency of re-learning.

[0078] B. Second embodiment: Next, a discrimination system 100 according to a second embodiment will be described. The discrimination system 100 according to the second embodiment differs from the first embodiment in that the learning model 200 for discriminating defect types in a printed image does not discriminate multiple defect types using a single model, but instead uses a learning model prepared for each defect type. FIG. 19 is an explanatory diagram showing how supervised machine learning is performed on the learning model 200 according to the second embodiment. In the second embodiment, eight defect types are assumed to be present in the printed image, and learning difference images 1 to 8 are prepared for each defect type along with labels 1 to 8 indicating the defect type. The learning model 200 used for learning is the same as the learning model shown in FIG. 3 of the first embodiment, but differs in that the ClassVN layer 250 has one channel (n1 = 1). Unlike the first embodiment, the second embodiment prepares a learning model 200 for each defect type, and performs the preparation process shown in FIG. 4 for each learning model 200. As a result, as shown in FIG. 19, eight learning models 1 to 8 are trained for the eight defect types.

[0079] Then, these learning models 1 to 8 are aggregated to form a classifier as shown in Fig. 20. When a differential image for inspection generated from an image printed on a printing medium P is input to the classifier, the differential image is input to all of the trained learning models 1 to 8, and each of the learning models 1 to 8 outputs one of the similarities Sclass_ConvVN1, Sclass_ConvVN2, and Sclass_ClassVN output by the three vector neuron layers 230, 240, and 250 as the similarity of the defect type trained by that learning model 1 to 8. The classifier is provided with an output unit 300, which receives the outputs of the eight learning models 1 to 8 and outputs the defect type discriminated by the learning model that output the highest similarity value as the defect type present in the printed image.

[0080] If none of the similarities of the defect types output by the learning models 1 to 8 exceeds a predetermined threshold ThS, the output unit 300 determines that there is no defect in the inspection image and outputs "No defect." Furthermore, if the user determines that the defect type discrimination result output by the discriminator is incorrect and the detected defect is a new defect, additional learning is performed, as in the first embodiment, to form a new learning model 9 through additional learning, which is added to the discriminator, and its output is connected to the output unit 300. In this way, new defect types can be handled, as in the first embodiment. Furthermore, as in the first embodiment, reinforcement learning may be performed to improve the accuracy of discrimination of each defect type. The difference images used as training data during such re-learning can be generated from a single difference image using the same method as in the first embodiment.

[0081] The discrimination system 100 of the second embodiment not only achieves the same effects as the first embodiment, but also has the advantages of being able to easily deal with new defect types and easily replacing trained learning models, because a learning model is prepared for each defect type and trained. Even if one learning model has learned a difference image with an incorrect label, it is sufficient to replace only that learning model.

[0082] C. Other Embodiments: (1) Another embodiment of the present disclosure will be described. One of the other embodiments is a defect discrimination device for printed images. This defect discrimination device for printed images includes: a reference image setting unit that prepares a reference image serving as a reference for an image in which defects are to be detected; a learning model that has been trained in advance through machine learning using a teacher image to be capable of outputting a similarity for each defect type; a target image acquisition unit that acquires an image of a printed matter and prepares a target image to be inspected; a classifier that uses the learning model for the target image to acquire a similarity of a defect present in the target image to the defect type and classifies the defect present in the target image as at least one of known defect types; and a learning unit that, when updating the learning model based on the result of the discrimination by the classifier, causes the learning model to perform machine learning on a defect type different from the determined defect type or a defect type associated with an unknown defect.

[0083] This print image defect discrimination device can easily discriminate various defects that may occur in a printed image. Moreover, if it is determined that learning of the learning model is necessary based on the results of discriminating defects present in the image to be inspected, machine learning is performed on the learning model for a defect type different from the discriminated defect type or a defect type associated with an unknown defect, making it possible to discriminate defects that could not be discriminated as known defects or unknown defects. Examples of cases where it is determined that learning of the learning model is necessary include when a user views the discrimination results and determines that learning is necessary, or when it is determined that the similarity to any of the previously learned defect types is not sufficiently high.

[0084] (2) Defects in printed images, such as banding, dripping, nozzle clogs, pinholes, foreign matter, distortion, cockling, and color variation, can be identified. These defects may occur in combination, but the similarity obtained from a trained learning model can be used to identify one of them, or multiple defects can be identified simultaneously. In the former case, the defect with the highest similarity can be identified as a defect. In the latter case, all defects with a similarity equal to or greater than a predetermined value can be identified as defects.

[0085] (3) For supervised machine learning, existing methods such as deep learning using neural networks can be adopted, and the configuration can be with or without vector neuron layers. The number of neural network layers and the number of pixels in the input image can also be set arbitrarily.

[0086] (4) The printed image defect discrimination device may be integrated with a printing device such as a printer that prints images, or may be configured as a standalone device that receives target images and discriminates defects. Parts of the device configuration may also be distributed across a network. For example, a separate device may be provided that performs machine learning in advance using teacher images associated with defect types for images containing defects that may occur during printing, and generates a trained learning model capable of outputting similarity for each defect type. The trained learning model may be accessed via a network, or a separate learning unit may be provided. When a discriminator fails to discriminate a defect in a target image as a known defect, the learning unit may be called via the network to perform machine learning on the defect that could not be discriminated as a known defect, and the learned model may be made available for use.

[0087] (5) In this configuration, the classifier may determine that the defect in the target image is an unknown defect if none of the similarities obtained for each of the known defects is equal to or greater than a predetermined threshold. This makes it possible to easily detect unknown defects. Of course, there may be cases where the printed image does not contain any defects, so none of the similarities obtained for each of the known defects is equal to or greater than a predetermined threshold. In such cases, the user can simply view the difference image used in the inspection and correct the determination result.

[0088] (6) In such a configuration, when the classifier determines that a defect present in the target image is an unknown defect, the learning unit may receive a designation of a new defect type corresponding to the unknown defect and additionally learn a new model that enables discrimination of the new defect type to the learning model. This allows new defects to be easily learned and subsequently made correctly discriminable. Moreover, in this case, since a new model is additionally learned rather than learning an already-learned learning model, discrimination by the existing learning model is not affected.

[0089] (7) In this configuration, the designation of a defect as a new defect may be performed by a user setting a label of any defect type. In this way, the user of the device can easily set a new defect label for an unknown defect. Of course, labels may be set for new defects by automatically assigning sequential numbers to the new defects.

[0090] (8) In such a configuration, when it is determined that the classifier has erroneously determined a defect present in the target image, the learning unit may perform reinforcement learning of the learning model by receiving a designation that the erroneously determined defect is a defect type associated with another one of the known defect types or an unknown defect. In this way, the learning model can be corrected to prevent erroneous determinations. Of course, reinforcement learning may be performed not only when it is determined that the classifier has erroneously determined a defect present in the target image, but also when it has correctly determined a defect.

[0091] (9) In this configuration, the designation of the defect as another one of the known defect types may be performed by selecting a defect type from among pre-learned defect types. In this way, the occurrence of erroneous determinations in which a pre-learned defect type is not identified can be reduced by reinforcement learning.

[0092] (10) In this configuration, the target image acquisition unit includes a reference image setting unit that prepares a reference image serving as a reference for an image in which defects are to be detected; an inspection image acquisition unit that captures a printed material on which an image corresponding to the reference image is printed to acquire an inspection image; and a difference image generation unit that generates a difference image by extracting the difference between the reference image and the inspection image as the target image. The learning model may be a learning model that uses the machine learning technique to learn using a difference image, which is the difference between an image containing a known defect in the printed material and the original image, as the training image. In this way, the difference image between the reference image and the inspection image is used as the target image, making it easy to learn and identify defects. The difference image between the reference image and the inspection image may be defined as the difference between the lightness (luminance) of both images calculated, or, if the image can be separated into hues such as RGB, it may be defined as the difference for each hue data. The reference image used to calculate the difference may be an image printed in a state adjusted to prevent defects, or it may be generated by image processing from the original image data for printing. Note that the target image does not have to be limited to a difference image. For example, the target image may be a scanned image of a printed matter on which a uniform image of a predetermined grayscale has been printed. Furthermore, the target image, reference image, test image, and other images are not limited to images printed on a print medium, but also include images that are handled in the form of image data.

[0093] (11) In this configuration, the learning unit may perform the machine learning using the difference images, perform a correction operation to generate multiple types of difference images from the difference images corresponding to defect types different from the determined defect type or defect types associated with unknown defects, and perform the machine learning using the generated multiple types of difference images. This allows for an increase in the types of supervised learning data based on a small number of difference images, thereby improving the effectiveness of the machine learning performed by the learning unit, such as incremental learning or reinforcement learning. Examples of such correction operations include rotating the difference images by a predetermined angle, for example, 90 degrees, to obtain each rotated image, translating the difference images by a predetermined distance, for example, ¼ of the image width, to obtain each translated image, or flipping the difference images to obtain an inverted image. Alternatively, instead of performing a correction operation, additional learning or reinforcement learning may be performed collectively after a number of similar unknown defects or misidentified defects have accumulated.

[0094] (12) In this configuration, during the correction operation, the learning unit may perform an enhancement process on the difference image corresponding to the explanatory information generated by the classifier during the discrimination, and perform the machine learning, such as incremental learning or reinforcement learning, using the difference image in which the defect is enhanced. This allows for efficient machine learning, such as incremental learning or reinforcement learning. As the explanatory information, if the learning model has a configuration capable of outputting explanatory information based on similarity, this information may be used, or the user may simply input explanatory information and use it.

[0095] (13) In this configuration, the differential image generation unit may include a preprocessing unit that performs preprocessing on at least one of the reference image and the inspection image so as to reduce the difference between the reference image and the inspection image excluding the defect. This allows the portion corresponding to the defect to be efficiently extracted as a differential image. Of course, if the reference image and the inspection image are acquired using the same scanner or camera, the differential image may be generated without performing such preprocessing.

[0096] (14) In this configuration, the preprocessing unit may perform at least one of level correction, geometric correction, and alignment on at least one of the reference image and the inspection image, thereby reducing noise in the difference image.

[0097] (15) In this configuration, the geometric correction may include enlarging, reducing, or rotating at least one of the reference image and the inspection image, thereby reducing geometric errors when generating the difference image.

[0098] (16) In this configuration, the alignment may be performed by pattern matching the reference image and the inspection image, thereby reducing errors due to misalignment when generating a difference image.

[0099] (17) In this configuration, the teacher image may include an image derived by a correction operation based on the teacher image. This increases the variety of teacher images and improves learning efficiency.

[0100] (18) Another embodiment of the present disclosure relates to a defect discrimination method for an inspection image. This defect discrimination method for an inspection image includes: preparing a learning model capable of outputting a similarity for each defect type by performing machine learning in advance using a teacher image associated with a defect type, which is an image containing defects that may occur during printing; acquiring an image of a printed matter to prepare a target image to be inspected; using the learning model for the target image to obtain a similarity between defects present in the target image and the defect type; and using the similarity to discriminate the defects present in the target image as at least one of known defect types. When updating the learning model based on the results of the discrimination, machine learning is performed on a defect type different from the discriminated defect type or a defect type associated with an unknown defect. This makes it easy to discriminate various defects that may occur in a printed image. Furthermore, when updating the learning model based on the results of discriminating defects present in the inspection target image, machine learning is performed on the learning model for a defect type different from the discriminated defect type or a defect type associated with an unknown defect. This makes it possible to discriminate defects that could not be discriminated as known defects or unknown defects.

[0101] (19) In each of the above embodiments, some of the configurations realized by hardware may be replaced with software. At least a portion of the configurations realized by software may also be realized by a discrete circuit configuration. Furthermore, when some or all of the functions of the present disclosure are realized by software, the software (computer program) may be provided in a form stored on a computer-readable recording medium. The term "computer-readable recording medium" is not limited to portable recording media such as floppy disks and CD-ROMs, but also includes internal storage devices within a computer, such as various RAMs and ROMs, and external storage devices fixed to a computer, such as a hard disk. In other words, the term "computer-readable recording medium" has a broad meaning, including any recording medium capable of fixing data packets, not just temporarily.

[0102] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0103] 10...printer, 20...information processing device, 21...display unit, 22...input device, 30...scanner, 100...defect discrimination system, 110...processor, 112...print processing unit, 114...classification processing unit, 116...learning / relearning unit, 118...differential image generation unit, 120...storage device, 130...interface circuit, 200...machine learning model, 210...convolutional layer, 211...differential image, 212...text, 215...approval button, 21 6...Additional learning button, 216a...Field, 217...Reinforcement learning button, 217a...Field, 220...Primary vector neuron layer, 230...First convolution vector neuron layer, 240...Second convolution vector neuron layer, 250...Classification vector neuron layer, 300...Output section, 310...Similarity calculation section, 320...Class determination section, 330...Text, 335...Image, 337...Cleaning button, 338...Exit button

Claims

1. an image including defects that may occur during printing, the image being a learning model that has been trained in advance through machine learning using teacher images associated with defect types, and that is capable of outputting a similarity for each defect type; a target image acquisition unit that acquires an image of a printed matter and prepares a target image to be inspected; a classifier that acquires a similarity of a defect present in the target image to the defect type by using the learning model for the target image, and classifies the defect present in the target image as at least one of known defect types; a learning unit that, when updating the learning model based on the result of the discrimination by the discriminator, causes the learning model to perform machine learning on a defect type different from the discriminated defect type or a defect type associated with an unknown defect; A print image defect detection device comprising:

2. 2. The printed image defect discrimination device according to claim 1, wherein the discriminator discriminates that the defect present in the target image is the unknown defect if none of the similarities obtained for each of the known defect types is equal to or greater than a predetermined threshold.

3. 3. The print image defect discrimination device according to claim 2, wherein, when the discriminator discriminates that a defect present in the target image is an unknown defect, the learning unit receives designation of a new defect type corresponding to the unknown defect and performs additional learning of a new model to the learning model that enables discrimination of the new defect type.

4. 4. The apparatus for determining defects in a printed image according to claim 3, wherein the designation of the defect as a new defect is performed by a user setting a label for an arbitrary defect type.

5. 2. The print image defect discrimination device according to claim 1, wherein, when it is determined that the discriminator has erroneously determined a defect present in the target image, the learning unit performs reinforcement learning of the learning model in response to a specification that the erroneously determined defect is another of the known defect types.

6. 6. The apparatus for determining defects in a printed image according to claim 5, wherein the designation of the defect as another one of the known defect types is performed by selecting a defect type from among defect types learned in advance.

7. The target image acquisition unit a reference image setting unit that prepares a reference image that serves as a reference for an image in which defects are to be detected; an inspection image acquisition unit that acquires an inspection image by capturing an image of a printed matter on which an image corresponding to the reference image is printed; a differential image generating unit that generates a differential image by extracting a difference between the reference image and the inspection image as the target image; Including, The learning model is a learning model obtained by performing the machine learning using, as the training image, a difference image that is a difference between an image including a known defect that has occurred in a printed matter and the original image. The apparatus for determining defects in a printed image according to any one of claims 1 to 6.

8. The learning unit performing the machine learning using the difference image; 8. The print image defect discrimination device according to claim 7, wherein a correction operation is performed to generate a plurality of types of difference images from the difference images corresponding to a defect type different from the determined defect type or a defect type associated with an unknown defect, and the machine learning is performed using the generated plurality of types of difference images.

9. The defect discrimination device for printed images described in claim 8, wherein the learning unit, during the correction operation, performs an enhancement process on the difference image corresponding to the explanatory information generated by the discriminator during the discrimination, and performs the machine learning using the difference image in which the defect is enhanced.

10. 10. The print image defect detection device according to claim 7, wherein the differential image generation unit includes a preprocessing unit that performs preprocessing on at least one of the reference image and the inspection image so as to reduce the difference between the reference image and the inspection image excluding the defect.

11. 11. The apparatus for detecting defects in a printed image according to claim 10, wherein the preprocessing unit performs at least one of level correction, geometric correction, and alignment on at least one of the reference image and the inspection image as the preprocessing.

12. 12. The apparatus for detecting defects in a printed image according to claim 11, wherein the geometric correction includes any one of enlarging, reducing, and rotating at least one of the reference image and the inspection image.

13. 12. The apparatus for detecting defects in a printed image according to claim 11, wherein the alignment is performed by pattern matching the reference image and the inspection image.

14. The print image defect detection device according to claim 1 , wherein the teacher image includes an image derived by a correction operation based on the teacher image.

15. preparing a learning model that has been trained to be able to output a similarity for each defect type by performing machine learning in advance using a teacher image that is an image including defects that may occur during printing and that is associated with a defect type; An image of the printed matter is acquired to prepare a target image to be inspected; using the learning model for the target image to obtain a similarity of the defect present in the target image to the defect type, and using the similarity, discriminating the defect present in the target image as at least one of known defect types; When updating the learning model based on the result of the determination, machine learning is performed on a defect type different from the determined defect type or a defect type associated with an unknown defect. A method for detecting defects in a print image.

Images