Image inspection system

Abstract

To enable inspecting a recording medium with excellent accuracy.SOLUTION: An image inspection system for inspecting an image recorded on a recording medium by a recording device comprises: a first estimation unit for acquiring a first estimation result by estimating a defect in an actual recorded image representing an inspection target using a first trained model that is generated by machine learning based on recorded images recorded in the recording medium and an evaluation result of whether the recorded images are normal or abnormal; and a second estimation unit for acquiring a second estimation result by estimating the defect in the actual recorded image using a second trained model that is generated by machine learning based on recording information that is different from the recorded images and that is related to at least one of the recording device or recording medium at the time of recording the recorded images, and an estimation result of whether the recorded images are normal or abnormal. The system detects a defect in a first recording medium on the basis of the first estimation result and the second estimation result.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an image inspection system.

Background Art

[0002] As a recording device for recording images and the like on a recording medium, an inkjet recording device that discharges a liquid such as ink onto the recording medium by a liquid discharge head to record images and the like is known. In such a recording device, the recording quality may deteriorate due to various factors such as displacement of the mounting position of the recording head and deterioration of ink discharge performance.

[0003] As a method for detecting defects associated with a decrease in recording quality such as density unevenness and image defects, there is a method of acquiring an image of a recorded object and analyzing the image to detect defects. Patent Document 1 discloses a method for reducing false detection when detecting defects in an image on a recorded object using machine learning for the defects of the recorded object. In this method, the image data of the recorded object or the processed data obtained by processing the image data is used as the input data for machine learning.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the images recorded by the recording device are diverse, such as a list of character information, a combination of geometric figures, and photographs of people and landscapes. Therefore, when only image data or its processed data is used as the input data as in the above method, the accuracy may decrease when detecting defects in images with properties that have not been learned by machine learning.

[0006] In view of the above problems, an object of the present invention is to inspect a recording medium with high precision.

Means for Solving the Problems

[0007] To achieve the above object, an image inspection system of the present invention is an image inspection system for inspecting an image recorded on a recording medium by a recording device, using a first learned model generated by machine learning based on a recorded image recorded on a recording medium and an evaluation result as to whether the recorded image is normal or abnormal, to estimate a defect in a recorded image to be inspected and obtain a first estimation result; a second estimation unit that uses a second learned model generated by machine learning based on recording information, which is information related to at least one of the recording device and the recording medium at the time of recording the recorded image and is different from the recorded image, and an evaluation result as to whether the recorded image is normal or abnormal, to estimate a defect in the recorded image to be inspected and obtain a second estimation result; comprising characterized in that a defect in the recorded image to be inspected is detected based on the first estimation result and the second estimation result.

Effects of the Invention

[0008] According to the present invention, a recording medium can be inspected with high precision.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Mode for Carrying Out the Invention

[0010] Hereinafter, with reference to the drawings, embodiments for carrying out the present invention will be exemplarily and detailedly described based on examples. Note that the dimensions, materials, shapes, relative arrangements, etc. of the components described in this embodiment should be appropriately changed according to the configuration of the apparatus to which the invention is applied and various conditions. That is, the scope of the present invention is not intended to be limited to the following embodiments.

[0011] In this specification, "recording" (which may also be referred to as "printing") refers not only to the case of forming significant information such as characters and figures, but also regardless of whether it is significant or not. Furthermore, it also represents the case of forming images such as landscapes, people, patterns, and patterns on a recording medium widely, or performing processing on the medium.

[0012] Also, the "recording medium" refers not only to paper used in general recording devices, but also widely represents things that can receive ink, such as cloth, plastic film, metal plate, glass, ceramics, wood, leather, etc.

[0013] Also, "ink" (which may also be referred to as "liquid") should be interpreted widely in the same way as the above definition of "recording (printing)". Therefore, it represents a liquid that can be used for forming images, patterns, patterns, etc., processing the recording medium, or processing the ink (for example, coagulating or insolubilizing the colorant in the ink applied to the recording medium) when applied on the recording medium.

[0014] <Processing System> First, the basic configuration of the processing system 100 according to the present invention will be described. The basic configuration shown below is merely an example, and the configuration content can be appropriately changed. The processing system 100 is an image inspection system that inspects defects in a recording medium on which an image is recorded (printed) by a recording device (printing device) such as a printer 600. FIG. 1 is a diagram showing the configuration of the processing system 100. The processing system 100 is composed of a cloud server 200, an edge server 300, and a device 400 connected by a local area network 102 and the Internet 104. The processing system 100 is composed of a cloud server 200, an edge server 300, and a device 400 connected by a local area network 102 and the Internet 104.

[0015] The device 400 includes various devices capable of network connection. For example, it includes a smartphone 500, a printer 600, client terminals 401 such as personal computers and workstations, a digital camera 402, and the like. However, the device 400 is not limited to these types, and may include, for example, home appliances such as refrigerators, televisions, and air conditioners.

[0016] The various devices 400 are interconnected by a local area network 102 and can be connected to the Internet 104 via a router 103 installed in the local area network 102. The router 103 is a device that connects the local area network 102 and the Internet 104, but it is also possible to configure the processing system 100 by providing the wireless LAN access point function that constitutes the local area network 102. In this case, in addition to connecting to the router 103 via a wired LAN, each device 400 can be configured to participate in the local area network 102 by accessing and connecting via a wireless LAN. Also, for example, the printer 600 and the client terminal 401 can be configured to be connected via a wired LAN, and the smartphone 500 and the digital camera 402 can be configured to be connected via a wireless LAN.

[0017] Each device 400 and the edge server 300 can communicate with each other via the Internet 104 connected via the router 103 and the cloud server 200. The edge server 300 and each device 400 can communicate with each other via the local area network 102. Also, each device 400 can communicate with each other via the local area network 102. Also, the smartphone 500 and the printer 600 can communicate by short-range wireless communication 101. As the short-range wireless communication 101, wireless communication conforming to the Bluetooth (registered trademark) standard or the NFC standard can be considered.

[0018] Note that the configuration of the above-described processing system 100 is merely an example, and a processing system 100 with a different configuration can also be used in the application of the present invention. For example, although an example in which the router 103 has an access point function has been shown, the access point may be configured by a device different from the router 103. Further, the connection between the edge server 300 and each device 400 may use connection means other than the local area network 102. For example, it may use wireless communication such as LPWA other than wireless LAN, ZIGBEE (registered trademark), Bluetooth (registered trademark), short-range wireless communication, etc., wired connection such as USB, or infrared communication.

[0019] FIG. 2 is a block diagram showing the configuration of the cloud server 200. In this configuration, the same hardware configuration as that of the cloud server 200 and the edge server 300 is used. Therefore, the configuration of the cloud server 200 will be described below, and the description of the configuration of the edge server 300 will be omitted.

[0020] The cloud server 200 includes a main board 210 that controls the entire device, a network connection unit 201, and a hard disk unit 202. The CPU 211 in the form of a microprocessor arranged on the main board 210 operates according to a control program stored in a program memory 213 connected via an internal bus 212 and the content of a data memory 214.

[0021] The CPU 211 controls the network connection unit 201 via a network control circuit 215 to connect to networks such as the Internet 104 and the local area network 102 and communicate with other devices. The CPU 211 can read and write data to and from the hard disk unit 202 connected via a hard disk control circuit 216.

[0022] The hard disk unit 202 stores an operating system to be loaded into the program memory 213 and used, control software for the cloud server 200 and the edge server 300, as well as various types of data.

[0023] The GPU 217 is connected to the main board 210 and can execute various arithmetic processes instead of the CPU 211. Since the GPU 217 can perform efficient arithmetic operations by processing more data in parallel, it is effective to use the GPU 217 for processing when learning is performed multiple times using a learning model such as deep learning. Therefore, in this configuration, in addition to the CPU 211, the GPU 217 is used for the processing by the learning unit 251 (see FIG. 5) described later. Specifically, when executing a learning program including a learning model, learning is performed by the CPU 211 and the GPU 217 cooperating to perform arithmetic operations. Note that the processing of the learning unit 251 may be performed by only one of the CPU 211 or the GPU 217. Also, the estimation unit 352 (see FIG. 5) may also use the GPU 217 in the same manner as the learning unit 251.

[0024] Also, in this configuration, it has been described that the cloud server 200 and the edge server 300 use a common hardware configuration, but in implementing the present invention, it is not necessarily limited to this configuration. For example, the cloud server 200 may be configured to be equipped with the GPU 217 while the edge server 300 is not, or may be configured to use GPUs 217 with different performances.

[0025] FIG. 3 is a diagram showing the device configuration of the printer 600 according to this configuration. The printer 600 is a recording device that prints on continuous paper (hereinafter referred to as roll paper) 3110 that can be continuously printed as a recording medium. In this configuration, the printer 600 includes a paper feeding device 3103 that conveys the roll paper 3110. Further, the printer 600 includes a printing unit 3111 that performs printing, a paper discharging device 3104 that winds up the roll paper 3110, and an operation panel 3101.

[0026] The paper feeding device 3103 is a device that supplies the roll paper 3110 to the printer 600. The paper feeding device 3103 rotates the paper tube of the roll paper 3110 around the rotating shaft 3112, and conveys the roll paper 3110 wound around the paper tube to the printer 600 at a constant speed via a plurality of rollers (such as conveying rollers and paper feeding rollers).

[0027] The paper discharging device 3104 is a device that winds up the roll paper 3110 conveyed from the printer 600 in a roll shape around the paper tube. The paper discharging device 3104, for example, as shown in FIG. 3, holds the roll paper 3110 wound around the paper tube of the rotating shaft 3113 in a roll shape. The paper discharging device 3104 is a device that rotates around the rotating shaft 3113 and winds up the roll paper 3110 conveyed to the paper tube at a constant speed via a plurality of rollers (for example, conveying rollers and paper discharging rollers) as the product of the roll paper on the rotating shaft 3113.

[0028] As a preparation before starting printing, the roll paper 3110 is pulled out from the paper feeding device 3103, passed through to the paper discharging device 3104, and set in the printer 600. As the setting operation of the roll paper 3100, first, set the roll paper 3110 in the paper feeding device 3103, and pass the leading end of the roll paper 3110 over the skew correction device 3109. Next, pass under the printing device 3102 of the printing unit 3111, pass under the drying device 3105, pass over the cooling device 3107 and the cooling device 3108. Then, pass through the connection scanner device 3106 and wind it around the paper discharging device 3104. After passing the roll paper 3110 through the printer 600 and setting it, input a printing job to the control PC 3114 of the printer 600. Then, after inputting the printing job, press the printing start button on the operation panel 3101 to start printing.

[0029] FIG. 4 is a block diagram showing the control configuration of the printer 600. The printer 600 includes a main board 610 that controls the entire apparatus, a wireless LAN unit 608, and a short-range wireless communication unit 606. The CPU 611 in the form of a microprocessor arranged on the main board 610 operates according to a control program stored in a program memory 613 in the form of a ROM and the content of a data memory 614 in the form of a RAM, which are connected via an internal bus 612.

[0030] The CPU 611 controls the scanner unit 615 to read a document and stores the image data of the document in an image memory 616 in the data memory 614. Also, the CPU 611 can control the printing unit 617 to print the image in the image memory 616 in the data memory 614 on a recording medium. The CPU 611 controls the wireless LAN unit 608 through a wireless LAN communication control unit 618 to perform wireless LAN communication with other communication terminal devices.

[0031] Further, the CPU 611 can detect a connection with other short-range wireless communication terminals or perform data transmission and reception with other short-range wireless communication terminals by controlling the short-range wireless communication unit 606 via a short-range wireless communication control circuit 620.

[0032] The CPU 611 can control the operation panel 605 to display the state of the printer 600 and a function selection menu or receive operations from the user by controlling an operation unit control circuit 621. The operation panel 605 is provided with a backlight, and the CPU 611 can control the lighting and extinguishing of the backlight via the operation unit control circuit 621. When the backlight is extinguished, the display on the operation panel 605 becomes difficult to see, but the power consumption of the printer 600 can be suppressed. In this configuration, each process of the above-described CPU 611 can also be performed by the GPU 621.

[0033] FIG. 5 is a diagram showing the software configuration of the processing system 100. In FIG. 5, only the software components related to the learning and estimation processes in this configuration are described, and other software modules are not shown. For example, the operating systems and various middleware operating on each device and server, applications for maintenance, etc. are omitted from the illustration.

[0034] The cloud server 200 includes a learning data generation unit 250, a learning unit 251, and a learning model 252. The learning data generation unit 250 is a module that generates learning data processable by the learning unit 251 from the data received from the outside. The learning data is a pair of input data 801 for the learning unit 251 and teacher data 802 indicating the correct answer of the learning result. The learning unit 251 is a program module that performs learning on the learning model 252 using the learning data received from the learning data generation unit 250. The learning model 252 accumulates the results of the learning performed by the learning unit 251. Here, an example of realizing the learning model 252 as a neural network will be described. By optimizing the weighting parameters between the nodes of the neural network, it is possible to classify the input data or determine the evaluation value. The accumulated learning model 252 is distributed to the edge server 300 as a learned model and used for the estimation process in the edge server 300.

[0035] The edge server 300 includes a data collection / providing unit 350, an estimation data generation unit 351, an estimation unit 352, a learned model 353, and an estimation result evaluation unit 354. The data collection / providing unit 350 receives data from the device 400 and data collected by the edge server 300 itself It is a module that transmits data as a data group for learning to the cloud server 200. The estimation data generation unit 351 is a module that generates estimation data that can be processed by the estimation unit 352 based on the data sent from the device 400. The estimation unit 352 is a program module that executes estimation using the learned model 353 based on the estimation data received from the estimation data generation unit 351. The estimation result evaluation unit 354 finally evaluates the estimation result received from the estimation unit 352 and returns it to the device 400. The data sent from the device 400 and generated by the estimation data generation unit 351 is the input data 801 of the estimation unit 352.

[0036] The learned model 353 is used for estimation performed on the edge server 300. Assume that the learned model 353 is also realized as a neural network in the same way as the learning model 252. However, as will be described later, the learned model 353 may be the same as the learning model 252, or may extract and use a part of the learning model 252. The learned model 353 stores the learning model 252 accumulated and distributed by the cloud server 200. The learned model 353 may distribute all of the learning model 252, or may extract and distribute only a part of the learning model 252 that is necessary for estimation on the edge server 300.

[0037] Device 400 includes an application unit 450, a data acquisition unit 451, a data transmission / reception unit 452, and a display control unit 453. The application unit 450 is a module that realizes various functions executed by the device 400 and is a module that utilizes a learning / estimation mechanism based on machine learning. The display control unit 453 is a module that controls the display of the application unit 450. The data acquisition unit 451 can acquire any data that the printer 600 may have, such as the image data acquired by the scanner unit 615 and stored in the image memory 616, and the information of the sensor attached to the printing unit 617 and stored in the data memory 614. The data transmission / reception unit 452 is a module that requests learning or estimation from the edge server 300. During learning, data used for learning is transmitted to the data collection / providing unit 350 of the edge server 300 according to a request from the application unit 450. Also, during estimation, data used for estimation is transmitted to the edge server 300 according to a request from the application unit 450, and the result is received and returned to the application unit 450.

[0038] Note that in this configuration, the learning model 252 learned by the cloud server 200 is distributed to the edge server 300 as the learned model 353 and used for estimation, but it is not limited to this form. Where to execute learning and estimation respectively on the cloud server 200, the edge server 300, and the device 400 may be determined according to the distribution of hardware resources, the amount of calculation, and the magnitude of data traffic. Alternatively, it may be configured to change dynamically according to the increase or decrease of the distribution of these resources, the amount of calculation, and the data traffic. When the entities performing learning and estimation are different, the estimation side can reduce the logic used only for estimation and the capacity of the learned model 353, or configure it to be executed more quickly.

[0039] Next, the input / output structure in the learning model 252 and the learned model 353 will be described in detail. FIGS. 6(a) and (b) are conceptual diagrams showing the input / output structure when using the learning model 252 and the learned model 353.

[0040] Fig. 6(a) shows the relationship between the learning model 252 and its input / output data during learning. The input data 801 is the data of the input layer of the learning model 252. The details of the input data 801 of the learning model 252 will be described later. As a result of evaluating (recognizing) the input data 801 using the learning model 252 which is a machine learning model, output data 803 is output. During learning, since teacher data 802 is given as the correct data of the evaluation result of the input data 801, by giving the output data 803 and the teacher data 802 to the loss function 804, the deviation amount L(805) from the correct answer of the recognition result is obtained. The coupling weight coefficients between the nodes of the neural network in the learning model 252 are updated so that the deviation amount L becomes small for a large number of learning data. The error backpropagation method is a method of adjusting the coupling weight coefficients between the nodes of each neural network so that the above error becomes small. Specific algorithms of machine learning include the nearest neighbor method, the naive Bayes method, decision trees, support vector machines, and the like. In addition, deep learning (deep learning) that generates the feature amounts and coupling weight coefficients for learning by itself using a neural network is also included. In this configuration, as the machine learning algorithm, the above-mentioned ones can be appropriately selected and used.

[0041] Fig. 6(b) shows the relationship between the learned model 353 and its input / output data during estimation. The input data 811 is the data of the input layer of the learned model 353. The details of the input data 801 of the learned model 353 will be described later. As a result of recognizing the input data 801 using the learned model 353 which is a machine learning model, output data 812 is output. During estimation, this output data 803 is used as the estimation result. Note that although the learned model 353 at the time of estimation has been described as having a neural network equivalent to the learning model 252 at the time of learning, it is also possible to prepare as the learned model 353 an extracted one of only the necessary parts for estimation. Thereby, it is possible to reduce the data amount of the learned model 353 or shorten the neural network processing time at the time of estimation.

[0042] In a recording apparatus such as a printer 600, the recording quality may deteriorate due to various factors, and defects may occur in the recorded material (printed matter). For example, in an inkjet recording apparatus that performs a recording operation by ejecting a liquid such as ink with a liquid ejection head, an error may occur in the attachment position of the recording head or the relative attachment position between a plurality of recording heads. This error causes a shift in the ink landing position on the recording medium, which is a factor in the deterioration of the recording quality. In addition, the recording head may have variations in ejection characteristics such as ejection amounts among a plurality of nozzles due to manufacturing errors or the like. This variation causes density unevenness, which is also a factor in the deterioration of the recording quality. Furthermore, when the ejection from the nozzles is not good, it is also a factor in the deterioration of the recording quality.

[0043] As a method for detecting defects in the recorded material due to such deterioration of the recording quality, conventionally, a special printing pattern has sometimes been used. For example, by using a reading device such as a scanner to read a special printing pattern and verifying whether the printing pattern is printed normally, it is also possible to detect a deterioration in the recording quality. However, with such a method, it is very difficult to comprehensively detect all the defects that may occur in the recorded material due to various factors. In addition, ink and time are consumed for printing the printing pattern. Therefore, in this configuration, in order to suppress a decrease in the productivity of the recording apparatus and perform high-precision defect detection of the recorded material, the machine learning results in the learning unit 251 and the learning model 252 are used. Next, a specific method for detecting defects in the printed matter of the printer 600 using the processing system 100 configured as described above will be described in a plurality of embodiments.

[0044] <Example 1> As Example 1 of the present invention, a method for estimating and detecting defects in the printed matter of the printer 600 by a processing system 100 including machine learning means will be described.

[0045] FIG. 7(a) and (b) are examples of image data obtained by the scanner unit 615 from the printed matter printed by the printing unit 617 and stored in the image memory 616. Hereinafter, such an image is referred to as "printed image data". That is, the printed image data is obtained by acquiring a recorded image recorded on a recording medium with a scanner or the like. The printed image data 750 in FIG. 7(a) is the printed image data of a normally printed printed matter. The printed image data 751 in FIG. 7(b) includes a defective portion 752 and is the printed image data of a printed matter that was not printed normally. Defects during printing (recording) include, for example, missing due to ink non-ejection, recording position deviation due to head attachment error, difference in ink ejection condition, color unevenness due to non-uniformity, etc. In the case of missing due to ink non-ejection, a white area such as the defective portion 752 in FIG. 7(b) occurs. Furthermore, there is also a possibility that a defect may occur in the printed matter due to a break in the roll paper 3110 itself. Although other defect examples are not shown, the defects detected by the present invention are not limited to those due to missing caused by ink non-ejection. The present invention estimates and detects defects in printed matter using a learned model that takes printed image data as input. However, the contents of printed matter are various, and there may be landscapes like FIG. 7(a), as well as illustrations, characters, barcodes, etc. For example, in the case of a printed matter with a geometric illustration, a white area such as the defective portion 752 in FIG. 7(b) may be included as a design and may be a normal area. That is, in a learned model that has learned the defective portion 752 in FIG. 7(b) as abnormal, there is a possibility of misjudging the white area of the illustration as a defect. Furthermore, generally speaking, it cannot be said for sure that the accuracy of a learned model by machine learning always guarantees 100%. From the above, in the present invention, in the estimation and detection of defects in printed matter using a learned model that takes printed image data as input, a method is provided to prevent misjudgment and obtain more accurate results by additionally using a learned model that takes data different from the printed image data as input.

[0046] The present invention estimates and detects defects in printed matter using a learned model that takes printed image data as input. However, the contents of printed matter are various, and there may be landscapes like FIG. 7(a), as well as illustrations, characters, barcodes, etc. For example, in the case of a printed matter with a geometric illustration, a white area such as the defective portion 752 in FIG. 7(b) may be included as a design and may be a normal area. That is, in a learned model that has learned the defective portion 752 in FIG. 7(b) as abnormal, there is a possibility of misjudging the white area of the illustration as a defect. Furthermore, generally speaking, it cannot be said for sure that the accuracy of a learned model by machine learning always guarantees 100%. From the above, in the present invention, in the estimation and detection of defects in printed matter using a learned model that takes printed image data as input, a method is provided to prevent misjudgment and obtain more accurate results by additionally using a learned model that takes data different from the printed image data as input.

[0047] In Example 1, the data different from the print image data used for detecting defects in the printed matter is data related to what can cause defects in the printed matter in the printer 600. More specifically, the data different from the print image data used for detecting defects in the printed matter is recording information related to the printer 600 and the roll paper 3110 which is a recording medium at the time of printing the printed matter. For example, for nozzle clogging which causes non-ejection, there are various factors such as a long print accumulation time, a long time of leaving the nozzles unprotected without a cover or the like, a low nozzle cleaning frequency, a high ink density, and the like. Therefore, the print accumulation time, the time of leaving the nozzles unprotected without a cover or the like, the number of nozzle cleaning times, the ink density, etc. can be used as input data.

[0048] Also, as a cause of the recording position deviation, there is a variation in the fixed position of the recording head itself of the printer 600. Therefore, for example, position information such as the detection sensor value of the position of the recording head can be used as input data. The position of the recording head can be obtained by installing a distance sensor on the device side and acquiring the distance from a certain component position where the recording head is located.

[0049] Also, as a cause of the recording position deviation, there are changes in the conveyance distance and speed of the medium (recording medium) such as the roll paper 3110. When the conveyance speed of the medium changes, it deviates from the ejection timing of each ink color that has been optimized in advance, resulting in a recording position deviation. As a cause of the change in the conveyance speed of the medium, there are defects in the components and control that make up the conveyance path. Therefore, sensor values that capture changes in the shape of the components, values of the environmental temperature (ambient temperature) and environmental humidity that can cause changes in the shape of the components, etc. can be used as input data.

[0050] As components constituting the conveyance path, for example, there are rollers for conveyance. By arranging a distance sensor or the like around the roller, it is possible to detect a change in the roller diameter. In particular, when the medium is roll paper, since the medium itself is conveyed while being wound around the paper feed side and the paper discharge side, the conveyance accuracy may be affected by changes in the medium properties due to differences in the type of medium and the amount of moisture contained due to humidity. Therefore, parameters for determining the medium type (such as paper type name, material, size, thickness, coating type, etc.) can be used as input data.

[0051] Also, the causes of differences in the ink ejection state may be the same as those of non-ejection. Here Examples of data different from the print image data among the data used as input data have been described. However, any data related to what can cause defects in the printed matter in the printer 600 other than the above can be used as input data.

[0052] As described above, in the first embodiment, since the machine learning results using two or more types of input data are utilized, two types of learning models 252, learned models 353, and estimation units 352 are provided. Hereinafter, among the learning models 252, those that take print image data as input will be referred to as the first learning model, and those that take data different from the print image data as input will be referred to as the second learning model for explanation. Similarly, among the learned models 353, those that take print image data as input will be referred to as the first learned model, and those that take data different from the print image data as input will be referred to as the second learned model for explanation. Similarly, among the estimation units 352, those that utilize the machine learning results of the first learned model will be referred to as the first estimation unit, and those that utilize the machine learning results of the second learned model will be referred to as the second estimation unit for explanation. Also, other matters related to machine learning will be distinguished and explained with "first" and "second" as necessary. However, the first learning model and the second learning model may constitute one learning model 252, and the estimation unit 352 may be a common single estimation unit.

[0053] (Defect Detection Flow of Printed Matter) Next, the processing flow of defect detection of the printed matter (actual printing result) of the printer 600 by the processing system 100 according to the first embodiment will be described. FIG. 8 is a flowchart showing an overall outline of defect detection of the processing system 100.

[0054] First, in step S801, using a first pre-trained model that takes as input the print image data, which is the image data obtained by scanning the actual print image (actual recorded image) of the actual printed matter (actual recording medium) printed by the printer 600, the presence or absence of defects in the print image data is estimated. That is, the first estimation unit uses the first pre-trained model to obtain a first estimation result as to whether there are defects in the actual recorded image of the actual printed matter to be inspected based on the print image data. When step S801 ends, the process proceeds to step S802.

[0055] In step S802, using a second pre-trained model that takes as input data different from the print image data described above as various parameters when printing the actual printed matter, the presence or absence of defects during printing is estimated. That is, the second estimation unit uses the second pre-trained model to obtain a second estimation result as to whether there are defects in the actual recorded image of the actual printed matter to be inspected based on data different from the print image data. As described above, the data different from the print image data is the recording information related to the printer 600 and the roll paper 3110, which is the recording medium. When step S802 ends, the process proceeds to step S803.

[0056] In step S803, based on the estimation results in steps S801 and S802, the presence or absence of defects in the printed matter is finally determined. Here, the estimation result evaluation unit 354 obtains a final estimation result as to whether there are defects in the printed matter based on the first estimation result and the second estimation result. When step S803 ends, the process proceeds to step S804.

[0057] In step S804, the printer is notified of the presence or absence of defects in the printed matter on which the print image data is based. The following describes in detail the learning to obtain each trained model for carrying out these steps, the estimation using each trained model, and the final determination of the presence or absence of defects using the results of both models. Note that, although FIG. 8 describes an example in which the two estimations are carried out in sequence, they may be carried out in parallel.

[0058] (Learning operation of the processing system) Next, the operation of the processing system 100 during learning will be described. Fig. 9 is a sequence diagram showing the overall operation of the processing system 100 during learning. This sequence begins when print image data is input. This is common to both the case of generating a first trained model using input data as input data and the case of generating a second trained model using input data that is different from the print image data at the time of printing of the printed matter on which the print image data is based.

[0059] In step S911, input data to be learned is acquired in the printer 600. The learning target when generating the first learned model is the print image data of all actual printed matter printed by the printer 600.

[0060] The learning target when generating the second trained model is various parameters when printing an actual printed matter that is the basis of the print image data input to the first trained model. The learning target when generating the second trained model is acquired in a format such as log data including such information for a period corresponding to the printing start time to the printing end time of the actual printed matter. It is desirable that the various parameter data are recorded at least once during the printing time. In addition, it is desirable that all the various parameter data are recorded at the same cycle, but the cycle may be different for each parameter. For example, when the printing speed is one sheet per second, it is desirable that the various parameter data are recorded at a cycle of once or more per second. In addition, printing time information of each printed matter may be included in the various parameter data, or may be acquired separately from the various parameter data and linked to the various parameter data later.

[0061] When input data is acquired, in step S912, a learning request is sent from the printer 600 to the edge server 300. Next, in step S913, a learning request is sent from the edge server 300 that has received the learning request from the printer 600 to the cloud server 200.

[0062] Upon receiving the learning request, in step S914, the cloud server 200 generates learning data by the learning data generation unit 250 from the received learning request. Details of the generation of the learning data will be described later. Next, in step S915, learning is executed by the learning unit 251 (the first learning unit and the second learning unit). Next, in step S916, based on the machine learning result, the learning model 252 (the first learning model and the second learning model) is updated and stored.

[0063] When the learning by the learning unit 251 and the accumulation of the learning result are completed, the cloud server 200 generates and distributes a learned model 353 (the first learned model and the second learned model) to be distributed from the learning model 252 to the edge server 300 in step S917. The edge server 300 that has received the learned model 353 distributed in step S917 reflects it in the learned model 353 it holds and stores it in step S918. As a result, the updated learned model 353 will be used for subsequent estimation requests.

[0064] When generating the first learned model and the second learned model, step S911 may be started at the same timing, or the generation of the second learned model may be started at the time of step S916 or step S918 when the generation of the first learned model is completed. The starting timing is preferably the time when the generation of a new piece of printed image data is completed, the time when the generation of the printed image data of grouped printed matter is completed, or the time when printed image data is accumulated for an arbitrary period such as one day, that is, the time when unlearned data is accumulated. If it is desired to always learn the data of a new printed matter and update the learned model, it is good to acquire it at the time when the generation of a new piece of printed image data is completed. Also, the timing of generating and updating the learned model may be automatic or manual.

[0065] (Learning data generation flow) Next, the learning data generation flow by the learning data generation unit 250 will be described. FIG. 10 is a flowchart showing the learning data generation flow processed by the learning data generation unit 250. When the learning data generation unit 250 acquires the input data in step S1001, it checks the content of the input data in step S1002 and determines whether the input data is the printed image data for generating the first learned model. If the input data is the printed image data, that is, if it is YES in step S1002, it proceeds to step S1003. If the input data is not the printed image data, that is, if it is NO in S1002, it proceeds to step S1006. Here, the case where the input data is not the printed image data is, for example, the case where the input data includes various parameter data for generating the second learned model.

[0066] In step S1003, the acquired printed image data is divided. By dividing the image, only the divided image data of the defective area in the whole image can be used as abnormal data, and the divided image data of the non-defective area can be used as normal data for learning. By dividing the image in this way, the learning data is expanded compared to the case where the printed image data itself is used as one piece of learning data. Also, if a defect is detected in the printed image data itself without division, it is not possible to clarify where the defect is in the printed image data. Therefore, by dividing the image, it becomes possible to grasp the position of the defective area in the whole image at the time of estimation. The divided image data becomes the input data 801 to the learning model 252 at the time of generating the first learned model. The details of the division process will be described later with reference to FIG. 12. When step S1003 ends, it proceeds to step S1004.

[0067] In step S1006, adjustment processing of the acquired various parameter data is performed. The data adjustment processing will be described later with reference to FIG. 13. The data after the adjustment processing becomes the input data 801 to the learning model 252 at the time of generating the second learned model. When step S1006 ends, the process proceeds to step S1004.

[0068] In step S1004, the learning data generation unit 250 associates a label with the ID of each data. The label, which is the associated data associated with the ID, becomes the teacher data 802 for the learning model 252. The association between the ID and the label will be described later with reference to FIG. 14. When step S1004 ends, the process proceeds to step S1005. In step S1005, the learning data generation unit 250 stores the associated data of each data, ID, and label as learning data.

[0069] (Processing flow of the learning data generation unit) Next, each processing flow (image segmentation processing and data adjustment processing) by the learning data generation unit 250 will be described. FIGS. 11(a) and (b) are flowcharts of each process processed by the learning data generation unit 250.

[0070] [Image segmentation processing flow] FIG. 11(a) is a processing flow of image segmentation. In step S1101, the learning data generation unit 250 sets the divided image size (horizontal width x, vertical width y) and the shift width z. Next, in step S1102, the learning data generation unit 250 sets the area of the divided image size as the cutout range starting from the upper left of the printed image data. Then, in step S1103, the divided image within the cutout range is created, assigned an ID, and saved. The ID may be any content that can be uniquely associated with each divided image by a numerical value or a character string.

[0071] In step S1104, the learning data generation unit 250 determines whether the right end of the clipping range in step S1103 has reached the right end of the printed image data. If the right end of the clipping range has not reached the right end of the printed image data, that is, if the answer in step S1104 is NO, the process proceeds to step S1105. If the right end of the clipping range has reached the right end of the printed image data, that is, if the answer in step S1104 is YES, the process proceeds to step S1106.

[0072] In step S1105, the learning data generation unit 250 shifts the clipping range to the right by a width z. Then, the process proceeds to step S1103, and the divided image data within the clipping range is saved again in step S1103. The learning data generation unit 250 repeats this process until the right end of the clipping range reaches the right end of the printed image data.

[0073] In step S1106, the learning data generation unit 250 determines whether the lower end of the clipping range has reached the lower end of the printed image data. If the lower end of the clipping range has not reached the lower end of the printed image data, that is, if the answer in step S1106 is NO, the process proceeds to step S1107. If the lower end of the clipping range has reached the lower end of the printed image data, that is, if the answer in step S1106 is YES, the process ends.

[0074] In step S1107, the learning data generation unit 250 shifts the clipping range downward by a width z and then shifts it to the left. Then, the process proceeds to step S1103, and the divided image data within the clipping range is saved again in step S1103. Finally, when the lower right end of the clipping range reaches the lower right end of the printed image data, the entire range of the printed image data is saved as divided image data, and the learning data generation unit 250 ends the process.

[0075] Here, the shift width z is set to the same value for both the right shift case and the downward shift case. However, if the shift width z is smaller than the divided image size (horizontal width x or vertical width y), different values can be used for the shift width in the right shift case and the downward shift case.

[0076] Also, here, the case where the shift width z is set as a value divisible by the horizontal width x and the vertical width y of the printed image data was taken as an example to describe the processing flow of image division, but a non-divisible value may also be used. In this case, as a result of shifting to the right or downward in the processes of step S1105 and step S1107, the range of the printed image data will be exceeded. In such a case, it is preferable to adopt a configuration in which the learning data generation unit 250 adds a process of correcting the shift width z in step S1105 and step S1107 so as not to exceed the size range of the printed image data.

[0077] [Data adjustment processing flow] FIG. 11(b) shows the processing flow of data adjustment. In step S1121, the learning data generation unit 250 arranges various parameters and performs data sorting. Specifically, the data is arranged in a format in which various parameters are arranged for each time. After arranging the various parameters, the process proceeds to step S1122.

[0078] In step S1122, the learning data generation unit 250 checks for the presence or absence of missing data at each printing time during the printing time when the data was collected. For example, if the recording period of a predetermined data is long relative to the cycle of the printing time for acquiring the data, there may be a case where the predetermined data is not present (not acquired) at a certain printing time. If there is missing or defective data in this way, that is, if the result in step S1122 is YES, the process proceeds to step S1124. If there is no missing data, that is, if the result in step S1123 is NO, the process proceeds to step S1123.

[0079] In step S1124, the learning data generation unit 250 performs data interpolation processing. In the data interpolation processing, a process of storing the missing data is performed. Details of the data interpolation processing will be described later. When the data interpolation processing ends in step S1124, the process proceeds to step S1123.

[0080] In step S1123, the learning data generation unit 250 deletes data other than the printing time, that is, data at times when printing is not performed and the printed matter cannot be evaluated. After that, the processing ends.

[0081] Next, an example of the image segmentation process by the learning data generation unit 250 will be described. FIG. 12 shows an application example of the image segmentation process. FIG. 12 shows a state in which one printed image data 1200 is divided into four first divided images 1210, second divided images 1211, third divided images 1212, and fourth divided images 1213.

[0082] The example shown in FIG. 12 is a case where the printed image data 1200 with a horizontal width X and a vertical width Y is processed by specifying a divided image size 1201 with a horizontal width x and a vertical width y and a shift width z, and four divided images (1210, 1211, 1212, 1213) are obtained in order. Although an example of obtaining four divided images is shown in FIG. 12, the number of obtained divided images varies depending on the balance between the size of the printed image data 1200, which is the overall image, and the horizontal width x, vertical width y, and shift width z, which are the sizes after division. For example, if X is 1000, Y is 500, x is 100, y is 100, and z is 10, 3731 divided images (91 in the X direction × 41 in the Y direction) can be obtained. The divided image size may be determined from a plurality of sizes with good accuracy during learning, or may be made larger in consideration of speeding up the estimation time using the same divided image size as during learning. Although details will be described later, if it is desired to present a more detailed position when presenting a defective portion of the estimation result, it is advisable to make the divided image size smaller. Also, since the resolution improves when presenting the result of detecting a defect as the shift width z is made smaller, the shift width z may be made smaller if it is desired to detect the defective position in detail.

[0083] Next, an example of the data adjustment process by the learning data generation unit 250 will be described. FIGS. 13(a) to (c) are application examples of the data adjustment process and are examples of input data in each step of the data adjustment process processed by the learning data generation unit 250.

[0084] FIG. 13(a) is a table showing an example of input data for which the process of step S1121 has been performed. As examples of parameters of input data that is different from print image data, print integration time, ink density, temperature, humidity, media thickness, and media coating are given. In addition to these parameters, the table shows the date and time and the name of the printed matter. In step S1121, the input data acquired from the printer 600 is first converted into a format in which the parameter values corresponding to the same time (the same date and time) are arranged as shown in FIG. 13(a). Although the number of parameters has been reduced for simplicity of explanation, in reality, it is not limited to those shown in FIG. 13(a), and all parameter data is arranged. For the name of the printed matter, if printing is in progress, the name of the printed matter is entered, and if printing is not in progress, a string indicating that there is no printed matter such as blank or "none" is entered. From the content of the name of the printed matter, it is possible to determine whether the various parameter data at each time is being printed. FIG. 13(a) shows an example in which one sheet is printed every 3 seconds and various parameter values are acquired every second. In this example, since there is no defective data at each printing time, the result in step S1122 is NO, and the process proceeds to step S1123 without performing data interpolation processing.

[0085] FIG. 13(b) is a table showing an example of input data for which step S1121 has been performed, and shows an example different from FIG. 13(a). In the example of FIG. 13(b), unlike the example of FIG. 13(a), there is defective data at a certain printing time. In this example, the print integration time is acquired at 5-second intervals, and the ink density, temperature, and humidity are acquired at 2-second intervals. The acquisition timings of the ink density and temperature are the same, but the acquisition timing of the humidity is different. Also, for the media thickness and coating, the values at the time of printing are input for all times.

[0086] If data for all parameters can be acquired at all times, there is no problem as learning data. However, if there are times when there is no data as shown in FIG. 13(b), it is preferable to adjust such missing values before learning. As a method for adjusting missing values, for parameters with missing values There is also a method of deleting data at times when there are missing values in any of the items, but in the first embodiment, interpolation is performed in order to increase the learning data as much as possible. Assuming that it is known that the printed integration time is added in seconds, the missing value can be calculated by adding the number of seconds of the difference from the time when there is valid data. For ink density, temperature, and humidity, since they are values with gradually changing properties, it is preferable to interpolate the missing values with the average value of the valid data before and after. When there is no data before or after, such as humidity, the data at the closest time can be used as it is. By performing such a complementary process, the data after the data adjustment process in Fig. 13(b) becomes equivalent to that in Fig. 13(a).

[0087] In this example, the average value process using the valid data before and after is used for processing missing values. However, as long as it is an interpolation method suitable for the data, other average value processing methods or interpolation processing methods other than the average value processing can also be applied. Also, although the interpolation of missing values and the like are described with reference to Fig. 13(b), if all data can be acquired at the same timing so that such processing is unnecessary, the data quality will be improved and the learning accuracy will be improved, which is more desirable. Also, although the case where the data becomes the same as that in Fig. 13(a) by performing the interpolation process in Fig. 13(b) is given, depending on the data before interpolation and the interpolation method, there may be cases where it does not become exactly the same.

[0088] Fig. 13(c) is the result of deleting data other than the printing time in step S1123, and it becomes learning data. Specifically, in the example of Fig. 13(a), when it is determined that the printed matter name is "none" and printing is not performed, the data at that time is deleted as unnecessary in the input data.

[0089] (ID and label linking process) Next, the ID and label linking process in step S1004 in the learning data generation flow of Fig. 10 will be described. Figs. 14(a) to (c) are explanatory diagrams of the ID and label linking process.

[0090] FIG. 14(a) shows, with dotted frames, three of the cropping ranges for creating divided images with respect to the printed image data 751. The cropping range 1401 shown in FIG. 14(a) is the upper left region that is cropped first in the division process, and the cropping ranges 1402 and 1403 are regions near the lower center that are cropped during the division process. Also, with respect to the cropping range 1402, the cropping range 1403 is a region shifted to the right by a shift width z. In this example, the cropping ranges 1402 and 1403 include the defective portion 752, which is a white area.

[0091] FIG. 14(b) is data associating IDs and labels for the divided images that are input data for generating the first learned model. The data shown in FIG. 14(b) includes, but is not limited to, the printed matter name, the starting coordinates that are the coordinate information of the divided image, the divided image size, and the divided image data name, in addition to the ID and the label. When the image division process is completed, an ID is assigned to each divided image and a label is given. The label is the content of the result of labeling whether each divided image is a normal region or an abnormal region having a defect. For example, the divided image of the cropping range 1401 is labeled as normal because it is a region without defects, and the divided images of the cropping ranges 1402 and 1403 are labeled as abnormal because they are regions including the defective portion 752. Thus, the label is the evaluation result of the defect of the printed matter and is used as the first teacher data of the first learning model. The labeling method can be manual labeling by visually checking each divided image, but automatic processing is also possible. As an automatic processing labeling method, the coordinate information of the defective region is managed before the image division process, and if the coordinate region of the defective region is included in the coordinate region of each divided image data, which can be known from the coordinate information of each divided image data, it can be labeled as abnormal.

[0092] In addition, for the ID, the printed matter name, the starting coordinates and size of each divided image, the media type and thickness of the printed matter, the coating, and the divided image data name are managed simultaneously. The linking data between the ID and information other than the label may be managed as separate data. Note that the above content is an example of the linking data between the ID and the label, and it is not necessary to include all of the items listed above, and it is also possible to include information other than the above. However, it is desirable to include at least the label and the divided image data name. For learning, it is sufficient to know the linking of the label information to each divided image data.

[0093] If coordinate information is included as data, it becomes possible to include the defective position when notifying the detection result of a defect. The coordinate information is not limited to the starting coordinates, and may be any content that uniquely determines the coordinates of each divided image. Also, instead of listing the linking of the ID and the label, the divided image data may be stored in a normal data storage folder and an abnormal data storage folder. In this case, it is advisable to include coordinate information in the file name of the divided image data. Also, binary data may be added to the linking data without saving it as an image file. From here on, it is described for the case of saving as an image file.

[0094] Figure 14(c) is the linking data of the ID and the label for various parameters that are the input data for generating the second learned model. The data shown in Figure 14(c) includes, but is not limited to, the printed matter name, the printing integration time, the ink density, the temperature, the humidity, the media thickness, and the coating in addition to the ID and the label. When the data adjustment process is completed, an ID is assigned instead of the time data, and a label is given. The label is labeled as normal if the printed image data obtained at each data acquisition has no overall defect like the printed image data 750, and abnormal if there is a defective area even in part like the printed image data 751. The label is the evaluation result of the defect of the printed matter as described above, and is also used as the second teacher data of the second learning model. Also, although an example is given in which the printed matter name is given as reference information, it is not essential.

[0095] (Input / Output Structure of the Learning Model during Learning) Next, the input / output structure of the learning model 252 during learning will be described. FIGS. 15(a) and (b) are diagrams showing the input / output structure of the learning model 252 during learning.

[0096] FIG. 15(a) shows the input / output structure of the first learning model for generating the first learned model. As the learning input data 801 (first input data) of the first learning model, all the divided image data of the full print image data is used. Also, as the teacher data 802 (first teacher data) of the first learning model, the label information included in the ID-label association data is used. Learning is performed so that the deviation amount L between the output when each divided image data is input to the learning model 252 and the label information, which is the teacher data 802 corresponding to each divided image, becomes minimum, and the learning model 252 is updated.

[0097] The deviation amount L can be, for example, the difference between the defect probability of the output data 803 and the defect probability of the teacher data 802. In this case, it is configured such that when all the divided image data is input as the input data 801, the output data 803 is given as a numerical value as the defect probability. And in the teacher data 802, by configuring it to be numerically obtained as a defect probability of 0% if the label information is normal and 100% if abnormal, the deviation amount L can be defined as described above. Note that the definition method of the deviation amount L described here is an example, and the deviation amount L only needs to be able to compare these two numerical values. Also, the correspondence between each divided image and the label can be obtained from the ID, etc.

[0098] FIG. 15(b) shows the input / output structure of the second learning model for generating the second learned model. As the learning input data 801 (second input data) of the second learning model, as the recording information related to the printer 600 and the roll paper 3110 which is a recording medium, all kinds of parameter data It is used. Also, as the teacher data 802 (second teacher data) of the second learning model, the label information included in the ID-label association data is used. That is, learning is performed so that the deviation amount L between the output when various parameter data is input to the learning model 252 and the label information which is the teacher data 802 corresponding to the various parameter data is minimized, and the learning model 252 is updated.

[0099] Similar to the first learning model, the deviation amount L can be the difference between the defect probability of the output data 803 and the defect probability of the teacher data 802. In this case, it is configured such that when various parameter data is input as the input data 801, the output data 803 is given as a numerical value for the defect probability. And in the teacher data 802, by configuring it so that the label information is digitized to a defect probability of 0% if normal and 100% if abnormal, the deviation amount L can be defined as described above. Note that the method of defining the deviation amount L described here is an example, and the deviation amount L only needs to be able to compare these two numerical values. The correspondence between various parameter data and the label can be obtained from the ID etc.

[0100] Also, the input data 801 for learning does not have to be all various parameter data. Therefore, the learning unit 251 can also create and save the learning model 252 for each combination of input data that does not input one or more of the various parameters. If there are such a plurality of learning models 252, even when not all various parameters have been obtained at the time of estimation, it is possible to detect a defect using only the obtained various parameters as the input data.

[0101] (Operation of the processing system at the time of estimation) Next, the operation of the processing system 100 at the time of estimation will be described. FIG. 16 is a sequence diagram showing the movement of the entire processing system 100 at the time of estimation. This sequence is common to the case of using the first learned model and the case of using the second learned model, and corresponds to the operations of steps S801 and S802 in the defect detection of printed matter.

[0102] In the estimation process by the processing system 100, first, in step S1601, input data to be estimated is acquired in the printer 600. The input data for the first learned model is printed image data printed by the printer 600. The input data for the second learned model is various parameters when printing an actual printed matter that is the basis of the printed image data input to the first learned model. The input data for the second learned model is acquired in a format such as log data including information on various parameters during the period from the start time to the end time of printing the actual printed matter.

[0103] It is desirable that the various parameter data be recorded at least once or more during the printing time. Also, similar to the learning time, it is desirable that all the various parameter data be recorded at the same cycle, but it may be different cycles for each parameter. Further, the printing time information of each printed matter may be included in the various parameter data, or may be acquired separately from the various parameter data and linked to the various parameter data later.

[0104] The data acquisition unit 451 determines whether to acquire the input data. By acquiring the printed image data and various parameter data generated each time one sheet is printed during printing, it becomes possible to detect defects in real time during printing. When the input data is acquired in step S1601, in step S1602, an estimation request is sent from the printer 600 to the edge server 300. When the edge server 300 receives the estimation request from the printer 600, in step S1603, the estimation data generation unit 351 generates estimation data. Details of the generation of the estimation data will be described later. Next, in step S1604, estimation is performed by the second learned model. The output data 812 of the estimation result is the probability that each divided image data includes a defect in the case of using the first learned model. Also, the output data 812 of the estimation result is the probability that there is a defect in the printed image data acquired based on the set of various parameter data to which an ID is assigned in the case of using the second learned model. The obtained estimation result is saved in step S1605, and the process ends. The output data 812 of the estimation result is the probability that each divided image data includes a defect in the case of using the first learned model. Also, the output data 812 of the estimation result is the probability that there is a defect in the printed image data acquired based on the set of various parameter data to which an ID is assigned in the case of using the second learned model. The obtained estimation result is saved in step S1605, and the process ends.

[0105] (Flowchart for generating data for estimation) Next, the flowchart for generating data for estimation by the data generation unit 351 for estimation will be described. FIG. 17 is a flowchart showing the flowchart for generating data for estimation that the data generation unit 351 for estimation processes. When the data generation unit 351 for estimation acquires input data in step S1701, it checks the content of the input data in step S1702 and determines whether the input data is printed image data for generating a first learned model. If the input data is printed image data, that is, if the result in step S1702 is YES, the process proceeds to step S1703. If the input data is not printed image data, that is, if the result in step S1702 is NO, the process proceeds to step S1707. Here, the case where the input data is not printed image data is, for example, the case where the input data includes various parameter data for generating a second learned model.

[0106] In step S1703, the data generation unit 351 for estimation acquires the divided image size information during learning from the data collection and provision unit 350 via the learning data generation unit 250. If, in step S918 shown in FIG. 9, the divided image size is saved together with the learned model 353 in the edge server 300, the data generation unit 351 for estimation acquires that information. When step S1703 ends, the process proceeds to step S1704.

[0107] In step S1704, the data generation unit 351 for estimation divides the data obtained by the same process as in step S1003 of FIG. 10. The divided image data becomes the input data 811 to the learned model 353. When step S1704 ends, the process proceeds to step S1705.

[0108] In step S1705, the estimation data generation unit 351 assigns and associates an ID and coordinate information to each of the divided image data. The data for associating the ID and the coordinate information will be described later. When step S1705 ends, the process proceeds to step S1706. In this case, in step S1706, the estimation data generation unit 351 stores the association data between the divided image data, the ID, and the coordinate information as estimation data.

[0109] In step S1707, the estimation data generation unit 315 performs adjustment processing on the acquired various parameter data. The data adjustment processing performed here is the same as the processing in step S1006 of FIG. 10. When step S1707 ends, the process proceeds to step S1708.

[0110] In step S1708, an ID is assigned in the same manner as in step S1004 of FIG. 10. When step S1708 ends, the process proceeds to step S1706. In this case, in step S1706, the estimation data generation unit 351 stores the association data between the various parameter data and the ID as estimation data.

[0111] Next, an example of the estimation data will be described. FIGS. 18(a) and (b) are examples of the estimation data.

[0112] FIG. 18(a) is an example of the estimation data for the first learned model. The data shown in FIG. 18(a) includes, but is not limited to, in addition to the ID, the printed matter name, the starting coordinates which are the coordinate information of the divided image, the divided image size, the divided image data name, and the estimation result. When the image segmentation process ends, an ID is assigned to each divided image and the coordinate information is associated. The coordinate information may be any content that uniquely determines the coordinates of each divided image, and the coordinate information may be included in the divided image data name as the ID. Regarding the printed matter name, if it is acquired simultaneously with the printed image data in step S1701, it is advisable to manage it in association with the ID. The estimation result column is an area for storing the probability that the printed matter obtained as the estimation result of each divided image includes a defect, and the probability is stored for each ID after the estimation execution in step S1604.

[0113] Figure 18(b) is an example of estimation data for the second learned model. The data shown in Figure 18(b) includes, but is not limited to, the printed matter name, printed integration time, ink density, temperature, humidity, media thickness, coating, and estimation result in addition to the ID. When the data adjustment process is completed, an ID is assigned. Regarding the printed matter name, if it is acquired simultaneously with various parameter data in step S1701, it is advisable to manage it in association with the ID. The estimation result column is an area for storing the probability that the printed matter obtained from the parameter data of each ID contains defects, and the probability is stored for each ID after the estimation execution in step S1604.

[0114] (Input / Output Structure of the Learned Model at the Time of Estimation) Next, the input / output structure of the learned model 353 at the time of estimation will be described. Figures 19(a) and (b) are diagrams showing the input / output structure of the learned model 353 at the time of estimation.

[0115] Figure 19(a) shows the input / output structure at the time of estimation by the first learned model. As shown in Figure 19(a), as the input data 811 (first estimation input data) for the estimation by the first learned model, all the divided image data of the printed image data is used. The output data 812 (first output data) obtained as the result of the estimation by the learned model 353 (first learned model) is the probability of defects estimated for each divided image data, and is stored in association with the ID of each divided image data.

[0116] Fig. 19(b) shows the input / output structure during estimation by the second learned model. As shown in Fig. 19(b), various parameter data are used as the input data 811 (second estimation input data) for estimation by the second learned model. For each type of input parameter data, a data set with the same publication name as that used during estimation by the first learned model is used from among the estimation data stored by the estimation data generation unit 351. If there are multiple data sets with the same publication name and multiple data sets of various parameters at the same time are to be acquired, estimation is performed for each data set. In the first embodiment, the publication name is configured on the premise that it is a name that uniquely determines the publication. However, when allowing duplication of the publication name, print time information, etc. can be included in the estimation data, and a data set with the same publication name and print time ID as that used during estimation by the first learned model can also be used.

[0117] The output data 812 (second output data) obtained as a result of estimation by the learned model 353 (second learned model) is the probability of defects estimated for the set of various parameter data, and is stored in association with the ID of the various parameter data. Here, if all the various parameters are not included in the various parameter data that becomes the input data 811, the estimation unit 352 selects the learned model 353 according to the content of the acquired various parameter data. As the learned model according to the content of the various parameter data, the one generated and stored by the learning unit 251 is used.

[0118] (Estimation result evaluation by the estimation result evaluation unit) Next, a method for evaluating the estimation result by the estimation result evaluation unit 354 as to whether the printed matter has defects will be described. The evaluation by the estimation result evaluation unit 354 here corresponds to the determination of the presence or absence of defects in step S803 of Fig. 8. Figs. 20(a) and (b) are explanatory diagrams of the processing content of the estimation result evaluation unit 354.

[0119] FIG. 20(a) shows an evaluation table for the final result of the estimation used by the estimation result evaluation unit 354 in processing. The estimation result evaluation unit 354 according to the first embodiment determines a final defect determination result from a first estimation result by a first learned model and a second estimation result by a second learned model for a certain printed image data (printed matter).

[0120] First, the estimation result evaluation unit 354 determines the final result of each estimation of the first estimation unit using the first learned model and the second estimation unit using the second learned model. The final result of each estimation is determined based on whether the defect probability obtained as output data by each is equal to or greater than a predetermined threshold. The final result of the estimation of the first estimation unit changes depending on whether the defect probability is equal to or greater than a first threshold (50% in this example). The estimation result evaluation unit 354 according to the first embodiment determines that the first estimation result is abnormal when the defect probability for the divided image data of the printed image data obtained by the first learned model is determined to be 50% or more for one or more divided image data. On the other hand, if the defect probability of all the divided image data is less than 50%, the first estimation result by the first learned model is determined. In this way, the estimation result evaluation unit 354 determines whether the first estimation result of the first learned model is normal or abnormal based on the defect probability that is the output data 812 of the first learned model.

[0121] The final result of the estimation by the second estimation unit changes depending on whether the defect probability is equal to or greater than a second threshold value (50% in this example). Further, the estimation result evaluation unit 354 changes the determination process of the estimation result by the second learned model according to the number of data sets of various parameter data at the time of printing of the printed matter that is the basis of the printed image data used at the time of estimation by the first learned model. When there is one data set of the corresponding various parameter data, if it is determined that the defect probability for that data set is 50% or more, the second estimation result is determined to be abnormal, and if it is less than 50%, the second estimation result is determined to be normal. When there are a plurality of data sets of the corresponding various parameter data, if it is determined that the defect probability is 50% or more for one or more data sets, it is determined to be abnormal, and if all are less than 50%, it is determined to be normal. In this way, the estimation result evaluation unit 354 determines whether the second estimation result of the second learned model is normal or abnormal based on the defect probability that is the output data 812 of the second learned model. In this example, 50% is set as a preferred example of the first threshold value and the second threshold value, but each numerical value can be freely changed and may be a value that can be set by the user.

[0122] Next, the estimation result evaluation unit 354 refers to the evaluation table and determines the final defect determination result from the combination of the estimation results by each learned model. The final defect determination result is notified to the printer 600 in S804, and the printer operates according to the notified content.

[0123] [When the first estimation result is normal and the second estimation result is normal] In the first embodiment, when the estimation results of both the first learned model and the second learned model are normal, the estimation result evaluation unit 354 determines the final result to be normal. In this case, printing can be continued while ensuring the printing quality. Therefore, in step S804, only the fact that the printed matter is normal is notified. Upon receiving this notification, the printer 600 may not do anything in particular, or may perform operations such as displaying that the printed matter is normal on the user interface, or saving the fact that it was normal as a history for the printed matter name.

[0124] [When the first estimation result is abnormal and the second estimation result is abnormal] Also, when the estimation results of both the first learned model and the second learned model are abnormal, the estimation result evaluation unit 354 determines that the final result is abnormal. In step S804, it notifies that there is an abnormality in the printed matter and detailed information, and also transmits data as necessary.

[0125] When the final result is abnormal, printing cannot be continued while ensuring the printing quality. Therefore, when an abnormality is notified, the received printer 600, if printing is in progress immediately stops the ongoing printing operation (recording operation). Also, when notifying the coordinate range of the defective portion in the printed image data, the printer 600 can visualize the defective portion in the printed image data from this information and present it to the user from the user interface etc. displayed on the operation panel. The coordinate range of the defective portion can be created from the coordinate information of the divided image data determined to have a defect probability of 50% or more.

[0126] FIG. 20(b) is an example of visualizing the defective portion in the printed image data by surrounding it with a dotted line frame. By displaying such an image together with a message such as "A printing defect has been detected" on the user interface, the user can recognize not only that a defect has occurred but also the defective portion. A user who sees this display can perform maintenance work so that the printer 600 can print normally after avoiding the defective printed matter.

[0127] Such maintenance work can also be automated. For example, when it is found that there is a defect in the corresponding printed matter, the printer 600 operates to automatically avoid the corresponding printed matter into a trash can etc. Furthermore, from the coordinate range of the divided image data determined to have a defect probability of 50% or more by the estimation using the first learned model, the nozzle area of the head that discharges to that area can be known. Therefore, automatic maintenance of the nozzles that may be the cause of the defect can be executed.

[0128] In addition, when using a model that can obtain the importance of feature quantities during estimation by the second learned model, it is possible to identify parameter data that contributed to the abnormality determination. Therefore, for example, when temperature or humidity contributes to the abnormality determination, it is possible to notify that the temperature or humidity is an abnormal value or the value itself. When parameters related to the medium contribute to the abnormality determination, it is possible to propose replacement of the medium or the like. When parameters related to the conveyance path contribute to the abnormality determination, it is possible to propose adjustment of the conveyance path of the medium or the like.

[0129] The notification content described here is an example, and any content that can be understood from the results of each learned model can be notified. Also, the operation of the printer 600 by the notification from the estimation result evaluation unit 354 is an example, and it is also possible to perform other operations or for the user to set from the user interface.

[0130] [When the first estimation result is abnormal and the second estimation result is normal] When the estimation result of the first learned model is abnormal and the estimation result of the second learned model is normal, the estimation result evaluation unit 354 determines that the final result has a possibility of being abnormal. In S804, it notifies that there is a possibility of abnormality in the printed matter and detailed information, and also transmits data if necessary. In this case, since there is a possibility that areas that are not defective in the printed image data are misdetected as defective, the printer 600 does not necessarily have to stop printing immediately even during printing. However, there is also a possibility that the defects in the printed image data are correctly detected. Therefore, similar to the case where both learned models determined an abnormality, it is advisable to display to the user the locations in the printed image data where there may be defects. The user can visually check whether there are defects, and if there are actually defects, perform maintenance work or the like.

[0131] [When the first estimation result is normal and the second estimation result is abnormal] When the first estimation result of the first learned model is normal and the second estimation result of the second learned model is abnormal, the estimation result evaluation unit 354 determines that the final result has a possibility of being abnormal. In step S804, it notifies that there is a possibility of abnormality and detailed information in the printed matter, and also transmits data as necessary. In this case, although the possibility of a defect in the printed matter itself is low, there is a possibility that various parameter data will soon affect the printed matter itself or that it will affect something other than the printed matter itself. When using a model that can obtain the importance of feature amounts during estimation by the second learned model, it is possible to specify the parameter data that contributed to the abnormality determination. Therefore, in S804, it notifies that there may soon be a defect in the printed matter and that there may be a possibility of device abnormality together with information on the parameter data that contributed to the abnormality determination. The printer 600 can display the received notification content to the user, or prompt component replacement when the parameter that contributed to the abnormality determination is a parameter caused by component consumption of the printer 600, etc.

[0132] <Example 2> Next, Example 2 according to the present invention will be described. Example 2 is different from Example 1 in the method for determining the final result in detecting a defect in a printed matter. Hereinafter, in the description of Example 2, the same components and processes as those in Example 1 are denoted by the same reference numerals and the description thereof is omitted, and only the characteristic configuration of Example 2 will be described.

[0133] ​In Example 1, when detecting a defect using the learned model 353, it was a process of always using both the first learned model and the second learned model. On the other hand, in Example 2, a case will be described where the defect is estimated by the second learned model and, depending on the conditions, the defect is estimated by the first learned model. Since the estimation by the first learned model involves image acquisition and image processing, the processing load becomes high. Therefore, depending on the printing speed, real-time detection of defects during printing may not be possible, or depending on the configuration of the processing system, it may be desirable to reduce the load. In such cases, it is preferable to adopt a configuration in which only the estimation of defects by the second learned model, which does not involve image acquisition and image processing, is always executed. Note that since the processing contents of the edge server 300 and the cloud server 200 in Example 2 and the contents of the sequence diagrams during learning and estimation of the processing system 100 are the same as those in the previous Example 1, the description is omitted here.

[0134] (Defect Detection Flow of Printed Matter) The processing flow of defect detection of the printed matter (actual printing result) of the printer 600 by the processing system 100 according to Example 2 will be described. FIG. 21 is a flowchart showing an overall outline of defect detection of the processing system 100 of Example 2.

[0135] First, in step S2101, the processing system 100 estimates the presence or absence of defects in the printed matter during printing using a second learned model that takes various parameters when the actual printed matter is printed, and obtains a second estimation result.

[0136] Next, based on the second estimation result in step S2102, it is determined whether there are defects in the printed matter. If the defect probability in the second estimation result by the second learned model is less than 50%, that is, if it is NO in step S2102, it is determined that the printed image data printed on the printed matter is normal and the process ends.

[0137] On the other hand, if there is one or more cases where the defect probability of the second estimation result by the second learned model is 50% or more, that is, if the answer in step S2102 is NO, it is determined that there may be an abnormality in the printed image data printed on the printed matter, and the process proceeds to step S2103.

[0138] In step S2103, the presence or absence of a defect in the printed matter is estimated by the first learned model that takes the printed image data as input, and the first estimation result is obtained. When the printer 600 constantly executes the acquisition of the printed image data, the printed image data saved at this time is acquired. However, when the printer 600 does not constantly execute and save the acquisition of the printed image data, it is necessary to create the printed image data to be input data at this time. When step S2103 ends, the process proceeds to step S2104.

[0139] Note that when the printer 600 is configured not to constantly execute and save the acquisition of the printed image data, it is preferable to obtain the estimation result by the second learned model before scanning the printed image data. That is, it is preferable that the estimation by the second learned model is completed while the recording medium is being conveyed from the printing unit that performs printing to the scanner unit that acquires the printed image data. Therefore, in such a configuration, in the hardware configuration of the printer 600, it is preferable to determine the conveyance distance and conveyance speed of the recording medium from the printing unit to the scanner unit in consideration of the estimation time by the second learned model. If the estimation by the second learned model starts immediately after printing in the printing unit and is completed before reaching the scanner unit, the printed image data can be quickly acquired by the scanner unit if the estimation result indicates a defect.

[0140] In step S2104, the estimation result evaluation unit 354 determines the final defect determination result through the same processing as in the first embodiment, and in step S2105, notifies the printer 600 of the result. In the second embodiment, when it is determined to be normal in step S2102, the processing flow is not to notify the printer 600, but this is not the only case. For example, it may be configured to notify the printer 600 that it is normal, and the printer 600 uses the fact that it is normal as data for history storage and the like.

[0141] <Embodiment 3> Next, Embodiment 3 according to the present invention will be described. In Embodiment 3, the method for determining the final result in the defect detection of the printed matter is different from those in Embodiments 1 and 2. Hereinafter, in the description of Embodiment 3, the same components and processes as those in Embodiment 1 are denoted by the same reference numerals and the description thereof is omitted, and only the characteristic components of Embodiment 3 will be described.

[0142] In Embodiment 3, contrary to Embodiment 2, first, the defect estimation by the first learned model is executed, and depending on the result, the defect estimation by the second learned model is performed. In Embodiment 1, the estimation of the presence or absence of a defect in the printed matter by the second learned model was always performed for the purpose of preventing false detection of a defect in the printed matter by the first learned model. That is, when the first estimation result by the first learned model is abnormal (there is a defect), additional estimation by the second learned model should be performed, but it is not essential when the estimation result by the first learned model is defect-free. Since the processing contents of the edge server 300 and the cloud server 200 in Embodiment 3, and the content of the sequence diagram during learning and estimation of the processing system 100 are the same as those in the previous Embodiment 1, the description thereof is omitted here.

[0143] (Defect detection flow of printed matter) The processing flow of defect detection of the printed matter (actual printing result) of the printer 600 by the processing system 100 according to Embodiment 3 will be described. FIG. 22 is a flowchart showing an overall outline of defect detection of the processing system 100 of Embodiment 3.

[0144] First, in step S2201, the processing system 100 estimates the presence or absence of defects in the printed matter during printing using a first pre-trained model that takes the printed image data as input, and obtains a first estimation result.

[0145] Next, based on the first estimation result in step S2202, it is determined whether there are any defects in the printed matter. If the defect probability in the first estimation result by the first pre-trained model is less than 50%, that is, if the result in step S2202 is NO, it is determined that the printed image data printed on the printed matter is normal and the process ends.

[0146] On the other hand, if there is one or more divided image data with a defect probability of 50% or more in the second estimation result by the second pre-trained model, that is, if the result in step S2202 is NO, it is determined that there may be an abnormality in the printed image data printed on the printed matter, and the process proceeds to step S2203.

[0147] In step S2203, the second pre-trained model that takes various parameters at the time of actually printing the printed matter as input estimates the presence or absence of defects in the printed matter and obtains a second estimation result. In the printer 600, data such as log data including various parameter data is always stored, and at this time, data such as log data including various parameter data at the time of printing the corresponding actual printed matter is obtained. When step S2203 ends, the process proceeds to step S2204.

[0148] In step S2204, the estimation result evaluation unit 354 determines the final defect determination result through the same process as in the first embodiment, and notifies the printer 600 of the result in step S2205. In the third embodiment, when it is determined to be normal in step S2202, the process flow is such that it is not notified to the printer 600, but this is not limited thereto. For example, it may be configured to notify the printer 600 that it is normal, and the printer 600 uses the fact that it was normal as data for history storage and the like.

[0149] As described above, according to each of the above-described configurations, in addition to the printed image data, based on the data regarding various parameters when actually printing a printed matter, in order to estimate the presence or absence of defects in the printed matter using a pre-trained machine learning model, it is possible to detect defects with high accuracy for a wider variety of printed matters. And since data different from the printed image data can be used for detecting defects in the printed matter, it is possible to suppress a decrease in the detection accuracy of defects even for printed image data having properties different from the printed image data used as input data during the learning of machine learning. As a result, it is possible to detect defects in the printed matter with high accuracy.

[0150] Note that although Examples 1, 2, and 3 have been described so far, it is free which configuration to adopt in the processing system 100. Also, as a configuration in which any one of the configurations can be selectively adopted, the user may be allowed to manually select, or the processing system 100 may automatically determine. When allowing the user to select, the processing system 100 switches the processing by allowing selection in the user interface displayed on the operation panel 605. Also, when automatically determining, for example, although basically it is the processing of Example 1, it may be configured to switch to Example 2 when the load of the entire processing system 100 is large.

[0151] Also, although a machine learning model was used for detecting defects based on various parameter data, when the parameters that become abnormal values when defects occur in the printed matter are clearly known, it is also conceivable to use only the values of the corresponding parameters for defect detection without using the machine learning model. With such a configuration, it is also possible to reduce the processing load by using two machine learning models. However, since there are many parameters that can cause defects in the printed matter and they are complicatedly related to each other, it is desirable to use a machine learning model. Also, among the input data of the second pre-trained model, for some parameters, if the defect probability is estimated and it is above a certain level, the defect probability of other parameters may be estimated.

[0152] <Other Embodiments> The present invention can also be implemented by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing a computer of the system or apparatus to read and execute the program. The computer may include one or more processors or circuits, and may include a network of multiple separate computers or multiple separate processors or circuits in order to read and execute computer-executable instructions.

[0153] The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). Further, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0154] The storage medium can also be referred to as a non-transitory computer-readable medium. The storage medium may include one or more hard disks (HD), random access memory (RAM), read-only memory (ROM), storage devices of a distributed computing system. Further, the storage medium may include an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD, registered trademark)), a flash memory device, and a memory card.

[0155] In the application of the present invention, the processing described as being performed by one apparatus in each of the above-described embodiments may be shared and executed by a plurality of apparatuses. Alternatively, the processing described as being performed by different apparatuses may be executed by one apparatus. In a computer system, how each function is realized by a hardware configuration can be flexibly changed.

[0156] The disclosure of this embodiment includes the following configurations. (Configuration 1) An image inspection system for inspecting an image recorded on a recording medium by a recording device, using a first learned model generated by machine learning based on a recorded image recorded on a recording medium and an evaluation result as to whether the recorded image is normal or abnormal, to estimate a defect in an actual recorded image to be inspected and obtain a first estimation result; a first estimation unit, recording information which is information related to at least one of the recording device and the recording medium at the time of recording the recorded image, different recording information from the recorded image, and a second learned model generated by machine learning based on an evaluation result as to whether the recorded image is normal or abnormal, to estimate a defect in the actual recorded image and obtain a second estimation result; a second estimation unit, comprising, An image inspection system characterized by detecting a defect in the actual recorded image based on the first estimation result and the second estimation result. (Configuration 2) The first estimation unit obtains a first probability that there is a defect in the actual recorded image, and obtains the first estimation result based on the first probability, The second estimation unit obtains a second probability that there is a defect in the actual recorded image, and obtains the second estimation result based on the second probability. The image inspection system according to Configuration 1, characterized in that. (Configuration 3) When the first probability is equal to or greater than a first threshold value, the first estimation unit sets the first estimation result as indicating that there is an abnormality in the actual recorded image, When the second probability is equal to or greater than a second threshold value, the second estimation unit sets the second estimation result as indicating that there is an abnormality in the actual recorded image. The image inspection system according to Configuration 2, characterized in that. (Configuration 4) The image inspection system according to Configuration 3, characterized in that the first threshold value and the second threshold value are 50%. (Configuration 5) The evaluation result is data in which the probability of a defect in the actual recorded image is set to 100% when the recorded image is normal, and the probability of a defect in the actual recorded image is set to 0% when the recorded image is abnormal, according to any one of Configurations 2 to 4 of the image inspection system described. (Configuration 6) It includes a data generation unit that divides the image data obtained from the actual recorded image into a plurality of parts to generate divided image data. The first estimation unit estimates whether each of the plurality of divided image data is normal or abnormal, and obtains the first estimation result based on the estimation results, according to any one of Configurations 1 to 5 of the image inspection system described. (Configuration 7) The data generation unit generates the divided image data by dividing the image data of the actual recorded image with a predetermined vertical width, a predetermined horizontal width, and a predetermined shift amount, according to the image inspection system described in Configuration 6. (Configuration 8) When the first estimation unit determines that one or more of the divided image data are abnormal, it determines that there is an abnormality in the actual recorded image for the first estimation result, according to the image inspection system described in Configuration 6 or 7. (Configuration 9) The recording information includes at least one of the ink density at the time of image recording, the environmental temperature, the environmental humidity, the thickness of the actual recording medium on which the actual recorded image is recorded, and the coating type of the actual recording medium, according to any one of Configurations 1 to 8 of the image inspection system described. (Configuration 10) The recording information includes at least one of the position information of the liquid ejection head that ejects the liquid provided in the recording device at the time of image recording, the number of cleaning times of the nozzles of the liquid ejection head, and the roller diameter that constitutes the conveyance path of the recording medium provided in the recording device, according to any one of Configurations 1 to 9 of the image inspection system described. (Configuration 11) The second estimation unit includes the recording information at a plurality of times at the time of image recording, according to any one of Configurations 1 to 10 of the image inspection system described. (Configuration 12) The second estimation unit estimates whether each of the recording information at a plurality of times is normal, and obtains the second estimation result based on the estimation results. The image inspection system according to Configuration 11, characterized in that. (Configuration 13) When it is determined that one or more of the recording information among the recording information at a plurality of times is abnormal, the second estimation unit determines that there is an abnormality in the actual recorded image as the second estimation result. The image inspection system according to Configuration 12, characterized in that. (Configuration 14) When the first estimation result is abnormal and the second estimation result is abnormal, the image inspection system according to any one of Configurations 1 to 13, characterized in that the user is notified that there is a defect in the actual recorded image. (Configuration 15) When there is a defect in the actual recorded image, the image inspection system according to Configuration 14, characterized in that the defective portion is notified to the user. (Configuration 16) When the first estimation result is abnormal and the second estimation result is abnormal, the image inspection system according to any one of Configurations 1 to 15, characterized in that the recording operation being executed by the recording device is stopped. (Configuration 17) When the first estimation result is abnormal and the second estimation result is normal, or when the first estimation result is normal and the second estimation result is abnormal, the user is notified that there may be a defect in the actual recorded image. The image inspection system according to any one of Configurations 1 to 16, characterized in that. (Configuration 18) When the first estimation result is abnormal and the second estimation result is normal, the image inspection system according to any one of Configurations 1 to 17, characterized in that the defective portion in the image data of the actual recorded image is notified to the user. (Configuration 19) The image inspection system according to any one of Configurations 1 to 18, characterized in that when the first estimation result is normal and the second estimation result is abnormal, information on parameter data that contributed to the abnormality determination is notified to the user. (Configuration 20) The image inspection system according to any one of Configurations 1 to 19, characterized in that when the first estimation result is abnormal, the second estimation unit acquires the second estimation result. (Configuration 21) The image inspection system according to any one of Configurations 1 to 19, characterized in that when the second estimation result is abnormal, the first estimation unit acquires the first estimation result.

Explanation of Signs

[0157] 100…Processing system (image inspection system), 352…Estimation unit (first estimation unit, second estimation unit), 353…Learned model (first learned model, second learned model), 600…Printer (recording device)

Claims

1. An image inspection system for inspecting an image recorded on a recording medium by a recording device, comprising: a first estimation unit that uses a first learned model generated by machine learning based on a recorded image recorded on a recording medium and an evaluation result indicating whether the recorded image is normal or abnormal, estimates a defect in a target recorded image, and obtains a first estimation result; a second estimation unit that uses a second learned model generated by machine learning based on recording information, which is information related to at least one of the recording device and the recording medium at the time of recording the recorded image and is different from the recorded image, and an evaluation result indicating whether the recorded image is normal or abnormal, estimates a defect in the target recorded image, and obtains a second estimation result; and comprising: an image inspection system, characterized in that a defect in the target recorded image is detected based on the first estimation result and the second estimation result.

2. The first estimation unit obtains a first probability that there is a defect in the target recorded image, and obtains the first estimation result based on the first probability. The second estimation unit obtains a second probability that there is a defect in the target recorded image, and obtains the second estimation result based on the second probability. The image inspection system according to claim 1, characterized in that.

3. The first estimation unit sets the first estimation result as indicating that there is an abnormality in the target recorded image when the first probability is equal to or greater than a first threshold value. The second estimation unit sets the second estimation result as indicating that there is an abnormality in the target recorded image when the second probability is equal to or greater than a second threshold value. The image inspection system according to claim 2, characterized in that.

4. The image inspection system according to claim 3, characterized in that the first threshold value and the second threshold value are 50%.

5. The evaluation result is data in which the probability that there is a defect in the target recorded image is set to 100% when the recorded image is normal, and the probability that there is a defect in the target recorded image is set to 0% when the recorded image is abnormal. The image inspection system according to claim 2, characterized in that.

6. comprising a data generation unit that divides image data obtained from the target recorded image into a plurality of parts to generate divided image data; The first estimation unit estimates whether each of the plurality of divided image data is normal or abnormal, and obtains the first estimation result based on the estimation results. The image inspection system according to claim 1, characterized in that.

7. The image inspection system according to claim 6, wherein the data generation unit generates the divided image data by dividing the image data of the actual recorded image with a predetermined vertical width, a predetermined horizontal width, and a predetermined shift amount.

8. The image inspection system according to claim 6, wherein when the first estimation unit determines that one or more of the divided image data are abnormal, the first estimation result is determined as abnormal in the actual recorded image.

9. The image inspection system according to claim 1, wherein the recording information includes at least one of ink density at the time of image recording, environmental temperature, environmental humidity, thickness of the actual recording medium on which the actual recorded image is recorded, and coating type of the actual recording medium.

10. The image inspection system according to claim 1, wherein the recording information includes at least one of position information of a liquid ejection head that ejects a liquid provided in the recording apparatus at the time of image recording, the number of cleaning times of nozzles of the liquid ejection head, and roller diameters that constitute a conveyance path of the actual recording medium provided in the recording apparatus.

11. The image inspection system according to claim 1, wherein the second estimation unit includes the recording information at a plurality of times at the time of image recording.

12. The image inspection system according to claim 11, wherein the second estimation unit estimates whether each of the recording information at a plurality of times is normal or not, and obtains the second estimation result based on the estimation results.

13. The image inspection system according to claim 12, wherein when the second estimation unit determines that one or more of the recording information at a plurality of times are abnormal, the second estimation result is determined as abnormal in the actual recorded image.

14. The image inspection system according to claim 1, wherein when the first estimation result is abnormal and the second estimation result is abnormal, the user is notified that there is a defect in the actual recorded image.

15. The image inspection system according to claim 14, wherein when there is a defect in the actual recorded image, the user is notified of the defective portion.

16. The image inspection system according to claim 1, wherein when the first estimation result is abnormal and the second estimation result is abnormal, the recording operation being executed by the recording apparatus is stopped.

17. The image inspection system according to claim 1, wherein when the first estimation result is abnormal and the second estimation result is normal, or when the first estimation result is normal and the second estimation result is abnormal, the user is notified that there may be a defect in the actual recorded image.

18. The image inspection system according to claim 1, wherein when the first estimation result is abnormal and the second estimation result is normal, the user is notified of the defective portion in the image data of the actual recorded image.

19. The image inspection system according to claim 1, wherein when the first estimation result is normal and the second estimation result is abnormal, the user is notified of the information of the parameter data that contributed to the abnormality determination.

20. The image inspection system according to claim 1, wherein the second estimation unit obtains the second estimation result when the first estimation result is abnormal.

21. The image inspection system according to claim 1, wherein the first estimation unit obtains the first estimation result when the second estimation result is abnormal.

Images