Detecting defective nozzles in digital printing systems

Abstract

The method includes receiving a first digital image (FDI) to be printed by a digital printing system (DPS) (10). During a training phase, a first set of synthetic images (SI) (112A, 112B, 114A, 114B, 116A, 116B) is generated for a first selected region (111) in the FDI, the first set of synthetic images (SI) having defects caused by a defective part (DP) (99) in the first selected region; a neural network (NN) (150) is trained to detect the defects using the first set of SI. During a subsequent detection phase, the NN is applied to identify suspect second regions (135, 145) in second digital images (SDI) (136, 146) acquired from the image generated by the DPS; for each of the second regions, a second set of SI (137, 147) having the DP forming the defect is generated; and within each of the second regions, the DP is identified by comparing the SDI with the second set of SI.

Description

[Technical Field]

[0001] The present invention relates generally to digital printing, and more particularly to a method and system for detecting defective nozzles in a digital printing system. [Background technology]

[0002] Various methods and systems are known in the art for correcting distortions by identifying defective parts in a printing system.

[0003] For example, U.S. Patent Application Publication No. 2019 / 0248153 describes a method for detecting defective print nozzles in an inkjet printer, including printing a multi-row print nozzle test chart consisting of horizontal rows of equidistant perpendicular lines periodically below one another, using only print nozzles in the print head that contribute to all rows of the test chart corresponding to the horizontal rows. Area coverage elements geometrically associated with the test chart are printed, and both elements are recorded by an image sensor and analyzed by a computer. The computer analyzes the recorded area coverage elements to detect printing defects and assigns the defects to areas of geometrically adjacent print nozzles. Analysis of the test chart within the area identifies the nozzles causing the defects. Defective print nozzles are detected based on a threshold, and the detected print nozzles are then compensated, and the sensor's influence is removed in the analysis of the recorded area coverage elements by shading compensation.

[0004] U.S. Patent No. 5,109,275 describes an apparatus for print signal correction and printer operation control for use in applications such as color copiers, which utilizes a neural network to convert input image signals, derived, for example, by scanning and analyzing an original image, into print density signals that are supplied to the printer. In addition, a sensed signal representing at least one internal environmental condition of the printer, such as temperature, is input to the neural network, whereby the output print density signal is automatically corrected for changes in the printer's internal environment.

[0005] U.S. Patent Application Publication No. 2019 / 0105895 describes a method for detecting defective printing nozzles in an inkjet printer with a computer. The method includes printing a multi-column nozzle test chart for detection purposes, where the test chart includes several horizontal columns of equidistant perpendicular lines printed periodically and positioned below each other. In every column of the nozzle test chart, only each printing nozzle of the print head periodically contributes to a first element of the nozzle test chart corresponding to a designated number of the horizontal column. An area cover element geometrically associated with the nozzle test chart is printed. Both elements are recorded by an image sensor, and both elements are evaluated by a computer. Defective printing nozzles are identified by evaluating the recorded nozzle test chart by the computer. Defects are assigned to printing nozzles in the nozzle test chart within the area cover element by the computer. Summary of the Invention [Means for solving the problem]

[0006] One embodiment of the present invention provides a method for detecting defective parts (DPs) in a digital printing system (DPS), the method including receiving a first digital image (FDI) printed by the DPS. During a training phase, (i) for one or more first selected regions in the FDI, a first set of one or more composite images is generated having defects caused by the DPs in the one or more first selected regions, and (ii) a neural network (NN) is trained using at least one of the composite images in the first set to detect the defects. During a detection phase following the training phase, (i) the trained NN is applied to identify one or more second regions suspected of having defects in a second digital image (SDI) obtained from the printed image generated by the DPS, (ii) for each second region, a second set of one or more composite images is generated having one or more DPs causing one or more of the defects, and (iii) at least the DPs are identified in each second region by comparing the SDI with the one or more composite images in the second set.

[0007] In some embodiments, the DPS includes a nozzle for directing printing fluid onto the substrate, and the DP includes a defective nozzle (DN) from among the nozzles, and the defect includes a missing nozzle failure (MNF) caused by a blocked orifice in the DN. In other embodiments, the method includes selecting a first selected region including features for training a neural network (NN) based on predefined selection criteria. In yet other embodiments, the NN includes a convolutional neural network (CNN).

[0008] In one embodiment, the CNN has an Inception V3 architecture. In another embodiment, at least one of the FDI and the SDI includes a product image. In yet another embodiment, the DPS includes a nozzle for directing printing fluid onto the substrate, the DP includes a partially clogged nozzle from among the nozzles, and the defect includes a positioning error caused by the partially clogged nozzle, which directs the printing fluid jetted at a deflected angle to be deposited on the substrate a short distance away from the intended deposition location.

[0009] According to one embodiment of the present invention, there is additionally provided a system for detecting defective parts (DPs) in a digital printing system (DPS), the system including an interface and a processor. The interface is configured to receive: (i) a first digital image (FDI) to be printed by the DPS, and (ii) a second digital image (SDI) acquired from the printed image generated by the DPS. During a training phase, the processor is configured to: (i) generate, for one or more first selected regions in the FDI, a first set of one or more composite images having defects caused by the DPs in the one or more first selected regions, and (ii) train a neural network (NN) to detect the defects using at least one of the composite images in the first set. In a detection phase following the training phase, the processor is configured to: (i) apply the trained NN to identify one or more second regions in the SDI that are suspected of having defects; (ii) generate, for each of the second regions, a second set of one or more composite images having one or more DPs that respectively cause one or more of the defects; and (iii) identify, in each of the second regions, at least the DPs by comparing between the SDI and the one or more composite images of the second set.

[0010] The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken in conjunction with the drawings in which: [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic side view of a digital printing system in accordance with one embodiment of the present invention; [Figure 2A] FIG. 1 is a schematic, pictorial diagram illustrating the training phase of a convolutional neural network (CNN) configured to detect missing nozzle failures (MNF), in accordance with an embodiment of the present invention; [Figure 2B] FIG. 1 is a simplified, pictorial diagram illustrating the effect of a partially clogged nozzle on a printed image and a technique for training a CNN to detect partially clogged nozzles, in accordance with one embodiment of the present invention. [Figure 3] FIG. 1 is a schematic, pictorial diagram illustrating a method for detecting MNF using a trained CNN, in accordance with an embodiment of the present invention. [Figure 4] 1 is a schematic, pictorial illustration of a CNN used to detect MNF in printed images, in accordance with one embodiment of the present invention; [Figure 5] 1 is a flowchart that schematically illustrates a method for detecting defective nozzles in a digital printing system using CNN, in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] overview Digital printing systems include components, such as nozzles, for ejecting printing fluid onto a substrate to generate an image thereon. In some cases, a nozzle may eject printing fluid inaccurately due to a clog or other defect therein, resulting in distortion of the printed image. While it is possible in principle to print a test job to detect defective nozzles, such test jobs reduce the production time of the printing system, involve extra costs for printing consumables, and generate waste. Moreover, such test jobs may be performed periodically and therefore cannot detect defective nozzles in real time.

[0013] Embodiments of the invention described herein below provide methods and systems for detecting defective nozzles in a digital printing system (DPS) that includes an array of such nozzles during manufacturing.

[0014] In some embodiments, the DPS includes a processor configured to detect defective nozzles using a convolutional neural network (CNN) applied to a first digital image (FDI) printed by the DPS and a second digital image (SDI) obtained from the printed image produced by the DPS. At least one of the FDI and the SDI are typically both product images printed during manufacturing.

[0015] In some embodiments, during a training phase of the CNN, the processor is configured to generate a first set of synthetic images (SIs) for selected regions within the FDI, each SI having a simulated missing nozzle failure (MNF) associated with a respective selected region caused by a respective defective nozzle in the array, and the processor is configured to train the CNN to detect MNFs using at least a portion of the first set of SIs.

[0016] In some embodiments, in a detection phase following the training phase, the processor is configured to apply the trained CNN to identify one or more regions within the SDI suspected of containing MNF. Based on a print plan of the SDI, the processor maintains a list of nozzles involved in printing the suspected regions. The processor is configured to generate, for each suspected region, a second set of SIs having one or more MNFs caused by one or more respective defective nozzles on the list.

[0017] In some embodiments, the processor is configured to, for each suspect region, identify at least its defective nozzles by comparing between the SDI and a respective SI of the second set.

[0018] The disclosed technology improves the quality of printed digital images by identifying defective nozzles in real time, thereby preventing MNF in subsequent printed images. In the context of this disclosure and claims, the term real-time identification refers to identifying defective nozzles by detecting MNF immediately after printing on a substrate. Moreover, by detecting MNF and identifying defective nozzles during production, the disclosed technology improves DPS utilization for manufacturing and reduces substrate and printing fluid waste.

[0019] System Description 1 is a schematic side view of a digital printing system 10 according to one embodiment of the present invention. In some embodiments, system 10 includes a rotating flexible blanket 44 that cycles through an imaging station 60, a drying station 64, an impression station 84, and a blanket treatment station 52. In the context of the present invention and in the claims, the terms "blanket" and "intermediate transfer member (ITM)" are used interchangeably to refer to a flexible member including one or more layers used as an intermediate member configured to receive an ink image and transfer the ink image to a target substrate, as described in detail below. Moreover, the embodiments of the present invention described below are also applicable to printing systems that use one or more drums as an ITM instead of or in addition to blanket 44.

[0020] In an operational mode, imaging station 60 is configured to form a mirror ink image, also referred to herein as an "ink image" (not shown) or simply "image," of digital image (DI) 42 on an upper run of the surface of blanket 44. The ink image is then transferred to a target substrate (e.g., paper, folding carton, multilayer polymer, or any suitable flexible packaging in the form of a sheet or continuous web) disposed beneath a lower run of blanket 44.

[0021] In the context of the present invention, the term "run" refers to the length or segment of blanket 44 between any two given rollers over which blanket 44 is guided.

[0022] In some embodiments, during installation, the blanket 44 may be attached edge to edge to form a continuous blanket loop (not shown). An example of a method and system for seam installation is described in detail in U.S. Provisional Application No. 62 / 532,400, the disclosure of which is incorporated herein by reference.

[0023] In some embodiments, the imaging station 60 typically includes multiple print bars 62, each mounted (e.g., using a slider) on a frame (not shown) positioned at a fixed height above the surface of the upper run of the blanket 44.

[0024] Reference is now made to inset FIG. 11, which shows print bars 62. In some embodiments, each print bar 62 includes a strip of print head (not shown) the same width as the print area on blanket 44, and an array of individually controllable print nozzles 99, each of which is configured to apply (e.g., by jetting and / or directing) printing fluid toward a predetermined location on blanket 44 as it is moved by system 10.

[0025] Referring again to the overview of FIG. 1 , in some embodiments, imaging station 60 may include any suitable number of print bars 62, and each print bar 62 may include a printing fluid, such as a different colored water-based ink. The inks typically have visible colors, such as, but not limited to, cyan, magenta, red, green, blue, yellow, black, and white. In the example of FIG. 1 , imaging station 60 includes seven print bars 62, but may include, for example, four print bars 62 having any selected color, such as cyan (C), magenta (M), yellow (Y), and black (K).

[0026] In some embodiments, the print heads are configured to eject ink droplets of different colors onto the surface of blanket 44 to form an ink image (not shown) on the surface of blanket 44 .

[0027] In some embodiments, the different print bars 62 are spaced apart from one another along an axis of movement, also referred to herein as the direction of blanket 44 movement, represented by arrow 94. In this configuration, precise spacing between print bars 62 and synchronization between the direction of ink droplets of each print bar 62 and the moving blanket 44 is essential to enable accurate placement of the image pattern.

[0028] In the context of this disclosure and in the claims, the terms "inter-color pattern placement," "pattern placement accuracy," "color-to-color registration," "C2C registration," "bar-to-bar registration," and "color registration" are used interchangeably to refer to any placement accuracy of two or more colors relative to one another.

[0029] In some embodiments, system 10 includes a heater, such as a hot gas or air blower 66, positioned between print bars 62 and configured to partially dry ink droplets deposited on the surface of blanket 44. This flow of warm air between print bars can help, for example, to reduce condensation on the surface of the print head, and / or to treat satellites (e.g., residue or small droplets dispersed around the main ink droplets), and / or to prevent clogging of the inkjet nozzles of the print head, and / or to prevent droplets of different color inks on blanket 44 from undesirably intermixing with each other. In some embodiments, system 10 includes a drying station 64 configured to blow warm air (or another gas) onto the surface of blanket 44. In some embodiments, the drying station includes an air blower 68 or any other suitable drying device. Additionally or alternatively, system 10 can include one or more lighting assemblies configured to emit infrared (IR) radiation to dry printing fluid (e.g., ink) applied to blanket 44. Such an IR radiation assembly may be implemented, for example, within imaging station 60 (instead of or in addition to blower 66), and / or within drying station 64 (instead of or in addition to blower 68), and / or at other locations along blanket 44.

[0030] At drying station 64, the ink image formed on blanket 44 is exposed to radiation and / or hot air to more thoroughly dry the ink, evaporating most or all of the liquid carrier and leaving only a layer of resin and colorant that is heated to the point where it becomes a tacky ink film.

[0031] In some embodiments, system 10 includes a blanket module 70 that includes a rotating ITM, such as blanket 44. In some embodiments, blanket module 70 includes one or more rollers 78, at least one of which includes an encoder (not shown) configured to record the position of blanket 44 in order to control the position of sections of blanket 44 relative to the respective print bars 62. In some embodiments, the encoder of roller 78 typically includes a rotary encoder configured to generate a rotary-based position signal indicative of the angular displacement of the respective roller. It should be noted that in the context of the present invention and claims, the terms "indicative of" and "indication" are used interchangeably.

[0032] Additionally or alternatively, blanket 44 may include an embedded encoder (not shown) for controlling the operation of various modules of system 10. One embodiment of an embedded encoder is described in detail, for example, in U.S. Provisional Application No. 62 / 689,852, the disclosure of which is incorporated herein by reference.

[0033] In some embodiments, blanket 44 is guided over rollers 76 and 78 and a powered tensioning roller, also referred to herein as dancer assembly 74. Dancer assembly 74 is configured to control the amount of slack in blanket 44, its movement being represented diagrammatically by a double-headed arrow. Furthermore, any stretching of blanket 44 with aging will not affect the ink image placement performance of system 10 and will simply require further slack removal by tensioning dancer assembly 74.

[0034] In some embodiments, dancer assembly 74 may be motorized. The configuration and operation of rollers 76 and 78 are described in further detail, for example, in U.S. Patent Application Publication No. 2017 / 0008272 and the aforementioned PCT International Publication No. WO2013 / 132424, the disclosures of which are incorporated herein by reference in their entireties.

[0035] In some embodiments, system 10 may include one or more tension sensors (not shown) positioned at one or more locations along blanket 44. The tension sensors may be integrated within blanket 44 or may include sensors external to blanket 44 using any other suitable technique for obtaining signals indicative of the mechanical tension applied to blanket 44. In some embodiments, processor 20 and additional controllers of system 10 are configured to receive signals generated by the tension sensors to monitor the tension applied to blanket 44 and to control the operation of dancer assembly 74.

[0036] At the printing station 84, the blanket 44 passes between an impression cylinder 82 and a pressure cylinder 90, which is configured to transport a compressible blanket.

[0037] In some embodiments, the system 10 includes a control console 12 configured to control multiple modules of the system 10, such as a blanket module 70, an image forming station 60 disposed above the blanket module 70, and a substrate transport module 80 disposed below the blanket module 70, including one or more printing stations as described below.

[0038] In some embodiments, console 12 includes a processor 20, typically a general-purpose processor, with appropriate front-end and interface circuitry for interfacing with and receiving signals from the controllers of dancer assemblies 74 and controller 54 via cable 57. Additionally or alternatively, console 12 may include any suitable type of application specific integrated circuit (ASIC) and / or digital signal processor (DSP) and / or any other suitable type of processing device configured to perform any type of processing on data processed within system 10.

[0039] In some embodiments, controller 54, shown diagrammatically as a single device, may include one or more electronic modules mounted in predetermined locations on system 10. At least one of the electronic modules of controller 54 may include an electronic device, such as a control circuit or processor (not shown), that is configured to control the various modules and stations of system 10. In some embodiments, processor 20 and control circuitry may be programmed with software to perform functions used by the printing system, and data for the software may be stored in memory 22. The software may be downloaded in electronic form to processor 20 and control circuitry, for example, via a network, or it may be provided on a non-transitory, tangible medium, such as an optical, magnetic, or electronic memory medium.

[0040] In some embodiments, console 12 includes a display 34 configured to display data and images received from processor 20 or input inserted by a user (not shown) using input device 40. In some embodiments, console 12 may have any other suitable configuration, for example, alternative configurations of console 12 and display 34 are described in detail in U.S. Pat. No. 9,229,664, the disclosure of which is incorporated herein by reference.

[0041] In some embodiments, console 12 includes a digital front-end module (DFEM) 100 configured to perform various computational tasks for system 10. DFEM 100 may include one or more processing and memory devices, such as, but not limited to, a raster image processor (RIP) and interface circuitry (not shown) for interfacing with processor 20 and / or other components of system 10. In the configuration depicted in FIG. 1 , DFEM 100 is incorporated into console 12 and interfaces with an operator (not shown) of system 10 using input devices 40 and display 34.

[0042] In other embodiments, DFEM 100 may include a stand-alone computer having input / output (I / O) devices for interfacing with an operator and console 12. In alternative embodiments, DFEM 100 may have any other suitable configuration.

[0043] In some embodiments, the processor 20 is configured to display on the display 34 various types of test patterns that may be stored in the DI 42 and / or memory 22, including one or more segments (not shown) of the DI 42.

[0044] In some embodiments, the blanket treatment station 52, also referred to herein as a cooling station, is configured to treat the blanket, for example, by cooling the blanket and / or applying a treatment fluid to the outer surface of the blanket 44 and / or cleaning the outer surface of the blanket 44. In the blanket treatment station 52, the temperature of the blanket 44 can be reduced to a desired temperature value before the blanket 44 enters the imaging station 60. Treatment can be performed by passing the blanket 44 over one or more rollers or blades configured to cool and / or clean and / or apply a treatment fluid to the outer surface of the blanket.

[0045] In some embodiments, the blanket treatment station 52 may be located adjacent to the imaging station 60 in addition to, or instead of, the location of the blanket treatment station 52 shown in Figure 1. In such embodiments, the blanket treatment station may include one or more bars adjacent to the print bar 62, where treatment fluid is applied to the blanket 44 by jets.

[0046] In some embodiments, processor 20 is configured to receive a signal indicative of the surface temperature of blanket 44, for example, from a temperature sensor (not shown), to monitor the temperature of blanket 44 and control the operation of blanket treatment station 52. Examples of such treatment stations are described, for example, in PCT International Publication Nos. WO 2013 / 132424 and WO 2017 / 208152, the disclosures of which are incorporated herein by reference in their entireties.

[0047] Additionally or alternatively, treatment fluid may be applied to blanket 44 by jetting prior to ink jetting at the imaging stations.

[0048] 1, station 52 is mounted between impression station 84 and imaging station 60, however, station 52 may be mounted adjacent blanket 44 in any other or additional suitable location or locations between impression station 84 and imaging station 60. As previously mentioned, station 52 may additionally or alternatively be mounted on a bar adjacent imaging station 60.

[0049] In the example of FIG. 1, impression cylinder 82 presses the ink image onto a target flexible substrate, such as an individual sheet 50 , which is conveyed by substrate transport module 80 from an input stack 86 to an output stack 88 via impression cylinder 82 .

[0050] In some embodiments, the lower run of blanket 44 selectively interacts with impression cylinder 82 at impression station 84 to impress an image pattern onto a target flexible substrate compressed between blanket 44 and impression cylinder 82 under the pressure of pressure cylinder 90. For the simplex printer shown in FIG. 1 (i.e., printing on one side of sheet 50), only one impression station 84 is required.

[0051] In other embodiments, module 80 may include two or more impression cylinders (not shown) to enable one or more duplex printing operations. A two-impression cylinder configuration also allows for single-sided printing to be performed at twice the speed of duplex printing. In addition, mixed lots of single-sided and double-sided prints can be printed. In alternative embodiments, a different configuration of module 80 may be used for printing on continuous web substrates. Detailed descriptions and various configurations of duplex printing systems and systems for printing on continuous web substrates are provided, for example, in U.S. Pat. Nos. 9,914,316 and 9,186,884, PCT International Publication No. WO 2013 / 132424, U.S. Patent Application Publication No. 2015 / 0054865, and U.S. Provisional Application No. 62 / 596,926, the disclosures of which are incorporated herein by reference in their entirety.

[0052] As briefly described above, a sheet 50 or continuous web substrate (not shown) is conveyed by module 80 from an input stack 86 and passes through a nip (not shown) located between impression cylinder 82 and pressure cylinder 90. Within the nip, the surface of blanket 44 carrying the ink image is pressed firmly against sheet 50 (or other suitable substrate) by, for example, a compressible blanket (not shown) of pressure cylinder 90, thereby imprinting the ink image onto the surface of sheet 50 and cleanly separating it from the surface of blanket 44. Sheet 50 is then conveyed to output stack 88.

[0053] 1, roller 78 is positioned on the upper run of blanket 44 and is configured to keep blanket 44 taut as it passes adjacent to imaging station 60. Furthermore, it is particularly important to control the speed of blanket 44 under imaging station 60 to obtain precise jetting and deposition of ink droplets onto the surface of blanket 44 by imaging station 60, thereby achieving placement of the ink image.

[0054] In some embodiments, the impression cylinder 82 is periodically engaged and disengaged from the blanket 44 to transfer the ink image from the moving blanket 44 to a target substrate passing between the blanket 44 and the impression cylinder 82. In some embodiments, the system 10 is configured to apply torque to the blanket 44 using the roller and dancer assembly described above to maintain the upper run taut and substantially isolate the upper run of the blanket 44 from being affected by mechanical vibrations occurring in the lower run.

[0055] In some embodiments, system 10 includes an image quality control station 55, also referred to herein as an automated quality control (AQM) system, which functions as a closed-loop inspection system integrated within system 10. In some embodiments, image quality control station 55 may be located adjacent impression cylinder 82, as shown in FIG. 1, or at any other suitable location within system 10.

[0056] In some embodiments, image quality control station 55 includes a camera (not shown) that is configured to acquire one or more digital images of the aforementioned ink images printed on sheet 50. In some embodiments, the camera may include any suitable image sensor, such as a contact image sensor (CIS) or a complementary metal oxide semiconductor (CMOS) image sensor, and a scanner that includes a slit having a width of about 1 meter or any other suitable width.

[0057] In the context of this disclosure and in the claims, the term "about" or "approximately" in connection with any numerical value or range indicates an appropriate dimensional tolerance that enables a part or collection of components to function for its intended purpose as described herein. For example, "about" or "approximately" may refer to a range of values ​​of ±20% of the recited value, e.g., "about 90%" may refer to a range of values ​​of 72% to 100%.

[0058] In some embodiments, station 55 may include a spectrophotometer (not shown) configured to monitor the quality of the ink printed on sheet 50 .

[0059] In some embodiments, digital images acquired by station 55 are transmitted to a processor, such as processor 20 or any other processor in station 55, which is configured to evaluate the quality of each printed image. Based on that evaluation and signals received from controller 54, processor 20 is configured to control the operation of the modules and stations of system 10. In the context of the present invention and claims, the term "processor" refers to any processing device, such as processor 20 or any other processor or controller connected to or integrated with station 55, that is configured to process signals received from the camera and / or spectrophotometer in station 55. It should be noted that the signal processing operations, control-related instructions, and other computational operations described herein may be performed by a single processor or distributed among multiple processors in one or more respective computers.

[0060] In some embodiments, station 55 is configured to inspect the quality of printed images and test patterns to monitor various attributes such as, but not limited to, perfect image registration with sheet 50, color-to-color (CTC) registration, printed geometry, image uniformity, color contour and linearity, and print nozzle functionality. In some embodiments, processor 20 is configured to automatically detect geometric distortions or other errors in one or more of the aforementioned attributes. For example, processor 20 is configured to compare between a design version of a given digital image (also referred to herein as a “master” or “source image”) and a digital image of a printed version of the given image, acquired by a camera.

[0061] In other embodiments, processor 20 may apply any suitable type of image processing software, for example, to a test pattern, to detect distortions indicative of the aforementioned errors. In some embodiments, processor 20 is configured to analyze the detected distortions to apply corrective action to the malfunctioning module and / or provide instructions to another module or station of system 10 to correct the detected distortions.

[0062] In some embodiments, system 10 may print test marks (not shown), for example, on the bevel or margin of sheets 50. By acquiring images of the test marks, station 55 is configured to measure various types of distortions, such as C2C registration errors, image-to-substrate registration, different widths between colors, referred to herein as “bar-to-bar width delta” or “color-to-color width difference,” various types of local distortions, and front-to-back registration errors (in duplex printing). In some embodiments, processor 20 is configured to (i) sort sheets 50 having distortions above a first predefined set of thresholds, for example, to a reject tray (not shown), (ii) initiate corrective action on sheets 50 having distortions above a second, lower, predefined set of thresholds, and (iii) output sheets 50 having insignificant distortions, for example, below the second set of thresholds, to output stack 88.

[0063] In some embodiments, the processor 20 is further configured to detect additional geometric distortions, such as expansion or contraction, skew, or wave distortion formed in at least one of an axis parallel to the axis of movement of the blanket 44 and an axis perpendicular thereto, for example, by analyzing the pattern of printed inspection marks.

[0064] In some embodiments, processor 20 is configured to detect deviations in the contour and linearity of printed colors based on signals received from the spectrophotometer of station 55 .

[0065] In some embodiments, processor 20 is configured to detect various types of defects based on signals acquired by station 55: (i) defects in the substrate (e.g., blanket 44 and / or sheet 50), such as scratches, pinholes, and damaged edges, and (ii) printing-related defects, such as irregular color spots, satellites, and splashes.

[0066] In some embodiments, processor 20 is configured to detect these defects by comparison between sections of the print and respective reference sections of the original design, also referred to herein as the master. Processor 20 is further configured to classify the defects and, based on the classification and one or more predetermined criteria, reject sheets 50 having defects that are not within the specified predetermined criteria.

[0067] In some embodiments, system 10 includes one or more suitable types of neural networks, which may be implemented within processor 20 and / or DFEM 100 and / or any other suitable processing device or module of system 10. One implementation of an exemplary type of neural network is described in detail below in FIG. 4, and methods for applying neural networks are described in detail below in FIGS. 2A, 2B, 3, and 5.

[0068] In some embodiments, the processor of station 55 is configured to determine whether to stop operation of system 10, for example, if the defect density exceeds a specified threshold. The processor of station 55 is further configured to initiate corrective action in one or more of the modules and stations of system 10, as described above. The corrective action may be performed on the fly (while system 10 continues the printing process) or offline by stopping printing operations and correcting the problem within the respective module and / or station of system 10. In other embodiments, any other processor or controller of system 10 (e.g., processor 20 or controller 54) is configured to initiate corrective action or stop operation of system 10 if the defect density exceeds a specified threshold.

[0069] Additionally or alternatively, processor 20 may be configured to receive signals, e.g., from station 55, indicative of additional types of defects and problems in the printing process of system 10. Based on these signals, processor 20 may be configured to automatically estimate the pattern placement accuracy and the level of additional types of defects not previously described. In other embodiments, any other suitable method for inspecting patterns printed on sheet 50 (or on any other substrate previously described) may also be used, e.g., using an external (e.g., offline) inspection system, or any type of measurement fixture and / or scanner. In these embodiments, based on information received from the external inspection system, processor 20 may be configured to initiate any appropriate corrective action and / or to stop operation of system 10.

[0070] The configuration of system 10 is provided in a simplified manner purely as an example to clarify the present invention. The components, modules, and stations described in the foregoing printing system 10, as well as additional components and configurations, are described in detail in, for example, U.S. Patent Nos. 9,327,496 and 9,186,884, PCT International Publication Nos. WO2013 / 132438, WO2013 / 132424, and WO2017 / 208152, and U.S. Patent Application Publication Nos. 2015 / 0118503 and 2017 / 0008272, the disclosures of which are incorporated herein by reference in their entirety.

[0071] The particular configuration of system 10 is shown as an example to illustrate the particular problem addressed by embodiments of the present invention and to demonstrate the application of these embodiments in enhancing the performance of such systems, but embodiments of the present invention are in no way limited to this particular type of example system, and the principles described herein may be applied to any other type of printing system as well.

[0072] Training a neural network to detect missing nozzle faults in digital printing FIG. 2A is a schematic, pictorial diagram illustrating the training phase of a convolutional neural network (CNN) configured to detect missing nozzle failures (MNF), in accordance with one embodiment of the present invention.

[0073] In some cases, a defective part (DP) in system 10 can cause defects, such as distortions, in the printed image. In this example, a defective nozzle (DN) from among nozzles 99 can cause a missing nozzle failure (MNF) in the image formed on blanket 44, and thus typically not only on blanket 44 but also on a corresponding sheet 50 (shown above in FIG. 1 ) or any other target substrate.

[0074] In some embodiments, a neural network (NN) may be used to detect one or more MNFs and identify one or more DNs causing MNF, as described in FIG. 5 below. The method includes two phases: (i) a training phase, in which the NN is trained using known input and output data, and (ii) a detection phase, which follows the training phase and is based on the training phase. In this example, a CNN, whose structure is described in detail in FIG. 4 below, may be used for this task, although, mutatis mutandis, any other suitable type of neural network may be used to detect MNFs and identify DNs causing MNF. Note that the architecture of the CNN was selected and optimized based on simulations and experiments performed by the inventors.

[0075] In some embodiments, the processor 20 is configured to receive a digital image, referred to herein as a RIP image or image 101, from, for example, a RIP of the DFEM 100, which is generated on the blanket 44 and transferred to the sheet 50, as described above in FIG. 1.

[0076] In the example of Figure 2A, image 101 is formed on moving blanket 44 so that as blanket 44 passes under print bars 62, ink droplets of colors selected for image 101 (e.g., cyan, magenta, yellow, and black, also referred to herein for brevity as CMYK) are directed by print bars 62 onto predetermined areas 111 of blanket 44 to form image 101. As shown in Figure 2A, as blanket 44 passes under cyan print bar 62, cyan ink is directed by nozzles 99A and 99B onto predetermined areas of blanket 44, referred to herein as patches 111A and 111B. The same process is performed for other print bars 62 (e.g., magenta (M) and yellow (Y) print bars 62).

[0077] In the context of this disclosure and in the claims, the terms "region" and "patch" are used interchangeably to refer to a section on a digital image (e.g., image 101 or any other image described herein).

[0078] Referring now to inset 102, an RGB (red, green, blue) palette 104 is shown, including red 105, green 106, blue 107, and combinations thereof. Also shown in inset 102 is a CMYK palette 103, including cyan ink (C) 108, magenta ink (M) 109, and yellow ink (Y) 110, and combinations thereof. Note that CMYK palette 103 includes all colors in RGB palette 104. Furthermore, the term "K" in the CMYK palette refers to the black color formed when mixing C, M, and Y inks, as shown in the center of CMYK palette 103. For example, green 106 is formed in the CMYK palette by mixing C 108 with Y 110, blue 107 is formed in the CMYK palette by mixing C 108 with M 109, and red 105 is formed in the CMYK palette by mixing M 109 with Y 110.

[0079] The following description of a method for generating a first set of one or more synthetic images (SI) for training a CNN to detect MNF is implemented by processor 20. However, the method may be implemented mutatis mutandis using a RIP of DFEM 100 or any other suitable processing unit or module of system 10, such as, but not limited to, a processor of quality control station 55.

[0080] In some embodiments, processor 20 is configured to generate (i) SIs 112A and 112B in patch 111A, (ii) SIs 114A and 114B in patch 111B, and (iii) SIs 116A and 116B in patch 111C. In some embodiments, processor 20 is configured to generate a composite MNF within each of the SIs. For example, to simulate a defective (e.g., blocked) nozzle 99A in SI 112A, cyan ink 108 is not applied to the substrate (e.g., blanket 44) in patch 111A, and thus SI 112A has column 118 with only yellow. Similarly, to simulate a blocked nozzle 99B in SI 114B, cyan ink 108 is not applied to the blanket 44 in patch 111B, and thus SI 114B has column 120 with only magenta. Note that the columns are generated as the blanket 44 moves in the direction of travel relative to the print bar 62, and the column positions within a given patch are derived from the positions of the respective nozzles relative to the direction of the given patch.

[0081] In some embodiments, processor 20 is configured to select the locations of patches 111 using any set of one or more predefined criteria. For example, each patch 111 may have 32x32 pixels, with at least a given amount (e.g., a percentage) of the pixels in the selected patch having a gray level less than 253 (on a 0-255 scale of gray levels). Note that each patch 111 location corresponds to one or more simulated defective (e.g., blocked) nozzles 99 intended to direct ink onto the surface of blanket 44 at the respective patch 111 location.

[0082] In some embodiments, the printed pixel size may be approximately 21 μm using a printing resolution of 1200 dots per inch (DPI), or any other suitable printing resolution. In such an embodiment, the 32×32 pixel patch 111 may have a size of approximately 0.672 mm×0.672 mm.

[0083] In some embodiments, processor 20 is configured to determine the number and locations of patches 111 to be distributed across image 101 based on any set of predefined criteria. For example, processor 20 may determine approximately 5000 patches 111 to be distributed within image 101 to cover approximately 80% of the width of image 101, e.g., orthogonal to the direction of movement of blanket 44, represented by arrow 94.

[0084] In some embodiments, processor 20 is configured to select patches 111 for training the CNN using any suitable criterion or set of criteria. For example, patch 111A can be used to train the CNN if, in patch 111A, (i) the original composite image (e.g., without a simulated MNF) and (ii) SI 112A (with a simulated MNF) have at least 10 pixel pairs with a gray level difference along a column of Y110 that is greater than about 15 gray levels.

[0085] In some embodiments, the number of SIs is derived based on, among other things, the number of patches and the average number of colors (e.g., RGB converted to CMYK or any other suitable combination of ink colors). For example, each patch 111 may have, on average, three different colors, and thus four SIs corresponding to a composite image for each CMYK color. In such an example embodiment, for 5000 patches, processor 20 generates approximately 20,000 SIs, and if only 80% of the patches are eligible for training the CNN, processor 20 can use 16,000 SIs (e.g., SIs 112A-116B) to train the CNN to detect one or more MNFs.

[0086] If 16,000 SIs are insufficient to achieve the required level of training, processor 20 may increase the number of patches 111 or select patches with more colors (and therefore more SIs).

[0087] FIG. 2B is a schematic, pictorial diagram illustrating the impact on printing of a partially clogged nozzle 99 and a technique for training a CNN to detect partially clogged nozzles, in accordance with one embodiment of the present invention.

[0088] In some embodiments, the techniques described above in FIG. 2A may be used to train a CNN and to detect additional faults that may occur in system 10, such as, but not limited to, a partially clogged nozzle.

[0089] 2B, after applying (e.g., ejecting) a droplet of ink from a nozzle 99 of print bar 62 (shown previously in FIG. 1), ink residue may remain on the surface of nozzle 99 and may solidify and produce undesirable clusters 72 of dried ink. Clusters 72 may partially block orifices 85 of nozzles 99, which, as described herein, may result in registration errors in the printed image.

[0090] In some embodiments, nozzle 99 is configured to eject ink droplets 75A and 75B toward an intended location 79 on the surface of blanket 44, along the Z axis of an XYZ coordinate system in this example. In some cases, the formation of cluster 72 can cause a deviation in the steering angle of droplets ejected toward blanket 44. As shown in the example of FIG. 2B, droplet 75A is deflected by cluster 72, and droplet 75B is expected to have a similar deviation as it passes through orifice 85. Note that the deviation in the steering angle depends, among other things, on the size and stiffness of cluster 72 and the location of cluster 72 on the surface of nozzle 99. The deviation in the steering angle causes droplet 75A to be deposited at a location 81 some distance away from the intended location 79 on the surface of blanket 44. In other words, cluster 72 causes a positioning error, which is measured by the distance 81 between the intended location of droplet 75A and the actual deposition location on the surface of blanket 44.

[0091] In the context of this disclosure and in the claims, the terms "partially clogged" and "partially blocked" are used interchangeably to refer to a nozzle 99 having a cluster 72 that does not completely block the orifice 85 of the nozzle 99 but deflects droplets 75A and 75B, as described above.

[0092] In some embodiments, during the training phase of the CNN, processor 20 is configured to select patches, such as patch 111 shown in FIG. 2A above, that are appropriate for detecting partially clogged nozzles 99 in print bar 62. Additionally, processor 20 is configured to generate a composite image showing the simulated landing locations of deflected ink droplets. For example, for a partially clogged nozzle 99, processor 20 is configured to generate a set of SIs in which the simulated landing locations are determined in a spherical coordinate system, the simulated landing locations including: (i) the radial distance of the landing location from the partially clogged nozzle, also referred to herein as r; (ii) the polar angle, also referred to herein as θ, measured from a fixed zenith direction (e.g., parallel to the Z axis and typically perpendicular to the surface of blanket 44) relative to the orifice 85 of the partially clogged nozzle 99; and (iii) the azimuthal angle of its orthogonal projection onto a reference plane that passes through the origin of the XYZ coordinate system and is perpendicular to the zenith. The azimuthal angle is measured from a fixed reference direction on that plane and is also referred to herein as φ.

[0093] In some embodiments, processor 20 is configured to estimate, for each SI, the distance between the intended landing position of the ink droplet (i.e., without the partial clog) and the actual landing position of the droplet (due to the partial clog). In the example of FIG. 2B, distance 81 was measured between intended position 79 and the actual landing position of droplet 75A. The intended and actual landing positions may be calculated in r, θ, φ coordinates in a spherical coordinate system and used by processor 20 to estimate the size and orientation of distance 81.

[0094] In some embodiments, the processor 20 is configured to select patches 111 for detecting partially clogged nozzles 99 using any suitable one or more criteria, such as irregular differences in gray levels within patches having an array of repeating structures (e.g., a grid of lines and spaces).

[0095] In some embodiments, processor 20 is configured to generate, for each selected patch 111, a set of synthetic images based on the aforementioned spherical coordinate system, including simulated landing locations of ink droplets deflected by a partially clogged nozzle. The number of SIs depends, among other things, on the number of distinct points selected within the spherical coordinate system, limited to the use case of interest, and satisfying computational capacity constraints. For example, processor 20 may select distinct points having a common radial distance estimated by the ejection force applied to droplets 75A and 75B, and a selected predefined number of polar and azimuthal angles.

[0096] Missing nozzle fault detection and defective nozzle identification using trained CNNs 3 is a schematic, pictorial diagram illustrating a method for detecting MNF using a trained CNN, according to one embodiment of the present invention. The following embodiment describes a method implementation using processor 20. However, the method may be implemented mutatis mutandis using DFEM 100 or its devices, or using any other suitable processing device or module of system 10.

[0097] In some embodiments, processor 20 receives DI 42, obtained by image quality control station 55 from a printed version of the digital image received from DFEM 100. Note that image 101 of FIG. 2A above has not yet been printed, and DI 42 is a digital image obtained from the printed image and, therefore, may include one or more distortions and / or defects, such as one or more MNFs caused by one or more defective nozzles. Moreover, while the CNN training described in FIG. 2A above was performed using image 101, a first digital image, DI 42 has a second digital image, which may be similar or different from the first digital image. In other words, CNN training may be performed on a given digital image, and detection of one or more MNFs and DNs described herein may be performed on a digital image obtained from a printed version of the given digital image or from a printed version of a different image. For example, image 101 may include a digital image of a dog (not shown), and DI 42 may include an image of the same dog (with or without imperfections introduced during the printing process), or, as shown in DI 42 of Figures 1 and 2 above, from a printed version of a digital image of an elephant.

[0098] In some embodiments, processor 20 is configured to apply the training dataset shown in FIG. 2A above (and / or the training dataset described in FIG. 2B above) to a suitable CNN architecture to identify one or more regions (e.g., regions 135 and 145) within DI 42 that are suspected of having the defects that the CNN was trained to detect (e.g., MNF and / or C2C positioning errors caused by partially clogged nozzle 99), as described in FIG. 2A above.

[0099] Note that by detecting potential MNFs within regions 135 and 145, the trained CNN reduces the number of suspected defective nozzles 99 within system 10. Thus, region 135 contains a known set of color pixels printed by a known nozzle 99 of a known print bar 62 of imaging station 60 of system 10.

[0100] 3, DI 136 shows a higher magnification of a section of an elephant ear shown in region 135 of DI 42. In some embodiments, processor 20 is configured to generate a set of one or more composite images, referred to herein as set 137, for region 135. Each SI in set 137 includes a simulation of one or more defective nozzles 99 selected from known print bars 62 and nozzles 99 used to apply one or more colors of ink to region 135.

[0101] In some embodiments, the SI of set 137 includes all combinations of nozzles 99 of print bars 62 that participate in forming DI 136 .

[0102] For example, DI 136 may be formed using four nozzles 99 in the cyan print bar 62, five nozzles 99 in the magenta print bar 62, and three nozzles 99 in the yellow print bar 62. Thus, set 137 may include up to (4·5·3=) 60 SIs, one SI for each nozzle 99. In some embodiments, the number of suspected defective nozzles, and therefore the number of SIs in set 137, may be reduced. For example, if MNF is visible and appears to occur in a particular color, e.g., cyan. In this example, set 137 may include only four SIs, each SI simulating one suspected defective nozzle 99 in the cyan print bar 62.

[0103] In some embodiments, processor 20 is configured to identify at least one defective nozzle from among the nozzles involved in forming region 135. In one embodiment, processor 20 is configured to compare between D1 136 and each SI of set 137. The comparison may be performed using a trained CNN or any other suitable image comparison technique.

[0104] In some embodiments, by comparing the DI 136 of region 135 with each SI of set 137, processor 20 is configured to: (i) identify one or more MNFs within suspect region 135, and (ii) associate each MNF with one or more respective defective nozzles 99 from among the nozzles involved in forming region 135.

[0105] A process for detecting defective nozzles based on the embodiments described in one or more of the above-described FIGS. 2A, 2B, and 3 may be summarized using the following example. In the example, nozzle 99A of cyan print bar 62 (shown in the above-described FIG. 2A) may be defective, e.g., blocked, and therefore unable to apply droplets of cyan ink to blanket 44 in region 135 of DI 42. During the training phase shown and described in the above-described FIG. 2A, processor 20 is configured to generate SI 112A, which shows a simulation of blocked nozzle 99A in patch 111A. A CNN is trained based on SI 112A of the above-described FIG. 2A and other SIs to detect areas in a digital image printed by system 10 that are suspected of having one or more MNFs caused by one or more respective defective nozzles 99. In the detection phase (shown in FIG. 3 ) following the training phase, the trained CNN is configured to detect areas 135 within DI 42 printed using nozzle 99A and additional nozzles 99, at least one of which is suspected to be a defective nozzle.

[0106] In some embodiments, processor 20 is configured to generate an SI to simulate each nozzle 99 in set 137 that is associated with region 135 and is suspected to be defective, as described above.

[0107] In some embodiments, the processor 20 is configured to detect a blocked nozzle 99A by comparing between the DI 136 and each SI in the set 137 to find a correlation between the DI 136 and the SI in the set 137 and simulating the MNF caused by the blocked nozzle 99A.

[0108] Similarly, DI 146 shows a higher magnification view of a section of the ivory tip shown in region 145 of DI 42. In some embodiments, processor 20 is configured to generate a set of one or more composite images, referred to herein as set 147, for region 145. Each SI in set 147 includes a simulation of one or more defective nozzles 99 selected from known print bars 62 and nozzles 99 used to apply one or more colors of ink to suspect region 145. Processor 20 is configured to apply the above-described process to region 135 to (i) detect one or more MNFs in region 145, and (ii) identify, within one or more respective print bars 62 of system 10, the one or more defective nozzles 99 causing the detected MNFs.

[0109] In another embodiment, processor 20 is configured to apply the technique described above in FIG. 2A to the trained CNN to detect regions suspected of having partially clogged nozzles 99. Those skilled in the art of digital printing will understand that similar mechanisms can typically cause partially clogged nozzles and completely blocked nozzles. Thus, in some cases, a given print bar 62 may cause both the MNF defect and the positioning error described above in FIGS. 2A and 2B, respectively. In this example, the CNN may output that region 145 is suspected of containing a positioning error that may be caused by cluster 72 partially blocking orifice 85 of nozzle 99, as shown and described in detail above in FIG. 2B.

[0110] In some embodiments, processor 20 receives, for example, from image quality control station 55, DI 146 showing a higher magnification of the aforementioned section of the ivory tip shown in region 145 of DI 42.

[0111] In some embodiments, processor 20 is configured to generate a supplemental set of one or more composite images for region 145 in addition to or instead of set 147. Each SI in the supplemental set includes a simulation of one or more positioning errors caused by stray droplets ejected by nozzle 99 suspected of being partially clogged, intended to apply droplets of one or more colors of ink to suspect region 145.

[0112] In some embodiments, processor 20 is configured to apply the above-described process for detecting MNF defects in regions 135 and 145 to (i) detect one or more positioning errors in region 145 and (ii) identify one or more partially clogged nozzles 99 within one or more respective print bars 62 of system 10 that fire stray droplets (such as droplet 75A in FIG. 2B described above) that are causing the detected positioning errors.

[0113] In this example, droplet deflection may occur within a particular nozzle 99 of cyan print bar 62 (shown previously in FIG. 2A ) that is intended to apply droplets of cyan ink to predefined sections of blanket 44. Thus, the previously described partially clogged nozzle 99 may result in C2C positioning errors between the cyan color and other colors applied to those predefined sections.

[0114] In other embodiments, processor 20 is configured to use CNN to detect suspect regions 135 and 145, for example, by applying the technique described above in FIG. 2A. Then, in the detection phase shown in FIG. 3, processor 20 is configured to apply the technique shown above in FIG. 2B to detect defects indicative of one or more partially clogged nozzles 99 within regions 135 and 145. In such embodiments, processor 20 may apply CNN only to regions 135 and 145 to generate SIs indicative of simulated partially clogged nozzles.

[0115] 2A, 2B, and 3 illustrate, by way of example, methods for identifying defective nozzles 99 (e.g., fully blocked or partially clogged) within print bar 62 of system 10. However, the techniques described herein can be used mutatis mutandis to identify other defective parts in system 10, or in any other suitable technique involving identifying one or more defective parts in any type of system used for printing or any other type of production process.

[0116] 4 is a schematic, pictorial diagram of a convolutional neural network (CNN) 150 used to detect MNF within DI 42, in accordance with one embodiment of the present invention. In some embodiments, elements of CNN 150 may be implemented using hardware or software, or any suitable combination thereof.

[0117] In some embodiments, processor 20 is configured to train CNN 150 to detect one or more MNFs within DI 42. Processor 20 is further configured to use the one or more detected MNFs to identify defective nozzles 99 (and / or other components) of system 10, as described above in Figures 2A, 2B, and 3, using the method described in Figure 5 below.

[0118] In some embodiments, CNN 150 has the Inception V3 architecture provided by Google (Mountain View, CA 94043, USA), but may have any other suitable type of neural network.

[0119] In some embodiments, CNN 150 may include multiple sections and modules, as described below.

[0120] In some embodiments, CNN 150 may include (e.g., within the aforementioned modules) a multi-layer convolutional neural network, each of the layers having an array of neurons.

[0121] In some embodiments, each neuron in CNN 150 calculates an output value by applying a particular function to input values ​​coming from the receptive field in the previous layer. The function applied to the input values ​​is determined by a vector of weights and biases (typically real numbers). The learning process in CNN 150 proceeds by iteratively adjusting these biases and weights.

[0122] The vector of weights and biases is referred to herein as the layer's filter and is defined by a particular size and shape (e.g., a particular shape) of the input. A notable feature of CNNs is that many neurons can share the same filter.

[0123] In some embodiments, CNN 150 is configured to receive input 160 including a 299x299x3 array of weights corresponding to 299x299 pixels of a respective digital image, with each pixel having three colors (e.g., RGB as described above in Figures 2A and 3).

[0124] Reference is now made to inset 151, which shows a legend for the layers and other elements used in the architecture of CNN 150. In some embodiments, CNN 150 includes a convolutional layer, labeled "Conv" in inset 151 and referred to herein as CL 152. The CNN includes an average pooling layer 153, labeled "Average Pool" in inset 151, and a max pooling layer 154, labeled "Max Pool" in inset 151. Pooling layers are configured to reduce the dimensionality of the data by combining the outputs of neuron clusters in one layer into a single neuron in the next layer.

[0125] The term "max pooling" refers to pooling that uses the maximum value from each cluster of neurons in the previous layer. Average pooling uses the average value from each cluster of neurons in the previous layer, and is therefore configured to convert the output tensor of a convolutional layer into a vector of weights, e.g., having any suitable number of scalars. In the context of the present invention and neural networks, the term "flatten" refers to converting a multidimensional tensor into a one-dimensional vector.

[0126] In some embodiments, CNN 150 includes multiple connections configured to connect between adjacent layers and / or modules and / or sections of CNN 150. Each of the connections is labeled "concat" in inset 151 and is referred to herein as connection 155.

[0127] In some embodiments, CNN 150 includes one or more dropout layers, referred to herein as dropout 156, which may be used in multiple layers as described herein. A softmax activation function may be used in one or more layers (e.g., in a classifier layer described below), referred to herein as softmax 158. The dropout layers and softmax activation function are labeled "dropout" and "Softmax," respectively, in inset 151.

[0128] In some embodiments, CNN 150 includes one or more fully connected layers (FCLs) 157, labeled "Fully connected" in inset 151. Note that "fully connected layer" refers to a neural network layer that connects every neuron in one layer to every neuron in another layer.

[0129] Referring again to the overview of Figure 4, in some embodiments, the CNN 150 includes a module 162 having a plurality of CLs 152 arranged in series and a max-pooling layer 154. The module 162 is configured to receive an input 160 and prepare the input weights for insertion into the module 164, which includes a plurality of CLs 152 and an average-pooling layer 153 arranged in a suitable structure, also referred to as "5X Inception Module A," with connections 155 for connecting between the structure of the layers.

[0130] In some embodiments, CNN 150 includes a grid size reduction module 166 that is configured to convert the input 160 299×299×3 array of weights to an output array of 8×8×2048 weights, referred to herein as output 180. In this example, grid size reduction module 166 includes multiple CLs 152, max pooling layers 154, and concatenation 155.

[0131] In other embodiments, the module 166 may have a different structure than that shown in FIG.

[0132] In some embodiments, the CNN 150 includes a module 168, also called a "4X Inception Module B," that includes multiple CLs 152 and average pooling layers 153 arranged in a suitable structure, with connections 155 for connecting between the structure of the layers.

[0133] In some embodiments, CNN 150 includes an auxiliary classifier 174, which is coupled to concatenation 172, the rightmost concatenation of module 168. Auxiliary classifier 174 includes multiple layers, such as one average pooling layer 153, two CLs 152, one FCL 157, and one softmax 158.

[0134] In some embodiments, the CNN 150 includes a grid size reduction module 170 that includes multiple CLs 152, a max pooling layer 154, and connections 155. In some embodiments, the CNN 150 includes a module 176, also referred to herein as a "2X Inception module C," that includes multiple CLs 152 and average pooling layers 153 arranged in a suitable structure and has connections 155 for connecting between the structure of the layers.

[0135] In some embodiments, CNN 150 includes tensor 178, which includes multiple layers, such as one average pooling layer 153, one dropout 156, one FCL 157, and one softmax 158. The output of CNN 150, tensor 178, has the structure previously described for output 180 (an 8x8x2048 array of weights).

[0136] The structure of the Inception V3 CNN architecture and in particular that of CNN 150, as well as examples of its use, are described in detail, for example, in Szegedy et al., "Rethinking the Inception Architecture for Computer Vision," computer vision and pattern recognition (CVPR) conference of the Computer Vision Foundation (CVF), pp. 2818-2826 (June 2016); and Sik-Ho Tsang, "Review: Inception-v3-1st Runner Up (Image Classification) in ILSVRC 2015," September 10, 2018, all of which are incorporated herein by reference.

[0137] This particular configuration of CNN 150 is presented as an example to illustrate certain problems addressed by embodiments of the present invention and to show the application of these embodiments in enhancing the performance of system 10 using CNN 150. However, embodiments of the present invention are in no way limited to this particular type of example CNN configuration, and the principles described herein may likewise be applied to other types of neural networks used to enhance the performance of such digital printing systems.

[0138] 5 is a flow chart that schematically illustrates a method for detecting defective nozzles within system 10, in accordance with one embodiment of the present invention. The method may be implemented using processor 20, as described below. However, the method may, mutatis mutandis, be implemented using DFEM 100 or its devices, or any other suitable processing device or module of system 10.

[0139] The method begins with a first raster image reception step 200, in which processor 20 receives an image 101, for example from the RIP of DFEM 100, as described above in Figure 2A. After step 200, the method has two phases: a training phase, and a detection phase following the training phase.

[0140] The detection phase begins with a patch selection step 202 in which processor 20 selects one or more (e.g., approximately 5000) patches (e.g., patches 111, 111A, 111B, and 111C) containing features of image 101 based on predefined criteria, as described above in FIG. 2A.

[0141] In a first composite image (SI) generation step 204, processor 20 generates, for each patch generated in step 202, a first set of one or more composite images (e.g., SIs 112A, 112B, 114A, 114B, 116A, and 116B of FIG. 2A above) having simulated missing nozzle failures (MNFs) within the respective patches (e.g., patches 111A, 111B, and 111C), as described above in FIG. 2A. Additionally or alternatively, processor 20 generates, for each patch generated in step 202, a first additional set of one or more composite images having simulated positioning errors that may result (due to partially clogged nozzles 99) within one or more respective patches 111, as described above in FIG. 2B.

[0142] In an SI selection step 206, the processor 20 selects one or more of the SIs of the first set (and / or the first additional set) generated in the above-described step 204 based on a second predefined criterion. The selected SIs are suitable for training a convolutional neural network (CNN), such as CNN 150, as described above in Figures 2A and 2B.

[0143] In a CNN training step 208, processor 20 trains CNN 150 using the first set (and / or first additional set) of synthetic images selected in step 206, as described above in Figures 2A and 2B. Step 208 concludes the training phase, after which CNN 150 is trained to detect defects, such as MNF and / or positioning errors, in images printed by system 10.

[0144] In a digital image receiving step 210, processor 20 receives a digital image obtained from an image printed by system 10, for example, from image quality control station 55. In some embodiments, processor 20 generates DI 42 based on the image received from image quality control station 55. In other embodiments, processor 20 receives DI 42 generated by image quality control station 55.

[0145] After completing step 210, the method begins the detection phase with region identification step 212 in which processor 20 applies CNN 150 within DI 42 to identify regions 135 and 145 suspected of having MNF and / or positioning errors, as described in detail above in FIG. 3 .

[0146] In a second SI generation step 214, processor 20 generates, for each of regions 135 and 145 detected in step 212, a second set of one or more composite images (e.g., sets 137 and 147 of FIG. 3 above) having simulated missing nozzle failures (MNFs) and / or positioning errors within the respective region (e.g., regions 135 and 145). As described in detail in FIG. 3 above, the number of SIs in sets 137 and 147 corresponds to the number of nozzles 99 involved in the images formed in regions 135 and 145. For example, sets 137 may each include up to 60 SIs to simulate defects in each of the 60 nozzles 99 used by system 10 in forming the image of region 135. In other words, each SI in set 137 has one simulated defective nozzle 99 to encompass all nozzles 99 used to apply ink to region 135 of DI 42.

[0147] In an MNF detection step 216, processor 20 detects one or more MNFs by comparing a second set of SIs (e.g., set 137) with the acquired DIs (e.g., DIs 136) of the corresponding suspect regions (e.g., region 135). In other embodiments, in step 216, processor 20 may detect one or more C2C positioning errors (in addition to or instead of MNFs) by comparing a second set of SIs (e.g., a supplemental set of one or more composite images described above in FIG. 3) with the acquired DIs (e.g., DIs 146) of the corresponding suspect regions (e.g., region 145). It should be noted that processor 20 may (i) use a training phase to train a CNN to detect both MNF and C2C positioning errors (caused by partially clogged nozzle(s) 99), (ii) apply the CNN to detect regions suspected of having MNF (e.g., regions 135 and 145), and (iii) apply the CNN trained to detect both MNF and C2C positioning errors to the suspect regions.

[0148] In a defective nozzle identification step 218, which concludes the detection phase and ends the method, processor 20 identifies one or more defective nozzles 99 by associating each MNF detected in step 216 with a respective defective nozzle 99 in system 10. As described above in FIG. 3 , processor 20 may compare DI 136 with each SI in set 137 to detect a defective nozzle (such as blocked nozzle 99A) by finding a correlation between DI 136 and a corresponding SI in set 137 that simulates an MNF caused by the defective nozzle (e.g., blocked nozzle 99A). Additionally or alternatively, in step 218, processor 20 identifies one or more partially blocked nozzles 99 by associating each C2C positioning error detected in step 216 with a respective partially blocked nozzle 99 in system 10.

[0149] In some embodiments, processor 20 may apply the trained CNN only to regions 135 and 145 detected based on the MNF training of the CNN to detect C2C positioning errors (as described above in step 216) and to identify one or more partially clogged nozzles 99 (as described above in step 218).

[0150] It will therefore be understood that the foregoing embodiments are cited by way of example, and that the present invention is not limited to what has been particularly shown and described herein above. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described above, as well as variations and modifications thereof not disclosed in the prior art that would occur to one skilled in the art upon reading the foregoing description. Documents incorporated by reference in this patent application are to be considered an integral part of this application, and only the definitions therein shall be considered, except to the extent that any term is defined in these incorporated documents in a manner that is inconsistent with a definition expressly or impliedly made herein.

Claims

1. 1. A method for detecting defective parts (DP) in a digital printing system (DPS), comprising: receiving a first digital image (FDI) to be printed by said DPS; During the training phase: generating, for one or more first selected regions within the FDI, a first set of one or more composite images having defects caused by the DP within the one or more first selected regions; training a neural network (NN) using at least one of the synthetic images of the first set to detect the defect; In the detection phase that follows the training phase: applying the trained neural network to identify one or more second regions suspected of having the defects in a second digital image (SDI) obtained from a printed image produced by the DPS; generating, for each of the second regions, a second set of one or more composite images representing one or more defects contained in one or more DPs within the respective region; identifying at least the DP within each of the second regions by comparing between the SDI and the one or more composite images of the second set; A method comprising:

2. The method of claim 1 , comprising selecting the first selected region containing features for training the NN based on predefined selection criteria.

3. The method of claim 1 , wherein the NN comprises a convolutional NN (CNN).

4. The method of claim 3 , wherein the CNN has an Inception V3 architecture.

5. The method of any of claims 1 to 3, wherein at least one of the FDI and SDI includes a product image.

6. The method of any one of claims 1 to 3, wherein the DPS includes a nozzle for directing printing fluid onto a substrate, the DP includes a defective nozzle (DN) from among the nozzles, and the defect includes a missing nozzle failure (MNF) caused by a blocked orifice of the DN.

7. 4. The method of claim 1, wherein the DPS includes a nozzle for directing printing fluid onto a substrate, the DP includes a partially clogged nozzle from among the nozzles, and the defect includes a positioning error caused by the partially clogged nozzle, the partially clogged nozzle directing printing fluid jetted at a deflected angle to be deposited on the substrate a short distance away from an intended deposition location.

8. 1. A system for detecting defective parts (DP) in a digital printing system (DPS), comprising: an interface configured to receive (i) a first digital image (FDI) to be printed by the DPS, and (ii) a second digital image (SDI) obtained from the printed image produced by the DPS; 1. A processor, comprising: In a training phase, the processor is configured to: (i) generate, for one or more first selected regions in the FDI, a first set of one or more composite images having defects caused by the DP in the one or more first selected regions; and (ii) train a neural network (NN) to detect the defects using at least one of the composite images in the first set; In a detection phase following the training phase, the processor is configured to: (i) apply the trained NN to identify one or more second regions in the SDI that are suspected of having the defects; (ii) generate, for each of the second regions, a second set of one or more composite images that represent one or more defects contained in one or more DPs within the respective region; and (iii) identify, in each of the second regions, at least the DPs by comparing between the SDI and the one or more composite images of the second set. Processor and A system comprising:

9. The system of claim 8 , wherein the processor is configured to select the first selected region containing features for training the NN based on predefined selection criteria.

10. The system of claim 8 , wherein the NN comprises a convolutional NN (CNN).

11. The system of claim 10 , wherein the CNN has an Inception V3 architecture.

12. The system of any of claims 8 to 10, wherein at least one of the FDI and SDI includes a product image.

13. The system of any one of claims 8 to 10, wherein the DPS includes a nozzle for directing printing fluid onto a substrate, the DP includes a defective nozzle (DN) from among the nozzles, and the defects detected by the NN include a missing nozzle failure (MNF) caused by a blocked orifice of the DN.

14. 11. The system of claim 8, wherein the DPS includes a nozzle for directing printing fluid onto a substrate, the DP includes a partially clogged nozzle from among the nozzles, and the defect includes a positioning error caused by the partially clogged nozzle, the partially clogged nozzle directing printing fluid jetted at a deflected angle to be deposited on the substrate a short distance away from an intended deposition location.

Images