Image processing device, image processing method and program

Abstract

To perform positioning of an inspection target image and a reference image with high accuracy even when a predetermined image is overprinted on a preprint paper.SOLUTION: An image processing device comprises: image obtaining means (201) which reads a printed matter with a print image printed on a preprint paper and generates an inspection target image of the printed matter; first positioning means (204) which performs whole positioning of the inspection target image with a reference image indicating a correct image of the inspection target image by projective transformation and generates a positioning image; and second positioning means (204) which performs positioning by non-rigid positioning for each local area of the positioning image. The second positioning means (204) uses at least one area of a preprint image area surrounding a preprint image printed on the preprint paper before printing the print image and a print image area surrounding the print image as the local area within the positioning image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to the inspection of printed matter. [Background technology]

[0002] Printed materials are inspected to ensure the quality of printed materials output from printing devices. In recent years, a method has become known for automated inspection systems that compares an image of the printed material scanned with a reference image (standard image). When performing inspections by comparing images in this way, it is important to perform high-precision image alignment, as image alignment has a significant impact on inspection accuracy.

[0003] Non-rigid registration, such as free-form deformations (FFD), is known as a highly accurate registration technique. Non-rigid registration allows for registration that includes not only image misalignment and rotation, but also local scaling and positional deviation. This makes free-form registration possible with high accuracy.

[0004] In non-rigid registration, multiple control points that control the shape of the image are arranged in a grid on the image, and the image is deformed by moving each control point one by one. In non-rigid registration, in order to deform the image to align it with the reference image, the image error is calculated and the positions of the control points are successively updated in the direction that minimizes the error.

[0005] Furthermore, if an abnormality such as a stain of the same color as a certain pattern is found near the image being inspected, the defect is treated as part of the pattern, and the positions of the control points are updated to minimize the error. As a result, the control points near the defect may be shifted to unexpected positions, reducing alignment accuracy. Therefore, in Patent Document 1, approximate lines for the rows and columns of the control points are calculated, and the positions of the control points that have shifted to unexpected positions are corrected based on the approximate lines. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2023-33152 Summary of the Invention [Problem to be solved by the invention]

[0007] In a use case where a printed product obtained by overprinting a printed image onto preprinted paper is inspected, the position of the printed image on the printed product may be misaligned with the preprinted image. When this occurs, a process is performed to align the position of the printed image with the preprinted image by moving each control point using non-rigid registration. However, the updated positions of control points updated based on the preprinted image and control points updated based on the printed image may differ. In this case, the approximation line calculated from these control points does not match the updated positions of the control points in either the preprinted image or the printed image, making it impossible to correct the control points near the defect to their correct positions. Furthermore, even control points that do not actually need to be corrected may be corrected, resulting in reduced registration accuracy.

[0008] The present disclosure aims to align an inspection target image and a reference image with high precision even when a predetermined image is overprinted on preprinted paper. [Means for solving the problem]

[0009] An image processing device according to one aspect of the present disclosure comprises an image acquisition means for reading a printed matter on which a print image is printed on preprint paper and generating an image of the printed matter to be inspected, a first alignment means for aligning the entire image to be inspected with a reference image that indicates a correct image of the image to be inspected using projective transformation to generate an aligned image, and a second alignment means for performing alignment using non-rigid alignment for each local region of the aligned image, wherein the second alignment means uses at least one of a preprint image region surrounding the preprint image printed on the preprint paper before printing of the print image and a print image region surrounding the print image as the local region in the aligned image. [Effects of the Invention]

[0010] According to the present disclosure, even when a predetermined image is overprinted onto preprinted paper, the inspection target image and the reference image can be aligned with high precision. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a configuration diagram of an inspection system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a schematic configuration of a software module of the inspection device of FIG. [Figure 3] 2 is a flowchart illustrating an inspection process executed by the inspection device of FIG. 1. [Figure 4] FIG. 4 is a diagram illustrating a reference image generated in the process of S302 in FIG. 3. [Figure 5] 4 is a flowchart illustrating the process of S302 in FIG. 3. [Figure 6] 4 is a flowchart illustrating the process of S306 in FIG. 3. [Figure 7] FIG. 2 is a diagram showing an example of a result display screen displayed on the UI panel of FIG. 1. [Figure 8] 7 is a flowchart illustrating the process of S601 in FIG. 6. [Figure 9]FIG. 7 is a diagram showing an example of a filter used in the process of S603 in FIG. [Figure 10] 1 is a schematic diagram showing the direction of distortion occurring in an image to be inspected, indicated by arrows; [Figure 11] 9A to 9C are diagrams illustrating a specific example of the alignment process in FIG. 8. [Figure 12] FIG. 10 is a diagram for explaining correction of the positions at which control points are arranged. [Figure 13] 9 is a flowchart illustrating the control point position correction processing in steps S806 and S807 of FIG. 8. [Figure 14] FIG. 14 is a diagram for explaining calculation of an approximation line in S1301 of FIG. [Figure 15] FIG. 10 is a diagram showing an example of a printed matter in which a plurality of print image areas are distributed. [Figure 16] 10 is a flowchart illustrating a second embodiment of the area designation process in S303 of FIG. 3. [Figure 17] 10A and 10B are diagrams illustrating a situation in which it is necessary to consider the control point interval in the second embodiment. [Figure 18] FIG. 11 is a diagram showing an example of arrangement of control points in the third embodiment. [Figure 19] 10 is a flowchart illustrating a third embodiment of the control point position correction processing in steps S806 and S807 in FIG. 8; DETAILED DESCRIPTION OF THE INVENTION

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present disclosure, and the combinations of features described in the following embodiments are not necessarily essential to the solutions of the present disclosure. Note that the same reference numerals are used to designate the same components.

[0013] [overview] Printed materials often have defects such as stains and missing colors. Because these defects degrade the quality of printed materials, they are inspected for defects. For example, one method of inspecting printed materials involves comparing an image of the object to be inspected, captured by a scanner, with a reference image prepared in advance. In this inspection, the alignment between the object to be inspected and the reference image affects the accuracy of the inspection. Therefore, highly accurate alignment is important. Non-rigid registration is a well-known high-precision alignment technique. In non-rigid registration, the object to be inspected is set to a predetermined coordinate system. A control point group consisting of multiple control points is then placed within the predetermined coordinate system. Each of the multiple control points is arranged in a grid pattern. By controlling the control point group to match the position of the reference image, at least one of the multiple control points is moved, and the predetermined coordinate system and the object to be inspected image are modified in accordance with this movement. In other words, the image is deformed by placing multiple control points on the image to control the shape of the image, and updating some of the control points so that each control point in the object to be inspected is aligned with the reference image. However, printed materials may contain images that are different from the printed image, such as preprint images. Even in such cases, conventional methods have been to generate an image to be inspected by scanning both the preprint image and the printed image together, and then compare the image to be inspected with a reference image. However, because the preprint image and the printed image have different images, the tendency for updating the control point positions differs between the preprint image and the printed image. Therefore, even if the control point positions are updated based on row and column approximation lines derived from some of the control points, the approximation lines may not be appropriate. In this case, control points that do not need to be updated may also be updated, which may result in reduced alignment accuracy when aligning the image to be inspected with the reference image. Therefore, in the present disclosure, if control points are to be corrected, an operation is performed to align the preprint image and the printed image with the reference image. Specifically, the image to be inspected is aligned with the reference image using projective transformation to generate an aligned image. Next, alignment is performed using non-rigid alignment for each local region of the aligned image.The local region is at least one of a preprint image region surrounding a preprint image printed on a preprinted sheet before printing the print image and a print image region surrounding the print image. With this configuration, even if the preprint image and the print image have different designs, they are aligned separately, making it possible to prevent a decrease in alignment accuracy. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0014] First Embodiment [Overall configuration] Fig. 1 is a configuration diagram of an inspection system 100 according to the first embodiment. In Fig. 1, the inspection system 100 includes a server 101, a printing device 102, and an inspection device 105. In the inspection system 100, the printing device 102 outputs a printed material based on print job data generated by the server 101, and the inspection device 105 inspects the printed material for defects.

[0015] The server 101 generates print job data and transmits the generated print job data to the printing device 102. A plurality of external devices (not shown) are communicably connected to the server 101 via a network. The server 101 receives requests to generate print job data from these external devices.

[0016] The printing device 102 forms an image on a print medium, such as paper, based on print job data received from the server 101. The print medium may be long paper. While the present embodiment describes a configuration in which the printing device 102 uses an electrophotographic printing method, the present invention is not limited to this configuration, and the printing device 102 may be configured to use other printing methods, such as offset printing or inkjet printing. The printing device 102 includes a paper feed unit 103. Paper is preloaded in the paper feed unit 103 by a user. The paper loaded in the paper feed unit 103 is preprinted paper on which a preprint image has been printed. Based on the print job data received from the server 101, the printing device 102 transports the paper loaded in the paper feed unit 103 along a transport path 104, forms an image on one or both sides of the paper, and outputs the printed matter with the image to the inspection device 105. However, in step S501 of acquiring a preprint image in FIG. 5 (described later), the paper loaded in the paper feed unit 103 is transported without forming an image and output to the inspection device 105.

[0017] The inspection device 105 includes a CPU 106, a RAM 107, a ROM 108, a main memory unit 109, an image reading unit 110, a printing device I / F 111, a general-purpose I / F 112, and a UI panel 113. The CPU 106, the RAM 107, the ROM 108, the main memory unit 109, the image reading unit 110, the printing device I / F 111, the general-purpose I / F 112, and the UI panel 113 are connected to one another via a main bus 114. The inspection device 105 also includes a conveying path 115 connected to the conveying path 104 of the printing device 102, an output tray 116, and an output tray 117.

[0018] The CPU 106 is a processor that controls the entire inspection device 105. The RAM 107 functions as the main memory, work area, etc. of the CPU 106. The ROM 108 stores multiple programs executed by the CPU 106. The main memory unit 109 stores applications executed by the CPU 106, data used for image processing, etc. The image reading unit 110 reads one or both sides of a preprinted sheet output from the printing device 102 or a printed material to be inspected, and generates a scanned image of the printed material. Specifically, the image reading unit 110 reads one or both sides of the printed material being conveyed using one or more reading sensors (not shown) provided near the conveying path 115. The reading sensors may be provided on only one side, or may be provided on both the front and back sides of the printed material being conveyed to simultaneously read both sides. In a configuration in which the reading sensor is provided on only one side of the printed matter, the printed matter having one side read is transported to a double-sided transport path (not shown) in the transport path 115, the printed matter is turned over, and the reading sensor reads the other side.

[0019] The printing device I / F 111 is connected to the printing device 102, synchronizes the timing of printing process with the printing device 102, and notifies each other of their operating statuses. The general-purpose I / F 112 is a serial bus interface such as USB or IEEE1394. For example, by connecting a USB memory to the general-purpose I / F 112, data such as logs stored in the main memory unit 109 can be written to the USB memory and taken out, or data stored in the USB memory can be imported into the inspection device 105. The UI panel 113 is, for example, a liquid crystal display (display unit). The UI panel 113 functions as a user interface for the inspection device 105, displaying the current status and settings to inform the user. The UI panel 113 is also a touch-panel liquid crystal display, and can accept instructions from the user by operating displayed buttons.

[0020] In the inspection device 105, the image reading unit 110 reads the preprinted paper output from the printing device 102 and generates a scanned image of the paper (hereinafter referred to as the "preprinted image"). In the inspection device 105, an image acquisition module 201 (described later in FIG. 2) combines the preprinted image and the printed image to generate a reference image, which serves as the correct image. In the inspection device 105, the image reading unit 110 reads the printed matter to be inspected output from the printing device 102 and generates a scanned image of the printed matter (hereinafter referred to as the "image to be inspected"). In the inspection device 105, an image inspection module 206 (described later in FIG. 2) compares the image to be inspected with the reference image to inspect the printed matter for defects. Defects in printed matter are those that reduce the quality of printed matter, such as stains caused by ink, toner, or other coloring materials adhering to unintended locations, or color loss, which occurs when insufficient coloring materials are adhering to locations where an image should be formed, resulting in a lighter color than intended. The inspection device 105 outputs printed matter that passes the inspection to an output tray 116, and outputs printed matter that does not pass the inspection to an output tray 117. In this way, only printed matter that is guaranteed to be of a certain quality can be collected in the output tray 116 as deliverables for delivery.

[0021] [Software module configuration] FIG. 2 is a block diagram showing a schematic configuration of software modules of the inspection device 105 of FIG. 1. The inspection device 105 includes various modules shown in FIG. 2 as software modules. The various modules include, for example, an image acquisition module 201, an image area setting module 202, an inspection process selection module 203, and an alignment process module 204. The various modules also include, for example, a processing parameter setting module 205, an image inspection module 206, and an inspection result output module 207. The processing by these various modules is realized by the CPU 106 reading out programs stored in the ROM 108 into the RAM 107 and executing them. The various modules will be described below.

[0022] The image acquisition module 201 acquires a preprint image or an image to be inspected from the image reading unit 110. The image acquisition module 201 also acquires a pre-registered print image from the RAM 107 or the main storage unit 109. The image acquisition module 201 then combines the acquired preprint image and print image to generate a reference image that serves as a correct image. The reference image includes the preprint image and the print image. Therefore, by referencing the reference image, it is possible to specify a preprint area, which is the area where the preprint image is generated, and a print image area, which is the area where the print image is generated. The image area setting module 202 then specifies the preprint image area and the print image area by referencing the reference image. The inspection process selection module 203 selects a defect detection process based on information input by the user to a selection screen (not shown) displayed on the UI panel 113. For example, the type of defect is selected on this selection screen. The inspection process selection module 203 selects a defect detection process for detecting the selected type of defect from among multiple defect detection processes that the image inspection module 206 can execute. The defect types include, for example, point-shaped defects and line-shaped (streak) defects. Note that the defect types are not limited to these and may include any type of defect, such as image unevenness or surface shape defects. If the user does not select a defect type, the inspection process selection module 203 selects a defect detection process set by default.

[0023] The alignment processing module 204 executes alignment processing to align the inspection target image with the reference image. This alignment processing will be described later with reference to FIG. 8. The processing parameter setting module 205 sets parameters to be used in the defect detection processing selected by the inspection processing selection module 203. The parameters include a filter for emphasizing the type of defect selected by the user and a defect discrimination threshold for discriminating between defects. The image inspection module 206 executes the defect detection processing selected by the inspection processing selection module 203. The inspection result output module 207 displays the inspection results on the UI panel 113.

[0024] [Overall Flowchart] 3 is a flowchart illustrating the inspection process executed by the inspection device 105 of FIG. 1. The inspection process of FIG. 3 is realized by the CPU 106 reading a program stored in the ROM 108 into the RAM 107 and executing it. The inspection process of FIG. 3 is executed when the user performs an operation to start the inspection process via the UI panel 113. Note that some or all of the functions of the steps in FIG. 3 may be realized by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of each process indicates that it is a step in the flowchart.

[0025] In S301, the CPU 106 performs inspection settings required for inspecting the image to be inspected, based on information input by the user on the selection screen displayed on the UI panel 113. For example, in S301, the inspection process selection module 203 selects one or more defect detection processes based on one or more types of defects selected by the user. In addition, the process parameter setting module 205 sets parameters to be used in each defect detection process selected by the inspection process selection module 203.

[0026] In S302, the CPU 106 executes reference image generation to generate a reference image that will be the correct image. The reference image will be described with reference to FIG. 4, and the generation of the reference image will be described with reference to FIG.

[0027] Fig. 4 is a diagram illustrating a reference image 430 generated in the processing of S302 in Fig. 3. The reference image 430 includes a plurality of pixels that make up print image information 401 and a plurality of pixels that make up a picture 402 that has been preprinted on preprinted paper. A print image area 403 is set so as to surround only the print image information 401. Furthermore, a preprint image area 404 is set so as to surround the picture 402.

[0028] [Details of S302 in Figure 3] FIG. 5 is a flowchart illustrating the processing of S302 in FIG. 3. The reference image generation processing of FIG. 5 is realized by the CPU 106 reading a program stored in ROM 108 into RAM 107 and executing it. The reference image generation processing of FIG. 5 is executed when the reference image generation processing of S302 in FIG. 3 is started. Note that some or all of the functions of the steps in FIG. 5 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates the step in the flowchart. In S501, the CPU 106 causes the image acquisition module 201 to read the preprint paper output from the printing device 102 with the image reading unit 110 and acquire a preprint image 410 corresponding to the preprint image. For example, FIG. 4B shows a preprint image 410 corresponding to the preprint image immediately after it is acquired from the image reading unit 110 by the CPU 106. Instead of acquiring the preprint image 410 corresponding to the preprint image from the image reading unit 110, the CPU 106 may acquire the preprint image 410 corresponding to the preprint image that has been stored in advance as data in the RAM 107 or the main memory unit 109.

[0029] In S502, the CPU 106 causes the image acquisition module 201 to acquire pre-registered print image information 401 from the RAM 107 or the main storage unit 109. An example of the print image information 401 is shown in a print image 400 corresponding to the print image in FIG. 4(a). The print image information 401 is information that the CPU 106 can acquire from the RAM 107 or the main storage unit 109. As shown in FIG. 4(a), it is assumed that an example of the print image information 401 is depicted in the print image 400 corresponding to the print image. In FIG. 4(a), a postal code, an address, and a name are depicted as examples of the print image information 401.

[0030] In S503, the CPU 106 generates a reference image 430, which is a correct image, by combining a preprint image 410 corresponding to the preprint image acquired by the image acquisition module 201 with a print image 400 corresponding to the print image. An example of the combining method will be described. The preprint image 410 corresponding to the preprint image in FIG. 4(b) is an image immediately after being scanned by the image reading unit 110. Therefore, the preprint image 410 may be rotated due to skew during paper transport, and may have a different resolution than the print image 400 corresponding to the print image. Therefore, as shown in FIG. 4(c), a converted image 420 is generated by performing projective transformation so that the four vertices of the preprint image 410 corresponding to the preprint image coincide with the four vertices of the print image 400 corresponding to the print image. Next, as shown in FIG. 4(d), multiple pixels constituting the print image information 401 of the print image 400 corresponding to the print image are overwritten on the converted image 420. In this manner, the reference image 430 in FIG. 4(d) is generated.

[0031] Returning to FIG. 3, in S303 of FIG. 3, the CPU 106 uses the image area setting module 202 to specify a preprint image area 404 and a print image area 403 in the reference image 430. FIG. 4D shows an example of the reference image 430 generated in S302. The reference image 430 includes a plurality of pixels constituting the print image information 401 of the print image 400 corresponding to the print image, and a plurality of pixels constituting a picture 402 preprinted on preprinted paper. A print image area 403 is set in an area including a plurality of pixels constituting the print image information 401. A preprint image area 404 is set in an area including a plurality of pixels constituting the picture 402. In S303, all of the plurality of pixels constituting the print image information 401 in the reference image 430 are assigned to the print image area 403. Also in S303, an area other than the print image area 403 is assigned to the preprint image area 404. The print image area 403 and the preprint image area 404 are each used in the control point position correction process described later with reference to Fig. 8. For example, the user may perform an operation to designate a specific location on the reference image 430 displayed on the UI panel 113 as the print image area 403. The CPU 106 can acquire location information that identifies the print image area 403 based on the user's operation to designate the print image area 403.

[0032] In S304, the CPU 106 acquires the inspection object image from the image reading unit 110 using the image acquisition module 201. Note that in S304, the image reading unit 110 may be configured to acquire the inspection object image that has been generated in advance and stored in the main storage unit 109.

[0033] In S305, the CPU 106 sets one defect detection process to be executed from among one or more defect detection processes selected by the inspection process selection module 203. Through the process of S305, for example, a defect detection process that is registered in advance to be executed with priority or a defect detection process corresponding to the type of defect initially selected by the user is set.

[0034] [Details of S306 in Figure 3] In S306, the CPU 106 executes defect detection processing. The defect detection processing will be described with reference to FIG. 6. FIG. 6 is a flowchart illustrating the processing of S306 in FIG. 3. The defect detection processing of FIG. 6 is implemented by the CPU 106 reading a program stored in the ROM 108 into the RAM 107 and executing it. The defect detection processing of FIG. 6 is executed when the processing of S306 in FIG. 3 is started. In other words, the defect detection processing of FIG. 6 is a subroutine of S306 and represents the flow of one defect detection processing. Therefore, each time the subroutine of S306 is called, the type of defect detection processing set in the processing of S305 is executed. Note that some or all of the functions of the steps in FIG. 6 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates the step in the flowchart. In S601, the CPU 106 executes alignment processing using the alignment processing module 204. The alignment processing is processing for aligning the inspection target image with the reference image. Details of the alignment process will be described later with reference to Fig. 8. In S602, CPU 106 compares the aligned inspection target image with the reference image using image inspection module 206 to generate a difference image. The difference image is an image generated by, for example, comparing the reference image and inspection target image pixel by pixel and acquiring pixel values, for example, difference values ​​between density values ​​for each RGB, for each pixel.

[0035] In S603, the CPU 106 causes the image inspection module 206 to perform filtering on the difference image generated in S602 to emphasize a specific shape. FIG. 9 shows examples of filters used in the processing of S603 in FIG. 6. For example, FIG. 9(a) shows a filter for emphasizing point defects. FIG. 9(b) shows a filter for emphasizing line defects. These filters are changed depending on the type of defect detection processing set in S305 or S308. For example, if the defect detection processing set in S305 or S308 is for detecting point defects, the filtering processing of S603 is performed using the filter of FIG. 9(a). On the other hand, if the defect detection processing set in S305 or S308 is for detecting line defects, the filtering processing of S603 is performed using the filter of FIG. 9(b). A filtered difference image is generated by the processing of S603. In S603, the filtering processing is performed using a filter corresponding to the type of defect detection processing set in the processing of S305.

[0036] In S604, CPU 106 causes image inspection module 206 to perform binarization processing on the filtered difference image. This generates an image (hereinafter referred to as a "difference binarized image") in which the pixel values ​​of pixels whose difference values ​​exceed the defect discrimination threshold are set to "1" and the pixel values ​​of pixels equal to or less than the defect discrimination threshold are set to "0." In S605, CPU 106 causes image inspection module 206 to use the difference binarized image to determine whether or not there are any pixels that exceed the defect discrimination threshold.

[0037] If it is determined in S605 that no pixels exceeding the defect discrimination threshold exist, it is determined that no defective portion exists, and the defect detection process ends. If it is determined in S605 that pixels exceeding the defect discrimination threshold exist, in S606, the CPU 106 causes the image inspection module 206 to store information about the detected defect in the RAM 107 or the main storage unit 109. Specifically, the CPU 106 causes the image inspection module 206 to store in the RAM 107 or the main storage unit 109 the type of defect detection process that detected the defective portion and the coordinates of the defective portion in association with each other. Thereafter, the defect detection process ends.

[0038] Returning to FIG. 3 , in S307, the CPU 106 determines whether or not the execution of all the set defect detection processes has been completed. If it is determined in S307 that the execution of any of the defect detection processes set in the process of S305 has not been completed, in S308 the CPU 106 sets one defect detection process to be executed from among the unexecuted defect detection processes, and the inspection process returns to S306. On the other hand, if it is determined in S307 that the execution of all the set defect detection processes has been completed, in S309 the CPU 106 causes the inspection result output module 207 to display a result display screen 701 of FIG. 7 , which shows the inspection results, on the UI panel 113. FIG. 7 is a diagram showing an example of the result display screen 701 displayed on the UI panel 113 of FIG. 1 . The result display screen 701 of FIG. 7 displays an inspection target image 702. For example, the words "point defect" are displayed near a defect 703 determined to be a point defect. Furthermore, the words "line defect" are displayed near a defect 704 determined to be a line defect. Coordinate information 705, 706 of each defect in the inspection target image 602 is also displayed. Note that the method of displaying the inspection results is not limited to the above-described method, and any display method that allows the user to recognize which of multiple defect detection processes a detected defect was detected by, such as displaying each defect type in a different color, may be used. When the processing of S309 is completed, the inspection processing ends. Note that in this embodiment, the defect detection processing has been described using examples of defect detection processing that detects point defects and defect detection processing that detects linear defects, but the types of defect detection processing are not limited to these. In other words, the present disclosure is applicable to any defect detection processing that can detect a defect desired by the user, and does not limit the type of defect detection processing.

[0039] [Details of S601 in Figure 6] Next, the details of the alignment process in S601 in FIG. 6 will be described with reference to FIGS. 8 and 11. FIG. 8 is a flowchart illustrating the process of S601 in FIG. 6. FIG. 11 is a diagram illustrating a specific example of the alignment process in FIG. 8. The alignment process in FIG. 8 is implemented by the CPU 106 reading a program stored in the ROM 108 into the RAM 107 and executing it. The alignment process in FIG. 8 is executed when the process of S601 in FIG. 6 is started. Note that some or all of the functions of the steps in FIG. 8 may be implemented by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of each process indicates that it is a step in the flowchart.

[0040] In this embodiment, an example will be described in which an aligned image to be inspected (hereinafter referred to as an "aligned image") I' is generated by aligning an image to be inspected I with a reference image T shown in Fig. 11(a). Furthermore, I(x, y), T(x, y), and I'(x, y) each represent a pixel value at coordinates (x, y).

[0041] In S801 of FIG. 8, the alignment processing module 204 performs initial alignment. In S801, the CPU 106 performs initial alignment by, for example, extracting feature points from the inspection target image I and the reference image T, and performing projective transformation so as to minimize the sum of the Euclidean distances between the feature points of the inspection target image I and the feature points of the reference image T. Any algorithm may be used to extract the feature points. For example, a common algorithm such as the Harris corner detection algorithm, template matching, or SIFT (Scale-Invariant Feature Transform) may be used. Furthermore, Mahalanobis distance may be used instead of Euclidean distance. Through the processing of S801, the inspection target image I is projectively transformed into the aligned image I'. Next, in S802, the alignment processing module 204 arranges control points (control point control means). Specifically, in S802, the alignment processing module 204 arranges L × M control points in a grid pattern on the inspection target image I (a scanned image of a printed material). Since L × M control points are arranged in a grid, the distance δ between the control points is calculated from L, M and the image size, as shown in Fig. 11(b). Also, as shown in Fig. 11(b), the coordinates of the control point in the lth row and mth column are calculated as p l,m(l=1, .., L, m=1, .., M). In S802, the registration processing module 204 specifies a preprint image corresponding region in the image to be inspected I as a region corresponding to the preprint image region 404 in the reference image T. In S802, the registration processing module 204 specifies a print image corresponding region in the image to be inspected I as a region corresponding to the print image region 403 in the reference image T. Note that it is assumed that a control point group consisting of multiple control points is arranged in a predetermined coordinate system. It is also assumed that the image to be inspected I, the reference image T, and the registered image I' are each set in the same predetermined coordinate system. Therefore, the local regions specified by the preprint image region 404, the preprint image corresponding region, the print image region 403, and the print image corresponding region are set in the same predetermined coordinate system. Therefore, the image to be inspected I is deformed in accordance with the movement of one control point in the control point group, and the local regions are also deformed in accordance with this deformation.

[0042] Next, in S803, the registration processing module 204 updates the positions where the control points are arranged. The update formula is shown in the following formula (1). μ represents a weighting coefficient, which may be a value such as 0.1, and may be changed according to the update speed of the control points. ∇ c is expressed by the following formula (2): ∇ c is the control point p in Figure 11(b). l,m A set of pixel locations in the neighborhood of l,m The differential value of the sum of squares of the difference between the pixel values ​​of the aligned image I' and the pixel values ​​of the reference image T is given by the first term of Equation (1). l,m are arranged in a grid pattern. Furthermore, the second term of Eq. (2) determines the control point p l,m The control points p l,m The arrangement of the control points is updated. Then, each time the row and column of the control points are changed, the process of updating the control points according to equation (1) is executed. Therefore, the control point p l,mIn a use case where a pixel with a print defect due to dirt or color loss is included among the multiple pixels in the vicinity of , the aligned image I' also contains pixels with such a print defect. Therefore, in this use case, the control point p l,m will cause an unexpected position shift. For example, the control point p l,m If there are pixels with print defects due to stains or color loss among the pixels in the vicinity of the control point p l,m are updated to be arranged along the pixel of the print defect. The locations where such pixels appear in the print result tend to differ between the print image corresponding area and the print image corresponding area. For this reason, in this embodiment, processing is performed to update the control points separately for the print image corresponding area and the print image corresponding area. Note that the control point p l,m A set of pixel locations in the neighborhood of l,m For example, the control point p l,m A range including eight control points around is specified.

[0043]

number

[0044]

number

[0045] In S804, the registration processing module 204 updates the pixels. The update formula is shown in formula (3) below. w(x,y) is expressed by formula (4) below. w(x,y) is a formula for calculating the coordinates in the registered image I' after the registration process of the coordinates (x,y) in the inspection target image I. The bases B0(t), B1(t), B2(t), and B3(t) in formula (4) below are expressed by formulas (5) to (8) below, respectively. Each of the bases B0(t), B1(t), B2(t), and B3(t) represents a B-spline basis function. Since B-spline basis functions have locality, when one control point among multiple control points is moved, the movement only affects neighboring control points. Furthermore, u, v, u', and v' shown in FIG. 11(c) are expressed by formulas (9) to (12) below, respectively.

[0046]

number

[0047]

number

[0048]

number

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053]

number

[0054]

number

[0055]

number

[0056] In this embodiment, the 16 grid points p(u, v), p(u+1, v), ..., p(u+3, v+3) are used to calculate the pixels in the aligned image I', but this is not limiting. For example, four grid points with close Euclidean distances (x, y) may be used.

[0057] Next, in S805, the registration processing module 204 determines whether pixel updating is complete. In S805, for example, the registration processing module 204 calculates the distance d between a pixel of the registered image I' and a pixel of the reference image T, and determines whether pixel updating is complete based on the distance d. The distance d is expressed by the following equation (13).

[0058]

number

[0059] In S805, if the distance d is equal to or less than a preset threshold, the registration processing module 204 determines that the pixel update is complete. On the other hand, if the distance d is not equal to or less than the preset threshold, the registration processing module 204 determines that the pixel update is not complete. Note that the distance d is the distance between a pixel of the registered image I' and a pixel of the reference image T, and therefore ideally should be zero, but in reality it is not zero. Therefore, a threshold is set, and if the distance d is equal to or less than the threshold, the registration processing module 204 completes the pixel update. The threshold is set, for example, to a distance that does not interfere with subsequent inspection processing.

[0060] If it is determined in S805 that the pixel updating is not complete, the alignment processing returns to S803. If it is determined in S805 that the pixel updating is complete, the alignment processing module 204 performs control point position correction processing in FIG. 13 (described later) to correct the positions of the control points in S806 and S807. Thereafter, in S808, the alignment processing module 204 generates an aligned image I' based on the corrected control points, and the alignment processing ends.

[0061] Next, the control point position correction processing executed by the registration processing module 204 in S806 and S807 will be described with reference to FIGS. 10, 12, 13, and 14. FIG. 10 is a schematic diagram showing the direction of distortion occurring in the inspection target image I with arrows. In FIG. 10, the direction along the short side of the paper is the main scanning direction. Also, in FIG. 10, the direction perpendicular to the main scanning direction and along the long side of the paper is the sub-scanning direction. Note that in the subsequent figures, although not shown, the main scanning direction and sub-scanning direction relative to the paper are the same as those in FIG. 10. Also, the direction of distortion occurring in the inspection target image I here represents the direction and magnitude of the positional deviation of the inspection target image I with respect to the reference image T. In other words, the direction of each arrow in FIG. 10 represents the direction of the positional deviation of the inspection target image I with respect to the reference image T. Also, the size of each arrow in FIG. 10 represents the magnitude of the positional deviation of the inspection target image I with respect to the reference image T. In FIG. 10, image 1000 is image I to be inspected, with a printed image printed in printed image corresponding region 1001 and only a preprinted image printed in preprinted image corresponding region 1002. FIG. 12 is a diagram for explaining the correction of the positions of control points. FIG. 12(a) is a schematic diagram showing an example of the positions of control points immediately after the control point update in S803 is completed. FIG. 12(b) is a schematic diagram showing the positions of control points after the control point position correction process in S806 and S807 is executed. As a result of the control point position correction process, only control point 1203 of the multiple control points is corrected to the position of control point 1213. In addition, a printed image corresponding region 1201 and a preprinted image corresponding region 1202 are specified for image 1200 in FIG. 12.

[0062] In the inspection device 105, when the image reading unit 110 reads the printout to be inspected output from the printing device 102 and generates the inspection target image I, the inspection target image I tends to have distortion. Specifically, as shown in FIG. 10 , the inspection target image I has different distortion tendencies in the printed image corresponding area 1001 and the preprint image corresponding area 1002. One cause of this is misalignment of the position at which the printing device 102 forms the printed image on the preprinted paper. Furthermore, focusing on the distortion in the preprint image corresponding area 1002, out of the printed image corresponding area 1001 and the preprint image corresponding area 1002, the preprint image in the preprint image corresponding area 1002 is uniformly distorted in the sub-scanning direction. This occurs because the paper transport speed during printing or scanning is not uniform. Furthermore, if the paper is transported at an angle, the inspection target image I in the printed image corresponding area 1001 may be distorted diagonally, but this does not change the tendency of the paper to be distorted in the sub-scanning direction. For the reasons described above, the direction of distortion changes linearly within a region. Therefore, when the control points are updated in S803, the control points within a region are aligned on a straight line. Therefore, locally shifted control points, such as control point 1203 in FIG. 12(a), should not occur. However, if a defect of the same color as the pattern exists near the pattern in the inspection image I, the defect is treated as part of the pattern when the control points are updated in S903, and the control points' positions are updated accordingly. As a result, the control points near the defect may be shifted to unexpected positions, potentially resulting in the defect being embedded in the pattern in the aligned image I'. Therefore, when aligning the position of the printed image with the position of the preprinted image, the alignment accuracy may be reduced.

[0063] In contrast, in this embodiment, a column approximation line is calculated based on multiple control points included in the preprint image corresponding region 1202 in Fig. 12 and arranged in the same column as one control point included in the preprint image corresponding region 1202. Furthermore, a row approximation line that intersects with the column approximation line is calculated based on multiple control points included in the preprint image corresponding region 1202 and arranged in the same row as the one control point. The position of the one control point is corrected based on the column approximation line and the row approximation line. Furthermore, a column approximation line is calculated based on multiple control points included in the print image corresponding region 1201 in Fig. 12 and arranged in the same column as one control point included in the print image corresponding region 1201. Furthermore, a row approximation line that intersects with the column approximation line is calculated based on multiple control points included in the print image corresponding region 1201 and arranged in the same row as the one control point. The position of the one control point is corrected based on the column approximation line and the row approximation line.

[0064] Next, the control point position correction process will be described with reference to FIGS. 13 and 14. FIG. 13 is a flowchart illustrating the control point position correction processes of S806 and S807 in FIG. 8. FIG. 13(a) is a flowchart illustrating the control point position correction process of S806 in FIG. 8, which targets the control points of the preprint image corresponding area 1202. The control point position correction process of FIG. 13(a) is implemented by the CPU 106 reading a program stored in the ROM 108 into the RAM 107 and executing it. The control point position correction process of FIG. 13(a) is executed when the process of S806 in FIG. 8 is started. Note that some or all of the functions of the steps in FIG. 13(a) may be implemented by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of each process indicates that it is a step in the flowchart.

[0065] [Details of S303 in Figure 3] The control point position correction process in Fig. 13(a) is executed for all control points arranged in the preprint image corresponding area corresponding to the area designated as the preprint image area in the process of designating the image area in S303 in Fig. 3. Below, as an example, a case will be described in which the process is executed for control point 1203 in Fig. 14 out of the L × M arranged control points. Note that control point 1203 is the control point in the lth row and mth column.

[0066] In S1301, the alignment processing module 204 calculates an approximation line using the control point 1203 in FIG. 14 (approximation line calculation means). Specifically, the alignment processing module 204 calculates an approximation line 1401 (column approximation line) in FIG. 14 based on the control point arranged in the same column as the control point 1203, i.e., among the multiple control points arranged in column m, that is, that is, among the multiple control points arranged in the preprint image corresponding area 1202. The alignment processing module 204 also calculates an approximation line 1402 (row approximation line) in FIG. 14 based on the control point arranged in the same row as the control point 1203, i.e., among the multiple control points arranged in row l, that is, that is, among the multiple control points arranged in the area specified as the preprint image corresponding area. In S1301, a regression line of the positions of multiple control points is used to calculate the approximation line. For example, in the approximation line 1401, the regression line that predicts the y coordinate from the x coordinate is calculated using the coordinates of all control points included in column m. However, because the slope of the regression line of the control points included in a column may diverge, a regression line that predicts the x coordinate from the y coordinate may also be calculated.

[0067] In S1302, the alignment processing module 204 calculates the intersection 1403 between the approximated lines 1401 and 1402. In S1303, the alignment processing module 204 calculates the distance from the control point 1203 to the intersection 1403. In S1303, the alignment processing module 204 determines whether the control point 1203 is a correction target based on the calculated distance. In S1303, if the calculated distance exceeds a predetermined value, the alignment processing module 204 determines that the control point 1203 is a correction target. On the other hand, if the calculated distance is equal to or less than the predetermined value, the alignment processing module 204 determines that the control point 1203 is not a correction target. If it is determined in S1303 that the control point 1203 is not a correction target, the alignment processing module 204 ends the correction processing of the control point 1203. If it is determined in S1303 that the control point 1203 is to be corrected, in S1304 the alignment processing module 204 corrects the position of the control point 1203 to the position of the intersection point 1403 calculated in S1302, and the correction processing for the control point 1203 ends.

[0068] [Details of S807 in Figure 8] FIG. 13(b) is a flowchart illustrating the control point position correction process for the control points of the print image corresponding area 1201 in S807 in FIG. 8. The control point position correction process in FIG. 13(b) is implemented by the CPU 106 reading a program stored in the ROM 108 into the RAM 107 and executing it. The control point position correction process in FIG. 13(b) is executed when the process of S807 in FIG. 8 is started. Note that some or all of the functions of the steps in FIG. 13(b) may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates a step in the flowchart.

[0069] The control point position correction process in Fig. 13(b) is executed for all control points arranged in the area specified as the print image area in the image area specification process in S303 in Fig. 3. The specific processing flow is the same as the control point position correction process targeted at the control points in the preprint image corresponding area, and therefore a detailed description thereof will be omitted.

[0070] According to the above-described embodiment, an approximated line 1401 is calculated based on the control point that is located in the area to which the control point 1203 is assigned, out of the multiple control points that are located in the same column as the control point 1203. Furthermore, an approximated line 1402 that intersects with the approximated line 1401 is calculated based on the control point that is located in the area to which the control point 1203 is assigned, out of the multiple control points that are located in the same row as the control point 1203. The position of the control point 1203 is then corrected based on the approximated lines 1401 and 1402. As a result, even if the position of the control point 1203 near the defect is updated to an unexpected position by the processing in S803 due to the presence of a defect of a color equivalent to that of a pattern near the pattern in the image to be inspected, the following processing is executed. That is, by the processing in S806 and S807, the position where the control point 1203 was located can be corrected to an appropriate position using the approximated lines 1401 and 1402. As a result, a decrease in alignment accuracy can be prevented. Specifically, even when a predetermined image is overprinted onto a preprinted sheet, it is possible to align the inspection target image and the reference image with high precision.

[0071] Furthermore, in the above-described embodiment, if the distance from the intersection 1403 of the approximated lines 1401 and 1402 to the control point 1203 exceeds a predetermined value, the position of the control point 1203 is corrected to the position of the intersection 1403. This makes it possible to correct the placement position of the control point 1203 so that it follows the arrangement of the control points that are placed in the area to which the control point 1203 is assigned, among the other control points that are placed in the same row and column as the control point 1203.

[0072] Although the present disclosure has been described using the above-described embodiment, the present invention is not limited to the above-described embodiment. For example, in the control point position correction process, an approximate curve may be used instead of an approximate straight line.

[0073] <Second embodiment> In the first embodiment, a printed matter in which the print image area is printed at the top of the paper and the preprint image area is printed at the bottom of the paper is used as an example, as shown in Fig. 4. In the second embodiment, the alignment process will be described using a printed matter in which the print image area is distributed at multiple positions on the paper as an example.

[0074] FIG. 15(a) shows an example of a printed matter in which preprint image areas 1501 and print image areas 1502 are alternately arranged. In the example of FIG. 15(a), a reference image 1500 includes a preprint image formed in the preprint image area 1501 and a print image formed in the print image area 1502. That is, the example shows a reference image 1500 of a printed matter in which two areas are alternately distributed from the top to the bottom of the paper. For example, in a printed matter such as an invoice, table lines, which are preprint images, and characters, which are print images, are printed alternately. On the other hand, FIG. 15(b) shows an example of a printed matter in which preprint image corresponding areas 1511 and print image corresponding areas 1512 are alternately arranged. In the example of FIG. 15(b), an inspection target image 1510 includes a preprint image formed in the preprint image corresponding area 1511 and a print image formed in the print image corresponding area 1512. 15(b), the print image corresponding area 1512 is misaligned in the main scanning direction with respect to the preprint image corresponding area 1511. Therefore, the print image formed in the print image corresponding area 1512 is misaligned in the main scanning direction with respect to the preprint image formed in the preprint image corresponding area 1511. In the second embodiment, a process for suitably aligning the inspection target image 1510 with the reference image 1500 will be described, even in such a case.

[0075] One method for specifying all print image areas in the reference image 1500 shown in FIG. 15(a) is to acquire print image areas on the reference image displayed on the UI panel 113 by user operation. However, when there are multiple locations where the user must specify areas, as in the case of reference image 1500, not only does it take time for the user to specify the areas by user operation, but there is also the possibility that the control points will not be assigned to the correct areas if the user forgets to specify them. As a result, the control point position correction process may not be performed correctly, and alignment accuracy may decrease. A feature of this embodiment is that the print image areas are specified based on the print image. Below, only the differences from the processing of the inspection system in the first embodiment will be described.

[0076] [Details of S303 in Figure 3] FIG. 16 is a flowchart illustrating a second embodiment of the area designation process in S303 of FIG. 3. The area designation process includes a process for designating a print image area and a process for designating a preprint image area. The area designation process in FIG. 16 is implemented by the CPU 106 reading a program stored in the ROM 108 into the RAM 107 and executing it. The area designation process in FIG. 16 is executed when the reference image generation process is completed. Note that some or all of the functions of the steps in FIG. 16 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates the step in the flowchart. In S1601, the CPU 106 acquires pixel positions that are paper white from the print image using the image area setting module 202. In S1602, the CPU 106 designates pixel positions that are not paper white as the print image corresponding area. In S1603, the CPU 106 designates pixel positions that are paper white as the preprint image corresponding area. By performing the above-described processing in steps S1601 to S1603, it is possible to specify only the area where the print image exists as the print image corresponding area.

[0077] Next, a method for arranging control points in a printed material in which print image corresponding areas are distributed at multiple positions on the paper will be described. To use control points to align the position of the image to be inspected 1510 in FIG. 15(b), at least one control point must be arranged in each of the print image corresponding area 1512 and the preprint image corresponding area 1511. For example, in FIG. 17, the print image corresponding area 1512 and the preprint image corresponding area 1511 each contain at least one control point 1701. A feature of this embodiment is that the control point interval 1702 is determined so that at least one control point is arranged in each area. For example, the control point interval 1702 in the sub-scanning direction is set to be smaller than the length of the smallest print image corresponding area in the sub-scanning direction.

[0078] In this embodiment, in step S802, where the control points are placed, the number of control points L is adjusted so that the control point spacing calculated from the number of control points in one row L and the image size is smaller than the length in the sub-scanning direction of the print image corresponding area with the smallest control point spacing.

[0079] In the above-described embodiment, the print image corresponding area is designated based on the print image, so that all areas in which print image information exists can be designated as print image corresponding areas.

[0080] In the above-described embodiment, the number L of control points is adjusted so that it is smaller than the length in the sub-scanning direction of the print image corresponding area with the smallest control point spacing calculated from the number L of control points in one column and the image size. The number M of control points is also adjusted so that it is smaller than the length in the main scanning direction of the print image corresponding area with the smallest control point spacing calculated from the number M of control points in one row and the image size. This allows at least one control point to be placed in each area.

[0081] <Third embodiment> In the third embodiment, a control point position correction process will be described when a control point is located near the boundary between the preprint image corresponding area and the print image corresponding area. This describes a case where misalignment occurs between the preprint image and the print image due to a misalignment in the position where the printing device 102 forms the print image on the preprint paper. FIG. 18 shows an example in which a control point, such as a control point 1801, is located near the boundary between the preprint image corresponding area 1511 and the print image corresponding area 1512 in the inspection target image 1510 in FIG. 17. The control point 1801 near the boundary between the preprint image corresponding area 1511 and the print image corresponding area 1512 is updated in the control point update process to control the misalignment between both the preprint image and the print image. Therefore, the position of the control point 1801 is not updated to match the misalignment between either the control point 1802 in the preprint image corresponding area or the control point 1803 in the print image corresponding area. Instead, the position of the control point 1801 is updated to approximately the middle of the misalignment direction between the two images. Therefore, in the control point position correction process, a more ideal approximate line can be calculated by not using control points 1801 near the area boundary in the calculation of the approximate line. Also, since the control points 1801 have been updated to optimal positions, not correcting the positions of the approximate line can improve the accuracy of the alignment correction. This embodiment is characterized by excluding control points near the area boundary from the calculation of the approximate line and not correcting the positions of control points near the area boundary.

[0082] [Details of S806 in Figure 8] FIG. 19(a) is a flowchart illustrating a third embodiment of the control point position correction process of S806 in FIG. 8. FIG. 19(a) is a variation of the control point position correction process shown in FIG. 13(a). The control point position correction process of FIG. 19(a) is implemented by the CPU 106 reading a program stored in the ROM 108 into the RAM 107 and executing it. The control point position correction process of FIG. 19(a) is executed when the process of S806 in FIG. 8 is started. Note that some or all of the functions of the steps in FIG. 19(a) may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates the step in the flowchart. Differences from the first embodiment will be described below.

[0083] In S1901, the CPU 106, using the registration processing module 204, calculates an approximation line for the control points in the preprint image corresponding area, excluding areas near the boundary of the preprint image corresponding area. The position of the control point is updated based on the differential value of the squared sum of the difference between the pixel values ​​of the registered image and the pixel values ​​of the reference image in the set of pixel positions near the control point, as shown in Equation (2). That is, if the distance between the boundary position of the preprint image corresponding area and a control point is closer than a predetermined distance, the control point is determined to be a control point near the boundary, and the control point is excluded from the calculation of the approximation line. In S1903, the CPU 106, using the registration processing module 204, determines whether the control point in the preprint image corresponding area is to be corrected. If the distance between the intersection point determined in S1902 and the control point used to calculate the approximation line in S1901 exceeds a predetermined value, the CPU 106 determines that the control point is to be corrected. If the distance between the intersection point obtained in S1902 and the control point is equal to or less than a predetermined value, the control point is determined not to be corrected.

[0084] [Details of S807 in Figure 8] Fig. 19(b) is a flowchart for explaining a third embodiment of the control point position correction process of S807 in Fig. 8. The specific processing flow is similar to the control point position correction process for the control points of the preprint image corresponding area, and therefore a description thereof will be omitted.

[0085] In the above-described embodiment, the CPU 106 calculates an approximation line of the control points in the preprint image corresponding area, excluding the area near the boundary of the preprint image corresponding area. The CPU 106 also calculates an approximation line of the control points in the print image corresponding area, excluding the area near the boundary of the print image corresponding area. This allows for calculation of a more ideal approximation line in each area. Furthermore, the CPU 106 determines that the control points used in calculating the approximation line are to be corrected, and determines that the control points not used in calculating the approximation line are not to be corrected. This prevents further correction of control points that have already been updated to optimal positions. As a result, a decrease in alignment accuracy can be prevented even when control points are located near the boundary between the preprint image corresponding area and the print image corresponding area.

[0086] [Another way to achieve this] The present disclosure may be applied to a system consisting of multiple devices, such as a host computer, an interface device, a reader, and a printer, or may be applied to an apparatus consisting of a single device, such as a copier or a facsimile machine.

[0087] (Other embodiments) Although various examples and embodiments of the present disclosure have been shown and described above, the spirit and scope of the present disclosure are not limited to the specific descriptions in this specification. The present disclosure is not limited to the above-described embodiments, and various modifications may be made. In addition, the present disclosure may be realized by appropriately combining parts of the above-described embodiments.

[0088] (Variation 1) For example, while an example has been described in which the configuration of each software module of the inspection apparatus 105 in FIG. 1 is realized by the CPU 106, the present invention is not limited to this. For example, the configuration of the software modules of the inspection apparatus 105 in FIG. 1 may be realized by the CPU (not shown) of the printing apparatus 102. Alternatively, the configuration of the software modules of the inspection apparatus 105 in FIG. 1 may be realized by a device external to the inspection system 100. Examples of such external devices include various terminals such as smartphones, tablet terminals, and personal computers. Alternatively, the configuration of the software modules of the inspection apparatus 105 in FIG. 1 may be realized by a cloud service that is connected to the inspection system 100 via the Internet (not shown) and can provide various services. Note that a device including some of the configurations of the inspection apparatus 105 may be referred to as an image processing apparatus.

[0089] (Variation 2) 4 has been described as an example in which the reference image 430 includes the print image area 403 and the preprint image area 404 from top to bottom of the paper, but the present invention is not limited to this. For example, a plurality of preprint image areas 404 may be set scattered on the paper, and a plurality of print image areas 403 may be set between the plurality of preprint image areas 404.

[0090] (Variation 3) 4, an example has been described in which text information such as a postal code, an address, and a name is drawn as the print image information 401, but the present invention is not limited to this. For example, a picture may be drawn as the print image information 401.

[0091] (Variation 4) 4, an example in which the image 402 is drawn in the preprint image area 404 has been described, but the present invention is not limited to this. For example, a ruled line or a frame may be drawn in the preprint image area 404 so as to surround the print image information 401.

[0092] (Variation 5) Furthermore, for example, although the configuration example in which the control points are arranged in a grid pattern has been described, the present invention is not limited to this, and the control points may be arranged in a mesh pattern, for example.

[0093] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The program may also be provided by recording it on a computer-readable storage medium.

[0094] The disclosure of the present embodiment includes configurations typified by the following image processing device, image processing method, and program.

[0095] <Configuration 1> an image acquisition means for reading a printed matter in which a print image is printed on a preprinted sheet and generating an inspection image of the printed matter; a first alignment means for aligning the entire image to be inspected with a reference image indicating a correct image of the image to be inspected by projective transformation to generate an aligned image; a second alignment means for performing alignment by non-rigid alignment for each local region of the aligned image; Equipped with The image processing device is characterized in that the second alignment means uses, as the local area in the aligned image, at least one of a preprint image area surrounding a preprint image printed on the preprint paper before printing the print image, and a print image area surrounding the print image.

[0096] <Configuration 2> 2. The image processing device according to claim 1, further comprising an inspection result output means for outputting an inspection result of the printed matter inspected based on an aligned image obtained by aligning the aligned image using the second alignment means and the reference image.

[0097] <Configuration 3> 3. The image processing device according to claim 2, further comprising a reference image generating means for generating the reference image based on first image information for generating the preprint image and second image information for generating the print image.

[0098] <Configuration 4> 4. The image processing device according to claim 3, further comprising a reference image generating means for overwriting pixels that do not become paper white among the plurality of pixels constituting the print image with a converted image that has been converted to align the four corners of the preprint image with the four corners of the print image, thereby generating the reference image.

[0099] <Configuration 5> a preprint image acquisition means for acquiring the first image information; a print image acquisition means for acquiring the second image information; Furthermore, 4. The image processing device according to claim 3, wherein the reference image generating means generates the reference image based on the first image information acquired by the preprint image acquiring means and the second image information acquired by the print image acquiring means.

[0100] <Configuration 6> 5. The image processing apparatus according to claim 4, further comprising an output preprint image acquisition unit that acquires the preprint image by reading the preprint paper.

[0101] <Configuration 7> the second alignment means further includes control point control means for controlling a control point group consisting of a plurality of control points arranged in the inspection object image, the plurality of control points are arranged in a grid pattern within a predetermined coordinate system; the inspection object image is set in the predetermined coordinate system; 6. The image processing device according to claim 4, wherein the control point control means controls the control point group to move at least one of the plurality of control points, thereby deforming the predetermined coordinate system and the image to be inspected.

[0102] <Configuration 8> 8. The image processing device according to claim 7, wherein the control point control means performs control to adjust the spacing between the plurality of control points based on the number of control points in the column direction in the control point group, the number of control points in the row direction in the control point group, and the image size of the aligned image.

[0103] <Configuration 9> the second alignment means further includes an update means for updating positions of control points to be updated in the control point group based on pixels of the reference image and pixels of the aligned image; 9. The image processing device according to claim 8, wherein the control point control means transforms the shape of the inspection target image into the shape of the aligned image in accordance with the update of the control points to be updated by the update means.

[0104] <Configuration 10> 10. The image processing device according to claim 9, wherein the second alignment means further comprises pixel update means for updating pixels of the image to be inspected to pixels of the aligned image in accordance with the update of the control points to be updated by the update means.

[0105] <Configuration 11> 11. The image processing device of claim 10, wherein the second alignment means further comprises approximation line calculation means for calculating, when one control point selected from the control point group is placed in the local region, a first approximation line along the column direction and a second approximation line along the row direction based on two or more control points selected from the control point group that are placed in the same region as the control point selected in the local region.

[0106] <Configuration 12> 12. The image processing device according to claim 11, wherein the approximation line calculation means excludes control points in the control point group that are located within a predetermined distance from a boundary of the local region from the calculation of each of the first approximation line and the second approximation line.

[0107] <Configuration 13> 12. The image processing apparatus according to claim 11, wherein the approximation line calculation means uses at least one of an approximation straight line and an approximation curve as the first approximation line and the second approximation line.

[0108] <Configuration 14> 12. The image processing device according to claim 11, wherein the second alignment means further comprises a correction target setting means for setting, as a correction target, a control point in the control point group that is located at a position that exceeds a predetermined distance from an intersection point between the first approximation line and the second approximation line.

[0109] <Configuration 15> 12. The image processing device according to claim 11, wherein the second alignment means further comprises a boundary correction target setting means for setting, as a correction target, a control point in the control point group that is located at a position that exceeds a predetermined distance from the boundary of the local region.

[0110] <Configuration 16> 15. The image processing apparatus according to claim 14, wherein the second alignment means further comprises correction means for correcting the position of the control point set for the correction target to the position of the intersection point.

[0111] <Configuration 17> 17. The image processing device according to claim 16, wherein the second alignment means further comprises image generation means that transforms the shape of the inspection object image into the shape of the aligned image in accordance with the change in the position of the control point corrected by the correction means.

[0112] <Configuration 18> 2. The image processing apparatus according to claim 1, further comprising an image area setting unit that specifies the preprint image area and the print image area based on a user setting area set by a user.

[0113] <Configuration 19> 19. The image processing device according to claim 18, wherein the image area setting means specifies the print image area based on an area including pixels that do not become paper white among the pixels that make up the print image, and specifies the preprint image area based on an area including pixels that become paper white among the pixels that make up the print image.

[0114] <Configuration 20> an image acquisition step of reading a printed matter in which a print image is printed on a preprinted sheet and generating an inspection image of the printed matter; a first alignment step of aligning the entire image to be inspected with a reference image indicating a correct image of the image to be inspected by projective transformation to generate an aligned image; a second registration step of performing registration by non-rigid registration for each local region of the registered image; Including, An image processing method characterized in that the second alignment step includes a step of using, as the local area in the aligned image, at least one of a preprint image area surrounding a preprint image printed on the preprint paper before printing the print image and a print image area surrounding the print image.

[0115] <Configuration 21> A program for causing a computer to execute each step of the image processing method according to claim 20. [Explanation of symbols]

[0116] 100 Inspection Systems 101 Server 102 Printing device 105 Inspection equipment 113 UI Panel

Claims

1. an image acquisition means for reading a printed matter in which a print image is printed on a preprinted sheet and generating an inspection image of the printed matter; a first alignment means for aligning the entire image to be inspected with a reference image that indicates a correct image of the image to be inspected by projective transformation to generate an aligned image; a second registration means for performing registration by non-rigid registration for each local region of the registered image; Equipped with An image processing device characterized in that the second alignment means uses, as the local area within the aligned image, at least one of a preprint image area surrounding a preprint image printed on the preprint paper before printing the print image, and a print image area surrounding the print image.

2. 2. The image processing device according to claim 1, further comprising an inspection result output means for outputting an inspection result of the printed matter inspected based on an aligned image obtained by aligning the aligned image by the second alignment means and the reference image.

3. 3. The image processing apparatus according to claim 2, further comprising a reference image generating means for generating the reference image based on first image information for generating the preprint image and second image information for generating the print image.

4. 4. The image processing device according to claim 3, further comprising a reference image generating means for overwriting pixels that do not become paper white among the plurality of pixels that make up the print image onto a converted image that has been converted to align the four corners of the preprint image with the four corners of the print image, thereby generating the reference image.

5. a preprint image acquisition means for acquiring the first image information; a print image acquisition means for acquiring the second image information; Furthermore, 4. The image processing apparatus according to claim 3, wherein the reference image generating means generates the reference image based on the first image information acquired by the preprint image acquiring means and the second image information acquired by the print image acquiring means.

6. 5. The image processing apparatus according to claim 4, further comprising an output preprint image acquisition unit that acquires the preprint image by reading the preprint paper.

7. the second alignment means further includes control point control means for controlling a control point group consisting of a plurality of control points arranged in the inspection object image, the plurality of control points are arranged in a grid pattern within a predetermined coordinate system; the inspection object image is set in the predetermined coordinate system; 6. The image processing device according to claim 4, wherein the control point control means controls the control point group to move at least one of the plurality of control points, thereby deforming the predetermined coordinate system and the image to be inspected.

8. 8. The image processing device according to claim 7, wherein the control point control means performs control to adjust the spacing between the plurality of control points based on the number of control points in the column direction in the control point group, the number of control points in the row direction in the control point group, and the image size of the aligned image.

9. the second alignment means further includes an update means for updating positions of control points to be updated in the control point group based on pixels of the reference image and pixels of the aligned image; 9. The image processing apparatus according to claim 8, wherein the control point control means transforms the shape of the inspection target image into the shape of the aligned image in accordance with the update of the control points to be updated by the update means.

10. 10. The image processing apparatus according to claim 9, wherein the second alignment means further comprises pixel update means for updating pixels of the inspection target image to pixels of the aligned image in accordance with the update of the control points to be updated by the update means.

11. 11. The image processing device according to claim 10, wherein the second alignment means further comprises an approximation line calculation means for calculating, when one control point selected from the control point group is placed in the local region, a first approximation line along the column direction and a second approximation line along the row direction based on two or more control points selected from the control point group that are placed in the same region as the control point selected in the local region.

12. 12. The image processing device according to claim 11, wherein the approximation line calculation means excludes control points in the control point group that are located within a predetermined distance from a boundary of the local region from the calculation of each of the first approximation line and the second approximation line.

13. 12. The image processing apparatus according to claim 11, wherein the approximation line calculation means uses at least one of an approximation straight line and an approximation curve as the first approximation line and the second approximation line.

14. 12. The image processing device according to claim 11, wherein the second alignment means further comprises a correction target setting means for setting, as a correction target, a control point in the control point group that is located at a position that exceeds a predetermined distance from an intersection of the first approximation line and the second approximation line.

15. 12. The image processing device according to claim 11, wherein the second alignment means further comprises a boundary correction target setting means for setting, as a correction target, a control point in the control point group that is located at a position that exceeds a predetermined distance from the boundary of the local region.

16. 15. The image processing apparatus according to claim 14, wherein the second alignment means further comprises correction means for correcting the position of the control point set for the correction target to the position of the intersection point.

17. 17. The image processing apparatus according to claim 16, wherein the second alignment means further comprises image generation means for transforming a shape of the inspection object image into a shape of an aligned image in accordance with a change in the position of the control point corrected by the correction means.

18. 2. The image processing apparatus according to claim 1, further comprising an image area setting unit that specifies the preprint image area and the print image area based on a user-defined area set by a user.

19. 19. The image processing device according to claim 18, wherein the image area setting means specifies the print image area based on an area including pixels that do not become paper white among the pixels that make up the print image, and specifies the preprint image area based on an area including pixels that become paper white among the pixels that make up the print image.

20. an image acquisition step of reading a printed matter in which a print image is printed on a preprinted sheet and generating an inspection image of the printed matter; a first alignment step of aligning the entire image to be inspected with a reference image indicating a correct image of the image to be inspected by projective transformation to generate an aligned image; a second registration step of performing registration by non-rigid registration for each local region of the registered image; Including, An image processing method characterized in that the second alignment step includes a step of using, as the local area in the aligned image, at least one of a preprint image area surrounding a preprint image printed on the preprint paper before printing the print image and a print image area surrounding the print image.

21. A program for causing a computer to execute each step of the image processing method according to claim 20.

Images