Computer program, image processing method, and image processing apparatus.

Abstract

Identify the positional relationships of objects between multiple images. [Solution] The computer program enables the computer to perform the following functions: acquire a first image containing a specific object and a second image containing a specific object; determine a plurality of feature points located at least one of the corners and edges of the specific object in the first image; perform a matching process for each of the plurality of feature points between a first local region of the first image containing the feature point and a second local region of the second image corresponding to the first local region; and generate a deformation amount map that shows the relative deformation amount between the specific object in the first image and the specific object in the second image for each of the plurality of unit regions of the first or second image, based on the results of the matching process performed for each of the plurality of feature points.

Description

Technical Field

[0001] This specification relates to a technique for specifying the positional relationship of objects between a plurality of images.

Background Art

[0002] In various processes, alignment between a plurality of images can be performed. Patent Document 1 discloses a technique for detecting defects in an image formed on a sheet by an image forming apparatus such as a printer or a copier. In this technique, together with a job image instructed to be printed by a user, a marker image for positioning is formed on the same sheet. An image reading unit reads the sheet surface and generates a read image. An inspection unit determines the position of the read image corresponding to the reference image based on the feature points of the job image and the marker image extracted from the read image to be inspected and the feature points of the job image and the marker image extracted from the reference image. The inspection unit compares the reference image and the read image after alignment, and detects an image area where the difference in pixel values is greater than or equal to a threshold value as a defect.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Specifying the positional relationship of objects between a plurality of images is not easy and there is room for improvement.

[0005] This specification discloses a technique for specifying the positional relationship of objects between a plurality of images.

Means for Solving the Problems

[0006] The technologies disclosed herein have been made to solve at least some of the problems described above and can be realized in the following examples of applications.

[0007] [Application Example 1] A computer program, An image acquisition function that acquires a first image containing a specific object and a second image containing the same specific object. A feature point determination function that determines a plurality of feature points located at least one of the corners and edges of the specific object in the first image, A matching function that performs a matching process for each of the multiple feature points between a first local region of the first image containing the feature point and a second local region of the second image corresponding to the first local region, A map generation function that generates a deformation amount map showing the relative deformation amount between a specific object in the first image and a specific object in the second image for each of a plurality of unit regions in the first or second image, based on the results of the matching process performed for each of the plurality of feature points, A computer program that enables a computer to realize something.

[0008] According to the above configuration, for each of the multiple feature points located at least one of the corners and edges of a specific object in the first image, a matching process is performed between the first local region of the first image containing the feature point and the second local region of the second image corresponding to the first local region. Based on the results of the matching process, a deformation amount map is generated that shows the relative deformation amount between the specific object in the first image and the specific object in the second image for each of the multiple unit regions. As a result, even if a nonlinear deformation occurs between the specific object in the first image and the specific object in the second image, the positional relationship of the specific object between the first image and the second image can be appropriately identified.

[0009] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, in the form of an image processing method and an image processing apparatus, a computer program for realizing the functions of such methods or apparatuses, a recording medium (e.g., a non-temporary recording medium) on which such computer programs are recorded, and so on. [Brief explanation of the drawing]

[0010] [Figure 1] An explanatory diagram showing an image processing apparatus as one embodiment. [Figure 2] A perspective view showing an example of the reading device 100. [Figure 3] Figure 1 shows examples of various images. [Figure 4] Flowchart of the inspection procedure in the first embodiment. [Figure 5] Diagram illustrating the first alignment process. [Figure 6] Figure 2 shows examples of various images. [Figure 7] Flowchart for generating deformation map. [Figure 8] Flowchart of local matching process. [Figure 9] A diagram illustrating template matching. [Figure 10] A flowchart for the map creation process. [Figure 11] Diagram illustrating the map creation process. [Figure 12] Conceptual diagram of the coordinate transformation map TM. [Figure 13] A diagram illustrating an example of a difference image (IMd). [Figure 14] Flowchart of the inspection procedure for the second embodiment. [Figure 15] A flowchart for the feature point determination process. [Figure 16] Flowchart of the local matching process in the second embodiment. [Figure 17] Flowchart of the first alignment process in modified example 1. [Figure 18] Diagram illustrating the first alignment process in modified example 2. [Figure 19] Flowchart of the printing process of the modified example. [Figure 20] Diagram showing an example of an image used in the modified example.

Embodiments for Carrying Out the Invention

[0011] A. First Embodiment A1. Device Configuration FIG. 1 is an explanatory diagram showing an image processing apparatus as an example. The image processing apparatus 200 is, for example, a personal computer. The image processing apparatus 200 executes an inspection process for a printed image. The image processing apparatus 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to perform data processing, and is, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile storage device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores the computer program PG and the data of the reference image IMt. The processor 210 realizes the inspection process described later by executing the computer program PG. As will be described in detail later, the reference image IMt is used in the inspection process.

[0013] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive user input, such as buttons, levers, or a touch panel superimposed on the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touchscreen. The user can input various requests and instructions to the image processing device 200 by operating the operation unit 250.

[0014] The communication interface 270 is an interface for communicating with other devices (for example, including one or more of the following: USB interface, wired LAN interface, IEEE 802.11 wireless interface, industrial camera interface (e.g., CameraLink, CoaXPress, etc.)). In this embodiment, the reading device 100 and the printing device 900 are connected to the communication interface 270. The printing device 900 is a so-called inkjet printer that prints an image on a printing medium such as cloth or paper by ejecting ink onto the medium. The reading device 100 generates read image data (scan data) representing the object by optically reading the object to be read. In this embodiment, the printing medium of the printing device 900 is a T-shirt, and the object to be read by the reading device 100 is a T-shirt with an image printed on it.

[0015] Figure 2 is a perspective view showing an example of the reading device 100. In the figure, the first direction Da and the second direction Db represent horizontal directions, and the third direction Dc represents the vertically upward direction. The first direction Da and the second direction Db are perpendicular to each other.

[0016] In this embodiment, the reading device 100 comprises a housing 190, a table 130, a support part 140 fixed to the upper surface of the table 130, a transport device 120, a reading sensor 180, and a control device 110. The control device 110, the transport device 120, and the reading sensor 180 are fixed to the housing 190.

[0017] The support portion 140 is a plate-shaped member that forms a flat top surface for supporting the object to be read (such a member is also called a platen). In the figure, a T-shirt 700 with a printed image IMp is placed on the support portion 140.

[0018] The conveying device 120 is configured to convey the table 130 in a direction parallel to the second direction Db. The configuration of the conveying device 120 may vary. Although not shown in the figures, in this embodiment the conveying device 120 has a rail that supports the table 130 so as to be slidable in a direction parallel to the second direction Db, a plurality of pulleys, a belt that is wrapped around the plurality of pulleys and partly fixed to the table 130, and an electric motor that rotates the pulleys. By rotating the pulleys with the electric motor, the table 130 (and by extension, the support part 140) moves in a direction parallel to the second direction Db. The conveying device 120 further has a position sensor 122 (for example, a rotary encoder) that detects the position of the table 130 on the conveying path.

[0019] The reading sensor 180 is positioned higher than the support unit 140 in the middle of the transport path Pth of the support unit 140. The reading sensor 180 includes a line sensor composed of multiple photoelectric conversion elements arranged in a direction intersecting the transport direction Db (in this embodiment, in a direction Da perpendicular to the transport direction Db) (for example, a Contact Image Sensor (CIS) or a Charge Coupled Device (CCD)). The reading sensor 180 faces downward. The reading sensor 180 can read the portion of the object supported by the support unit 140 that is located below the reading sensor 180.

[0020] When reading the T-shirt 700, the reading device 100 transports the table 130 in a direction parallel to the second direction Db. The reading sensor 180 repeatedly reads the T-shirt 700 during transport. As a result, the reading sensor 180 can read approximately the entire portion of the T-shirt 700 that is supported by the support portion 140.

[0021] The control device 110 is an electrical circuit configured to control the transport device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates scan data by controlling the transport device 120 and the reading sensor 180.

[0022] A2. Printing process In this embodiment, an image is printed on a T-shirt 700 (Figure 2). The image printing is performed, for example, as part of a T-shirt sales service. For example, the service provider prints an image on the T-shirt 700 using a printing device 900 according to the customer's order. The image data for printing may be provided by the customer or prepared by the service provider.

[0023] Figure 3 is the first figure showing examples of various images. Figure 3(A) shows an image represented by printable image data (referred to as the reference image IMt). In this embodiment, the data of the reference image IMt is bitmap data representing the color values ​​(here, the gradation values ​​of red R, green G, and blue B (for example, values ​​between zero and 255)) of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. In the example of Figure 3(A), the reference image IMt is an image representing a specific object Ob. In this embodiment, the specific object Ob is a design to be printed on the chest of a T-shirt 700.

[0024] Figure 2 shows a printed image IMp formed by printing a reference image IMt by a printing device 900 on the surface of a T-shirt 700. The printed image IMp may have various defects due to, for example, errors during printing. For example, due to an abnormality in ink ejection, a portion of the image may be missing in the printed image IMp. The image processing device 200 detects defects in the printed image IMp through an inspection process described later. In this embodiment, the data of the reference image IMt is used in the inspection process. The data of the reference image IMt used for printing is stored in the storage device 215 (for example, a non-volatile storage device 230) of the image processing device 200 for the inspection process (Figure 1).

[0025] A3. Inspection process Figure 4 is a flowchart of the inspection procedure in the first embodiment. For inspection, the T-shirt 700 is placed on the support part 140 of the reading device 100 (Figure 2) so that the printed image IMp is visible. In this embodiment, the operator places the T-shirt 700 on the support part 140. Alternatively, a machine (e.g., a robotic arm) may place the T-shirt 700 on the support part 140. After the T-shirt 700 is placed, an instruction to start the inspection process is input to the image processing device 200 (Figure 1). In this embodiment, the operator inputs the instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the instruction to start. The instruction to start the inspection process may be input to the image processing device 200 via the communication interface 270 by a device other than the image processing device 200.

[0026] The processor 210 of the image processing device 200 executes inspection processing according to the computer program PG. In S100, the processor 210 acquires data of the reference image IMt (Figure 3(A)) used for printing on the T-shirt 700. In this embodiment, the data of the reference image IMt stored in the non-volatile storage device 230 is acquired.

[0027] In S105, the processor 210 causes the reader 100 to read the T-shirt 700 and acquires data of the read image IMc. Specifically, the processor 210 supplies a read instruction to the reader 100. The control device 110 of the reader 100 reads the T-shirt 700 by controlling the reading sensor 180 and the transport device 120 in response to the read instruction. The control device 110 generates data (scan data) of the read image IMc representing the read T-shirt 700. The processor 210 of the image processing device 200 acquires the data of the read image IMc from the reader 100.

[0028] Figure 3(B) shows the read image IMc. The read image IMc in Figure 3(B) is an image of T-shirt 700 containing the printed image IMp.

[0029] In S110, the processor 210 extracts a predetermined specific range SA from the read image IMc that corresponds to the chest area of ​​the T-shirt 700, and acquires the target image IMs.

[0030] Figure 3(C) shows the target image IMs. In this embodiment, the data of the target image IMs is bitmap data representing the color values ​​(here, the gradation values ​​of red R, green G, and blue B, for example, values ​​between zero and 255) of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. The target image IMs represents the portion of the read image IMc in Figure 3(C) that includes the printed image IMp (Figure 3(B)).

[0031] As described above, the printed image IMp is the image obtained by printing the reference image IMt (Figure 3(A)). For this reason, the target images IMs contain the same specific objects Ob as the reference image IMt. Here, in order to distinguish between the specific objects Ob in the target images IMs and the specific objects Ob in the reference image IMt described above, the specific objects Ob in the target images IMs are also called target objects Obs, and the specific objects Ob in the reference image IMt are also called reference objects Obt.

[0032] Returning to Figure 4, the explanation continues. In S120, the processor 210 performs a first alignment process. Figure 5 is an explanatory diagram of the first alignment process. Figure 5(A) shows a flowchart of the first alignment process. In S210, the processor 210 performs a feature point detection process to detect multiple feature points of the reference image IMt (referred to as reference feature points Tt) and multiple feature points of the target image IMs (referred to as target feature points Ts).

[0033] Specifically, the processor 210 detects multiple reference feature points Tt by analyzing the grayscale-converted reference image IMt using known methods. The processor 210 also detects multiple target feature points Ts by analyzing the grayscale-converted target image IMs using known methods.

[0034] In this embodiment, a method called Accelerated-KAZE (A-KAZE) is used for feature point detection and feature quantity calculation (described later). A-KAZE is disclosed, for example, in the following paper. "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013"

[0035] Note that feature point detection may also be performed using other methods, such as searching for extrema (maximum and minimum values) using DoG (Difference-of-Gaussian), Harris corner detection, corner detection using FAST (Features from Accelerated Segment Test), SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc.

[0036] Figure 5(B) shows an example of a reference feature point Tt detected in the reference image IMt and a target feature point Ts detected in the target image IMs. As shown in the figure, points indicating characteristic parts of an object or its components, such as corners and edges, are detected as feature points Tt and Ts. Although not shown in the figure, in reality, many more feature points Tt and Ts may be detected, including internal edges of objects (for example, tens or hundreds).

[0037] In S220, the processor 210 calculates feature quantities Ft and Fs for multiple reference feature points Tt and multiple target feature points Ts. The feature quantity Ft for the reference feature point Tt may be various pieces of information describing the features of the reference feature point Tt, and the feature quantity Fs for the target feature point Ts may be various pieces of information describing the features of the target feature point Ts. The feature quantities Ft and Fs are calculated, for example, to change according to the distribution of color values ​​of multiple pixels surrounding the feature points Tt and Ts. In this embodiment, the processor 210 uses the grayscale-converted reference image IMt and target image IMs to calculate the A-KAZE feature descriptor and direction as feature quantities Ft and Fs. In A-KAZE technology, the feature descriptor is rotationally invariant. In order to obtain a rotationally invariant feature descriptor, the direction of the feature point is also calculated. The direction of the feature point indicates the direction of the luminance gradient in the neighborhood region centered on the feature point (also called the gradient direction or dominant direction).

[0038] Once the feature quantities Ft and Fs are calculated, the processor 210 performs feature point matching to determine multiple matching pairs. One matching pair consists of a reference feature point Tt in the reference image IMt and a target feature point Ts in the target image IMs that corresponds to that reference feature point Tt.

[0039] A known algorithm is used for feature point matching (matching algorithm). An example of a matching algorithm is described below. For example, the processor 210 calculates the distance dF between two feature points Tt and Ts for all combinations of one reference feature point Tt of interest and all target feature points Ts. The distance dF is calculated using the feature quantities Ft and Fs of the two feature points Tt and Ts, and the smaller the distance dF, the higher the similarity of the two feature quantities Ft and Fs. When the distance dF is small (i.e., the similarity is high), it is likely that the reference feature point Tt and the target feature point Ts represent similar parts of two images IMt and IMs (for example, the same part of object Ob). In this embodiment, the feature quantities Ft and Fs include A-KAZE feature descriptors. These feature descriptors are represented by binary vectors (vectors consisting of one or more binary elements). To this end, in this embodiment, the processor 210 calculates the Hamming distance of the feature descriptors contained in the feature quantities Ft and Fs as the distance dF. The processor 210 determines the combination with the smallest distance dF among all combinations as a matching pair that includes one reference feature point Tt of interest. However, if there are no combinations for which a distance dF below a predetermined threshold is calculated, a matching pair that includes one reference feature point Tt of interest is not determined in order to avoid erroneous matching. The processor 210 determines multiple matching pairs by performing the above process on each of the reference feature points Tt as a reference feature point of interest.

[0040] Figure 5(B) shows examples of several determined matching pairs. Each of the lines ML in the figure represents a matching pair. Each line ML connects the reference feature point Tt and the target feature point Ts that form the matching pair.

[0041] Once feature point matching is complete, processor 210 executes a transformation matrix determination process in S240 based on the determined matching pairs. The transformation matrix determination process determines a transformation matrix that defines the correspondence between coordinates on the reference image IMt and coordinates on the target image IMs. In this embodiment, an affine transformation matrix Mtx is used as the transformation matrix.

[0042] Figure 5(C) shows the coordinates COt on the reference image IMt, the coordinates COs on the target image IMs, and the affine transformation matrix Mtx that associates these coordinates COt and COs. In the figure, the coordinates COt and COs and the matrix Mtx are represented in a homogeneous coordinate system. Each coordinate COt and COs is represented as a three-dimensional vector having positions Xt and Xs in the first direction Dx on images IMt and IMs, positions Yt and Ys in the second direction Dy, and a third component of 1. The affine transformation matrix Mtx is a 3x3 matrix. As shown in the figure, the affine transformation matrix Mtx is represented by six parameters af in a 2x3 arrangement and three components (0,0,1) in the third row. Such an affine transformation matrix Mtx can represent rotation, scaling, reduction, translation, and skew. The affine transformation matrix Mtx can be calculated according to known methods by using three or more pairs of coordinates COt and COs, i.e., three or more matching pairs.

[0043] Furthermore, the transformation matrix to be determined may be a homography transformation matrix or other transformation matrices instead of the affine transformation matrix Mtx. Also, the coordinate correspondence may be represented in various formats, such as a lookup table, instead of the matrix Mtx.

[0044] In S250, the processor 210 performs a linear transformation on the target images IMs using the affine transformation matrix Mtx to generate the transformed target image IMu. Specifically, the processor 210 uses the affine transformation matrix Mtx to transform the coordinate system of the target images IMs to the coordinate system of the reference image IMt, thereby adjusting the orientation, size, and position of the target images IMs so that they roughly match those of the reference image IMt. The first alignment process is completed when the linear transformation is finished.

[0045] Figure 6 is the second figure showing examples of various images. Figure 6(A) shows the reference image IMt, and Figure 6(B) shows the transformed target image IMu. Through linear transformation, the orientation, size, direction Dx, and position of the target object Obu in the target image IMu in Figure 6(B) roughly match the orientation, size, direction Dx, and position of the reference object Obt in the reference image IMt in Figure 6(A). However, the target object Obu in Figure 6(B) has local distortions in areas such as those indicated by dashed lines, and does not perfectly match the reference object Obt in the reference image IMt. For this reason, the target object Obu in the transformed target image IMu and the reference object Obt in the reference image IMt include areas where they are locally misaligned. The T-shirt 700 is made of cloth material and, unlike a rigid body, is elastic and easily deformed. The distortion of the target object Obu in the target image IMu is due to this deformation of the T-shirt 700. In the following, the converted target image IMU will also be referred to simply as the target image IMU.

[0046] Returning to Figure 4, let's continue the explanation. In S130, the processor 210 performs a second alignment process. The second alignment process non-linearly transforms the target image IMu to correct local distortions of the target object Obu in the target image IMu, thereby achieving a more accurate match between the reference object Obt in the reference image IMt and the target object Obu in the target image IMu.

[0047] The second alignment process in S130 includes the processes in S132-S136 shown in Figure 4. In S132, the processor 210 executes a deformation amount map generation process. The deformation amount map generation process generates a deformation amount map VM that shows the local deformation amount of the target object Obu of the target image IMu caused by the distortion described above, for each unit region (pixel in this embodiment).

[0048] Figure 7 is a flowchart of the deformation amount map generation process. In S310, the processor 210 performs a grayscale conversion on the reference image IMt to generate a grayscale reference image IMt. A known relationship can be used as the correspondence between color values ​​(e.g., RGB values) and grayscale color values ​​(e.g., the correspondence between RGB values ​​in the sRGB color system and luminance values ​​Y in the YCbCr color system).

[0049] In S315, the processor 210 detects the corners of the reference object Obt in the reference image IMt and determines the first type feature point Pc. Corners are detected by performing a known corner detection process on the grayscale reference image IMt. Known corner detection processes include, for example, Harris corner detection and FAST (Features from Accelerated Segment Test) corner detection. A corner is, for example, the intersection of two edges with different directions. The processor 210 determines the detected corner as the first type feature point Pc. Figure 6(A) illustrates the corner detected in the reference image IMt, i.e., the determined first type feature point Pc.

[0050] In S320, the processor 210 performs a first local matching process. The first local matching process is a local matching process performed using multiple first-kind feature points Pc of the reference image IMt as the feature points to be processed. The local matching process matches the local region of the reference image IMt containing the feature points to be processed with the local region of the target image IMu that corresponds to the local region of the reference image IMt.

[0051] Figure 8 is a flowchart of the local matching process. In S410, one feature point of interest Pn is selected from among multiple feature points to be processed. In the first local matching process, the multiple feature points to be processed are multiple Type I feature points Pc (corners of the reference object Obt) of the reference image IMt, so the feature point of interest Pn is selected from among the multiple Type I feature points Pc.

[0052] In S415, the processor 210 determines a local region of the reference image IMt that contains the feature point of interest Pn (hereinafter also referred to as the reference local region LAt). The reference local region LAt is a local region of the reference image IMt that contains the feature point of interest Pn. In this embodiment, the reference local region LAt is a rectangular region of a predetermined size centered on the feature point of interest Pn. The size of the reference local region LAt is set to be sufficiently small compared to the reference image IMt. For example, the vertical and horizontal lengths (number of pixels) of the reference local region LAt are set to approximately 0.5%-10% of the vertical and horizontal lengths (number of pixels) of the reference image IMt. In this embodiment, while the size of the reference image IMt is 2500 pixels vertically × 2500 pixels horizontally, the size of the reference local region LAt is set to 50 pixels vertically × 50 pixels horizontally.

[0053] In S420, the processor 210 determines a local region of the target image IMu corresponding to the reference local region LAt (hereinafter also referred to as the target local region LAu). The target local region LAu is a local region of the target image IMu that includes a corresponding point Pu in the target image IMu that corresponds to a notable feature point Pn in the reference image IMt. A corresponding point in the target image IMu is a point in the target image IMu that is located at the same coordinates as the notable feature point Pn in the reference image IMt. In the reference image IMt, the notable feature point Pn (for example, a first-kind feature point Pc) is located at a corner of the reference object Obt, but in the target image IMu, the corresponding point Pu may be offset from the corner of the target object Obt. Such offsets are due to the local distortion of the T-shirt 700 described above.

[0054] In this embodiment, the target local region LAu is a rectangular region of a predetermined size centered on the corresponding point Pu. The size of the target local region LAu is set to be sufficiently small compared to the target image IMu, and larger than the reference local region LAt. For example, the vertical and horizontal lengths (number of pixels) of the target local region LAu are set to approximately 1%-15% of the vertical and horizontal lengths (number of pixels) of the target image IMu. Since the size of the reference image IMt and the size of the target image IMu are almost the same, the vertical and horizontal lengths (number of pixels) of the target local region LAu are also approximately 1%-15% of the vertical and horizontal lengths (number of pixels) of the reference image IMt.

[0055] Furthermore, the vertical and horizontal lengths (number of pixels) of the target local region LAu are set to approximately 150%-300% of the vertical and horizontal lengths (number of pixels) of the reference local region LAt. For example, considering the material of T-shirt 700, the larger the expected amount of deformation due to local distortion, the larger the size of the target local region LAu is set relative to the reference image IMt. In this embodiment, while the size of the reference image IMt and the target image IMu is 2500 pixels vertically x 2500 pixels horizontally, the size of the target local region LAu is set to 200 pixels vertically x 200 pixels horizontally.

[0056] In S425, the processor 210 performs template matching on the target local region LAu using the reference local region LAt as a template. Template matching is a process that searches for the position, size, and angle with the highest agreement rate between the template and the target local region LAu by, for example, changing the position, size, and angle of the template (reference local region LAt) relative to the target local region LAu. In this embodiment, since the reference image IMt and the target image IMu are in general agreement by the first alignment process, template matching is performed by changing only the position of the template (reference local region LAt) relative to the target local region LAu. For this reason, the matching position in which the reference local region LAt matches the target local region LAu is determined by template matching. A known method is used for the template matching algorithm. In this embodiment, the processor 210 calculates the agreement rate using a method called Zero-means Normalized Cross-Correlation (ZNCC), which can accurately match even if the brightness levels are different.

[0057] In S430, the processor 210 generates a score map SM as result information of template matching. Figure 9 is an explanatory diagram of template matching. For example, if template matching is performed on the target local region LAu in Figure 9(B) using the reference local region LAt in Figure 9(A) as a template, a match will be made to the matching position shown by the dashed line in Figure 9(B). An example of a score map SM is shown in Figure 9(C). The score map SM is an image that shows the positional agreement rate (score) of the reference local region LAt with respect to the target local region LAu. For example, the score map SM is an image of the same size as the target local region LAu, and each pixel of the score map SM corresponds one-to-one with each pixel of the target local region LAu. The score calculated when a feature point of interest Pn (e.g., a first-kind feature point Pc) at the center of the reference local region LAt is at a specific position in the target local region LAu is recorded as the value of the pixel at that specific position in the score map SM. The score map SMg in Figure 9(C) is a binarized map after the score values ​​have been binarized. Pixels with scores above a threshold are shown in black, and pixels with scores below a threshold are shown in white. Pixels with scores above a threshold are called object pixels in the score map SM, and regions composed of consecutive object pixels are called objects in the score map SM.

[0058] When template matching is successful, the score will be significantly larger in the vicinity of the single correct matching location than in other regions. For this reason, in this case, as shown in the score map SMg in Figure 9(C), there is only one object MO in the score map SM, and the size of that single object MO is relatively small.

[0059] When template matching fails, there may be multiple objects MO included in the score map SM. Figures 9(D)-(F) show an example of when template matching is not performed correctly. Using the reference local region LAtx in Figure 9(D) as a template, template matching is performed on the target local region LAux in Figure 9(E). The object Obux of the target local region LAux has two parts that are similar to the object Obtx of the template, the reference local region LAtx. For this reason, in this case, the reference local region LAtx can match to the two matching positions shown by the dashed lines in Figure 9(E). Since the reference local region LAtx should match to only one matching position, one of the two matching positions is correct, but the other is incorrect. Therefore, in this example, template matching may fail.

[0060] The score map SMx generated in this example is shown in Figure 9(F). The score map SMg in Figure 9(C) contains two objects MO1 and MO2, which may lead to a matching failure.

[0061] Furthermore, if there are many potential matching locations and template matching fails, one object in the score map SM may contain the locations of multiple candidates. In this case, the area of ​​one object in the score map SM will be larger than when template matching is performed correctly. Therefore, although not illustrated, matching may also fail if the area of ​​one object in the score map SM is excessively large.

[0062] In S435, the processor 210 determines whether the number of objects in the binarized score map SM is one, and whether the area of ​​that single object is less than or equal to a predetermined threshold THo. The threshold THo is experimentally determined to allow for the determination of whether or not the matching was successful.

[0063] If the score map SM has only one object and the area of ​​that single object is less than or equal to a predetermined threshold THo (S435: YES), then template matching is considered successful. In this case, in S440, the processor 210 determines and records the amount of deformation of the feature point of interest Pn based on the results of template matching. As shown in Figure 9(B), if template matching is performed appropriately, the position of the reference local region LAt relative to the target local region LAu is determined by template matching. Since the feature point of interest Pn (for example, the first type feature point Pc) is located at the center of the reference local region LAt, the position of the feature point of interest Pn relative to the target local region LAu is also determined by template matching (Figure 9(B)). In the target local region LAu, the vector Mv, which starts at the corresponding point Pu and ends at the feature point of interest Pn determined by template matching, indicates the amount of positional displacement of the feature point of interest Pn due to the local distortion described above. Therefore, it can be said that the vector Mv indicates the amount of local deformation of the target object Obu printed on the T-shirt 700 at the feature point of interest Pn. Furthermore, since this vector Mv is based on the feature points of the reference object Obt in the reference image IMt, it can also be said that it represents the relative deformation amount of the target object Obu with respect to the reference object Obt. Hereafter, vector Mv will also be called deformation amount Mv. As can be seen from the above explanation, deformation amount Mv is a vector quantity that has magnitude and direction. The processor 210 calculates the deformation amount Mv and records it in the non-volatile memory device 230 in association with the feature point of interest Pn. After recording the deformation amount Mv, the processor 210 proceeds to processing S460.

[0064] If the score map SM has multiple objects, or if the area of ​​one object is greater than the threshold THo (S435:NO), template matching is considered to have failed. In this case, at S450, processor 210 determines whether the number of template matching retries is less than the upper limit Rmax. The upper limit Rmax is set to, for example, 2-5 times.

[0065] If the number of retries is less than the upper limit Rmax (S450: YES), the processor 210 updates the local regions LAt and LAu in S455. Specifically, the processor 210 increases the size of the reference local region LAt and the target local region LAu compared to before the update. For example, the number of pixels in the vertical and horizontal directions of the local regions LAt and LAu are increased by approximately 1.2 to 2.0 times, respectively.

[0066] When local regions LAt and LAu are updated, processor 210 returns to S425 and retries template matching using the updated local regions LAt and LAu. In this way, processor 210 performs the nth template matching (where n is a natural number greater than or equal to 1), and if the nth matching process fails, it updates the sizes of local regions LAt and LAu and performs the (n+1)th template matching.

[0067] If the number of retries is at the upper limit Rmax (S450: NO), the processor 210 proceeds to S460 without recording the deformation amount Mv of the feature point of interest Pn.

[0068] In S460, the processor 210 determines whether all feature points of the target (e.g., first-kind feature points Pc) have been processed as feature points of interest. If there are any unprocessed feature points of the target (S460: NO), the processor 210 returns to S410. If all feature points of the target have been processed (S460: YES), the processor 210 terminates the local matching process.

[0069] Returning to Figure 7, let's continue the explanation. Once the first local matching process in S320 is complete, the processor 210 performs edge extraction on the reference image IMt in S325 to generate the edge image EI. For edge detection, a detection method called the Canny method is used, for example. However, the edge detection method may be any of the various other methods. For example, it may be a method that uses filters to calculate edge strength, such as a Laplacian filter or a Sobel filter, or a method that uses a machine learning model that has been trained to detect edges.

[0070] Figure 6(C) shows an example of a generated edge image EI. In the edge image EI in Figure 6(C), the black lines indicate edges composed of edge pixels. Edge pixels are represented by large pixel values ​​(e.g., the maximum value of 255), while non-edge pixels that do not represent edges are represented by small pixel values ​​(e.g., the minimum value of zero). In this way, by generating the edge image EI, multiple edge pixels that constitute the reference object Obt of the reference image IMt are identified.

[0071] In S330, the processor 210 removes the edges in the vicinity of the Type 1 feature point Pc that was successfully matched in the first local matching process from the edge image EI. The Type 1 feature point Pc that was successfully matched in the first local matching process is the Type 1 feature point Pc for which the deformation amount Mv was recorded in S440 of the process in Figure 8, which uses this feature point as the feature point of interest. Specifically, as shown in Figure 6(C), the processor 210 identifies a rectangular neighborhood region NA in the edge image EI centered on multiple Type 1 feature points Pc of the reference image IMt. The size of the neighborhood region NA (number of pixels in the vertical and horizontal directions) is sufficiently smaller than that of the reference image IMt. In this embodiment, the size of the reference image IMt is 2500 pixels vertically × 2500 pixels horizontally, while the size of the neighborhood region NA is set to 200 pixels vertically × 200 pixels horizontally. The processor 210 removes the edges in the neighborhood region NA by changing all edge pixels within the neighborhood region NA to non-edge pixels. This generates the processed edge image EIs shown in Figure 6(D). In the processed edge image EIs in Figure 6(D), the edges within the neighboring region NA described above have been removed.

[0072] In S335, the processor 210 identifies contour lines (i.e., lines indicating edges) EL composed of edge pixels by performing contour tracking on the processed edge image EIs. Contour tracking can be performed using known methods. In this embodiment, the findContours function of OpenCV (Open Source Computer Vision Library) was used for contour tracking.

[0073] In S340, the processor 210 determines multiple feature points Pe along the identified contour line EL. That is, multiple feature points Pe are determined on edges other than corners. To distinguish the feature points Pe determined in this step from the first type feature points Pc determined at corners, they are also called second type feature points Pe. For example, the processor 210 determines second type feature points Pe along the contour line EL at predetermined intervals (e.g., tens to hundreds of pixels). In the processed edge image EIs in Figure 6(D), the determined multiple second type feature points Pe are shown as black circles.

[0074] As can be seen from the explanation in S330-S350, multiple Type II feature points Pe are not determined within the neighborhood range (neighborhood region NA) of a Type I feature point Pc, but are determined outside the neighborhood range of the Type I feature point Pc.

[0075] In S350, the processor 210 identifies two or more Type II feature points Pe whose distance from each other is less than or equal to a threshold THr, from among the multiple Type II feature points Pe that have been determined. The processor 210 identifies two or more Type II feature points Pe whose distance from each other is less than or equal to a threshold THr, for example, by searching using a data structure called a Kd tree (k-dimensional tree). The threshold THr is set to a distance of, for example, a few pixels to a few tens of pixels. In Figure 6(D), for example, two Type II feature points Pe enclosed by the dashed circle CA are identified.

[0076] In S350, the processor 210 aggregates two or more identified Type II feature points Pe into a single feature point. That is, the processor 210 determines a single Type II feature point Pe based on the two or more Type II feature points Pe. In this embodiment, the processor 210 selects only one of the two or more Type II feature points Pe whose distance from each other is less than or equal to the threshold THr as a valid Type II feature point Pe, and deems the remaining feature points invalid. Alternatively, for example, the centroid of two or more Type II feature points Pe whose distance from each other is less than or equal to the threshold THr may be calculated, and this centroid may be selected as a new Type II feature point Pe.

[0077] In S355, the processor 210 executes the second local matching process. The second local matching process is a local matching process performed using multiple Type II feature points Pe of the reference image IMt as the feature points to be processed. Specifically, the local matching process shown in Figure 8 above is executed using multiple Type II feature points Pe as the feature points to be processed. In other words, the second local matching process is the same as the first local matching process, except that Type II feature points Pe are used instead of Type I feature points Pc as the feature points to be processed. As a result of the second local matching process, the deformation amount Mv of the feature points that successfully matched among the multiple Type II feature points Pe is recorded in the non-volatile memory device 230 (S440 in Figure 8).

[0078] By the time the process proceeds to S360, the deformation amounts Mv of the feature points that have been successfully matched from among the multiple first-kind feature points Pc and multiple second-kind feature points Pe are recorded in the non-volatile storage device 230. In S360, the processor 210 uses these deformation amounts Mv to perform a map creation process that generates a deformation amount map VM.

[0079] Figure 10 is a flowchart of the map creation process. In S510, the processor 210 prepares the initialized deformation amount map VM and the count map CT in memory (for example, the volatile storage device 220). In this embodiment, the deformation amount map VM is a map that records the deformation amount for each pixel. The count map CT is a map that records the number of times the weighted deformation amount has been added (hereinafter also called the number of additions) for each pixel in the map creation process, as will be described later. The data of the initialized deformation amount map VM and the count map CT are image data in which the value of each pixel is set to 0, and the image data has the same size as the target image IMu (number of pixels in the vertical and horizontal directions).

[0080] In S515, one feature point of interest Pmi (where i is the identifier of the feature point) is selected from the feature points to be processed. The feature points to be processed include, among multiple first-kind feature points Pc, the feature point for which the deformation amount Mv was recorded in the first local matching process described above, and among multiple second-kind feature points Pe, the feature point for which the deformation amount Mv was recorded in the second local matching process described above. In S520, the processor 210 obtains the deformation amount Mv of the feature point of interest Pmi from the non-volatile storage device 230.

[0081] In S525, the processor 210 determines the propagation range At of the deformation amount Mv of the feature point of interest Pmi, and the weighted deformation amount to be added to each pixel in the propagation range. Figure 11 is an explanatory diagram of the map creation process. Figure 11(A) is a perspective view showing the distribution of weights Wt. In the perspective view, the first direction Dx and the second direction Dy are taken horizontally in relation to the image coordinates, and the weight direction Dw is taken vertically. As shown in Figure 11(A), the weight Wt takes its maximum value Wmax at the coordinates (Xp, Yp) of the feature point of interest Pmi. The maximum value Wmax is, for example, 1. The weight Wt of each pixel is set to be larger as the distance r between the pixel and the feature point of interest Pmi decreases, and smaller as the distance r increases, approaching 0. Specifically, the weight Wt is calculated using the distance r according to the following equation (1). As can be seen from equation (1), a two-dimensional Gaussian distribution with a peak height of 1 and a standard deviation of σ is adopted for the weight Wt. σ is a fixed value determined experimentally.

[0082]

number

[0083] Then, the weighted deformation amount Vi(r) to be added to the pixel located at a distance r from a feature point of interest Pmi is calculated according to the following equation (2). Mvi represents the deformation amount Mvi of the feature point of interest Pmi with identifier i. That is, the weighted deformation amount Vi(r) is the value obtained by multiplying the deformation amount Mvi of the feature point of interest Pmi by the weight Wt, and if this value is less than 1, it is ignored (set to 0).

[0084]

number

[0085] The propagation range TA is the range in which a weighted deformation amount Vi(r) different from 0 is calculated. Figure 11(B) shows a conceptual diagram of the deformation amount map VM. The rectangular grid of the deformation amount map VM in Figure 11(B) represents the pixels (also called unit regions) of the deformation amount map VM, and corresponds one-to-one with the pixels of the target image IMu. Figure 11(B) shows the propagation range At1 (dashed line) when the feature point of interest Pmi is at the position of pixel Pm1, and the propagation range At2 (solid line) when the feature point of interest Pmi is at the position of pixel Pm2 in Figure 11(B). In this way, the propagation range At1 of one feature point may overlap with the propagation range At2 of another feature point.

[0086] Once the propagation range At of the feature point of interest Pmi and the weighted deformation amount Vi(r) for each pixel in the propagation range At are determined, the processor 210 adds the determined weighted deformation amount Vi(r) to the value of each pixel in the determined propagation range At in S530.

[0087] In S535, the processor 210 counts up the value (i.e., the number of additions) of each pixel in the determined propagation range At in the count map CT. That is, 1 is added to the value of each pixel in the determined propagation range At in the count map CT. Figure 11(C) shows a conceptual diagram of the count map CT. The rectangular grid of the count map CT in Figure 11(C) represents the pixels of the count map CT, and there is a one-to-one correspondence with the pixels of the deformation amount map VM in Figure 11(B). For example, if the feature point of interest Pmi is at the position of pixel Pm1 in Figure 11(B), 1 is added to the value of the pixel in the count map CT corresponding to the propagation range At1 (dashed line) in Figure 11(B). If the feature point of interest Pmi is at the position of pixel Pm2 in Figure 11(B), 1 is added to the value of the pixel in the count map CT corresponding to the propagation range At2 (solid line) in Figure 11(B). For this reason, in Figure 11(C), the pixel values ​​in the region where propagation range At1 and propagation range At2 overlap are 2.

[0088] In S540, the processor 210 determines whether all feature points of the target object have been processed as feature points of interest. If there are any unprocessed feature points of the target object (S540: NO), the processor 210 returns to S515. If all feature points of the target object have been processed (S540: YES), the processor 210 proceeds to S545.

[0089] In step S545, the processor 210 divides the value of each pixel in the deformation amount map VM (i.e., the sum of the weighted deformation amounts of two or more feature points) by the number of additions recorded in the count map CT. This averages the sum of the weighted deformation amounts of each pixel in the deformation amount map VM. Note that the processing in S545 is performed only for pixels where the number of additions recorded in the count map CT is 2 or more, and not for pixels where it is 0 or 1.

[0090] Once processing S545 is complete, the deformation amount map VM is finished. The final value of each pixel recorded in the completed deformation amount map VM is also called the local deformation amount LV. Processor 210 finishes the map creation process shown in Figure 11. Once the map creation process in Figure 11 is finished, the deformation amount map generation process shown in Figure 7 is finished.

[0091] Returning to Figure 4, let's continue the explanation. Once the deformation amount map generation process in S132 is completed, in S134, the processor 210 generates a coordinate transformation map TM based on the deformation amount map VM. The coordinate transformation map TM is a map that is referenced when performing a nonlinear transformation on the target image IMu. The coordinate transformation map TM defines the correspondence between the coordinates (Xa, Ya) before the nonlinear transformation and the coordinates (Xb, Yb) after the nonlinear transformation.

[0092] Figure 12 is a conceptual diagram of the coordinate transformation map TM. The data of the coordinate transformation map TM is image data with the same size (number of pixels in the vertical and horizontal directions) as the target image IMu, similar to the data of the deformation amount map VM and the count map CT. As shown in Figure 12, the coordinates of each pixel in the coordinate transformation map TM represent the coordinates (Xb, Yb) after nonlinear transformation. The value of each pixel in the coordinate transformation map TM represents the coordinates (Xa, Ya) before the nonlinear transformation. For example, the value (Xa, Ya) of the pixel at coordinates (Xb, Yb) in the coordinate transformation map TM can be expressed by the following equations (3) and (4), where (ΔX, ΔY) is the local deformation amount LV (vector quantity) at coordinates (Xb, Yb) recorded in the coordinate transformation map TM.

[0093] Xa = Xb + ΔX …(3) Ya = Yb + ΔY …(4)

[0094] The processor 210 generates the coordinate transformation map TM by calculating the values ​​of all pixels in the coordinate transformation map TM by referring to the deformation amount map VM.

[0095] In S136, the processor 210 performs a nonlinear transformation on the target image IMu using the coordinate transformation map TM. This generates the nonlinearly transformed target image IMf. Specifically, the processor 210 identifies the coordinates of the pixels in the target image IMu corresponding to each pixel in the nonlinearly transformed target image IMf by referring to the coordinate transformation map TM. The processor 210 obtains the values ​​(RGB values) of the identified pixels from the target image IMu. This generates the nonlinearly transformed target image IMf. Figure 6(E) shows an example of the nonlinearly transformed target image IMf. In the nonlinearly transformed target image IMf, the local distortion (deformation) that occurred in the target image IMu (Figure 6(B)) is eliminated. That is, the target image IMu is transformed so that the reference object Obt of the reference image IMt and the target object Obf of the transformed target image IMf match in position and shape. As a result, the reference object Obt of the reference image IMt and the target object Obf of the target image IMf are aligned so that they match with high precision when the reference image IMt and the target image IMf are superimposed.

[0096] In S150, the processor 210 inspects the printed image IMp. Specifically, the processor 210 generates a difference image IMd between the reference image IMt and the nonlinearly transformed target image IMs.

[0097] Figure 13 shows an example of a difference image IMd. Figure 13(A) shows difference image IMd1, which is generated when the printed image IMp does not contain defects, and Figure 13(B) shows difference image IMd2, which is generated when the printed image IMp contains defects. The difference image IMd represents the difference in color values ​​(for example, the absolute value of the difference in luminance values) between the reference image IMt in Figure 6(A) and the target image IMf in Figure 6(E), pixel by pixel. In the difference image IMd, the parts showing large differences are the parts where there is a difference between the reference object Obt of the reference image IMt and the target object Obf of the target image IMf. In this way, by calculating the difference image IMd, it is possible to detect the difference between the reference object Obt of the reference image IMt and the target object Obf of the target image IMf, i.e., defects in the printed image IMp. If the printed image IMp does not contain defects, the difference image IMd1 does not contain any parts showing large differences (Figure 13(A)). If the printed image IMp contains defects, the differential image IMd2 will include the corresponding Err portion (the portion indicating a large difference) (Figure 13(B)).

[0098] In S160, the processor 210 outputs the inspection results. There are various methods for outputting the inspection results. In this embodiment, the processor 210 displays the difference image IMd on the display unit 240 (Figure 1). By observing the difference image IMd displayed on the display unit 240, the operator can easily recognize defects in the printed image IMp. Alternatively, the processor 210 may output data representing the inspection results to a storage device (for example, a non-volatile storage device 230, or an external storage device connected to the image processing device 200). After S160, the processor 210 terminates the inspection process.

[0099] According to the embodiment described above, the processor 210 acquires a reference image IMt (Figure 6(A)) containing a specific object (reference object Obt) (S100 in Figure 4). The processor 210 acquires a target image IMu (Figure 6(B)) containing a specific object (target object Obu) (S105-S120 in Figure 4). The processor 210 determines a plurality of feature points Pc and Pe located at the corners and edges of the reference object Obt in the reference image IMt (S315 in Figure 4, S340 in Figure 7). For each of the plurality of feature points Pc and Pe, the processor 210 performs template matching between the reference local region LAt of the reference image IMt containing the feature point and the target local region LAu of the target image IMu corresponding to the reference local region LAt (S320, S355 in Figure 7, S425 in Figure 8). The processor 210 generates a deformation amount map VM that shows the local deformation amount LV for each of the multiple pixels of the target image IMu based on the results of the local matching process (S360 in Figure 7, Figure 10). As a result, even if a nonlinear deformation occurs between the reference object Obt of the reference image IMt and the target object Obu of the target image IMu, the positional relationship of the objects between the reference image IMt and the target image IMu can be appropriately determined. Therefore, for example, as in the above embodiment, by taking the difference between the target image IMu and the reference image IMt, the object alignment between the target image IMu and the reference image IMt can be performed with high accuracy when inspecting the printed target object Obu of the target image IMu. In particular, when the target image IMu is an image obtained by optically reading an object that has low rigidity and is easily deformed, such as a T-shirt 700, as in this embodiment, there is a high possibility that the target object Obu of the target image IMu will be locally deformed, so generating the deformation amount map VM in this embodiment is effective.

[0100] Furthermore, according to this embodiment, the processor 210 linearly transforms the target image IMs so that the difference in position and size between the reference object Obt of the reference image IMt and the target object Obs of the target image IMs is reduced (S120 in Figure 4, S250 in Figure 5(A)). After the linear transformation, the processor 210 uses the linearly transformed target image IMu to perform a local matching process (S320 and S355 in Figure 7) and generates a deformation amount map VM based on the results of the local matching process (S360 in Figure 7, Figure 10). As a result, even if the position and size of the reference object Obt of the reference image IMt and the target object Obs of the target image IMs differ significantly, the deformation amount map VM can be appropriately generated by using the linearly transformed target image IMu after linearly transforming the target image IMs.

[0101] Furthermore, according to this embodiment, the processor 210 determines multiple matching pairs from among multiple reference feature points Tt of the reference image IMt and multiple target feature points Ts of the target image IMs based on a predetermined matching algorithm (S230 in Figure 5(A), Figure 5(B)). Based on the multiple matching pairs, the processor 210 calculates an affine transformation matrix Mtx for performing a linear transformation (S240 in Figure 5(A), Figure 5(C)), and performs a linear transformation of the target image IMs using the affine transformation matrix Mtx (S250 in Figure 5(A)). As a result, the target image IMs can be appropriately linearly transformed using so-called feature point matching. Therefore, prior to the generation of the deformation amount map VM, a rough alignment can be performed between the reference object Obt of the reference image IMt and the target object Obu of the linearly transformed target image IMu.

[0102] Furthermore, according to this embodiment, the processor 210 uses a deformation amount map VM to nonlinearly transform the target image IMu so that the position and shape of the reference object Obt of the reference image IMt and the target object Obf of the nonlinearly transformed target image IMf match (S136 in Figure 4). After this nonlinear transformation, the processor 210 detects the difference between the reference object Obt of the reference image IMt and the target object Obf of the target image IMf (S150 in Figure 4). As a result, even if the target object Obu of the target image IMu has undergone nonlinear deformation (distortion) due to the deformation of the T-shirt 700, the difference between the reference object Obt of the reference image IMt and the target object Obf of the target image IMf can be detected with high accuracy by using the nonlinearly transformed target image IMf in which such deformation has been corrected.

[0103] Furthermore, according to this embodiment, the deformation amount map generation process (Figure 10) includes a process (440 in Figure 8) to calculate the deformation amount Mv of each of the multiple feature points (first-type feature point Pc and second-type feature point Pe) based on the results of template matching (S425 in Figure 8) of each of the multiple feature points Pc and Pe, and a process (S520-S545 in Figure 10) to calculate the local deformation amount LV corresponding to each pixel using a weight Wt (Equation (1)) which is set to be larger the shorter the distance between each pixel and the feature points Pc and Pe, and the deformation amount Mv of the feature points Pc and Pe. As a result, the local deformation amount LV of each pixel of the target image IMu can be calculated using the deformation amounts Mv of the multiple feature points Pc and Pe. Therefore, a deformation amount map VM showing the local deformation amount LV for each pixel can be appropriately generated.

[0104] Furthermore, according to this embodiment, the processor 210 determines the feature points Pc and Pe of the reference image IMt by analyzing the reference image IMt according to a predetermined algorithm (S315, S325-S350 in Figure 7). For example, the first type feature point Pc is detected using a known algorithm such as Harris corner detection. The second type feature point Pe is determined using an algorithm consisting of the series of processes in S325-S350. As a result, the feature points Pc and Pe of the reference image IMt are determined without the user having to specify feature points in advance, for example, thus reducing the burden on the user.

[0105] Furthermore, according to this embodiment, the processor 210 performs the nth template matching (n is a natural number greater than or equal to 1) between the reference local region LAt and the target local region LAu (S425 in Figure 8). If it is determined that the nth matching process has failed (NO in S435 in Figure 8), both local regions LAt and LAu are changed to larger regions than those used in the nth matching process (S455 in Figure 8), and the (n+1)th template matching process is performed after the changes to local regions LAt and LAu (S425 in Figure 8). As a result, the success rate of template matching can be improved. For example, if the reference local region LAt does not contain enough features to distinguish it from other regions, increasing the size of the reference local region LAt increases the likelihood of successful template matching. Also, if the target local region LAu does not contain the portion corresponding to the notable feature point Pn of the reference local region LAt due to the large distortion of the target image IMu, increasing the size of the target local region LAu increases the likelihood of successful template matching.

[0106] On the other hand, if the local regions LAt and LAu become excessively large, it can no longer be considered local template matching, and even if template matching is successful, it may become impossible to properly calculate the local deformation amount Mv. In this embodiment, by setting an upper limit Rmax on the number of retries, it is possible to suppress the local regions LAt and LAu from becoming excessively large.

[0107] Furthermore, according to this embodiment, the processor 210 determines a plurality of first-type feature points Pc located at the corners of the reference object Obt (S315 in Figure 7), and performs template matching for each of the plurality of first-type feature points Pc (S320 in Figure 7). Subsequently, the processor 210 determines a plurality of second-type feature points Pe located at the edges of the reference object Obt (S325-S350 in Figure 7), and performs template matching for each of the plurality of second-type feature points Pe (S355 in Figure 7). First-type feature points Pc at corners where multiple edge lines intersect are more distinctive than second-type feature points Pe at edges other than corners, and are therefore more suitable for local matching. For example, corners where multiple edge lines intersect are easier to uniquely determine, so the matching position by template matching can be determined with high accuracy. In this embodiment, template matching is preferentially performed on first-type feature points Pc that are more suitable for matching, so the deformation amount map VM can be generated efficiently.

[0108] Furthermore, according to this embodiment, the processor 210 does not determine the second type feature point Pe within the neighboring region NA containing the first type feature point Pc that has been successfully template-matched, but determines the second type feature point Pe outside the neighboring region NA (S330-S340 in Figure 7, Figure 6(D)). The local deformation amount in the neighborhood of the first type feature point Pc that has been successfully template-matched can be grasped by template matching using the first type feature point Pc. For this reason, it is highly likely that placing the second type feature point Pe in the neighborhood of the first type feature point Pc that has been successfully template-matched would be redundant. In this embodiment, since the second type feature point Pe is not determined within the neighboring region NA containing the first type feature point Pc, the second type feature point Pe can be determined without redundancy. Therefore, redundant calculations can be suppressed, and the deformation amount map VM can be generated efficiently.

[0109] Furthermore, according to this embodiment, the processor 210 identifies multiple edge pixels that constitute the edges of the reference object Obt in the reference image IMt (S325 in Figure 7), and determines multiple type II feature points Pe along the contour line EL formed by the multiple edge pixels (S345 in Figure 7, Figure 6(D)). As a result, multiple type II feature points Pe located on edges other than the corners of the reference object Obt in the reference image IMt can be appropriately determined.

[0110] Furthermore, according to this embodiment, if the distance between two or more Type II feature points Pe is less than or equal to the threshold THr, one Type II feature point Pe is determined based on the two or more Type II feature points (S350 in Figure 7). As a result, unnecessary Type II feature points Pe can be reduced so that they are placed efficiently. Therefore, unnecessary calculations can be further suppressed, and the deformation amount map VM can be generated even more efficiently.

[0111] As can be seen from the above explanation, the reference image IMt in this embodiment is an example of the first image, and the linearly transformed target image IMu is an example of the second image. The reference object Obt and the target object Obu are examples of specific objects. The reference local region LAt is an example of the first local region, and the target local region LAu is an example of the second local region. Furthermore, the reference feature point Tt of the reference image IMt is an example of the first feature point, and the target feature point Ts of the target image IMs is an example of the second feature point. Also, template matching is an example of a matching process, and the neighboring region NA is an example of a specific range.

[0112] B. Second Example In the second embodiment, the content of the inspection process differs from that of the first embodiment. The other components of the second embodiment are the same as those of the first embodiment. Figure 14 is a flowchart of the inspection procedure in the second embodiment.

[0113] In the flowchart of Figure 14, the same processes as in Figure 4 are denoted by the same reference numerals, while processes that differ from those in Figure 4 have "B" appended to the end of their reference numerals. In the inspection process of Figure 14, steps S100-S120 are the same as in Figure 4. In the inspection process of Figure 14, steps S130B-S160B are executed instead of steps S130-S160 in Figure 4. The inspection process of the second embodiment will be described below, focusing on the differences from Figure 4.

[0114] Once the first alignment process in S120 is completed, in S130B of Figure 14, the processor 210 executes a feature point determination process. The feature point determination process determines multiple first-type feature points Pc and second-type feature points Pe on the reference image IMt.

[0115] Figure 15 is a flowchart of the feature point determination process. In the flowchart of Figure 15, processes identical to those in Figure 7 are denoted by the same symbols as in Figure 7, and processes different from those in Figure 7 are denoted by the letter "B" at the end of their symbols.

[0116] Processes S310 and S315 are the same as those shown in Figure 7. Processes S310 and S315 determine multiple first-kind feature points Pc located at the corners of the reference object Obt of the reference image IMt, as shown in Figure 6(A).

[0117] S320 is the same process as S320 in Figure 7. The process in S320 generates the edge image EI in Figure 6(C). In the feature point determination process in Figure 15, the first local matching process of S320 in Figure 7 is not performed.

[0118] In S330B, similar to S330 in Figure 7, the processor 210 removes the edges in the vicinity of the first type feature point Pc from the edge image EI. For example, edges within the rectangular neighborhood region NA (Figure 6(D)) centered on the first type feature point Pc are removed. However, in the feature point determination process in Figure 15, the first local matching process is not performed, so all the edges in the vicinity of the determined first type feature point Pc are removed from the edge image EI.

[0119] S335-S350 is the same process as S335-S350 in Figure 7. The process in S335-S350 determines multiple Type II feature points Pe in the processed edge image EIs, as shown in Figure 6(D). Specifically, multiple Type II feature points Pe located on edges of the reference object Obt in the reference image IMt, excluding the corners, are determined. Once the Type II feature points Pe are determined, the feature point determination process in Figure 15 is terminated. At this point, multiple Type I feature points Pc and multiple Type II feature points Pe are determined in the reference image IMt.

[0120] Returning to Figure 14, let's continue the explanation. Once the feature point determination process in S130B is complete, the processor 210 executes a local matching process in S140B. The local matching process in the second embodiment uses the determined feature points Pc and Pe to perform local matching and detects whether or not there is an abnormality in the target object Obu of the target image IMu, or in other words, whether or not there is an abnormality in the printed image IMp of the T-shirt 700.

[0121] Figure 16 is a flowchart of the local matching process in the second embodiment. In the flowchart of Figure 16, processes identical to those in Figure 8 are denoted by the same reference numerals as in Figure 8, and processes different from those in Figure 8 are denoted by the letter "B" at the end of their reference numerals. The local matching process in the second embodiment is performed using a plurality of first-type feature points Pc and a plurality of second-type feature points Pe, which have been determined in the feature point determination process, as the feature points to be processed.

[0122] S410-S430 is the same process as S410-S430 in Figure 8. The S410-S430 process performs template matching between a reference local region LAt containing the feature points of interest selected from the feature points to be processed, and a target local region LAu corresponding to the reference local region LAt, similar to the local matching process in the first embodiment. A score map SM of the template matching is then generated.

[0123] In S435, similar to S435 in Figure 8, the processor 210 determines whether the number of objects in the binarized score map SM is one, and whether the area of ​​that single object is less than or equal to a predetermined threshold THo. In other words, the processor 210 determines whether template matching was successful.

[0124] If the score map SM has multiple objects, or if the area of ​​one object is greater than the threshold THo (S435:NO), in other words, if template matching fails, the processor 210 proceeds to S480B.

[0125] In the S480B, the processor 210 determines the inspection result to be abnormal. If template matching fails, there is a high probability that there is a significant difference (for example, an abnormality in the target object Obu) between the image in the reference local region LAt and the image in the target local region LAu. In other words, in this case, there is a high probability that there is an abnormality in the printed image IMp of the T-shirt 700. For example, in this embodiment, parameters related to template matching, such as the threshold THo and the sizes of the reference local region LAt and the target local region LAu, have been experimentally adjusted so that it is possible to detect whether or not there is an abnormality in the target image IMu based on whether or not template matching fails.

[0126] If the score map SM has only one object and the area of ​​that single object is less than or equal to a predetermined threshold THo (S435: YES), that is, if template matching is successful, in S440B the processor 210 determines the deformation amount Mv of the feature point of interest Pn based on the results of template matching. The method for determining the deformation amount Mv is the same as the method described in S440 of Figure 8 of the first embodiment.

[0127] In the S450B, the processor 210 determines whether the calculated deformation amount Mv is greater than or equal to the threshold THv. In this embodiment, since the deformation amount Mv is a vector quantity, the processor 210 calculates the length of the vector representing the deformation amount Mv and determines whether the length of the vector is greater than or equal to the threshold THv.

[0128] If the calculated deformation amount Mv is greater than or equal to the threshold THv (S435: YES), the processor 210 determines the inspection result to be abnormal in S480B. In this case, it is considered that there is a significant difference (for example, an abnormality in the target object Obu) between the image in the reference local region LAt and the image in the target local region LAu. For example, if the deformation amount Mv between the image in the first local region and the image in the second local region is excessively large, for example, if the deformation exceeds the amount expected to be due to local distortion of the T-shirt 700, there is a high possibility that there is an abnormality in the target object Obu of the target image IMu. In other words, in this case, there is a high possibility that there is an abnormality in the printed image IMp of the T-shirt 700. For example, in this embodiment, the size of the threshold THv is experimentally adjusted so that it is possible to detect whether or not there is an abnormality in the target image IMu based on whether or not the deformation amount Mv is greater than or equal to the threshold THv.

[0129] If the calculated deformation amount Mv is less than the threshold THv (S435: NO), the processor 210 determines in S460 whether or not all feature points of the processing target (first-type feature point Pc and second-type feature point Pe) have been processed as feature points of interest. If there are any unprocessed feature points of the processing target (S460: NO), the processor 210 returns to S410.

[0130] If all feature points of the target to be processed have been processed (S460:YES), the processor 210 determines the inspection result to be normal in S470B.

[0131] Once the test result is determined in S470B or S480B, the processor 210 terminates the local matching process.

[0132] When the local matching process in S140B of Figure 14 is completed, in S160B, the processor 210 outputs the inspection result. There are various methods for outputting the inspection result. In this embodiment, the processor 210 displays on the display unit 240 (Figure 1) whether there is no abnormality or an abnormality. If an abnormality is found, the processor 210 may also display on the display unit 240 information indicating the location of the feature point of interest at the time the abnormality was found. Alternatively, the processor 210 may output data representing the inspection result to a storage device (for example, a non-volatile storage device 230, or an external storage device connected to the image processing device 200). After S160B, the processor 210 terminates the inspection process.

[0133] According to the second embodiment described above, the processor 210 acquires a reference image IMt (Figure 6(A)) containing a specific object (reference object Obt) (S100 in Figure 14). The processor 210 acquires a target image IMu (Figure 6(B)) containing a specific object (target object Obu) (S105-S120 in Figure 14). The processor 210 determines a plurality of feature points Pc and Pe located at the corners and edges of the reference object Obt in the reference image IMt (S130B in Figure 14, S315 and S340 in Figure 15). For each of the plurality of feature points Pc and Pe, the processor 210 performs template matching between the reference local region LAt of the reference image IMt containing the feature point and the target local region LAu of the target image IMu corresponding to the reference local region LAt (S425 in Figure 16). The processor 210 detects the difference (specifically, anomalies in the target object Obu) between the image in the reference local region LAt and the image in the target local region LAu based on the results of template matching (S435, S450B, S480B in Figure 16). As a result, the difference between the reference object Obt in the reference image IMt and the target object Obu in the target image IMu can be detected using local template matching. For example, anomalies in the target object Obu in the target image IMu can be detected without generating a deformation amount map VM (S360 in Figure 7), performing a nonlinear transformation (S136 in Figure 4), or generating a difference image IMd (Figure 13) as in the first embodiment.

[0134] C. Variations (1) In each of the above embodiments, the first alignment process is performed by feature point matching (Figure 5). However, the first alignment process may be performed by other methods. Below, the first alignment process performed using template matching will be described as Modification 1, and the first alignment process performed using alignment information input by the user will be described as Modification 2.

[0135] Figure 17 is a flowchart of the first alignment process in Modification 1. At S610, the processor 210 uses the entire reference image IMt as a template and performs template matching on the target images IMs. The processor 210 uses a known method for the template matching algorithm. This searches for the position, size, and angle of the reference image IMt relative to the target images IMs so that the agreement rate between the reference image IMt and the target images IMs is highest. As a result of template matching, the processor 210 obtains the position, size, and angle of the reference image IMt relative to the target images IMs with the highest agreement rate.

[0136] In S620, the processor 210 determines a transformation matrix for performing a linear transformation based on the results of template matching. Specifically, the processor 210 determines an affine transformation matrix Mtx that defines the correspondence between coordinates on the reference image IMt and coordinates on the target images IMs, similar to the embodiment. In S610, the affine transformation matrix Mtx is determined to realize a linear transformation that follows the position, size, and angle of the reference image IMt for the target images IMs with the highest matching rate obtained.

[0137] In S630, similar to S250 in Figure 5(A), the processor 210 performs a linear transformation on the target images IMs using the affine transformation matrix Mtx to generate the transformed target image IMu. Once the linear transformation is complete, the first alignment process in Figure 17 is terminated.

[0138] According to this modification, by using template matching and linearly transforming the target image IMs, it is possible to generate a transformed target image IMu such that the reference object Obt of the reference image IMt and the target object Obu of the transformed target image IMu are roughly the same.

[0139] Figure 18 is an explanatory diagram of the first alignment process of Modification 2. Figure 18(A) shows a flowchart of the first alignment process of Modification 2. In S710, the processor 210 displays the input screen W1 on the display unit 240 to obtain correspondence information from the user. The correspondence information is information regarding the correspondence between the reference image IMt and the target image IMs. In one example, the correspondence information specifies three or more pairs of corresponding points. A pair of corresponding points is a pair of points in the target image IMs and points in the target image IMu that correspond to each other.

[0140] Figure 18(B) shows an example of the input screen W1. The input screen W1 in Figure 18(B) includes a message MS prompting the user to specify three or more pairs of corresponding points, a reference image IMt, target images IMs, a complete button BT1, and a redo button BT2. The user operates the operation unit 250 (e.g., a pointing device such as a mouse) to move the cursor CS on the input screen W1, specifying one point on the reference image IMt (e.g., point UPt) and a corresponding point on the target images IMs (e.g., UPs), thereby specifying one pair of corresponding points. After the user has specified three or more pairs of corresponding points, the user presses the complete button BT1. The processor 210 obtains correspondence relationship information to specify three or more pairs of corresponding points based on the information entered by the user before the complete button BT1 is pressed.

[0141] In S720, the processor 210 determines a transformation matrix for performing a linear transformation based on correspondence information obtained from the user. Specifically, the processor 210 determines an affine transformation matrix Mtx that defines the correspondence between coordinates on the reference image IMt and coordinates on the target image IMs, similar to the embodiment. In the same method used in S240 of the embodiment to determine the affine transformation matrix Mtx based on multiple matching pairs, the affine transformation matrix Mtx is determined based on three or more pairs of corresponding points indicated by the correspondence information.

[0142] In S730, similar to S250 in Figure 5(A), the processor 210 performs a linear transformation on the target images IMs using the affine transformation matrix Mtx to generate the transformed target image IMu. Once the linear transformation is complete, the first alignment process in Figure 18 is terminated.

[0143] According to this modified version, by using correspondence information obtained from the user and performing a linear transformation on the target image IMs, it is possible to generate a transformed target image IMu such that the reference object Obt of the reference image IMt and the target object Obu of the transformed target image IMu roughly match.

[0144] The correspondence information obtained from the user is not limited to three or more pairs of corresponding points; it may also be other types of information. For example, in an input screen W1 that includes a reference image IMt and a target image IMs, the target image IMu can be moved, rotated, and scaled according to the user's operation. The user moves, rotates, and scales the target image IMu on the input screen to adjust its position, size, and angle so that it overlaps the reference image IMt as completely as possible. The processor 210 may obtain the position, size, and angle of the target image IMu adjusted by the user as correspondence information. In this case, the processor 210 should determine the affine transformation matrix Mtx based on the position, size, and angle of the target image IMu adjusted by the user.

[0145] Furthermore, in the above embodiments and modifications 1 and 2, the first alignment process linearly transforms the target image IMs and aligns the target object Obu of the linearly transformed target image IMu with the reference object Obt of the reference image IMt. Alternatively, the first alignment process may involve linearly transforming the reference image IMt and aligning the reference object of the linearly transformed reference image with the object Obs of the target image IMs. In this case, the processor 210 uses the linearly transformed reference image and the target image IMs to perform the second alignment process.

[0146] (2) In the first embodiment described above, the deformation amount map VM is used to perform an inspection process of the printed image IMp of the T-shirt 700. The deformation amount map VM may be used for other purposes as well. As an example, a printing process will be described in which a nonlinear transformation is performed on the printed image using the deformation amount map VM, and printing is performed using the data of the nonlinearly transformed printed image.

[0147] In this modified example, for instance, an additional print image PRI is printed on a T-shirt S700 that has already been printed with a reference image IMt. For example, it is conceivable that an additional print image PRI is printed on a T-shirt S700 that has already been printed with an image at the factory, according to the user's preference at each store. The printing process involves printing the additional print image PRI onto the T-shirt S700 using a garment printer, a so-called inkjet type printer, which prints an image onto fabric using ink.

[0148] Figure 19 is a flowchart of the printing process for a modified example. In S100C, similar to S100 in Figure 4, the processor 210 obtains the data of the reference image IMt used for printing on the T-shirt 700 from the non-volatile storage device 230.

[0149] Figure 20 shows an example of an image used in a modified example. Figure 20(A) shows an example of a reference image IMt. The reference image IMt in Figure 20(A) is the same image as the reference image IMt in Figures 3(A) and 6(A) of the first embodiment.

[0150] In S105C, similar to S105 in Figure 4, the processor 210 obtains data of the read image IMc by having the reader read the T-shirt 700. However, in this modified example, the T-shirt 700 is placed on the platen of the garment printer. The reader is a digital camera fixed in a position that allows it to read the T-shirt 700 placed on the platen from above. The processor 210 supplies a reading instruction to the reader. In response to the reading instruction, the reader generates data of the read image (captured data) by reading (photographing) the T-shirt 700 placed on the platen from above. The processor 210 of the image processing device 200 obtains the data of the read image from the reader. This is how the read image IMc in Figure 3(B) is obtained. The read image IMc in Figure 3(B) is an image showing the T-shirt 700 including the printed image IMp.

[0151] In S110C, similar to S110 in Figure 4, the processor 210 extracts a predetermined specific range SA from the read image IMc that corresponds to the chest area of ​​the T-shirt 700, and obtains the target image IMs. This obtains the target image IMs shown in Figure 3(C). The target image IMs represents the portion of the read image IMc in Figure 3(C) that includes the printed image IMp (Figure 3(B)).

[0152] In S120C, the processor 210 performs a first alignment process. The first alignment process is the same as the process in S120 in Figure 4 of the first embodiment (Figure 5). The first alignment process generates the linearly transformed target image IMu shown in Figure 6(B).

[0153] In S125C, the processor 210 executes a deformation amount map generation process. This deformation amount map generation process is the same as the deformation amount map generation process in S132 of Figure 4 of the first embodiment (Figure 7). The deformation amount map generation process generates a deformation amount map VM.

[0154] In S130C, the processor 210 generates a coordinate transformation map TM (Figure 12) using a deformation amount map VM. As described in the first embodiment, the coordinate transformation map TM defines the correspondence between coordinates (Xa, Ya) before nonlinear transformation and coordinates (Xb, Yb) after nonlinear transformation. However, in the first embodiment, the coordinate transformation map TM is used to transform a distorted target image IMu to match an undistorted reference image IMt, whereas in the second embodiment, the coordinate transformation map TM is used to transform an undistorted printed image (described later) to match a distorted target image IMu. For this purpose, in the second embodiment, the pixel values ​​(Xa, Ya) of coordinates (Xb, Yb) in the coordinate transformation map TM are expressed by the following equations (5) and (6), where (ΔX, ΔY) is the local deformation amount LV (vector quantity) at coordinates (Xb, Yb) recorded in the coordinate transformation map TM.

[0155] Xa = Xb - ΔX …(5) Ya = Yb - ΔY …(6)

[0156] The processor 210 generates the coordinate transformation map TM by calculating the values ​​of all pixels in the coordinate transformation map TM by referring to the deformation amount map VM.

[0157] In S135C, the processor 210 obtains data for the print image PRI to be printed in S135. Since the print image PRI data is, for example, pre-generated and stored in the non-volatile storage device 230, the processor 210 obtains the print image PRI data from the non-volatile storage device 230.

[0158] Figure 20(B) shows an example of a print image PRI. The print image PRI includes additional objects to be printed (also called print object AOs). The print image PRI in Figure 20(B) does not contain any elements to be printed other than the print object AOs. The positional relationship of the print image PRI with respect to the reference image IMt is defined. That is, the position where the print object AOs of the print image PRI should be printed relative to the reference object Obt shown in the reference image IMt is defined. Specifically, in this modified example, the print image PRI has the same size (number of pixels in the vertical and horizontal directions) as the reference image IMt. The print object AOs of the print image PRI are printed relative to the reference object Obt of the reference image IMt in such a positional relationship that the four vertices of the rectangular print image PRI and the four vertices of the rectangular reference image IMt overlap. Figure 20(A) shows the positional relationship between the reference object Obt of the reference image IMt and the print object AO of the print image PRI by indicating the print object AO of the print image PRI with a dashed line. Figure 20(B) shows the positional relationship between the reference object Obt of the reference image IMt and the print object AO of the print image PRI, with the reference object Obt of the reference image IMt indicated by a dashed line.

[0159] In S140C, the processor 210 performs a nonlinear transformation on the print image PRI using the coordinate transformation map TM generated in S130C. This generates a nonlinearly transformed print image PRIu. Specifically, the processor 210 identifies the coordinates of the pixels in the print image PRI corresponding to each pixel of the nonlinearly transformed print image PRIu by referring to the coordinate transformation map TM. The processor 210 obtains the values ​​(RGB values) of the identified pixels from the print image PRI. This generates a nonlinearly transformed print image PRIu.

[0160] Figure 20(C) shows an example of a nonlinearly transformed print image PRIu. In Figure 20(C), the target object Obu with local distortion in the target image IMu (Figure 6(B)) is shown by a dashed line. As shown in Figure 20(C), in the nonlinearly transformed print image PRIu, the local distortion occurring in the target image IMu (Figure 6(B)) is added to the undistorted print image PRI (Figure 20(B)). In other words, the print object AOu of the nonlinearly transformed print image PRIu is transformed to match the target object Obu of the distorted target image IMu.

[0161] In the S150C, the processor 210 generates print data using the nonlinearly transformed print image PRIu data. Specifically, the processor 210 generates print data by executing known color conversion and halftone processing on the nonlinearly transformed print image PRIu. The color conversion process converts the value of each pixel in the print image PRIu data (RGB image data) from RGB color system color values ​​to CMYK color system color values, which are composed of ink colors, using a lookup table. Halftone processing is performed using known methods such as dithering and error diffusion.

[0162] In the S160C, the processor 210 performs printing using the generated print data. Specifically, the processor 210 generates a print job containing the print data and sends the print job to the garment printer. Upon receiving the print job, the garment printer prints the image onto the T-shirt 700 placed on the platen according to the print job.

[0163] Figure 20(D) shows the printing result on the T-shirt 700. As shown in Figure 20(D), the T-shirt 700 has the print object AOu of the nonlinearly transformed print image PRIu printed on top of the already printed image IMp. As shown in Figure 20(D), the positional relationship between the target object Obs already printed on the T-shirt 700 and the newly printed image (print object AOu) on the T-shirt 700 matches the defined positional relationship between the reference object Obt of the reference image IMt and the print object AO of the print image PRI.

[0164] According to the modified example described above, the processor 210 nonlinearly transforms the print image PRI using a deformation amount map VM (S140C in Figure 19). The processor 210 uses the data of the nonlinearly transformed print image PRIu to form a print image (print object AOs) on the T-shirt 700 (S150C-S160C in Figure 19). As a result, printing is performed so that the positional relationship between the target object Obs already printed on the T-shirt 700 and the newly printed print object AOu on the T-shirt 700 matches the defined positional relationship between the reference object Obt of the reference image IMt and the print object AO of the print image PRI. Therefore, even if the T-shirt 700 is placed in a state where local distortion has occurred, the print object AOu can be printed in a manner that is consistent with the distorted target object Obs on the T-shirt 700.

[0165] In this modified example, the formation of the nonlinearly transformed print image PRIu is performed using a garment printer, but the formation of the nonlinearly transformed image may be performed using other image-forming devices. Other image-forming devices may include, for example, a sewing machine that forms an image on fabric by embroidering on the fabric with thread according to embroidery data. Furthermore, if a sheet-like material such as resin or paper is used as the medium on which the image is formed instead of fabric such as a T-shirt, the image-forming device may be a cutting machine that expresses the image in the shape of the sheet-like material by cutting the sheet-like material with a blade.

[0166] (3) In the first embodiment described above, the deformation amount map VM is a map in which the local deformation amount LV is recorded for each pixel, with each pixel serving as a unit region. However, the unit region of the deformation amount map VM may be a region containing multiple pixels. For example, a region UT1 containing four pixels arranged in a 2x2 grid as shown in Figure 11(B) may be used as a unit region, and the local deformation amount LV may be recorded for each unit region. Alternatively, a region UT2 containing nine pixels arranged in a 3x3 grid as shown in Figure 11(B) may be used as a unit region, and the local deformation amount LV may be recorded for each unit region.

[0167] In this case, the number of unit regions in the deformation amount map VM is less than the number of pixels in the reference image IMt and the target image IMu. In this case, when the processor 210 creates the coordinate transformation map TM (Figure 12), it determines the local deformation amount of each pixel of the target image IMu to be transformed using one or more local deformation amounts LV for each unit region recorded in the deformation amount map VM, according to a predetermined interpolation algorithm. The processor 210 creates the coordinate transformation map TM using the determined local deformation amounts for each pixel, and performs a nonlinear transformation of the target image IMu using the coordinate transformation map TM. The predetermined interpolation algorithm may be any of the known algorithms. For example, the bicubic method, the bilinear method, or the nearest neighbor method can be used as the predetermined interpolation algorithm. This modification reduces the amount of memory used to store the deformation amount map VM and the computational cost required to generate the deformation amount map VM.

[0168] (4) The first and second embodiments described above may be combined. For example, if the inspection process of the second embodiment determines that there are no abnormalities (S470B in Figure 16), the processor 210 may use the deformation amount Mv determined in S440B in Figure 16 to execute the map generation process in Figure 10 to generate a deformation amount map VM. After that, the processor 210 may execute S134, S136, S150, and S160 in Figure 4. In this case, for example, abnormalities that could not be detected in the inspection process of the second embodiment can be detected using the inspection process method of the first embodiment. This can improve the accuracy of abnormality detection in the inspection process.

[0169] (5) In the first embodiment described above, a deformation amount map VM is generated using a reference image IMt used for printing on the T-shirt 700 and target images IMs obtained by optically reading the printed T-shirt 700. In the second embodiment described above, anomaly detection based on local template matching is performed using a reference image IMt used for printing on the T-shirt 700 and target images IMs obtained by optically reading the printed T-shirt 700. However, the method is not limited to these, and for example, deformation amount map VM generation and difference detection based on local template matching may be performed using various images.

[0170] For example, a deformation map VM may be generated using an examination image A (e.g., an X-ray or CT scan) obtained by photographing the human body in a first period, and an examination image B obtained by photographing the same human body in a second period later than the first period. In this case, due to subtle differences in the human body's posture at the time of photography, local deformation may occur in the object in image B relative to the object in image A (the human body). Even in this case, by correcting the local deformation of the object in image B and the object in image A using the deformation map VM, the alignment of the object in image B and the object in image A can be performed with high accuracy. As a result, for example, by taking the difference between the object in image B and the object in image A after alignment, differences in the objects (e.g., changes in the human body's condition) can be detected with high accuracy.

[0171] Alternatively, the detection processes of the first and second embodiments may be performed to detect abnormalities (scratches, damage) in easily deformable industrial products such as rubber or other elastic materials. In this case, for example, a deformation amount map VM may be generated using an image C obtained by photographing a normal product without abnormalities or an image based on a design drawing, and an image D obtained by photographing the product to be inspected. In this case, even if local deformation occurs in the product, the local deformation of the object (product) in image D can be corrected using the deformation amount map VM, thereby enabling accurate alignment of the object in image C and the object in image D. This allows for accurate detection of differences in objects (e.g., scratches or damage to the product) by taking the difference between the object in image C and the object in image D after alignment.

[0172] Furthermore, in each of the above embodiments, the object to be inspected is not limited to T-shirts 700, but may be various types of clothing (for example, various shirts such as polo shirts, coats, slacks, etc.). Also, the object to be inspected is not limited to clothing, but may be various fabric products such as bags, hats, handkerchiefs, etc. Also, the object to be inspected is not limited to fabric products, but may be products made from various materials such as paper, film, leather, etc.

[0173] Furthermore, the reference image and target image in each of the above embodiments may be reversed. For example, the inspection process in each of the above embodiments may be performed using an image obtained by optically reading the T-shirt 700 as the reference image and an image used for printing on the T-shirt 700 as the target image.

[0174] As can be seen from the above explanation, generally, deformation amount maps (VMs) can be generated and differences detected based on local template matching can be performed using a first image containing a specific object (such as a T-shirt 700, human body, or industrial product) and a second image containing the same specific object.

[0175] (6) The first alignment process in each of the above embodiments (S120 in Figures 4 and 14) may be omitted. The first alignment process is unnecessary if it can be guaranteed that the position and size of the reference object Obt in the reference image IMt and the target object Obs in the target image IMs are roughly the same, by strictly controlling the placement position of the object when optically reading the object to be inspected (T-shirt 700 or industrial product).

[0176] (7) In each of the above embodiments, the first type feature points Pc and the second type feature points Pe of the reference image IMt are determined by analyzing the reference image IMt according to a predetermined algorithm (for example, Harris corner detection). However, the processor 210 may also determine the first type feature points Pc and the second type feature points Pe of the reference image IMt based on instructions input by the user. For example, in the inspection process, the processor 210 displays a predetermined input screen including the reference image IMt on the display unit 240 and obtains instructions from the user via the input screen to specify the positions of the first type feature points Pc and the second type feature points Pe. The processor 210 may then determine the first type feature points Pc and the second type feature points Pe based on these instructions.

[0177] Alternatively, for example, before the inspection process, the user may create coordinate data indicating the positions of the first type feature point Pc and the second type feature point Pe, and store this coordinate data in the non-volatile storage device 230 in association with the reference image IMt. In this case, during the inspection process, the processor 210 may determine the first type feature point Pc and the second type feature point Pe by referring to the coordinate data already stored in the non-volatile storage device 230.

[0178] (8) In each of the above embodiments, template matching is used as the local matching process (S425 in Figures 8 and 16). Alternatively, so-called feature point matching may be used as the local matching process. In this case, for example, the processor 210 detects multiple feature points in the target local region LAu using a known method such as A-KAZE. The processor 210 calculates the feature quantity of the feature point of interest in the reference local region LAt and the feature quantities of multiple feature points in the target local region LAu. The processor 210 determines the feature point in the target local region LAu that is most similar to the feature quantity of the feature point of interest as the corresponding feature point. The processor 210 may also calculate the deformation amount Mv of the feature point of interest based on the coordinates of the feature point of interest and the corresponding feature point.

[0179] Furthermore, in each of the above embodiments, when performing template matching as a local matching process, the processor 210 uses the reference local region LAt as the template to perform template matching. Alternatively, the processor 210 may use the target local region LAu as the template to perform template matching.

[0180] (9) In the first embodiment described above, when generating the deformation amount map VM, the local deformation amount LV for each unit region (e.g., a pixel) is determined using the deformation amount Mv of each feature point Pc, Pe and the weight Wt (equation (1), etc.). However, the processor 210 may determine the local deformation amount LV for each unit region without using the weight Wt. For example, the processor 210 may adopt the deformation amount Mv of the feature point with the shortest distance from the unit region as the local deformation amount LV for that unit region.

[0181] (10) In the local matching process of the first embodiment described above, if it is determined that template matching has failed (S435:NO in Figure 8), the sizes of the local regions LAt and LAu are increased and the template matching is retried (S450, S455 in Figure 8). Alternatively, if it is determined that template matching has failed, the processor 210 may proceed immediately to S460 in Figure 8 without performing a retry.

[0182] (11) In the local matching process of the second embodiment described above, if it is determined that template matching has failed (S435:NO in Figure 16), the template matching retry is not performed. Alternatively, if the processor 210 determines that template matching has failed, it may increase the size of the local regions LAt and LAu and then retry the template matching, as in the first embodiment. The processor 210 may then determine the inspection result to be abnormal if template matching is not successful even after performing up to the upper limit number of retries Rmax.

[0183] (13) In the local matching process of the first embodiment described above, if it is determined that template matching has failed (S435:NO in Figure 8), the sizes of both local regions LAt and LAu are increased and the template matching is retried (S450, S455 in Figure 8). Alternatively, if it is determined that template matching has failed, the processor 210 may increase only the size of the reference local region LAt and maintain the size of the target local region LAu while retrying the template matching. Or, if it is determined that template matching has failed, the processor 210 may maintain the size of the reference local region LAt and increase only the size of the target local region LAu while retrying the template matching.

[0184] (14) In each of the above embodiments, both a first-kind feature point Pc located at a corner and a second-kind feature point Pe are determined as feature points for local template matching. However, the processor 210 may determine only the first-kind feature points Pc located at a corner if it can determine a sufficient number of feature points using only the first-kind feature points Pc located at a corner. Alternatively, the processor 210 may determine only the feature points Pe located at an edge without performing corner detection.

[0185] Furthermore, in each of the above embodiments, the processor 210 does not determine a Type 2 feature point Pe within the neighboring region NA containing the Type 1 feature point Pc (S330 in Figure 7, S330B in Figure 15). Alternatively, the processor 210 may determine a Type 2 feature point Pe in the vicinity of the Type 1 feature point Pc. Also, in each of the above embodiments, two or more Type 2 feature points Pe whose distance from each other is less than or equal to the threshold THr are aggregated into a single feature point (S340 and S350 in Figures 7 and 15). Alternatively, even if there are two or more Type 2 feature points Pe whose distance from each other is less than or equal to the threshold THr, the processor 210 may perform local template matching at each feature point without aggregating them into a single feature point.

[0186] (15) The image processing device 200 is not limited to a personal computer, but may be any other device (for example, a smartphone, a tablet computer, a control device incorporated into a reader, etc.). Alternatively, multiple devices (for example, computers) that can communicate with each other via a network may each share a portion of the data processing function of the image processing device and, as a whole, provide the data processing function (a system equipped with these devices corresponds to the image processing device).

[0187] (16) In the second embodiment described above, it is determined that there is an abnormality in the target object Obu within the target local region LAu when template matching between the reference local region LAt and the target local region LAu fails, and when the deformation amount Mv calculated using template matching between the reference local region LAt and the target local region LAu is the threshold THv (S435, S450B, S480B in Figure 16). However, the determination of whether or not there is an abnormality in the target object Obu within the target local region LAu may be made by other methods based on the results of template matching between the reference local region LAt and the target local region LAu. For example, the processor 210 may determine that there is an abnormality in the target object Obu when template matching between the judgment reference local region LAt and the target local region LAu fails, and may not perform a determination based on the deformation amount Mv. Alternatively, the processor 210 may determine that there is an abnormality in the target object Obu when the deformation amount Mv calculated using template matching between the reference local region LAt and the target local region LAu is the threshold THv, and may not perform a determination based on whether or not the template matching was successful. Alternatively, the processor 210 may determine whether or not there is an anomaly in the target object Obu by analyzing the score map of template matching between the reference local region LAt and the target local region LAu.

[0188] (17) The second embodiment can be realized in the following form. [Mode] A computer program, An image acquisition function that acquires a first image containing a specific object and a second image containing the same specific object. A feature point determination function that determines a plurality of feature points located at least one of the corners and edges of the specific object in the first image, A matching function that performs a matching process for each of the multiple feature points between a first local region of the first image containing the feature point and a second local region of the second image containing a corresponding point to the feature point, A difference detection function that detects the difference between the image in the first local region and the image in the second local region based on the results of the matching process, A computer program that enables a computer to realize something.

[0189] According to the above embodiment, for each of a plurality of feature points located at least one of the corners and edges of a specific object in the first image, a matching process is performed between a first local region of the first image containing the feature point and a second local region of the second image corresponding to the first local region. Based on the result of the matching process, the difference between the image in the first local region and the image in the second local region is detected. As a result, a local difference between a specific object in the first image and a specific object in the second image can be detected using local matching processing.

[0190] Furthermore, the above embodiments can be realized in various forms, for example, in the form of a method for detecting differences, a detection device, a computer program for realizing the functions of these methods or devices, a recording medium (for example, a non-temporary recording medium) on which the computer program is recorded, and so on.

[0191] In each of the above embodiments and modifications, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, feature point matching and template matching processes may be performed by dedicated hardware circuits such as Application Specific Integrated Circuits (ASICs).

[0192] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0193] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of symbols]

[0194] 100…Reading device, 110…Control device, 120…Transport device, 122…Position sensor, 130…Table, 140…Support unit, 180…Reading sensor, 200…Image processing device, 210…Processor, 215…Storage device, 220…Volatile storage device, 230…Non-volatile storage device, 240…Display unit, 250…Operation unit, 270…Communication interface, 900…Printing device, EI…Edge image, EIs…Processed edge image, IMc…Read image, IMd…Difference image, IMf, IMs, IMu…Target image, IMt…Reference image, LAt…Reference local region, LAu…Target local region, Obf, Obs, Obu…Target object, Obt…Reference object, PG…Computer program, PRI, PRIs, PRIu…Printed image, Pc…Type 1 feature point, Pe…Type 2 feature point, SM…Score map, TM…Coordinate transformation map, VM…Deformation amount map

Claims

1. It is a computer program, An image acquisition function that acquires a first image containing a specific object and a second image containing the same specific object. A feature point determination function that determines a plurality of feature points located at least one of the corners and edges of the specific object in the first image, A matching function that performs a matching process for each of the multiple feature points between a first local region of the first image containing the feature point and a second local region of the second image corresponding to the first local region, A map generation function that generates a deformation amount map showing the relative deformation amount between a specific object in the first image and a specific object in the second image for each of a plurality of unit regions in the first or second image, based on the results of the matching process performed for each of the plurality of feature points, A computer program that enables a computer to realize something.

2. A computer program according to claim 1, further, A computer is provided with a transformation function that linearly transforms at least one of the first image and the second image so that the difference in position and size between the specific object in the first image and the specific object in the second image is reduced. A computer program wherein, after the linear transformation, the matching function performs the matching process, and the map generation function generates the deformation amount map based on the results of the matching process.

3. A computer program according to claim 2, The aforementioned conversion function is From among the multiple first feature points of the first image and the multiple second feature points of the second image, a plurality of matching pairs consisting of the first feature point and the second feature point are determined based on a predetermined matching algorithm. Based on the aforementioned multiple matching pairs, a transformation matrix for performing the linear transformation is calculated. A computer program that performs a linear transformation on at least one of the first image and the second image using the transformation matrix.

4. A computer program according to claim 2, The aforementioned conversion function is Using one of the first image and the second image as a template, template matching is performed on the other of the first image and the second image. Based on the results of the template matching, a transformation matrix for performing the linear transformation is calculated. A computer program that performs a linear transformation on at least one of the first image and the second image using the transformation matrix.

5. A computer program according to claim 2, The aforementioned conversion function is The user provides correspondence information regarding the correspondence between the first image and the second image. Based on the aforementioned correspondence information, a transformation matrix for performing the linear transformation is calculated. A computer program that performs a linear transformation on at least one of the first image and the second image using the transformation matrix.

6. A computer program according to claim 1, further, A transformation function that uses the deformation amount map to transform at least one of the first image and the second image so that the position and shape of the specific object in the first image and the specific object in the second image match, After the conversion, a detection function is provided to detect the difference between the specific object in the first image and the specific object in the second image. A computer program that enables a computer to realize something.

7. A computer program according to claim 1, One of the first image and the second image is a read image obtained by optically reading the object on which the specific object is formed, The other of the first image and the second image is a reference image showing the specific object whose positional relationship with the third image is defined. The aforementioned computer program, further, A transformation function that transforms the third image using the deformation amount map, An image forming function that forms the third image on the object using the converted data of the third image, To make this a reality on a computer, The image forming function is a computer program that forms the third image on the object such that the positional relationship between the specific object formed on the object and the third image formed on the object matches the defined positional relationship between the specific object and the third image in the reference image.

8. A computer program according to claim 1, The process for generating the aforementioned deformation map is as follows: A process to calculate the deformation amount for each of the multiple feature points based on the results of the matching process for each of the multiple feature points, A process for calculating the deformation amount corresponding to the unit region, using a weight that is set to be larger the shorter the distance between the unit region and the feature point, and the deformation amount of the feature point. A computer program that includes [this].

9. A computer program according to claim 6 or 7, The number of unit regions in the deformation amount map is less than the number of pixels in the first image and the second image. The aforementioned conversion function is The amount of deformation for each pixel of the image to be transformed is determined according to a predetermined interpolation algorithm using one or more of the deformation amounts recorded in the deformation amount map. A computer program that performs the transformation of the image to be transformed using the determined transformation amount for each pixel.

10. A computer program according to claim 1, The feature point determination function is a computer program that determines a plurality of feature points of the first image by analyzing the first image according to a predetermined algorithm.

11. A computer program according to claim 1, The matching function described above is The nth matching process (where n is a natural number greater than or equal to 1) between the first local region and the second local region is performed. If it is determined that the nth matching process has failed, at least one of the first local region and the second local region is changed to a larger region than the one used during the nth matching process. A computer program that performs the (n+1)th matching process between the first local region and the second local region after modifying at least one of the first local region and the second local region.

12. A computer program according to claim 11, The matching function is a computer program that, if the nth matching attempt fails, changes both the first local region and the second local region to a larger region than the one used during the nth matching attempt.

13. A computer program according to claim 1, The feature point determination function determines a plurality of first-kind feature points located at the corners of the specific object. The matching function performs the matching process for each of the plurality of first type feature points. After the matching process is performed for each of the plurality of first type feature points, The feature point determination function determines a plurality of Type II feature points located on the edges of the specific object, The matching function is a computer program that performs the matching for each of the plurality of type 2 feature points.

14. A computer program according to claim 13, The feature point determination function is a computer program that does not determine the second type of feature point within a specific range that includes the first type of feature point for which the matching process was successful, but determines the second type of feature point outside the specific range.

15. A computer program according to claim 13, The aforementioned feature point determination function is, Identify multiple edge pixels that constitute the edge of the aforementioned specific object, A computer program that determines a plurality of second type feature points along a line formed by the plurality of edge pixels.

16. A computer program according to claim 15, The feature point determination function is a computer program that determines one of the second type feature points based on two or more of the second type feature points determined along the line, if the distance between two or more of the second type feature points is less than or equal to a threshold.

17. An image processing method, Image acquisition steps include obtaining a first image containing a specific object and a second image containing the same specific object. A feature point determination step of determining a plurality of feature points located at least one of the corners and edges of the specific object in the first image, A matching step is performed for each of the plurality of feature points, which involves matching a first local region of the first image containing the feature point with a second local region of the second image corresponding to the first local region. A map generation step generates a deformation amount map that shows the relative deformation amount between a specific object in the first image and a specific object in the second image for each of a plurality of unit regions in the first image or the second image, based on the results of the matching process performed for each of the plurality of feature points, An image processing method comprising:

18. An image processing device, An image acquisition unit that acquires a first image containing a specific object and a second image containing the same specific object, A feature point determination unit that determines a plurality of feature points located at least one of the corners and edges of the specific object in the first image, A matching unit performs a matching process for each of the multiple feature points between a first local region of the first image containing the feature point and a second local region of the second image corresponding to the first local region. A map generation unit generates a deformation amount map that shows the relative deformation amount between a specific object in the first image and a specific object in the second image for each of a plurality of unit regions in the first or second image, based on the results of the matching process performed for each of the plurality of feature points, An image processing device equipped with the following features.

Images