Program and data processing device

Abstract

The present invention performs alignment between a plurality of images.　A plurality of candidate feature point pairs, each consisting of a pair of a feature point in a read image and a feature point in a reference image, are acquired. A plurality of candidate feature point pairs satisfying a selection condition are selected as a plurality of feature point pairs. The selection condition includes a first condition for selecting a target combination which is a combination of N candidate feature point pairs (N is 2 or 3) as N feature point pairs. The first condition is determined using one or more types of parameters among the four types of parameters: the ratio of the length of two sides, the length of the sides, the interior angle, and the angle formed by the line segment connecting two feature points and the direction associated with one of the two feature points. A correspondence between the coordinates on the read image and the coordinates on the reference image is determined using the plurality of feature point pairs.

Description

Program and data processing device

[0001] This specification relates to registration between multiple images.

[0002] In various processes, alignment between multiple images can be performed. Patent Document 1 discloses a technology for detecting defects in an image formed on paper by an image forming device such as a printer or copier. In this technology, a marker image for position determination is formed on the same paper along with a job image instructed to be printed by a user. An image reading unit reads the paper surface to generate a read image. An inspection unit determines the position of the read image corresponding to the reference image based on each feature point of the job image and marker image extracted from the read image to be inspected and each feature point of the job image and marker image extracted from the reference image. The inspection unit compares the reference image and the read image after alignment and detects image areas where the difference in pixel values ​​is greater than or equal to a threshold as defects.

[0003] JP 2018-112440 A

[0004] Aligning multiple images is not easy and requires some ingenuity.

[0005] This specification discloses a technique for performing registration between multiple images.

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] A program including a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in a read image and feature points in a reference image, by using feature amounts of each of the plurality of feature points in the read image and feature amounts of each of the plurality of feature points in the reference image, and a pair selection function that selects, from the plurality of candidate feature point pairs, a plurality of candidate feature point pairs that satisfy a selection condition as a plurality of feature point pairs, wherein the selection condition is to select, as N feature point pairs, target combinations that are combinations of N (N is 2 or 3) candidate feature point pairs. a first condition for determining a correspondence between coordinates on the scanned image and coordinates on the reference image by using the plurality of feature point pairs, the first condition being determined using one or more of four parameters: a ratio of lengths of two sides of a triangle formed by three feature points; a side length of the triangle; an interior angle of the triangle; and an angle formed between a line segment connecting two feature points and a direction associated with one of the two feature points.

[0008] According to this configuration, a plurality of candidate feature point pairs that satisfy a selection condition are selected as a plurality of feature point pairs from a plurality of candidate feature point pairs, and the selection condition includes a first condition for selecting a target combination, which is a combination of N (N is 2 or 3) candidate feature point pairs, as N feature point pairs, and the first condition is determined using one or more of four types of parameters: the ratio of the lengths of the two sides of a triangle formed by the three feature points, the length of the side of the triangle, the interior angle of the triangle, and the angle between the line segment connecting the two feature points and the direction associated with the feature amount of one of the two feature points, thereby making it possible to properly align the read image with the reference image.

[0009] The technology disclosed in this specification can be realized in various forms, such as a data processing method and a data processing device, a computer program for realizing the functions of the method or device, a recording medium (e.g., a non-temporary recording medium) on which the computer program is recorded, and the like.

[0010] 14 is an explanatory diagram showing a data processing device according to an embodiment. It is a perspective view showing an example of a reading device 100. (A) is a diagram showing an example of an image represented by image data for printing. (B) is a diagram showing an example of a read image. It is a flowchart showing an example of an inspection process. It is a flowchart showing an example of a registration process. It is a flowchart showing an example of a feature point matching process. (A)-(D) are diagrams showing examples of images processed by feature point matching. It is a flowchart showing an example of a selection process. (A) and (B) are diagrams showing an example of a calculation of a representative color value. It is a flowchart showing an example of a selection process for a candidate feature point pair M. It is a diagram showing an example of a feature point group N. (A) and (B) are diagrams showing an example of a triangle formed by candidates MA, MB, and MC. It is a flowchart showing an example of a process for determining combination condition CC. It is a flowchart showing an example of a process for determining shape condition CW1. (A)-(D) are diagrams showing examples of the results of the process shown in FIG. 14. It is a diagram showing an example of an expression format for the coordinate correspondence relationship. (A) and (B) are diagrams showing examples of difference images. 13 is a flowchart illustrating an example of a process for acquiring reference image data.

[0011] A. First Embodiment: A1. Device Configuration: FIG. 1 is an explanatory diagram showing a data processing device as one embodiment. The data processing device 200 is, for example, a personal computer. The data processing device 200 executes an inspection process for a printed image. The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to process data, and is, for example, a central processing unit (CPU) or a system on a chip (SoC). The volatile storage device 220 is, for example, a dynamic random access memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores data for a first program 231 and a second program 232. The second program 232 is used in another embodiment described later.

[0013] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel overlaid on the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250.

[0014] The communication interface 270 is an interface for communicating with other devices (for example, it includes one or more of a USB interface, a wired LAN interface, an IEEE802.11 wireless interface, and an industrial camera interface (for example, CameraLink, CoaXPress, etc.)). In this embodiment, the communication interface 270 connects the reading device 100 and the printing device 900. The printing device 900 is a so-called inkjet printing device that prints an image on a printing medium such as cloth or paper by ejecting ink onto the printing medium. The reading device 100 optically reads the object to be read and generates read image data representing the object. In the following description, the printing medium is a T-shirt, and the reading device 100 reads the T-shirt on which an image is printed.

[0015] 2 is a perspective view showing an example of the reading device 100. In the figure, the first direction Da and the second direction Db indicate horizontal directions, and the third direction Dc indicates a vertically upward direction. The first direction Da and the second direction Db are perpendicular to each other.

[0016] In this embodiment, the reading device 100 includes a housing 190, a table 130, a support portion 140 fixed to the upper surface of the table 130, a conveying device 120, a reading sensor 180, and a control device 110. The control device 110, the conveying device 120, and the reading sensor 180 are fixed to the housing 190.

[0017] The support unit 140 is a plate-like member that forms a flat upper surface for supporting an object to be read (such a member is also called a platen). In the drawing, a T-shirt 700 having a printed image IMpp thereon is placed on the support unit 140.

[0018] The conveying device 120 is configured to convey the table 130 in a direction parallel to the second direction Db. The conveying device 120 may have various configurations. Although not shown, in this embodiment, the conveying device 120 includes a rail that supports the table 130 so that the table 130 is slidable in a direction parallel to the second direction Db, multiple pulleys, a belt that is wound around the multiple pulleys and has a portion fixed to the table 130, and an electric motor that rotates the pulley. As the electric motor rotates the pulley, the table 130 (and thus the support portion 140) moves in a direction parallel to the second direction Db. The conveying device 120 also includes a position sensor 122 (e.g., a rotary encoder) that detects the position of the table 130 on the conveying path.

[0019] The reading sensor 180 is disposed at a position higher than the support unit 140 and midway along the transport path PTh of the support unit 140. The reading sensor 180 includes a line sensor (for example, a contact image sensor (CIS) or a charge coupled device (CCD)) that is composed of multiple photoelectric conversion elements arranged in a direction intersecting the transport direction Db (in this embodiment, a direction Da perpendicular to the transport direction Db). The reading sensor 180 faces downward. The reading sensor 180 can read the portion of the object supported by the support unit 140 that is located below the reading sensor 180.

[0020] When reading T-shirt 700, reading device 100 transports table 130 in a direction parallel to second direction Db. Reading sensor 180 repeatedly reads T-shirt 700 during transportation. This allows reading sensor 180 to read almost the entire portion of T-shirt 700 that is supported by support portion 140.

[0021] The control device 110 is an electric circuit configured to control the conveying device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates read image data by controlling the conveying device 120 and the reading sensor 180.

[0022] A2. Printing Process: In this embodiment, an image is printed on a T-shirt 700 (FIG. 2). The printing of the image is performed, for example, as part of a T-shirt sales service. The T-shirt sales service may include on-demand printing. A customer places an order for on-demand printing with a service provider. In response to the customer's order, the service provider prints the image on the T-shirt 700 using image data provided by the customer.

[0023] FIG. 3A is a diagram showing an example of an image represented by image data for printing (referred to as target image IMp). In this embodiment, the data of target image IMp is bitmap data representing the color values ​​(here, the gradation values ​​(e.g., values ​​greater than or equal to zero and less than or equal to 255) of red R, green G, and blue B) of multiple pixels arranged in a matrix along a first direction Dx and a second direction Dy. In the example of FIG. 3A, target image IMp represents a background BG and four rectangular objects OB1-OB4. Objects OB1-OB4 are represented in different colors. For example, the color of the first object OB1 is red, the color of the second object OB2 is green, the color of the third object OB3 is blue, and the color of the fourth object OB4 is gray. In this way, target image IMp may include multiple objects having different hues.

[0024] Although not shown in the drawings, the target image IMp is printed using the data processing device 200 and the printing device 900. Alternatively, the target image IMp may be printed using another device.

[0025] Various errors may occur during printing of the target image IMp. The printed image may have various defects due to errors. For example, a portion of the printed image may be missing due to an ink ejection error. The data processing device 200 detects defects in the printed image through an inspection process described below. In this embodiment, the data of the target image IMp is used in the inspection process. After printing the target image IMp, the data of the target image IMp is stored in the storage device 215 (e.g., the non-volatile storage device 230) of the data processing device 200 for the inspection process.

[0026] A3. Inspection Process: FIG. 4 is a flowchart illustrating an example of the inspection process. For inspection, the T-shirt 700 is placed on the support unit 140 of the reading device 100 (FIG. 2) so that the printed image IMpp is visible. In this embodiment, an operator places the T-shirt 700 on the support unit 140. Alternatively, a machine (e.g., a robot arm) may place the T-shirt 700 on the support unit 140. After the T-shirt 700 is placed, an instruction to start the inspection process is input to the data processing device 200 (FIG. 1). In this embodiment, the operator inputs the instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the start instruction. Note that the start instruction may be input to the data processing device 200 via the communication interface 270 by a device other than the data processing device 200.

[0027] The processor 210 of the data processing device 200 executes the inspection process in accordance with the first program 231. In S110, the T-shirt 700 is photographed. The processor 210 supplies a reading instruction to the reading device 100. The control device 110 of the reading device 100 reads the T-shirt 700 by controlling the reading sensor 180 and the conveying device 120 in response to the reading instruction. The control device 110 generates read image data representing the read T-shirt 700. The processor 210 of the data processing device 200 acquires the read image data from the control device 110 of the reading device 100 and stores the acquired read image data in the storage device 215 (e.g., the non-volatile storage device 230).

[0028] FIG. 3B is a diagram showing an example of a scanned image. In this embodiment, the data of the scanned image IMs is bitmap data representing the color values ​​(here, the gradation values ​​(e.g., values ​​greater than or equal to zero and less than or equal to 255) of red R, green G, and blue B) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. The scanned image IMs in the figure represents a portion of the T-shirt 700 that includes the printed image IMpp. Here, the printed image IMpp is assumed to be an image obtained by printing the target image IMp (FIG. 3A). The printed image IMpp, like the target image IMp, represents the background BG and objects OB1-OB4.

[0029] In S120 (FIG. 4), the processor 210 aligns the reference image with the scanned image. In this embodiment, the target image IMp (FIG. 3A) is used as the reference image (hereinafter, the target image IMp will also be referred to as the reference image IMt).

[0030] The orientation of the T-shirt 700 relative to the reading sensor 180 may be in various directions. Therefore, the orientation of the objects OB1-OB4 in the image (i.e., the angle of rotation of the objects) may differ between the reference image IMt and the read image IMs. Furthermore, the pixel density (also called resolution) for the same object may differ between the reference image IMt and the read image IMs. In other words, the scale for the object may differ between the reference image IMt and the read image IMs.

[0031] 5 is a flowchart illustrating an example of the alignment process. In this embodiment, the processor 210 obtains a plurality of pairs of feature points by feature point matching, and determines a correspondence between coordinates on the reference image IMt and coordinates on the scanned image IMs using the obtained plurality of pairs.

[0032] In S220, the processor 210 performs feature point matching. FIG. 6 is a flowchart illustrating an example of the feature point matching process. In S305, the processor 210 generates a gray reference image IMtg and a gray scanned image IMsg by grayscale conversion of the reference image IMt and the scanned image IMs. A known relationship can be used as the correspondence between color gradation values ​​and grayscale gradation values ​​(for example, the correspondence between RGB values ​​in an RGB color space and a luminance value Y in a YCbCr color space).

[0033] In S310, the processor 210 extracts feature points Tt from the gray reference image IMtg. FIGS. 7A-7D are diagrams showing examples of images processed by feature point matching. FIG. 7A shows examples of feature points detected from images IMtg and IMsg. Each of the multiple black dots on the gray reference image IMtg represents a feature point Tt (also referred to as a reference feature point Tt) detected from the gray reference image IMtg. As shown in the figure, points indicating characteristic portions of an object, such as corners or edges, are detected as feature points Tt. Such feature points Tt are also called key points. Although not shown, in practice, many more feature points Tt may be detected (e.g., several tens or several hundreds). The reference feature points Tt detected from the gray reference image IMtg correspond to feature points indicating the same portions at the same coordinates on the reference image IMt (FIG. 3A).

[0034] Various methods may be used to detect feature points. In this embodiment, a technique called Accelerated-KAZE (A-KAZE) is used, which detects key points and calculates feature descriptors for each key point. The A-KAZE technique is disclosed, for example, in the following paper: "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013"

[0035] The processor 210 detects a plurality of feature points Tt by analyzing the gray reference image IMtg according to the A-KAZE technique.

[0036] In S315 (FIG. 6), the processor 210 extracts a plurality of feature points Ts from the gray-scanned image IMsg. The plurality of black dots on the gray-scanned image IMsg in FIG. 7A each represent a feature point Ts detected from the gray-scanned image IMsg (also referred to as a read feature point Ts). The processor 210 detects a plurality of feature points Ts by analyzing the gray-scanned image IMsg according to the A-KAZE technique. Although not shown, in practice, many more feature points Ts may be detected (for example, several tens or several hundreds). The read feature points Ts detected from the gray-scanned image IMsg correspond to feature points indicating the same portions at the same coordinates on the read image IMs (FIG. 3B).

[0037] In S320 (FIG. 6), the processor 210 calculates a feature value Ft for each of the multiple reference feature points Tt. The feature value Ft may be various information describing the features of the feature point Tt. For example, the feature value Ft is calculated so that it changes depending on the distribution of color values ​​of multiple pixels surrounding the feature point Tt. In this embodiment, the processor 210 uses the gray reference image IMtg to calculate an A-KAZE feature descriptor as the feature value Ft. In the A-KAZE technique, the feature descriptor is rotation-invariant. To obtain a rotation-invariant feature descriptor, the orientation of the feature point is also calculated. The orientation of the feature point indicates the direction of the brightness gradient in a neighborhood centered on the feature point (also called the gradient direction or dominant direction). The processor 210 calculates the feature value Ft and the orientation of the reference feature point Tt.

[0038] In S325, the processor 210 calculates the feature value Fs for each of the plurality of read feature points Ts. In this embodiment, the processor 210 uses the gray read image IMsg to calculate the feature descriptor (i.e., the feature value Fs) and the direction of the read feature points Ts according to the A-KAZE technique.

[0039] The outline of the processing after S325 is as follows: The processor 210 calculates the distance dF between the feature amounts Ft and Fs for all combinations of the reference feature point Tt and the read feature point Ts (S350). Then, the processor 210 acquires the pair of feature points Tt and Ts showing the distance dF less than the distance threshold dFth as the candidate feature point pair M (S355: Yes, S360).

[0040] Specifically, the process is as follows: The processor 210 executes loop processing S330 (including S335-S360) between start L31s and end L31e for each of the multiple reference feature points Tt. In S335, the processor 210 selects the unprocessed reference feature point Tt as the target feature point Tti, which is the feature point to be processed. The processor 210 executes loop processing S340 (including S345-S360) between start L32s and end L32e for each of the multiple read feature points Ts. In S345, the processor 210 selects the unprocessed read feature point Ts as the target feature point Tsj, which is the feature point to be processed.

[0041] In S350, the processor 210 calculates the distance dF between the two feature quantities Fti and Fsj of the two interest feature points Tti and Tsj. The distance dF is calculated so that a small distance dF indicates a high degree of similarity between the two feature quantities Fti and Fsj. A high degree of similarity (i.e., a small distance dF) indicates that the two interest feature points Tti and Tsj represent similar portions of the two images IMtg and IMsg (e.g., corresponding portions of the same object). In this embodiment, the feature quantities Ft and Fs are A-KAZE feature descriptors and are represented by binary vectors (vectors consisting of one or more binary elements). In this case, the processor 210 may calculate the Hamming distance as the distance dF.

[0042] In S355, the processor 210 determines whether the distance dF is less than the distance threshold dFth. If the distance dF is small, the feature points Tti and Tsj are likely to represent similar portions of the images IMt and IMs (e.g., the same portion of the same object). If the distance dF is less than the distance threshold dFth (S355: Yes), in S360, the processor 210 acquires the pair of target feature points Tti and Tsj as a candidate feature point pair M (also referred to as a candidate feature point pair M). A feature point pair is a pair of a reference feature point Tt and a read feature point Ts used to determine the correspondence between coordinates. The corresponding feature points Tt and Ts are also referred to as a matching pair. After S360, the processor 210 ends the loop processing S340 for the target feature point Tsj. If the distance dF is equal to or greater than the distance threshold dFth (S355: No), the processor 210 skips S360 and ends the loop process S340 for the target feature point Tsj. The processor 210 then repeats the loop process S340 and the loop process S330 to perform the processes of S350-S360 for each of a plurality of combinations of the target feature point Tsj and the target feature point Tti.

[0043] 7(B) shows an example of a candidate feature point pair M. Each of the multiple lines RL in the figure indicates a candidate feature point pair M. Each line RL connects feature points Tt and Ts that form the candidate feature point pair M. As shown in the figure, a pair of feature points Tt and Ts that indicate the same part may be selected as the candidate feature point pair M. For example, feature point Tt1 that indicates the upper right corner of the first object OB1 in the gray reference image IMtg may be associated with feature point Ts1 that indicates the upper right corner of the first object OB1 in the gray read image IMsg.

[0044] Furthermore, a pair of feature points Tt, Ts indicating mutually different portions may be selected as a candidate feature point pair M. For example, feature point Tt1 indicating the upper right corner of a first object OB1 in the gray reference image IMtg may be associated with feature point Ts2 indicating the upper right corner of a third object OB3 in the gray read image IMsg. Each of rectangular objects OB1-OB4 has four corners. These corners are locally similar. As a result, feature point Tt indicating one corner of an object in the gray reference image IMtg may be associated with feature point Ts indicating another corner of the same object in the gray read image IMsg, or with feature point Ts indicating a corner of another object. In this way, when each image IMtg and IMsg represents multiple locally similar portions, feature points Tt, Ts indicating mutually different portions may be associated.

[0045] Furthermore, in this embodiment, as described above, the feature quantities Ft and Fs are rotation-invariant. When feature quantities are rotation-invariant, similar feature quantities can be calculated from the same portion of an object regardless of the angle of rotation of the object in the image. By using the distance dF between the rotation-invariant feature quantities Ft and Fs, the processor 210 can match pairs of feature points representing the same portion of the object even if the rotation angles of the same object are different between the two images IMtg and IMsg. However, if each image IMtg and IMsg represents multiple locally similar portions, feature points Tt and Ts representing different portions may be matched. For example, feature point Tt1 representing the upper right corner of the first object OB1 in the gray reference image IMtg may be matched with feature point Ts3 representing the lower right corner of the first object OB1 in the gray read image IMsg.

[0046] Note that the larger the distance threshold dFth ( FIG. 6 : S355), the larger the total number of suitable candidate feature point pairs M may be. However, the total number of inappropriate candidates M may also be larger. The larger the total number of suitable candidates M, the smaller the error in the coordinate correspondence, which will be described later. If the total number of inappropriate candidates M is large, the error in the coordinate correspondence may increase. The distance threshold dFth may be determined experimentally in advance so that the error in the coordinate correspondence is tolerable.

[0047] When processing of all combinations of reference feature points Tt and read feature points Ts has been completed, in S365 the processor 210 stores data representing multiple candidate feature point pairs M in the storage device 215 (e.g., the non-volatile storage device 230). Then, the processor 210 ends the processing in Fig. 6, i.e., the processing of S220 in Fig. 5.

[0048] In S230, the processor 210 executes a feature point pair selection process using color information. This selection process uses color information to select an appropriate candidate M from a plurality of candidate feature point pairs M (i.e., the candidate feature point pair M is verified). Fig. 8 is a flowchart showing an example of the selection process. The processor 210 executes a loop process S410 (including S420-S465) between start L4s and end L4e for each of the plurality of candidate feature point pairs M.

[0049] In S420, the processor 210 selects the unprocessed candidate M as a candidate of interest Mi, which is the candidate to be processed.

[0050] In S425, the processor 210 calculates a representative color value Ct of the first partial region Pt including the reference feature point Tt of the focused candidate Mi (the representative color value Ct is also referred to as the reference representative color value Ct). FIGS. 9A and 9B are diagrams showing an example of calculating a representative color value. FIG. 9A shows a portion of the reference image IMt including the feature point Tt. The figure also shows the first partial region Pt including the feature point Tt. The processor 210 calculates the reference representative color value Ct using multiple color values ​​of multiple pixels in the first partial region Pt. The method for calculating the representative color value Ct may be any of various methods for calculating a color representative of the first partial region Pt. In this embodiment, the processor 210 calculates the average value of each of the RGB components in the first partial region Pt as the gradation value of each of the RGB components of the representative color value Ct. Note that various summary statistics (e.g., median, mode, etc.) representing the magnitude of the color value may be used instead of the average value. Furthermore, the configuration of the first partial region Pt (specifically, the shape of the first partial region Pt and the relative position of the first partial region Pt with respect to the position of the feature point Tt) may be various configurations that enable calculation of the representative color value Ct representing the color of the portion indicated by the feature point Tt. In this embodiment, the first partial region Pt is a region centered on the feature point Tt, for example, a region of P rows and Q columns and P*Q pixels. In order to reduce the dependency of the representative color value Ct on the angle of rotation of the object, P=Q is preferable (for example, P=Q=5). Instead of a region of P rows and Q columns, the first partial region Pt may be a region whose distance from the feature point Tt is equal to or less than a distance threshold.

[0051] In S430 (FIG. 8), the processor 210 calculates a representative color value Cs of the second partial region Ps including the read feature point Ts of the target candidate Mi (the representative color value Cs is also referred to as the read representative color value Cs). FIG. 9B shows a portion of the read image IMs including the feature point Ts. The figure also shows the second partial region Ps including the feature point Ts. The configuration of the second partial region Ps is the same as the configuration of the first partial region Pt. The second partial region Ps is an area of ​​P rows and Q columns, P*Q pixels, centered on the feature point Ts. The calculation method of the read representative color value Cs is the same as the calculation method of the reference representative color value Ct. The processor 210 calculates the average values ​​of red (R), green (G), and blue (B) within the second partial region Ps as the read representative color value Cs.

[0052] In S435 (FIG. 8), the processor 210 calculates a first hue Ht, a first saturation St, and a first luminance Vt from the reference representative color value Ct, and calculates a second hue Hs, a second saturation Ss, and a second luminance Vs from the read representative color value Cs. A known relationship can be used as the correspondence between the representative color value, hue, saturation, and luminance (e.g., the correspondence between RGB values ​​in an RGB color space and HSV values ​​in an HSV color space). The first hue Ht, the first saturation St, and the first luminance Vt are each examples of representative color values ​​for the first partial region Pt, similar to the reference representative color value Ct. The second hue Hs, the second saturation Ss, and the second luminance Vs are each examples of representative color values ​​for the second partial region Ps, similar to the read representative color value Cs.

[0053] In S440, the processor 210 determines whether a first saturation condition CSt, which indicates that the first saturation St is higher than the saturation threshold Sth, is satisfied. If the first saturation St is higher than the saturation threshold Sth (S440: Yes), the processor 210 determines in S445 whether a hue condition CH, which indicates that the absolute value of the difference between the first hue Ht and the second hue Hs is less than a hue difference threshold dHth, is satisfied. The absolute value of the difference between the hues Ht and Hs (also referred to as the hue difference dH) is the value corresponding to the smaller angle difference between the first hue Ht and the second hue Hs on the hue circle. If the target candidate Mi is an appropriate pair of feature points Tt and Ts that indicate the same part, the hue difference dH may be a small value. If the hue difference dH is less than the hue difference threshold dHth (S445: Yes), in S460, the processor 210 selects the focused candidate Mi as a candidate to be retained. If the hue difference dH is equal to or greater than the hue difference threshold dHth (S445: No), the hues may differ significantly between the first partial region Pt and the second partial region Ps. In other words, the feature points Tt and Ts of the focused candidate Mi are likely to represent different parts. In S465, the processor 210 excludes the focused candidate Mi from the candidate feature points pairs.

[0054] Thus, when the first saturation St is higher than the saturation threshold Sth (S440: Yes), the processor 210 excludes from the candidates a focused candidate Mi having a hue difference dH equal to or greater than the hue difference threshold dHth. FIG. 7C shows an example of a candidate feature point pair M remaining after the processing of FIG. 8. A pair of feature points Tt and Ts indicating portions with different hues may be excluded. For example, the hues differ between the first object OB1 and the third object OB3. The line RLa shown in FIG. 7B indicates a pair of the reference feature point Tt of the first object OB1 and the read feature point Ts of the third object OB3. As shown in FIG. 7C, this pair is excluded (FIG. 8: S440 (Yes), S445 (No), S465).

[0055] If the first saturation St is equal to or less than the saturation threshold Sth (S440: No), then in S450, the processor 210 determines whether a second saturation condition CSs is satisfied, which indicates that the second saturation Ss is higher than the saturation threshold Sth. If the second saturation Ss is higher than the saturation threshold Sth (S450: Yes), then the saturation may differ significantly between the first partial region Pt and the second partial region Ps. That is, the feature points Tt and Ts of the focused candidate Mi are likely to represent different portions. In S465, the processor 210 excludes the focused candidate Mi from the candidates for the feature point pair.

[0056] If the second saturation Ss is equal to or less than the saturation threshold Sth (S450: No), in S455, the processor 210 determines whether a luminance condition CV is satisfied, indicating that the absolute value of the difference between the first luminance Vt and the second luminance Vs is less than the luminance difference threshold dVth. If the focused candidate Mi is an appropriate pair of feature points Tt and Ts that represent the same part, the absolute value of the difference between the luminance Vt and Vs (also referred to as the luminance difference dV) may be small. If the focused candidate Mi is an inappropriate pair of feature points Tt and Ts that represent different parts, the luminance difference dV may be large. In this way, even if the saturations St and Ss are low, the luminances Vt and Vs can appropriately represent the color of the part represented by the feature points Tt and Ts.

[0057] If the brightness difference dV is less than the brightness difference threshold dVth (S455: Yes), the processor 210 selects the focused candidate Mi as a candidate to be retained in S460. If the brightness difference dV is equal to or greater than the brightness difference threshold dVth (S455: No), the processor 210 excludes the focused candidate Mi from the candidate feature points pair M in S465.

[0058] Note that the more lenient the conditions for retaining the focused candidate Mi, the greater the total number of suitable candidates M that remain without being excluded. However, the total number of inappropriate candidates may also be greater. The greater the total number of suitable candidates, the smaller the error in the coordinate correspondence, which will be described later. If the total number of inappropriate candidates is large, the error in the coordinate correspondence may increase. The parameters Sth, dHth, and dVth used in S440-S455 may be experimentally determined in advance so that the error in the coordinate correspondence is tolerable.

[0059] For example, the larger the hue difference threshold dHth (S445), the more lenient the conditions for retaining the focused candidate Mi. Even if the focused candidate Mi is an appropriate pair of feature points Tt, Ts that represent a highly saturated color portion, the hue difference dH may be a value greater than zero. A hue difference threshold dHth greater than zero allows such a hue difference dH. The hue difference threshold dHth may be set to a value greater than zero and smaller than the maximum possible value of the hue difference dH.

[0060] Furthermore, the larger the brightness difference threshold dVth (S455), the more lenient the conditions for retaining the focused candidate Mi. Even if the focused candidate Mi is an appropriate pair of feature points Tt, Ts that represent a low-saturation color portion, the brightness difference dV may be a value greater than zero. A brightness difference threshold dVth greater than zero allows such a brightness difference dV. The brightness difference threshold dVth may be set to a value greater than zero and smaller than the maximum possible value of the brightness difference dV.

[0061] Furthermore, if the saturation threshold value Sth (S440) is small, a focused candidate Mi with a low first saturation St is processed in S445. Even if the reference representative color value Ct ( FIG. 9A ) is a chromatic color, if the first saturation St is low, the hue of the corresponding region of the scanned image IMs representing the printed image is likely to differ from the first hue Ht. That is, if the first saturation St is low, the error in the second hue Hs and, ultimately, the error in the hue difference dH may be large. If the error in the hue difference dH is large, an appropriate focused candidate Mi may be erroneously excluded (S445: No), and an inappropriate focused candidate Mi may be erroneously retained (S445: Yes). As a result, the error in the coordinate correspondence may increase. The saturation threshold value Sth may be experimentally determined in advance so as to mitigate the influence of the error in the hue difference dH on the error in the coordinate correspondence. The saturation threshold Sth may be set to a value greater than zero and less than the maximum possible value of saturation.

[0062] After S460 or S465, the processor 210 proceeds to S420 and executes the loop process S410 for the next focused candidate Mi. When the loop process S410 for all candidates M has been completed, in S470 the processor 210 stores data representing the candidate feature points pairs M to be retained in the storage device 215 (e.g., the non-volatile storage device 230). Then, the processor 210 ends the process of FIG. 8, i.e., the process of S230 in FIG. 5.

[0063] In S240, the processor 210 executes a selection process for a candidate feature point pair M using a combination of multiple candidates M. This selection process selects an appropriate candidate M from the remaining multiple candidates M using the combination of multiple candidates M (i.e., the candidate feature point pair M is verified).

[0064] FIG. 10 is a flowchart illustrating an example of a process for selecting candidate feature point pairs M. The process in FIG. 10 is outlined as follows: The processor 210 selects three candidate feature point pairs MA, MB, and MC (S520-S560). The second candidate MB and the third candidate MC are selected from the area between a first radius r1 and a second radius r2, centered on the reference feature point Tt of the first candidate MA (details will be described later). Hereinafter, the selected combination of candidates MA, MB, and MC will also be referred to as the candidate combination MU. The processor 210 uses the candidate combination MU to determine whether the combination condition CC is satisfied (S565). If the combination condition CC is satisfied (S570: Yes), the processor 210 selects the candidate combination MU as the target combination MT (S573) and uses the target combination MT to determine whether the shape condition CW1 is satisfied (S575). If the shape condition CW1 is satisfied (S580: Yes), the processor 210 selects three candidates MA, MBA, and MC of the target combination MT as candidates to be retained (S585). In this embodiment, the multiple candidate feature point pairs M retained by the process of FIG. 10 are used to determine the correspondence between coordinates.

[0065] Specifically, the process is as follows: The processor 210 executes the loop process S510 (including S520-S585) between the start L51s and the end L51e for each of the multiple candidate feature points pairs M. In S520, the processor 210 selects an unprocessed candidate M as a first candidate MA.

[0066] In S525, the processor 210 selects a group of feature points N within a range between a first radius r1 and a second radius r2, with the feature point Tt of the first candidate MA at its center. FIG. 11 is a diagram showing an example of the group of feature points N. The figure shows a reference image IMt and multiple reference feature points Tt. One feature point Tt is selected as the first candidate MA. The first circle C1 is a circle with a first radius r1 and centered on the feature point Tt of the first candidate MA. The second circle C2 is a circle with a second radius r2 and centered on the feature point Tt of the first candidate MA. The selection range SR is the area between the first circle C1 and the second circle C2 (including the portions on the circles C1 and C2). The processor 210 selects the multiple feature points Tt included in the selection range SR as the group of feature points N.

[0067] As will be described later, the processor 210 uses the three reference feature points Tt and the three read feature points Ts of the three candidates MA, MB, and MC to determine whether the combination of the candidates MA, MB, and MC should be retained. Specifically, the processor 210 determines that the combination of the three candidates MA, MB, and MC should be retained if certain conditions are met, including a small difference between the shape of the triangle formed by the three reference feature points Tt and the shape of the triangle formed by the three read feature points Ts. If the length of any of the three sides of the triangle is significantly longer or shorter than the other sides, the error in the shape comparison between the two triangles may increase. The candidates MB and MC to be combined with the first candidate MA are selected from the group of feature points N. The selection range SR (here, radii r1 and r2) is experimentally determined in advance so that the combination of the candidates MA, MB, and MC forms a shape suitable for judgment. For example, the first radius r1 may be 1% or more and 20% or less of the size of the reference image IMt (e.g., the length in the first direction Dx or the second direction Dy), and the second radius r2 may be 30% or more and 70% or less of the size of the reference image IMt.

[0068] After S525 (FIG. 10), the processor 210 selects a second candidate MB from the feature point group N (S540) and selects a third candidate MB (S560). Specifically, the processor 210 executes the loop process S530 (including S540-S585) between start L52s and end L52e for each of the multiple candidates M in the feature point group N. In S540, the processor 210 selects an unprocessed candidate M from the feature point group N as the second candidate MB. Hereinafter, the second candidate MB is assumed to be the j-th candidate in the feature point group N (also referred to as second candidate MB[j]). The index j is selected, for example, from a range of zero or greater and less than the total number NN of candidates M in the feature point group N. After S540, the processor 210 executes a loop process S550 (including S560-S585) between start L53s and end L53e for each of the multiple candidates M of the feature point group N. In S560, the processor 210 selects an unprocessed candidate M from the feature point group N as the third candidate MC. Hereinafter, the third candidate MC is assumed to be the kth candidate among the feature point group N (represented as the third candidate MC[k]). The index k is selected, for example, from a range equal to or greater than j+1 and less than the total number NN of candidates M of the feature point group N. As a result, a candidate M different from the second candidate MB is selected as the third candidate MC. In this way, the processor 210 selects a combination of three candidates MA, MB, and MC (i.e., a candidate combination MU).

[0069] In S565, the processor 210 uses the candidate combination MU to determine whether a combination condition CC is satisfied. The combination condition CC is a condition for selecting the candidate combination MU as a target combination MT for determining the shape condition CW1, which will be described later. The determination of the combination condition CC is performed using a triangle formed by the candidates MA, MB, and MC of the candidate combination MU.

[0070] 12(A) and 12(B) are diagrams showing examples of triangles formed by candidates MA, MB, and MC. Each candidate MA, MB, and MC represents a pair of a reference minutiae point Tt and a read minutiae point Ts. FIG. 12(A) shows a triangle TRt formed by reference minutiae points Tta, Ttb, and Ttc of candidates MA, MB, and MC (referred to as reference triangle TRt). FIG. 12(B) shows a triangle TRs formed by read minutiae points Tsa, Tsb, and Tsc of candidates MA, MB, and MC (referred to as read triangle TRs). The positions of one or more minutiae may differ between the reference triangle TRt and the read triangle TRs.

[0071] Each figure shows symbols indicating the sides, side lengths, and interior angle sizes of a triangle. Symbols beginning with S (e.g., Sabt) indicate sides. The two letters following the S indicate two candidates M connected by the side (a, b, and c indicate candidates MA, MB, and MC, respectively). The suffix t or s of the symbol indicates the reference triangle TRt or the read triangle TRs, respectively. For example, the side Sabt ( FIG. 12(A) ) is the side connecting the reference feature points Tta and Ttb of the reference triangle TRt. The symbol obtained by replacing the leading S of the symbol indicating a side with L indicates the length of that side. For example, the length Labt indicates the length of the side Sabt. Symbols beginning with A (e.g., Aat) indicate the size of the interior angle of a triangle. The single letter following the A indicates the candidate M that forms the vertex of the interior angle (a, b, and c indicate candidates MA, MB, and MC, respectively). The suffix t or s at the end of the symbol indicates the reference triangle TRt or the read triangle TRs, respectively. For example, the interior angle Aat (FIG. 12(A)) indicates the size of the interior angle of the reference triangle TRt, with the reference minutiae Tta as its vertex. Symbols based on the above rules will also be used in other figures described later. Note that the direction Oct in FIG. 12(A) indicates the direction of the minutiae Ttc (FIG. 6: S320), and the direction Ocs in FIG. 12(B) indicates the direction of the minutiae Tsc (FIG. 6: S325).

[0072] FIG. 13 is a flowchart showing an example of the process for determining the combination condition CC (FIG. 10: S565). FIG. 13 shows the process for determining whether one triangle satisfies the individual combination condition. The processor 210 executes the process of FIG. 13 for each of the reference triangle TRt (FIG. 12(A)) and the read triangle TRs (FIG. 12(B)). If both the reference triangle TRt and the read triangle TRs satisfy the individual combination condition, the processor 210 determines that the combination condition CC is satisfied. The process of FIG. 13 will be described below using the reference triangle TRt as an example. In the description of the process of FIG. 13, symbols without the final letter (t or s) will be used to indicate the sides, the lengths of the sides, and the size of the interior angles.

[0073] In S610, the processor 210 determines whether the three candidates MA, MB, and MC are different from one another. If two or more candidates are the same candidate (S610: No), in S640, the processor 210 determines that the individual combination condition is not satisfied, and ends the processing of FIG. 13.

[0074] If the three candidates MA, MB, and MC are different from one another (S610: Yes), in S615, the processor 210 determines whether a first length condition CL1 is satisfied. The first length condition CL1 indicates that the first length Lab is within an allowable length range LR, which is greater than a lower limit Lth1 and less than an upper limit Lth2 (the allowable length range LR is also simply referred to as the length range LR). If the first length Lab is outside the length range LR (S615: No), the processor 210 proceeds to S640. The length range LR (here, the lower limit Lth1 and the upper limit Lth2) is experimentally determined in advance, similar to the radii r1 and r2 of the selection range SR (FIG. 11), so that a combination of the three candidates MA, MB, and MC forms a figure suitable for determination. For example, the lower limit Lth1 may be equal to the first radius r1, and the upper limit Lth2 may be equal to the second radius r2.

[0075] If the first length Lab is within the length range LR (S615: Yes), the processor 210 determines in S620 whether a second length condition CL2 is satisfied, which indicates that the second length Lac is within the length range LR. If the second length Lac is outside the length range LR (S620: No), the processor 210 proceeds to S640.

[0076] If the second length Lac is within the length range LR (S620: Yes), the processor 210 determines in S625 whether the first interior angle condition CA is satisfied. The first interior angle condition CA indicates that the interior angle Aa is within an allowable interior angle range AR, which is greater than the lower limit Ath1 and less than the upper limit Ath2 (the allowable interior angle range AR is also simply referred to as the interior angle range AR). If the interior angle Aa is outside the interior angle range AR (S625: No), the processor 210 proceeds to S640. The interior angle range AR (here, the lower limit Ath1 and the upper limit Ath2) is experimentally determined in advance, similar to the radii r1 and r2 of the selection range SR (FIG. 11), so that a suitable figure is formed by combining the three candidates MA, MB, and MC. For example, the lower limit Ath1 may be a value between 10 degrees and 60 degrees. The upper limit Ath2 may be a value not less than 90 degrees and not more than 160 degrees.

[0077] If the interior angle Aa is within the interior angle range AR (S625: Yes), in S630, the processor 210 determines whether the scalene condition CQ is satisfied. The scalene condition CQ indicates that the lengths of the three sides of a triangle (here, the reference triangle TRt) are different from one another. That is, the scalene condition CQ indicates that the shape of the triangle is different from any of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle. An isosceles triangle and an equilateral triangle maintain their original shape even if two sides of the same length are swapped. If the reference triangle TRt is an isosceles triangle or an equilateral triangle, even if two appropriate read feature points Ts corresponding to two reference feature points Tt are swapped, the difference between the shape of the reference triangle TRt and the shape of the read triangle TRs becomes small. As a result, the candidates MA, MB, and MC, including two inappropriate candidates M, may be erroneously determined to be retained for determining the coordinate correspondence. The same applies when the reference triangle TRt is a triangle similar to an isosceles triangle or a triangle similar to an equilateral triangle. Therefore, in this embodiment, if the scalene condition CQ is not satisfied (S630: No), the processor 210 determines in S640 that the individual combination condition is not satisfied, and ends the processing of Figure 13. If the scalene condition CQ is satisfied (S630: Yes), the processor 210 determines in S635 that the individual combination condition is satisfied, and ends the processing of Figure 13.

[0078] The specific configuration of the inequality condition CQ may be various. In this embodiment, the inequality condition CQ is that all of the following conditions CQ1-CQ3 are satisfied: (CQ1) The absolute value of the difference between the lengths Lab and Lac is greater than the length difference threshold dLth. (CQ2) The absolute value of the difference between the lengths Lab and Lbc is greater than the length difference threshold dLth. (CQ3) The absolute value of the difference between the lengths Lbc and Lac is greater than the length difference threshold dLth. Each of the conditions CQ1-CQ3 indicates that the difference in the lengths of two sides is large. The inequality condition CQ defined by the conditions CQ1-CQ3 indicates that the triangle does not contain two sides whose difference in length is less than or equal to the length difference threshold dLth. The length difference threshold dLth, like the radii r1 and r2 of the selection range SR (FIG. 11), is experimentally determined in advance so that a shape suitable for judgment is formed by combining the three candidates MA, MB, and MC. For example, the length difference threshold dLth may be a value that is 2% or more and 10% or less of the upper limit Lth2 referred to in S615 and S620 (FIG. 13).

[0079] In S565 (FIG. 10), the processor 210 executes the process of FIG. 13 for each of the reference triangle TRt and the read triangle TRs. If both the reference triangle TRt and the read triangle TRs satisfy the individual combination conditions, the processor 210 determines that the candidate combination MU satisfies the combination condition CC. If one or both of the reference triangle TRt and the read triangle TRs do not satisfy the individual combination conditions, the processor 210 determines that the candidate combination MU does not satisfy the combination condition CC.

[0080] In S570, the processor 210 branches the process according to the determination result of S565. If the combination condition CC is satisfied (S570: Yes), in S573, the processor 210 selects the candidate combination MU as the target combination MT. In S575, the processor 210 uses the target combination MT to determine whether the shape condition CW1 is satisfied.

[0081] FIG. 14 is a flowchart showing an example of the process for determining the shape condition CW1. When each of the three candidates MA, MB, and MC (FIGS. 12A and 12B) of the target combination MT indicates an appropriate pair of feature points Tt and Ts, the shape of the read triangle TRs is approximately the same as the shape of the reference triangle TRt. Furthermore, when an object rotates in the image, the orientation of the feature point (e.g., the orientation Oct of feature point Ttc) rotates along with the object indicated by the feature point. The shape condition CW1 is configured taking the above properties into consideration. In this embodiment, the shape condition CW1 is determined to be satisfied when all three conditions CR, CD, and CF, described below, are satisfied.

[0082] In S710, the processor 210 calculates the ratio between the lengths of the two sides of the triangles TRt and TRs. Specifically, the processor 210 calculates the following two ratios Rt and Rs: Rt = Labt / Lact Rs = Labs / Lacs These ratios Rt and Rs are invariant to the scale and rotation of the objects in the images IMtg and IMs.

[0083] In S715, the processor 210 determines whether a ratio condition CR indicating that the ratio Rt / Rs is close to 1 is satisfied. Because the ratios Rt and Rs are invariant to the scale and rotation of the objects in the images IMt and IMs, the determination result of the ratio condition CR is invariant to the scale and rotation. If the shape of the read triangle TRs is the same as the shape of the reference triangle TRt, the ratio Rt / Rs = 1. Even if each of the candidates MA, MB, and MC indicates an appropriate pair of feature points Tt and Ts, the ratio Rt / Rs may deviate from 1. The ratio condition CR is configured to allow such deviation. For example, the ratio condition CR may be that the ratio Rt / Rs is greater than a lower limit Rth1 and less than an upper limit Rth2 (where Rth1 < 1 < Rth2).

[0084] If the ratio condition CR is satisfied (S715: Yes), in S720, the processor 210 determines whether a second interior angle condition CD, which indicates that the interior angle Abs of the read triangle TRs is close to the interior angle Abt of the reference triangle TRt, is satisfied. The interior angles Abt and Abs are invariant to the scale and rotation of the objects in the images IMt and IMs. Therefore, the determination result of the second interior angle condition CD is invariant to the scale and rotation.

[0085] When the shape of the read triangle TRs is the same as the shape of the reference triangle TRt, the interior angle Abs is the same as the interior angle Abt. Even if each of the candidates MA, MB, and MC indicates an appropriate pair of feature points Tt and Ts, the interior angle Abs may deviate from the interior angle Abt. The second interior angle condition CD is configured to allow such deviation. For example, the second interior angle condition CD may be that the absolute value of the difference between the interior angle Abs and the interior angle Abt is less than an interior angle difference threshold dAth (where dAth>0).

[0086] If the second interior angle condition CD is satisfied (S720: Yes), in S725, the processor 210 calculates the angle between the direction of the feature point and the side. Specifically, the processor 210 calculates the following two angles Zt and Zs: Zt = AG(Oct, Sact) Zs = AG(Ocs, Sacs) AG is a function for deriving the angle. The directions Oct and Ocs are the directions Oct and Ocs of the feature points Ttc and Tsc ( FIGS. 12(A) and 12(B) ). The directions Oct and Ocs of the feature points Ttc and Tsc may be the directions calculated for calculating the feature quantities Ft and Fs (S320, S325 ( FIG. 6 )). The angle Zt is the angle between the side Sact and the direction Oct. The angle Zs is the angle between the side Sacs and the direction Ocs.

[0087] In this embodiment, the directions Oct and Ocs of the feature points Ttc and Tsc are gradient directions calculated according to the A-KAZE technique. When an object rotates in the images IMtg and IMs, the directions Oct and Ocs rotate together with the object represented by the feature points Ttc and Tsc in the images IMtg and IMs. The angles Zt and Zs are invariant to the scale and rotation of the object in the images IMt and IMs.

[0088] In S730, the processor 210 determines whether the angle condition CF, which indicates that the angle Zs of the read triangle TRs is close to the angle Zt of the reference triangle TRt, is satisfied. Because the angles Zt and Zs are invariant to the scale and rotation of the objects in the images IMtg and IMs, the determination result of the angle condition CF is invariant to scale and rotation. If the shape of the read triangle TRs is the same as the shape of the reference triangle TRt and the feature points Ttc and Tsc indicate the same part of the same object, the angle Zs will be approximately the same as the angle Zt. Even if each of the candidates MA, MB, and MC indicates an appropriate pair of feature points Tt and Ts, the angle Zs may deviate from the angle Zt. The angle condition CF is configured to allow such deviation. For example, the angle condition CF may be that the absolute value of the difference between the angles Zt and Zs is less than an angle difference threshold dZth (where dZth>0).

[0089] If all of the conditions CR, CD, and CF are satisfied (S715: Yes, S720: Yes, and S730: Yes), it is estimated that all of the candidates MA, MB, and MC are suitable pairs. In this case, in S735, the processor 210 determines that the shape condition CW1 is satisfied, and ends the processing of FIG. 14 .

[0090] If one or more of the conditions CR, CD, and CF are not satisfied, one or more of the candidates MA, MB, and MC, M, are estimated to be an inappropriate pair. If one or more of S715: No, S720: No, and S730: No are satisfied, in S740, the processor 210 determines that the shape condition CW1 is not satisfied, and ends the processing of FIG. 14 .

[0091] Figures 15(A) to 15(D) are diagrams showing examples of the results of the processing in Figure 14. Figure 15(A) shows an example of a reference triangle TRt, and Figures 15(B) to 15(D) show examples of a read triangle TRs associated with the reference triangle TRt.

[0092] 15B shows a case where the candidates MA, MB, and MC are appropriate, and the scale is the same between the reference image IMt and the scanned image IMs, but the rotation angle is different. In this case, the rotation invariant conditions CR, CD, and CF (FIG. 14) are satisfied, and therefore the shape condition CW1 is also satisfied.

[0093] 15C shows a case where the candidates MA, MB, and MC are appropriate, and the reference image IMt and the scanned image IMs have the same rotation angle but different scales. In this case, the scale-invariant conditions CR, CD, and CF (FIG. 14) are satisfied, and therefore the shape condition CW1 is also satisfied.

[0094] Although not shown in the figure, even if the candidates MA, MB, and MC are appropriate and both the scale and the angle of rotation differ between the reference image IMt and the read image IMs, the conditions CR, CD, and CF (Figure 14) are met, and therefore the shape condition CW1 is met.

[0095] 15(D) shows a case where candidates MA and MB are appropriate, but the third candidate MC is inappropriate. Specifically, feature point Tsc is different from the appropriate feature point Tscr. If the combination of candidates MA, MB, and MC includes an inappropriate candidate M, the determination result of one or more of conditions CR, CD, and CF ( FIG. 14 ) may be No (i.e., the shape condition CW1 is likely not satisfied).

[0096] Note that the more lenient the conditions CR, CD, and CF ( FIG. 14 ) are, the greater the total number of suitable candidate combinations MA, MB, and MC that satisfy the shape condition CW1. However, the greater the total number of inappropriate candidate combinations MA, MB, and MC that satisfy the shape condition CW1. The greater the total number of suitable candidates, the smaller the error in the coordinate correspondence, as described below. If the total number of inappropriate candidates is large, the error in the coordinate correspondence may increase. The conditions CR, CD, and CF (in this embodiment, parameters Rth1, Rth2, dAth, and dZth) may be experimentally determined in advance so that the error in the coordinate correspondence is tolerable. The lower limit Rth1 (S715) may be various values ​​less than 1, for example, 0.85 or more and less than 1. The upper limit Rth2 may be various values ​​greater than 1, for example, greater than 1 and 1.15 or less. The interior angle difference threshold dAth (S720) may be any value greater than zero, for example, greater than zero and less than or equal to 10 degrees. The angle difference threshold dZth (S730) may be any value greater than zero, for example, greater than zero and less than or equal to 10 degrees.

[0097] 14, i.e., after S575 (FIG. 10), in S580, the processor 210 branches the process according to the determination result of S575. If the shape condition CW1 is satisfied (S580: Yes), in S585, the processor 210 selects the candidates MA, MB, and MC of the target combination MT as candidates to be retained. After S585, the processor 210 ends the loop process S550 for the current combination of candidates MA, MB, and MC.

[0098] If the combination condition CC is not satisfied (S570: No) or if the shape condition CW1 is not satisfied (S580: No), the processor 210 skips S585 and terminates the loop processing S550 for the current combination of candidates MA, MB, and MC.

[0099] Thereafter, the processor 210 executes a loop process S550 using each of the plurality of third candidates MC, a loop process S530 using each of the plurality of second candidate MBs, and a loop process S510 using each of the plurality of first candidates MA. After completing the repetition of the loop processes S510, S530, and S550, the processor 210 stores data representing the candidate feature points pairs M to be retained in the storage device 215 (e.g., the non-volatile storage device 230) in S590. The processor 210 then ends the process of FIG. 10, i.e., the process of S240 in FIG. 5.

[0100] FIG. 7(D) shows an example of candidate feature points pairs remaining after the process of FIG. 10 . As shown in FIGS. 7(C) and 7(D), pairs of inappropriate feature points Tt, Ts can be eliminated. For example, the line RLb shown in FIG. 7(C) represents a pair of the reference feature point Tt at the top right and the read feature point Ts at the bottom right of the fourth object OB4. When combined with other pairs, such a pair cannot satisfy the shape condition CW1, and can be eliminated from the candidate feature points pairs ( FIG. 10 : S580: No). As described with reference to FIGS. 15(A) to 15(D), a combination of candidates MA, MB, and MC that form an appropriate reference triangle TRt and read triangle TRs can remain as candidates. A combination of candidates MA, MB, and MC that form an inappropriate reference triangle TRt and read triangle TRs is not retained as a candidate.

[0101] Note that even if a combination of candidates MA, MB, and MC including a suitable candidate M does not satisfy the shape condition CW1, the suitable candidate M may satisfy the shape condition CW1 by being combined with another suitable candidate M. Thus, in the process of FIG. 10 , even if the determination result of S570 or S580 is No, the candidates MA, MB, and MC are not immediately eliminated. The processor 210 selects multiple candidates M selected by one or more S585 through repeated loop processes S510, S530, and S550 as candidates M to be retained. The processor 210 eliminates candidates M that have not been selected even once by S585 through repeated loop processes S510, S530, and S550.

[0102] After the processing of FIG. 10 , i.e., after S240 of FIG. 5 , in S250, the processor 210 determines the correspondence between coordinates on the reference image IMt and coordinates on the scanned image IMs. FIG. 16 is a diagram showing an example of a representation format for the coordinate correspondence. The diagram shows coordinates COt on the reference image IMt, coordinates COs on the scanned image IMs, and a matrix Mtx that associates these coordinates COt and COs. In this embodiment, the matrix Mtx represents a so-called affine transformation. In the diagram, the coordinates COt and COs and the matrix Mtx are expressed in a homogeneous coordinate system. Each of the coordinates COt and COs is represented by a three-dimensional vector having positions Xt and Xs in the first direction Dx on the images IMt and IMs, positions Yt and Ys in the second direction Dy, and a third component of 1. The matrix Mtx is a 3-row, 3-column matrix. As shown in the figure, the matrix Mtx is represented by six parameters a-f in two rows and three columns, and three elements (0, 0, 1) in the third row. Such a matrix Mtx can represent rotation, enlargement, reduction, translation, and skew. The matrix Mtx can be calculated by using three or more pairs of coordinates COt, COs (i.e., three or more pairs of feature points).

[0103] The processor 210 determines the matrix Mtx (here, six parameters a-f) using a plurality of feature points pairs MP (i.e., a plurality of candidate feature points pairs M remaining in S220-S240 of FIG. 5). Various methods may be used to determine the matrix Mtx. For example, the processor 210 may determine the matrix Mtx according to a method called Random Sample Consensus (RANSAC). Note that in this embodiment, as described with reference to FIGS. 7(A)-7(D) and 15(A)-15(D), it is highly likely that inappropriate candidates for the feature points pairs MP have been eliminated. Therefore, the processor 210 may calculate the matrix Mtx using all the remaining feature points pairs MP. Various methods may be used for the calculation (e.g., the least squares method). Note that, for example, a function from OpenCV (Open Source Computer Vision Library) may be used to determine the matrix Mtx.

[0104] Upon completion of S250 (FIG. 5), the process of FIG. 5, i.e., S120 in FIG. 4, ends. In S130, the processor 210 inspects the printed image. The inspection method may be any of various methods using the coordinate correspondence (FIG. 16). In this embodiment, the processor 210 generates differential image data using the coordinate correspondence. FIGS. 17A and 17B are diagrams showing examples of differential images. FIG. 17A shows a case where the printed image has no defects, while FIG. 17B shows a case where the printed image has defects (here, missing bits Err). On the left side of each diagram, scanned images IMs and IMs2 and a reference image IMt arranged on the scanned images IMs and IMs2 according to the coordinate correspondence are shown. Images IMd and IMd2 on the right side of each diagram represent examples of differential images between the scanned images IMs and IMs2 and the reference image IMt. The processor 210 generates data for difference images IMd and IMd2 that represent the color value differences (e.g., absolute values ​​of the brightness value differences) between the scanned images IMs and IMs2 and the reference image IMt at positions that correspond by the coordinate correspondence. The scanned image IMs in FIG. 17A represents a printed image without defects. Therefore, the difference image IMd does not have any areas that show large differences. The scanned image IMs2 in FIG. 17B represents a printed image that has a missing part Err. Therefore, in the difference image IMd2, the area corresponding to the missing part Err shows a larger difference than the other areas.

[0105] In S140 (FIG. 4), the processor 210 outputs the inspection results. Various methods for outputting the inspection results are possible. In this embodiment, the processor 210 displays a difference image on the display unit 240 (FIG. 1). By observing the display unit 240, the operator can easily recognize defects in the printed image. Alternatively, the processor 210 may output data representing the inspection results to a storage device (e.g., the non-volatile storage device 230 or an external storage device connected to the data processing device 200). The data representing the inspection results is thereby stored in the storage device. The data representing the inspection results can be used for various processes (e.g., inspection processing of the entire T-shirt 700). After S140, the processor 210 ends the inspection processing.

[0106] As described above, in this embodiment, the processor 210 executes the following processing in accordance with the first program 231. In S355-S360 of FIG. 6 , the processor 210 uses the feature values ​​Fs of the plurality of read feature points Ts and the feature values ​​Ft of the plurality of reference feature points Tt to acquire a plurality of candidate feature point pairs M, which are pairs of feature points Ts and feature points Tt. The read feature points Ts are feature points in the gray read image IMsg ( FIG. 7 ), i.e., feature points in the read image IMs. The reference feature points Tt are feature points in the gray reference image IMtg, i.e., feature points in the reference image IMt.

[0107] 5 (including S240), the processor 210 selects, from the plurality of candidate feature points pairs M, a plurality of candidate feature points pairs M that satisfy a selection condition CP as a plurality of feature points pairs MP. The selection condition CP includes the condition for selecting the candidate feature points pairs M in S240. The condition in S240 includes the shape condition CW1 of S575 in FIG. 10. The shape condition CW1 is an example of a first condition for selecting the target combination MT, which is a combination of three candidates MA, MB, and MC, as three feature points pairs MP (hereinafter, the shape condition CW1 will also be referred to as the first condition CW1).

[0108] The first condition CW1 includes the conditions CR, CD, and CF in FIG. 14. The ratio condition CR (S715) is determined using the ratio Rt, Rs of the lengths of two sides of the triangles TRt, TRs formed by the three candidates MA, MB, and MC (FIGS. 12A and 12B). Specifically, the ratio Rt is the ratio of the lengths Labt, Lact of the two sides Sabt, Sact of the reference triangle TRt. The ratio Rs is the ratio of the lengths Labs, Lacs of the two sides Sabs, Sacs of the read triangle TRs. The second interior angle condition CD (S720) is determined using the interior angles Abt, Abs of the triangles TRt, TRs. The angle condition CF (S730) is determined using the angles Zt, Zs. Angle Zt ( FIG. 12A ) is the angle between a side Sact (i.e., a line segment) connecting the two reference feature points Tta and Ttc and a direction Oct associated with one of the two reference feature points Tta and Ttc, Ttc. Angle Zs ( FIG. 12B ) is the angle between a side Sacs (i.e., a line segment) connecting the two read feature points Tsa and Tsc and a direction Ocs associated with one of the two read feature points Tsa and Tsc, Tsc. As such, in this embodiment, the first condition CW1 is determined using the ratios Rt and Rs, the interior angles Abt and Abs, and the angles Zt and Zs.

[0109] In S250 (FIG. 5), the processor 210 uses the plurality of feature point pairs MP (i.e., the plurality of remaining candidate feature point pairs M) to determine a correspondence (in this embodiment, a matrix Mtx) between the coordinates COs on the scanned image IMs and the coordinates COt on the reference image IMt.

[0110] Thus, in this embodiment, from the plurality of candidate feature point pairs M, a plurality of candidate feature point pairs M that satisfy the selection condition CP are selected as a plurality of feature point pairs MP. The selection condition CP includes a first condition CW1 for selecting a target combination MT, which is a combination of N (in this embodiment, N=3) candidates MA, MB, and MC, as the N feature point pairs MP. The first condition CW1 is determined using the ratios Rt, Rs, the interior angles Abt, Abs, and the angles Zt, Zs. Therefore, the processor 210 can appropriately align the scanned image IMs and the reference image IMt.

[0111] In this embodiment, as described in S110 (FIG. 4), the scanned image IMs is represented by image data generated by optically scanning the printed image IMpp. As described in FIG. 3A, the reference image IMt is represented by image data for printing. As described above, the processor 210 can appropriately align the scanned image IMs of the printed image IMpp with the reference image IMt represented by the image data for printing.

[0112] Furthermore, in this embodiment, in S240 of Fig. 5, the processor 210 executes the processing of Fig. 10. In the processing of Fig. 10, the processor 210 selects a combination of N (in this embodiment, N = 3) candidate feature points pairs M as a target combination MT. Specifically, the processing of Fig. 10 includes S565 to S573. In S565 to S573, the processor 210 selects a candidate combination MU, which is a combination of N candidate feature points pairs M, as the target combination MT. In S565, which is included in S565 to S573, the processor 210 determines a combination condition CC for selecting the candidate combination MU as the target combination MT. The combination condition CC includes conditions CL1, CL2, and CA of Fig. 13.

[0113] The first length condition CL1 (S615) and the second length condition CL2 (S620) indicate that the lengths Lab and Lac of the line segments connecting the two feature points of the two candidate feature point pairs M included in the candidate combination MU are within the allowable length range LR. Specifically, in S615, the lengths Labt ( FIG. 12A ) and Labs ( FIG. 12B ) are evaluated. The length Labt is the length of the line segment (side Sabt) connecting the two reference feature points Tta and Ttb. The length Labs is the length of the line segment (side Sabs) connecting the two read feature points Tsa and Tsb. In S620, the lengths Lact ( FIG. 12A ) and Lacs ( FIG. 12B ) are evaluated. The length Lact is the length of the line segment (side Sact) connecting the two reference minutiae Tta and Ttc. The length Lacs is the length of the line segment (side Sacs) connecting the two read minutiae Tsa and Tsc.

[0114] The first interior angle condition CA (S625) indicates that the interior angle Aa of the triangle formed by the three feature points of the three candidates MA, MB, and MC included in the candidate combination MU is within the allowable interior angle range AR. Specifically, the interior angle Aat ( FIG. 12A ) and the interior angle Aas ( FIG. 12B ) are evaluated. The interior angle Aat is the size of the interior angle of the reference triangle TRt formed by the three reference feature points Tta, Ttb, and Ttc. The interior angle Aas is the size of the interior angle of the read triangle TRs formed by the three read feature points Tsa, Tsb, and Tsc.

[0115] Thus, the combination condition CC includes conditions CL1 and CL2 for the length of the line segment connecting two feature points, and condition CA for the interior angle of the triangle formed by three feature points. Therefore, the processor 210 can select an appropriate candidate combination MU as the target combination MT.

[0116] In this embodiment, the combination condition CC (FIG. 10: S565) includes the scalene condition CQ (S630) of FIG. 13. The scalene condition CQ indicates that the triangle formed by the three feature points of the three candidates MA, MB, and MC included in the candidate combination MU does not contain two edges whose length difference is equal to or less than the length difference threshold dLth. Specifically, three edge pairs (Sabt-Sact, Sabt-Sbct, and Sbct-Sact) are obtained from the three edges Sabt, Sact, and Sbct of the reference triangle TRt (FIG. 12(A)). If the absolute values ​​of the differences in length of each of the three edge pairs are greater than the length difference threshold dLth, the processor 210 determines that the reference triangle TRt satisfies the scalene condition CQ. The scalene condition CQ is satisfied when the shape of the reference triangle TRt is different from any of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle. The same is true for the read triangle TRs ( FIG. 12(B) ). In this way, the combination condition CC includes the scalene condition CQ. Therefore, the processor 210 can reduce the possibility of erroneously selecting an inappropriate candidate combination MU as the target combination MT.

[0117] In this embodiment, the selection conditions CP (FIG. 5) further include conditions for selecting candidate feature point pairs M in S230. The conditions in S230 include the conditions CSt, CH, CSs, and CV in S425-S455 in FIG. 8. The conditions CSt, CH, CSs, and CV are determined using the saturation St, hue Ht, and luminance Vt of the first partial region Pt (FIG. 9A) and the saturation Ss, hue Hs, and luminance Vs of the second partial region Ps (FIG. 9B). The saturation St, hue Ht, and luminance Vt are examples of the first representative color value CJt of the first partial region Pt that includes the feature point Tt in the reference image IMt. The saturation Ss, hue Hs, and luminance Vs are examples of the second representative color value CJs of the second partial region Ps that includes the feature point Ts in the scanned image IMs. The conditions CSt, CH, CSs, and CV together are an example of a second condition for selecting a candidate feature points pair M as a feature points pair MP (hereinafter, the conditions CSt, CH, CSs, and CV together are referred to as the second condition CW2).

[0118] Thus, in this embodiment, the selection condition CP further includes a second condition CW2 for selecting a candidate feature point pair M as a feature point pair MP. The second condition CW2 is determined using a first representative color value CJt (including St, Ht, and Vt) of a first partial region Pt that includes feature point Tt in the reference image IMt, and a second representative color value CJs (including Ss, Hs, and Vs) of a second partial region Ps that includes feature point Ts in the scanned image IMs. Therefore, the processor 210 can select an appropriate candidate feature point pair M as a feature point pair MP.

[0119] In this embodiment, the first representative color value CJt ( FIG. 9A ) indicates a first saturation St and a first hue Ht, and the second representative color value CJs ( FIG. 9B ) indicates a second hue Hs. As shown in S440 and S445 of FIG. 8 , if the first representative color value CJt indicates a saturation St higher than the saturation threshold Sth ( S440: Yes), the processor 210 determines whether the second condition CW2 is satisfied ( S445 ) using the hue Ht indicated by the first representative color value CJt and the hue Hs indicated by the second representative color value CJs. As described above, if the first saturation St of the first representative color value CJt is higher than the saturation threshold Sth, the error in the second hue Hs of the second representative color value CJs may be small. Therefore, the processor 210 can appropriately determine whether the second condition CW2 is satisfied.

[0120] In this embodiment, the first representative color value CJt ( FIG. 9A ) indicates a first saturation St and a first luminance Vt, and the second representative color value CJs ( FIG. 9B ) indicates a second saturation Ss and a second luminance Vs. As shown in S440, S450, and S455 of FIG. 8 , if the saturation St of the first representative color value CJt and the saturation Ss of the second representative color value CJs are equal to or less than the saturation threshold Sth ( S440: No, S450: No), the processor 210 determines whether the second condition CW2 is satisfied ( S455 ) using the luminance Vt indicated by the first representative color value CJt and the luminance Vs indicated by the second representative color value CJs. As described above, even if the saturations St and Ss are low, the luminances Vt and Vs can appropriately represent the color of the portion indicated by the characteristic points Tt and Ts. Therefore, the processor 210 can appropriately determine the second condition CW2.

[0121] Furthermore, in this embodiment, as shown in FIG. 5 , S240 is executed after S230. In S230 ( FIG. 8 ), the processor 210 selects multiple candidate feature points pairs M that satisfy the second condition CW2. In S240 ( FIG. 10 ), the processor 210 uses the multiple candidate feature points pairs M selected in S230 to select multiple candidate feature points pairs M that satisfy the first condition CW1. As described in FIG. 10 , the processor 210 determines whether the first condition CW1 is met (S575) for each of multiple target combinations MT. When selecting a target combination MT (three candidate feature points pairs M) from p (p is an integer greater than or equal to 3) candidate feature points pairs M, the total number of target combinations MT is represented by combination (pC3). The total number of target combinations MT increases rapidly as the total number p of candidate feature points pairs M increases. If S240 were executed before S230, the total number of target combinations MT would be large, and the calculation load of S240 (determination of the first condition CW1) would increase. In this embodiment, the total number of candidate feature points pairs M that can be included in the target combination MT is reduced by S230, so the calculation load can be reduced.

[0122] B. Second Example: FIG. 18 is a flowchart showing an example of a process for acquiring reference image data. Unlike the above-described examples, in this example, image data generated by optically reading an image printed using reference image data is acquired as reference image data. This acquisition process may be performed in various cases. For example, printing of an image on a T-shirt 700 may be performed multiple times using the same image data for printing. In this case, reference image data may be acquired by reading the image printed on the T-shirt 700. Hereinafter, it is assumed that the data processing device 200, printing device 900, and reading device 100 of FIG. 1 are used in the acquisition process. It is assumed that the data of the target image IMp (FIG. 3(A)) is an image for printing.

[0123] In this embodiment, the operator operates the operation unit 250 to input an instruction to start the acquisition process to the data processing device 200. In response to the start instruction, the processor 210 executes the acquisition process in accordance with the second program 232.

[0124] In S810, the processor 210 causes the reading device 100 to print an image onto the T-shirt 700 using image data for obtaining a reference image (here, data for the target image IMp (FIG. 3A)). The target image IMp is printed on the T-shirt 700, as in the T-shirt 700 of FIG. 2. Hereinafter, it is assumed that the printed image on the T-shirt 700 has no defects. Note that the worker may confirm that the printed image has no defects by observing the printed image on the T-shirt 700. If the printed image has defects, the worker may cause the data processing device 200 to execute S810 again to obtain a printed image without defects.

[0125] In S820, the T-shirt 700 is placed on the support unit 140 of the reading device 100 (FIG. 2) so that the printed image is visible. In this embodiment, a worker places the T-shirt 700 on the support unit 140. Alternatively, a machine (e.g., a robot arm) may place the T-shirt 700 on the support unit 140. After placing the T-shirt 700, the worker inputs a proceeding instruction by operating the operation unit 250. The processor 210 supplies a reading instruction to the reading device 100 in response to the proceeding instruction. The control device 110 of the reading device 100 reads the T-shirt 700 in response to the reading instruction. The reading process is performed in the same manner as S110 in FIG. 4. The control device 110 generates data of a read image representing the read T-shirt 700 and supplies the generated data to the data processing device 200. The processor 210 of the data processing device 200 stores the acquired scanned image data as reference image data in the storage device 215 (e.g., the non-volatile storage device 230). Although not shown, the reference image represents a portion of the T-shirt 700 that includes a printed image, similar to the scanned image IMs in FIG. 3B.

[0126] The data of the reference image acquired by the process of Fig. 18 is used in place of the data of the reference image IMt in the inspection process of Fig. 4. In this way, the reference image may be represented by image data generated by optically reading a printed image. In this case, too, the processor 210 can appropriately align the read image IMs and the reference image, as in the first embodiment.

[0127] 18 and the instruction to proceed in S820 may be input to the data processing device 200 via the communication interface 270 by a device other than the data processing device 200.

[0128] C. Third Example: FIG. 19 is a diagram showing another example of the scalene condition (FIG. 13: S630). The diagram shows S630b, which is executed instead of S630 in FIG. 13. In S630b, processor 210 determines whether scalene condition CQb is satisfied. Similar to scalene condition CQ (FIG. 13), scalene condition CQb indicates that the lengths of the three sides of the triangle are different from one another. That is, scalene condition CQb indicates that the shape of the triangle is different from any of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle.

[0129] In this embodiment, the unequal sides condition CQb is that all of the following conditions CQb1-CQb3 are satisfied: (CQb1) The ratio Lab / Lac of the lengths Lab and Lac is outside the ratio range PR. (CQb2) The ratio Lab / Lbc of the lengths Lab and Lbc is outside the ratio range PR. (CQb3) The ratio Lbc / Lac of the lengths Lbc and Lac is outside the ratio range PR. Here, the ratio range PR is equal to or greater than the lower limit RRth1 and equal to or less than the upper limit RRth2 (where RRth1<1<RRth2).

[0130] Each of the conditions CQb1-CQb3 indicates that the ratio of the lengths of the two sides is different from 1. The inequality condition CQb indicates that a triangle (e.g., the reference triangle TRt or the read triangle TRs) does not contain two sides whose length ratio is within a ratio range PR that includes 1. The lower limit RRth1 and the upper limit RRth2, like the radii r1 and r2 of the selection range SR (FIG. 11), are experimentally determined in advance so that a figure suitable for judgment is formed by combining the three candidates MA, MB, and MC. For example, the lower limit RRth1 may be a value between 0.9 and 0.98. The upper limit RRth2 may be a value between 1.02 and 1.1.

[0131] If the scalene condition CQb is satisfied (S630b: Yes), the processor 210 proceeds to S635 (FIG. 13) and determines that the individual combination condition is satisfied. If the scalene condition CQb is not satisfied (S630b: No), the processor 210 proceeds to S640 and determines that the individual combination condition is not satisfied.

[0132] As described above, in this embodiment, the combination condition CC (FIG. 10: S565) includes the scalene condition CQb (FIG. 19: S630b). Therefore, similar to the embodiment of FIG. 13, the processor 210 can reduce the possibility of erroneously selecting an inappropriate candidate combination MU as the target combination MT.

[0133] D. Modifications: (1) The first condition CW1 ( FIG. 10 : S575) is not limited to the condition described in FIG. 14 , but may be any of various conditions indicating that the target combination MT, which is a combination of N (N is 2 or 3) candidate feature point pairs M, is composed of appropriate N candidate feature point pairs M. For example, the reading device 100 may be configured so that the scale of the object is approximately the same between the reference image IMt and the read image IMs. In this case, the first condition CW1 may further include a third length condition indicating that the length of a specific side of the triangle is within an acceptable range. The side length may be, for example, the lengths Lbct and Lbcs of the sides Sbct and Sbcs ( FIGS. 12(A) and 12(B)). When the scale is approximately the same between the reference image IMt and the read image IMs, the third length condition can easily eliminate target combinations MT that include inappropriate candidate feature point pairs M. The allowable range of the third length condition may be experimentally determined in advance, similar to the allowable length range LR in Fig. 13 (for example, the allowable range of the third length condition may be the same as the allowable length range LR). In this way, the first condition CW1 may be determined using the lengths of the sides of the triangle.

[0134] The reading device 100 may also be configured so that the angle of rotation of the object between the reference image IMt and the read image IMs is approximately the same. In this case, in S320 and S325 of FIG. 6, feature quantities Ft and Fs that are not rotation-invariant may be calculated (e.g., BRIEF (Binary Robust Independent Elementary Features)). Then, calculation of the orientation of feature points may be omitted. Conditions using the orientation of feature points (e.g., angle condition CF) may be omitted.

[0135] The first condition CW1 may be a condition (e.g., a third length condition) that can be determined using two candidate feature point pairs M. In this case, the candidate combination MU and the target combination MT may be a combination of two candidate feature point pairs M. The first condition CW1 is preferably determined using one or more of four parameters: the ratio of the lengths of two sides of a triangle formed by three feature points (e.g., ratios Rt and Rs), the lengths of the sides of the triangle (e.g., lengths Lbct and Lbcs), the interior angles of the triangle (e.g., interior angles Abt and Abs), and the angle between the line segment connecting the two feature points and the direction associated with one of the two feature points (e.g., angles Zt and Zs).

[0136] (2) The combination condition CC ( FIG. 10 : S565) is not limited to the conditions described in FIGS. 13 and 19 , and may be various conditions indicating that a combination of N (N is 2 or 3) candidate feature point pairs M forms a figure suitable for determining the first condition CW1. For example, the combination condition CC may include a condition indicating that the lengths Lbct and Lbcs ( FIGS. 12(A) and 12(B)) are within the allowable length range LR. The scalene condition CQ ( FIG. 13 ) may be composed of one or two conditions arbitrarily selected in advance from the conditions CQ1 to CQ3. The scalene condition CQb ( FIG. 19 ) may be composed of one or two conditions arbitrarily selected in advance from the conditions CQb1 to CQb3. The combination condition CC may be composed of one or more conditions arbitrarily selected in advance from the first length condition CL1, the second length condition CL2, the first interior angle condition CA, the scalene side condition CQ, and the scalene side condition CQb. Note that S565-S570 in FIG. 10 (i.e., the determination of the combination condition CC) may be omitted.

[0137] (3) The feature point pair selection process using color information (FIG. 5: S230) is not limited to the process of FIG. 8 and may be various processes. For example, instead of the processes of S425 and S435, the processor 210 may calculate the hue of each of multiple pixels in the first partial region Pt and calculate a representative hue of the first partial region Pt using the multiple hues. The same applies to saturation and luminance. The same applies to the representative color value of the second partial region Ps. Furthermore, the second condition CW2 may be various conditions that use the first representative color value of the first partial region Pt and the second representative color value of the second partial region Ps. For example, the second condition CW2 may be either a hue condition CH or a luminance condition CV. Preferably, the second condition CW2 indicates that the first representative color value of the first partial region Pt is similar to the second representative color value of the second partial region Ps.

[0138] (4) The alignment process may be various processes instead of the process of Fig. 5. For example, S240 may be executed before S230. Furthermore, S230 (i.e., the determination of the second condition CW2) may be omitted.

[0139] (5) The detection method for the feature points Tt and Ts (i.e., key points) may be any of various methods for detecting points representing portions of an object in an image, instead of the methods described in S310 and S315 (FIG. 6). The detection method may be selected in advance from, for example, a search for extrema (maximum and minimum) using Difference-of-Gaussian (DoG), Harris corner detection, Features from Accelerated Segment Test (FAST) corner detection, Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented Fast and Rotated Brief (ORB).

[0140] The calculation method of the feature quantities Ft and Fs (i.e., the feature descriptor) may be various methods for calculating information describing the features of keypoints, instead of the methods described in S320 and S325 (FIG. 6). The algorithm for calculating the feature descriptor may be selected in advance from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, and A-KAZE. Furthermore, the calculation method of the distance dF between the feature quantities Ft and Fs may be various methods suitable for the data structure of the feature quantities Ft and Fs. When the feature quantities Ft and Fs are expressed by binary vectors, the distance dF may be the Hamming distance. Instead of the Hamming distance, the distance dF may be various other distances (e.g., norms such as the L1 norm and the L2 norm (also known as the Euclidean distance)). The norm is applicable to a variety of feature descriptors.

[0141] (6) Processes other than the process of determining the correspondence between coordinates using a plurality of feature points Tt and Ts may be executed according to a program other than the first program 231. For example, the detection of feature points Tt and Ts and the calculation of feature amounts Ft and Ts (FIG. 6: S305-S325) may be executed according to a program other than the first program 231. S130 and S140 in FIG. 4 may be executed according to a program other than the first program 231. Furthermore, processes other than the process of determining the correspondence between coordinates using a plurality of feature points Tt and Ts may be executed by a device other than the data processing device 200.

[0142] (7) The print medium is not limited to the T-shirt 700 and may be various types of clothing (for example, various shirts such as polo shirts, coats, slacks, etc.). The print medium may be various types of fabric such as clothing or bags. The print medium is not limited to fabric and may be various media such as paper, film, leather, etc. Furthermore, the printing device 900 may be a printing device of another type (for example, a laser type) instead of an inkjet type printing device.

[0143] (8) The coordinate correspondence ( FIG. 16 ) may represent various transformations such as homography transformation instead of affine transformation. Also, the coordinate correspondence may be represented in various formats such as a lookup table instead of the matrix Mtx.

[0144] The coordinate correspondence may be used in various processes, not limited to inspection. For example, in the machining of metal parts, the position and orientation of the metal part relative to the tool may be misaligned. Here, the position and orientation of the metal part relative to the tool may be determined by analyzing an image of the metal part captured by a digital camera attached to the tool. For example, the processor 210 may determine the reference position of the metal part on the captured image (and thus the position and orientation of the metal part relative to the tool) by aligning the captured image with a reference image representing a portion of the metal part that indicates the reference position. The alignment process may employ the process of the above embodiment ( FIG. 5 ) or the process of the above modification. In this way, the scanned image may be an image representing an object scanned by a line sensor such as the scanning sensor 180 ( FIG. 2 ), or an image representing an object scanned by an area sensor such as a digital camera. Furthermore, the reference image may be various images (e.g., a prepared image representing a specific portion of an object) instead of a print-related image. In either case, the image may be a grayscale image instead of a color image.

[0145] (9) The data processing device that determines the correspondence between coordinates is not limited to a personal computer (e.g., data processing device 200 (FIG. 1)), but may be various other devices (e.g., a smartphone, a tablet computer, a control device incorporated in a reading device, etc.). Furthermore, multiple devices (e.g., computers) that can communicate with each other via a network may share part of the data processing function of the data processing device and collectively provide the data processing function (a system including these devices corresponds to the data processing device).

[0146] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the process of S250 in Fig. 5 may be executed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0147] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used while stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0148] The above-described examples and modifications can be combined as appropriate. The above-described examples and modifications are provided to facilitate understanding of the present disclosure and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof.

[0149] 100...reading device, 110...control device, 120...conveying device, 122...position sensor, 130...table, 140...supporting section, 180...reading sensor, 190...casing, 200...data processing device, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 231...first program, 232...second program, 240...display section, 250...operation section, 270...communication interface, 700...T-shirt, 900...printing device, Abs, Abt...interior angle, AR...allowable interior angle range, CC...combination condition, CJt...first representative color value, CJs...second representative color value, COs, COt...coordinates, CP...selection condition , CW1...first condition, CW2...second condition, Ocs,Oct...direction, Fs,Ft...feature amount, Hs,Ht...hue, IMs,IMs2...scanned image, IMt...reference image, Labs, Labt, Lacs, Lact, Lbcs, Lbct...length, Sabs, Sabt, Sacs, Sact, Sbcs, Sbct...side, LR...allowable length range, Rs,Rt...ratio, PR...ratio range, M...candidate feature point pair, MP...feature point pair, MT...target combination, Pt...first partial region, Ps...second partial region, Ss,St...saturation, Sth...saturation threshold, TRs,TRt...triangle, Ts,Tt...feature point, Vs,Vt...luminance, Zs,Zt...angle

Claims

1. A program comprising: a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in a scanned image and feature points in a reference image, by using feature amounts of each of a plurality of feature points in the scanned image and a plurality of feature points in a reference image; a pair selection function that selects a plurality of candidate feature point pairs that satisfy a selection condition as a plurality of feature point pairs from the plurality of candidate feature point pairs, the selection condition including a first condition for selecting a target combination, which is a combination of N (N is 2 or 3) candidate feature point pairs, as N feature point pairs, the first condition being determined using one or more of four types of parameters: a ratio of lengths of two sides of a triangle formed by three feature points, a side length of the triangle, an interior angle of the triangle, and an angle between a line segment connecting two feature points and a direction associated with one of the two feature points; and a determination function that determines a correspondence relationship between coordinates on the scanned image and coordinates on the reference image by using the plurality of feature point pairs. A program that enables a computer to achieve this.

2. A program as described in claim 1, wherein the read image is represented by image data generated by optically reading a printed image, and the reference image is represented by image data for printing or image data generated by optically reading a printed image using reference image data.

3. A program as claimed in claim 1 or 2, further comprising: causing a computer to realize a combination selection function for selecting a combination of N candidate feature points pairs as the target combination; and wherein the combination conditions for selecting the combination of the N candidate feature points pairs as the target combination include one or both of: an interior angle of a triangle formed by three feature points included in the N candidate feature points pairs is within an allowable interior angle range; and a length of a line segment connecting two feature points included in the N candidate feature points pairs is within an allowable length range.

4. A program as claimed in claim 1 or 2, further comprising: causing a computer to realize a combination selection function for selecting a combination of N candidate feature points pairs as the target combination; and wherein the combination conditions for selecting the combination of the N candidate feature points pairs as the target combination include one or both of: a triangle formed by three feature points included in the N candidate feature points pairs does not include two sides having a length difference equal to or less than a difference threshold; and the triangle does not include two sides having a length ratio within a ratio range including 1.

5. A program as described in claim 1 or 2, wherein the selection conditions further include a second condition for selecting the candidate feature point pair as the feature point pair, and the second condition is determined using a first representative color value of a first partial area including a feature point in the reference image and a second representative color value of a second partial area including a feature point in the read image.

6. A program as described in claim 5, wherein the first representative color value indicates saturation and hue, the second representative color value indicates hue, and the pair selection function, when the first representative color value indicates the saturation higher than a saturation threshold value, determines whether the second condition is met using the hue indicated by the first representative color value and the hue indicated by the second representative color value.

7. A program as described in claim 5, wherein the first representative color value indicates saturation and luminance, the second representative color value indicates saturation and luminance, and the pair selection function determines whether the second condition is met using the luminance indicated by the first representative color value and the luminance indicated by the second representative color value when the saturation of the first representative color value and the saturation of the second representative color value are equal to or lower than a saturation threshold value.

8. A program according to claim 5, wherein the pair selection function selects a plurality of candidate feature points pairs that satisfy the first condition using a plurality of candidate feature points pairs that satisfy the second condition.

9. A data processing device comprising: a candidate acquisition unit that acquires a plurality of candidate feature point pairs, which are pairs of feature points in the read image and feature points in the reference image, by using feature amounts of each of a plurality of feature points in the read image and a plurality of feature points in a reference image; a pair selection unit that selects a plurality of candidate feature point pairs that satisfy a selection condition as a plurality of feature point pairs from the plurality of candidate feature point pairs, the selection condition including a first condition for selecting a target combination, which is a combination of N (N is 2 or 3) candidate feature point pairs, as N feature point pairs, the first condition being determined using one or more of four types of parameters: a ratio of lengths of two sides of a triangle formed by three feature points, a side length of the triangle, an interior angle of the triangle, and an angle between a line segment connecting two feature points and a direction associated with one of the two feature points; and a determination unit that determines a correspondence relationship between coordinates on the read image and coordinates on the reference image by using the plurality of feature point pairs. A data processing device comprising:

Images