Program and data processing apparatus

Abstract

To perform positioning between a plurality of images.SOLUTION: A plurality of candidate feature point pairs that are pairs of feature points in a read image and feature points in a reference image are acquired. The plurality of candidate feature point pairs satisfying a selection condition is selected as a plurality of feature point pairs. The selection condition includes a first condition for selecting a target combination that is a combination of N (N is 2 or 3) candidate feature point pairs as N feature point pairs. The first condition is determined by using one or more parameters out of four parameters of the ratio of the lengths of the two sides, the length of the sides, an internal angle, and an angle formed by a line segment connecting the two feature points and a direction corresponding to one of the two feature points. A plurality of feature point pairs is used to determine the correspondence between the coordinates on the read image and the coordinates on the reference image.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] This specification relates to the alignment between a plurality of images.

Background Art

[0002] In various processes, alignment between a plurality of images can be performed. Patent Document 1 discloses a technique for detecting defects in an image formed on a sheet by an image forming apparatus such as a printer or a copier. In this technique, together with a job image instructed to be printed by a user, a marker image for positioning is formed on the same sheet. An image reading unit reads the sheet surface and generates a read image. An inspection unit determines the position of the read image corresponding to the reference image based on each feature point of the job image and the marker image extracted from the read image to be inspected and each feature point of the job image and the marker image extracted from the reference image. The inspection unit compares the reference image and the read image after alignment, and detects an image area where the difference in pixel values is greater than or equal to a threshold value as a defect.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Alignment between a plurality of images is not easy and there is room for improvement.

[0005] This specification discloses a technique for aligning a plurality of images.

Means for Solving the Problems

[0006] The technique disclosed in this specification can be realized as the following application examples.

[0007] [Application Example 1] A program that uses the feature amounts of each of a plurality of feature points in a read image and the feature amounts of each of a plurality of feature points in a reference image to obtain a plurality of candidate feature point pairs that are pairs of the feature points in the read image and the feature points in the reference image; a pair selection function that selects, as a plurality of feature point pairs, a plurality of candidate feature point pairs that satisfy a selection condition from the plurality of candidate feature point pairs, where the selection condition includes a first condition for selecting, as N feature point pairs, a target combination that is a combination of N (N is 2 or 3) candidate feature point pairs, and the first condition is determined using one or more of four types of parameters: the ratio of the lengths of two sides of a triangle formed by three feature points, the length of a side of the triangle, the interior angle of the triangle, and the angle formed by a line segment connecting two feature points and a direction associated with one of the two feature points; the pair selection function; and a determination function that determines a correspondence relationship between the coordinates on the read image and the coordinates on the reference image using the plurality of feature point pairs, realized by a computer, program.

[0008] According to this configuration, a plurality of candidate feature point pairs that satisfy the selection condition are selected as a plurality of feature point pairs from the plurality of candidate feature point pairs. The selection condition includes a first condition for selecting, as N feature point pairs, a target combination that is a combination of N (N is 2 or 3) candidate feature point pairs. The first condition is determined using one or more of four types of parameters: the ratio of the lengths of two sides of a triangle formed by three feature points, the length of a side of the triangle, the interior angle of the triangle, and the angle formed by a line segment connecting two feature points and a direction associated with one of the two feature points. Therefore, the alignment between the read image and the reference image can be appropriately performed.

[0009] Note that the technology disclosed in this specification can be realized in various forms, for example, in the form of a data processing method and a data processing device, a computer program for realizing the functions of those methods or devices, a recording medium (e.g., a non-transitory recording medium) recording the computer program, etc.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Mode for Carrying Out the Invention

[0011] A. First Embodiment: A1. Device Configuration: FIG. 1 is an explanatory diagram showing a data processing device as an example. The data processing device 200 is, for example, a personal computer. The data processing device 200 executes an inspection process of a printed image. The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to perform data processing, and is, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile storage device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores data of the first program 231 and the second program 232 respectively. The second program 232 is used in another embodiment described later.

[0013] The display unit 240 is a device configured to display an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive an operation by a user, such as a button, a lever, or a touch panel disposed on top of the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250.

[0014] The communication interface 270 is an interface for communicating with other devices (including, for example, one or more of a USB interface, a wired LAN interface, a wireless interface of IEEE802.11, an interface of an industrial camera (such as CameraLink, CoaXPress, etc.)). In this embodiment, a reading device 100 and a printing device 900 are connected to the communication interface 270. The printing device 900 is a so-called inkjet printing device, which prints an image on a printing medium such as cloth or paper by ejecting ink onto the printing medium. The reading device 100 generates data of a reading image representing an object by optically reading the object to be read. Hereinafter, the printing medium is a T-shirt, and the reading device 100 reads the T-shirt with an image printed thereon.

[0015] FIG. 2 is a perspective view showing an example of the reading device 100. The first direction Da and the second direction Db in the figure indicate horizontal directions, and the third direction Dc indicates the vertically upward direction. The first direction Da and the second direction Db are perpendicular to each other.

[0016] In this embodiment, the reading device 100 includes a housing 190, a table 130, a support portion 140 fixed to the upper surface of the table 130, a conveying device 120, a reading sensor 180, and a control device 110. The control device 110, the conveying device 120, and the reading sensor 180 are fixed to the housing 190.

[0017] The support portion 140 is a plate-like member that forms a flat upper surface for supporting the object to be read (such a member is also called a platen). In the figure, a T-shirt 700 having a printed image IMpp is placed on the support portion 140.

[0018] The conveying device 120 is configured to convey the table 130 in a direction parallel to the second direction Db. The configuration of the conveying device 120 may be various configurations. Although illustration is omitted, in this embodiment, the conveying device 120 includes a rail that slidably supports the table 130 in a direction parallel to the second direction Db, a plurality of pulleys, a belt that is wound around the plurality of pulleys and partially fixed to the table 130, and an electric motor that rotates the pulley. When the electric motor rotates the pulley, the table 130 (and thus the support portion 140) moves in a direction parallel to the second direction Db. The conveying device 120 further includes a position sensor 122 (for example, a rotary encoder) that detects the position of the table 130 on the conveying path.

[0019] The reading sensor 180 is disposed at a position higher than the support portion 140 in the middle of the conveying path PTh of the support portion 140. The reading sensor 180 includes a line sensor composed of a plurality of photoelectric conversion elements arranged in a direction intersecting the conveying direction Db (in this embodiment, the direction Da perpendicular to the conveying direction Db) (for example, a Contact Image Sensor (CIS) or a Charge Coupled Device (CCD)). The reading sensor 180 faces downward. The reading sensor 180 can read a portion of the object supported by the support portion 140 that is located below the reading sensor 180.

[0020] When reading the T-shirt 700, the reading device 100 conveys the table 130 in a direction parallel to the second direction Db. The reading sensor 180 repeatedly reads the T-shirt 700 during conveyance. Thereby, the reading sensor 180 can read approximately the entire portion of the T-shirt 700 that is supported by the support portion 140.

[0021] The control device 110 is an electric circuit configured to control the transport device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates data of a read image by controlling the transport device 120 and the reading sensor 180.

[0022] A2. Printing process: In this embodiment, an image is printed on the T-shirt 700 (FIG. 2). The printing of the image is performed, for example, as part of a T-shirt sales service. The T-shirt sales service may include on-demand printing. A customer orders on-demand printing from a service provider. The service provider prints an image on the T-shirt 700 using the image data provided by the customer in response to the customer's order.

[0023] FIG. 3(A) is a diagram showing an example of an image represented by image data for printing (referred to as a target image IMp). In this embodiment, the data of the target image IMp is bitmap data representing the respective color values (here, the gradation values of red R, green G, and blue B (for example, values of zero or more and 255 or less)) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. In the example of FIG. 3(A), the target image IMp represents a background BG and four rectangular objects OB1 - OB4. The objects OB1 - OB4 are represented in different colors. For example, the color of the first object OB1 is red, the color of the second object OB2 is green, the color of the third object OB3 is blue, and the color of the fourth object OB4 is gray. Thus, the target image IMp may include a plurality of objects having different hues.

[0024] Although illustration is omitted, the printing of the target image IMp is performed using the data processing device 200 and the printing device 900. Alternatively, the printing of the target image IMp may be performed using other devices.

[0025] In the printing of the target image IMp, various errors can occur. The printed image may have various defects due to errors. For example, due to abnormal ink ejection, a part of the image may be missing in the printed image. The data processing device 200 detects the defects of the printed image through the inspection process described later. In this embodiment, the data of the target image IMp is used in the inspection process. After the printing of the target image IMp, the data of the target image IMp is stored in the storage device 215 (for example, the non-volatile storage device 230) of the data processing device 200 for the inspection process.

[0026] A3. Inspection process: FIG. 4 is a flowchart showing an example of the inspection process. For inspection, the T-shirt 700 is placed on the support portion 140 of the reading device 100 (FIG. 2) so that the printed image IMpp can be seen. In this embodiment, an operator places the T-shirt 700 on the support portion 140. Alternatively, a machine (for example, a robotic arm) may place the T-shirt 700 on the support portion 140. After the placement of the T-shirt 700, an instruction to start the inspection process is input to the data processing device 200 (FIG. 1). In this embodiment, the operator inputs the inspection start instruction by operating the operation unit 250. The processor 210 starts the inspection process in response to the start instruction. Note that the start instruction may be input to the data processing device 200 via the communication interface 270 by another device different from the data processing device 200.

[0027] The processor 210 of the data processing apparatus 200 executes an inspection process according to the first program 231. In S110, the T-shirt 700 is photographed. The processor 210 supplies a reading instruction to the reading apparatus 100. The control apparatus 110 of the reading apparatus 100 controls the reading sensor 180 and the conveying apparatus 120 in response to the reading instruction to read the T-shirt 700. The control apparatus 110 generates data of a reading image representing the read T-shirt 700. The processor 210 of the data processing apparatus 200 acquires the data of the reading image from the control apparatus 110 of the reading apparatus 100, and stores the acquired data of the reading image in the storage apparatus 215 (for example, the nonvolatile storage apparatus 230).

[0028] FIG. 3(B) is a diagram showing an example of a reading image. In the present embodiment, the data of the reading image IMs is bitmap data representing the respective color values (here, the respective gradation values of red R, green G, and blue B (for example, values of 0 or more and 255 or less)) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. The reading image IMs in the figure represents a portion including the printed image IMpp of the T-shirt 700. Here, it is assumed that the printed image IMpp is an image obtained by printing the target image IMp (FIG. 3(A)). The printed image IMpp represents a background BG and objects OB1 - OB4 in the same manner as the target image IMp.

[0029] In S120 (FIG. 4), the processor 210 performs alignment between the reference image and the reading image. In the present embodiment, the target image IMp (FIG. 3(A)) is used as the reference image (hereinafter, the target image IMp is also referred to as the reference image IMt).

[0030] The orientation of the T-shirt 700 with respect to the reading sensor 180 can be various orientations. Therefore, between the reference image IMt and the read image IMs, the orientations of the objects OB1-OB4 in the image (i.e., the rotation angles of the objects) can be different. Also, between the reference image IMt and the read image IMs, the pixel density (also called resolution) for the same object can be different. That is, between the reference image IMt and the read image IMs, the scale for the object can be different.

[0031] FIG. 5 is a flowchart showing an example of the alignment process. In this embodiment, the processor 210 obtains a plurality of pairs of feature points by feature point matching, and uses the obtained plurality of pairs to determine the correspondence between the coordinates on the reference image IMt and the coordinates on the read image IMs.

[0032] In S220, the processor 210 performs feature point matching. FIG. 6 is a flowchart showing an example of the feature point matching process. In S305, the processor 210 generates a gray reference image IMtg and a gray read image IMsg by gray scale conversion of the reference image IMt and the read image IMs. As the correspondence between the color tone value and the gray scale tone value, a known relationship can be adopted (for example, the correspondence between the RGB values in the RGB color space and the luminance value Y in the YCbCr color space).

[0033] In S310, the processor 210 extracts feature points Tt from the gray reference image IMtg. FIGS. 7(A)-7(D) are diagrams showing examples of images processed by feature point matching. FIG. 7(A) shows an example of feature points detected from the images IMtg and IMsg. A plurality of black dots on the gray reference image IMtg each indicate a feature point Tt detected from the gray reference image IMtg (also referred to as a reference feature point Tt). As shown in the figure, points indicating characteristic portions such as corners and ends of an object are detected as the feature points Tt. Such feature points Tt are also called keypoints. Although not shown, actually, more feature points Tt can be detected (for example, on the order of several tens or several hundreds). Note that the reference feature points Tt detected from the gray reference image IMtg correspond to feature points indicating the same portion at the same coordinates on the reference image IMt (FIG. 3(A)).

[0034] The method for detecting feature points may be various methods. In this embodiment, a technique called Accelerated-KAZE (A-KAZE) that performs detection of keypoints and calculation of feature descriptors for each keypoint is used. The A-KAZE technique is disclosed, for example, in the following paper. “Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013”

[0035] The processor 210 detects a plurality of feature points Tt by analyzing the gray reference image IMtg according to the A-KAZE technique.

[0036] In S315 (FIG. 6), the processor 210 extracts a plurality of feature points Ts from the gray-scale read image IMsg. The plurality of black dots on the gray-scale read image IMsg in FIG. 7(A) each indicate a feature point Ts detected from the gray-scale read image IMsg (also referred to as a read feature point Ts). The processor 210 detects a plurality of feature points Ts by analyzing the gray-scale read image IMsg according to the A-KAZE technique. Although not shown, actually, more feature points Ts can be detected (for example, about several tens or several hundreds). Note that the read feature point Ts detected from the gray-scale read image IMsg corresponds to the feature point indicating the same portion at the same coordinates on the read image IMs (FIG. 3(B)).

[0037] In S320 (FIG. 6), the processor 210 calculates a feature amount Ft for each of the plurality of reference feature points Tt. The feature amount Ft may be various information describing the features of the feature point Tt. The feature amount Ft is calculated, for example, so as to change according to the distribution of the color values of a plurality of pixels around the feature point Tt. In the present embodiment, the processor 210 uses the gray-scale reference image IMtg to calculate an A-KAZE feature descriptor as the feature amount Ft. In the A-KAZE technique, the feature descriptor is rotation invariant. And in order to obtain a rotation-invariant feature descriptor, the direction of the feature point is also calculated. The direction of the feature point indicates the direction of the luminance gradient in the vicinity area centered on the feature point (also referred to as the gradient direction or the dominant direction). The processor 210 calculates the feature amount Ft and the direction of the reference feature point Tt.

[0038] In S325, the processor 210 calculates a feature amount Fs for each of the plurality of read feature points Ts. In the present embodiment, the processor 210 uses the gray-scale read image IMsg to calculate a feature descriptor (that is, the feature amount Fs) and the direction of the read feature point Ts according to the A-KAZE technique.

[0039] The outline of the processing after S325 is as follows. For each combination of the reference feature point Tt and the read feature point Ts, the processor 210 calculates the distance dF between the feature amounts Ft and Fs (S350). Then, a pair of feature points Tt and Ts indicating a distance dF less than the distance threshold dFth is acquired as a candidate M for the feature point pair (S355: Yes, S360).

[0040] Specifically, it is as follows. The processor 210 executes a loop process S330 (including S335 - S360) between the start L31s and the end L31e for each of the plurality of reference feature points Tt. In S335, the processor 210 selects an unprocessed reference feature point Tt as a feature point of interest Tti that is a feature point to be processed. The processor 210 executes a loop process S340 (including S345 - S360) between the start L32s and the end L32e for each of the plurality of read feature points Ts. In S345, the processor 210 selects an unprocessed read feature point Ts as a feature point of interest Tsj that is a feature point to be processed.

[0041] In S350, the processor 210 calculates the distance dF between the two feature amounts Fti and Fsj of the two feature points of interest Tti and Tsj. The distance dF is calculated such that a smaller distance dF indicates a higher similarity between the two feature amounts Fti and Fsj. A high similarity (i.e., a small distance dF) indicates that the two feature points of interest Tti and Tsj represent similar portions (e.g., corresponding portions of the same object) in the two images IMtg and IMsg. In this embodiment, the feature amounts Ft and Fs are A-KAZE feature descriptors and are represented by binary vectors (vectors composed of one or more binary elements). In this case, the processor 210 may calculate the Hamming distance as the distance dF.

[0042] In S355, the processor 210 determines whether the distance dF is less than the distance threshold dFth. When the distance dF is small, the feature points Tti and Tsj are likely to indicate similar parts (e.g., the same part of the same object) in the images IMt and IMs. If the distance dF is less than the distance threshold dFth (S355: Yes), in S360, the processor 210 obtains the pair of the target feature points Tti and Tsj as a candidate M for the feature point pair (also referred to as the candidate feature point pair M). The feature point pair is a pair of the reference feature point Tt and the read feature point Ts used for determining the coordinate correspondence. The feature points Tt and Ts associated with each other are also called a matching pair. After S360, the processor 210 ends the loop process S340 for the target feature point Tsj. If the distance dF is greater than or equal to the distance threshold dFth (S355: No), the processor 210 skips S360 and ends the loop process S340 for the target feature point Tsj. Then, the processor 210 executes the processes of S350 - S360 for each of the multiple combinations of the target feature point Tsj and the target feature point Tti by repeating the loop process S340 and the loop process S330.

[0043] Figure 7(B) shows an example of the candidate feature point pair M. The multiple lines RL in the figure each indicate the candidate feature point pair M. Each line RL connects the feature points Tt and Ts that form the candidate feature point pair M. As shown in the figure, a pair of feature points Tt and Ts indicating the same part of each other can be selected as the candidate feature point pair M. For example, the feature point Tt1 indicating the upper right corner of the first object OB1 in the gray reference image IMtg can be associated with the feature point Ts1 indicating the upper right corner of the first object OB1 in the gray read image IMsg.

[0044] In addition, a pair of feature points Tt and Ts indicating different parts from each other can be selected as a candidate feature point pair M. For example, a feature point Tt1 indicating the upper right corner of the first object OB1 in the gray reference image IMtg can be associated with a feature point Ts2 indicating the upper right corner of the third object OB3 in the gray read image IMsg. The rectangular objects OB1 - OB4 each have four corners. These corners are locally similar. As a result, a feature point Tt indicating one corner of an object in the gray reference image IMtg can be associated with a feature point Ts indicating another corner of the same object in the gray read image IMsg, or a feature point Ts indicating a corner of another object. Thus, when each of the images IMtg and IMsg represents a plurality of locally similar parts, feature points Tt and Ts indicating different parts from each other can be associated with each other.

[0045] Also, in this embodiment, as described above, the feature quantities Ft and Fs are rotation-invariant. When the feature quantity is rotation-invariant, similar feature quantities can be calculated from the same part of the object regardless of the rotation angle of the object within the image. By using the distance dF between the rotation-invariant feature quantities Ft and Fs, the processor 210 can associate pairs of feature points indicating the same part of the object even when the rotation angles of the same object are different between the two images IMtg and IMsg. However, when each of the images IMtg and IMsg represents a plurality of locally similar parts, feature points Tt and Ts indicating different parts from each other can be associated with each other. For example, a feature point Tt1 indicating the upper right corner of the first object OB1 in the gray reference image IMtg can be associated with a feature point Ts3 indicating the lower right corner of the first object OB1 in the gray read image IMsg.

[0046] Incidentally, the larger the distance threshold dFth (Fig. 6: S355), the greater the total number of appropriate candidate feature point pairs M can be. However, the total number of inappropriate candidates M can also be large. The greater the total number of appropriate candidates M, the lower the error in the coordinate correspondence relationship described later. When the total number of inappropriate candidates M is large, the error in the coordinate correspondence relationship can increase. The distance threshold dFth may be determined experimentally in advance so that the error in the coordinate correspondence relationship is acceptable.

[0047] When the processing for all combinations of the reference feature point Tt and the read feature point Ts is completed, in S365, the processor 210 stores the data representing the plurality of candidate feature point pairs M in the storage device 215 (for example, the non-volatile storage device 230). Then, the processor 210 ends the processing of Fig. 6, that is, the processing of S220 in Fig. 5.

[0048] In S230, the processor 210 executes a selection process for feature point pairs using color information. This selection process selects appropriate candidates M using color information from the plurality of candidate feature point pairs M (that is, the candidate feature point pairs M are verified). Fig. 8 is a flowchart showing an example of the selection process. The processor 210 executes loop processing S410 (including S420 - S465) between the start L4s and the end L4e for each of the plurality of candidate feature point pairs M.

[0049] In S420, the processor 210 selects an unprocessed candidate M as the candidate of interest Mi to be processed.

[0050] In S425, the processor 210 calculates a representative color value Ct of the first partial region Pt including the reference feature point Tt of the candidate of interest Mi (the representative color value Ct is also referred to as the reference representative color value Ct). FIGS. 9(A) and 9(B) are diagrams showing examples of calculating the representative color value. FIG. 9(A) shows a part of the reference image IMt including the feature point Tt. In the figure, the first partial region Pt including the feature point Tt is shown. The processor 210 calculates the reference representative color value Ct using the color values of a plurality of pixels in the first partial region Pt. The method of calculating the representative color value Ct may be various methods of calculating the color representing the first partial region Pt. In this embodiment, the processor 210 calculates the average value of each of RGB in the first partial region Pt as each gradation value of RGB of the representative color value Ct. Note that, instead of the average value, various summary statistics representing the magnitude of the color value (for example, median, mode, etc.) may be used. Further, the configuration of the first partial region Pt (specifically, the shape of the first partial region Pt and the relative position of the first partial region Pt with respect to the position of the feature point Tt) may be various configurations capable of calculating the representative color value Ct representing the color of the portion indicated by the feature point Tt. In this embodiment, the first partial region Pt is a region centered on the feature point Tt, for example, a region of P*Q pixels of P rows and Q columns. In order to reduce the dependency of the representative color value Ct on the rotation angle of the object, P = Q is preferable (for example, P = Q = 5). Instead of the region of P rows and Q columns, the first partial region Pt may be a region where the distance from the feature point Tt is equal to or less than a distance threshold.

[0051] In S430 (FIG. 8), the processor 210 calculates the representative color value Cs of the second partial region Ps including the read feature point Ts of the candidate of interest Mi (the representative color value Cs is also referred to as the read representative color value Cs). FIG. 9(B) shows a part of the read image IMs including the feature point Ts. In the figure, the second partial region Ps including the feature point Ts is shown. The configuration of the second partial region Ps is the same as the configuration of the first partial region Pt. The second partial region Ps is a region of P*Q pixels in P rows and Q columns centered on the feature point Ts. The method for calculating the read representative color value Cs is the same as the method for calculating the reference representative color value Ct. The processor 210 calculates the average value of red R, green G, and blue B in the second partial region Ps as the read representative color value Cs.

[0052] In S435 (FIG. 8), the processor 210 calculates the first hue Ht, the first saturation St, and the first brightness Vt from the reference representative color value Ct, and calculates the second hue Hs, the second saturation Ss, and the second brightness Vs from the read representative color value Cs. As the correspondence relationship between the representative color value, the hue, the saturation, and the brightness, a known relationship can be adopted (for example, the correspondence relationship between the RGB values in the RGB color space and the HSV values in the HSV color space). Note that the first hue Ht, the first saturation St, and the first brightness Vt are, respectively, examples of the representative color value of the first partial region Pt, similar to the reference representative color value Ct. The second hue Hs, the second saturation Ss, and the second brightness Vs are, respectively, examples of the representative color value of the second partial region Ps, similar to the read representative color value Cs.

[0053] In S440, the processor 210 determines whether a first chroma condition CSt indicating that the first chroma St is higher than the chroma threshold Sth is satisfied. When the first chroma St is higher than the chroma threshold Sth (S440: Yes), in S445, the processor 210 determines whether a hue condition CH indicating that the absolute value of the difference between the first hue Ht and the second hue Hs is less than the hue difference threshold dHth is satisfied. As the absolute value of the difference between the hues Ht and Hs (also referred to as the hue difference dH), the value corresponding to the smaller angular difference between the first hue Ht and the second hue Hs on the hue circle is adopted. When the feature point pair Tt, Ts where the candidate of interest Mi indicates the same part with each other, the hue difference dH can be a small value. When the hue difference dH is less than the hue difference threshold dHth (S445: Yes), in S460, the processor 210 selects the candidate of interest Mi as a candidate to be retained. When the hue difference dH is greater than or equal to the hue difference threshold dHth (S445: No), between the first partial region Pt and the second partial region Ps, the hues can be significantly different. That is, the feature points Tt, Ts of the candidate of interest Mi are likely to indicate different parts from each other. The processor 210 excludes the candidate of interest Mi from the candidates of the feature point pair in S465.

[0054] Thus, when the first chroma St is higher than the chroma threshold Sth (S440: Yes), the processor 210 excludes the candidate of interest Mi having a hue difference dH greater than or equal to the hue difference threshold dHth from the candidates. FIG. 7(C) shows an example of the candidate feature point pair M remaining by the process of FIG. 8. The pair of feature points Tt, Ts indicating parts having different hues from each other can be excluded. For example, between the first object OB1 and the third object OB3, the hues are different. The line RLa shown in FIG. 7(B) indicates the pair of the reference feature point Tt of the first object OB1 and the read feature point Ts of the third object OB3. As shown in FIG. 7(C), this pair is excluded (FIG. 8: S440(Yes), S445(No), S465).

[0055] When the first saturation St is less than or equal to the saturation threshold value Sth (S440: No), in S450, the processor 210 determines whether a second saturation condition CSs indicating that the second saturation Ss is higher than the saturation threshold value Sth is satisfied. When the second saturation Ss is higher than the saturation threshold value Sth (S450: Yes), the saturations between the first partial region Pt and the second partial region Ps can be significantly different. That is, the feature points Tt and Ts of the candidate of interest Mi are likely to indicate different parts from each other. In S465, the processor 210 excludes the candidate of interest Mi from the candidates of the feature point pair.

[0056] When the second saturation Ss is less than or equal to the saturation threshold value Sth (S450: No), in S455, the processor 210 determines whether a luminance condition CV indicating that the absolute value of the difference between the first luminance Vt and the second luminance Vs is less than the luminance difference threshold value dVth is satisfied. When the candidate of interest Mi is an appropriate pair of feature points Tt and Ts indicating the same part from each other, the absolute value of the difference between the luminances Vt and Vs (also referred to as the luminance difference dV) can be a small value. When the candidate of interest Mi is an inappropriate pair of feature points Tt and Ts indicating different parts from each other, the luminance difference dV can be a large value. Thus, even when the saturations St and Ss are low, the luminances Vt and Vs can appropriately represent the colors of the parts indicated by the feature points Tt and Ts.

[0057] When the luminance difference dV is less than the luminance difference threshold value dVth (S455: Yes), in S460, the processor 210 selects the candidate of interest Mi as a candidate to be retained. When the luminance difference dV is greater than or equal to the luminance difference threshold value dVth (S455: No), in S465, the processor 210 excludes the candidate of interest Mi from the candidate feature point pair M.

[0058] Note that, the looser the conditions for leaving the candidate of interest Mi, the larger the total number of appropriate candidates M that remain without being excluded can be. However, the total number of inappropriate candidates can also be large. The larger the total number of appropriate candidates, the lower the error in the correspondence relationship of coordinates described later. When the total number of inappropriate candidates is large, the error in the correspondence relationship of coordinates can increase. The parameters Sth, dHth, and dVth used in S440 - S455 may be determined experimentally in advance so that the error in the correspondence relationship of coordinates is acceptable.

[0059] For example, the larger the chromatic difference threshold dHth (S445), the looser the conditions for leaving the candidate of interest Mi. Even if the candidate of interest Mi is an appropriate pair of feature points Tt and Ts representing a portion of a highly saturated color, the chromatic difference dH can be a value greater than zero. A chromatic difference threshold dHth greater than zero allows such a chromatic difference dH. The chromatic difference threshold dHth may be set to a value greater than zero and less than the maximum possible value of the chromatic difference dH.

[0060] Also, the larger the luminance difference threshold dVth (S455), the looser the conditions for leaving the candidate of interest Mi. Even if the candidate of interest Mi is an appropriate pair of feature points Tt and Ts representing a portion of a low - saturation color, the luminance difference dV can be a value greater than zero. A luminance difference threshold dVth greater than zero allows such a luminance difference dV. The luminance difference threshold dVth may be set to a value greater than zero and less than the maximum possible value of the luminance difference dV.

[0061] Also, when the saturation threshold Sth (S440) is small, the attention candidate Mi showing a low first saturation St is processed at S445. Even if the reference representative color value Ct (Fig. 9(A)) is a chromatic color, when the first saturation St is low, the hue of the corresponding region of the read image IMs representing the printed image is likely to be different from the first hue Ht. That is, when the first saturation St is low, the error of the second hue Hs, and thus the error of the color difference dH, can become large. When the error of the color difference dH is large, an appropriate attention candidate Mi may be erroneously excluded (S445: No), and an inappropriate attention candidate Mi may be erroneously left (S445: Yes). As a result, the error of the coordinate correspondence relationship can increase. The saturation threshold Sth may be determined experimentally in advance so that the influence of the error of the color difference dH on the error of the coordinate correspondence relationship is mitigated. The saturation threshold Sth may be set to a value greater than zero and less than the maximum possible value of the saturation.

[0062] After S460 or S465, the processor 210 proceeds to S420 and executes the loop process S410 of the next attention candidate Mi. When the loop process S410 of all candidates M ends, at S470, the processor 210 stores the data representing the candidate feature point pair M to be left in the storage device 215 (e.g., the non-volatile storage device 230). Then, the processor 210 ends the process of Fig. 8, that is, the process of S230 of Fig. 5.

[0063] At S240, the processor 210 executes a selection process of the candidate feature point pair M that uses combinations of a plurality of candidates M. This selection process selects an appropriate candidate M (i.e., the candidate feature point pair M is verified) using combinations of a plurality of candidates M from the remaining plurality of candidates M.

[0064] FIG. 10 is a flowchart showing an example of the selection process of candidate feature point pairs M. The outline of the process in FIG. 10 is as follows. The processor 210 selects three candidate feature point pairs MA, MB, and MC (S520 - S560). The second candidate MB and the third candidate MC are selected from the region between the first radius r1 and the second radius r2 centered on the reference feature point Tt of the first candidate MA (details will be described later). Hereinafter, the combination of the selected candidates MA, MB, and MC is also referred to as a candidate combination MU. The processor 210 determines whether the combination condition CC is satisfied using the candidate combination MU (S565). When the combination condition CC is satisfied (S570: Yes), the processor 210 selects the candidate combination MU as the target combination MT (S573), and determines whether the shape condition CW1 is satisfied using the target combination MT (S575). When the shape condition CW1 is satisfied (S580: Yes), the processor 210 selects the three candidates MA, MBA, and MC of the target combination MT as candidates to be left (S585). In this embodiment, a plurality of candidate feature point pairs M left by the process in FIG. 10 are used for determining the coordinate correspondence.

[0065] Specifically, it is as follows. The processor 210 executes the loop process S510 (including S520 - S585) between the start L51s and the end L51e for each of the plurality of candidate feature point pairs M. In S520, the processor 210 selects the unprocessed candidate M as the first candidate MA.

[0066] In S525, the processor 210 selects a group of feature points N within the range between a first radius r1 and a second radius r2 centered on the feature point Tt of the first candidate MA. FIG. 11 is a diagram showing an example of the group of feature points N. In the figure, a reference image IMt and a plurality of reference feature points Tt are shown. One feature point Tt is selected as the first candidate MA. The first circle C1 is a circle with a first radius r1 centered on the feature point Tt of the first candidate MA. The second circle C2 is a circle with a second radius r2 centered on the feature point Tt of the first candidate MA. The selection range SR is the area between the first circle C1 and the second circle C2 (including the portions on the circles C1 and C2). The processor 210 selects a plurality of feature points Tt included in the selection range SR as the group of feature points N.

[0067] As will be described later, the processor 210 uses the three reference feature points Tt and the three read feature points Ts of the three candidates MA, MB, and MC to determine whether to leave the combinations of the candidates MA, MB, and MC. Specifically, when a condition including that the difference between the shape of the triangle formed by the three reference feature points Tt and the shape of the triangle formed by the three read feature points Ts is small is satisfied, the processor 210 determines that the combinations of the three candidates MA, MBA, and MC should be left. If the length of any one of the three sides of the triangle is significantly longer or significantly shorter than the lengths of the other sides, the error in the comparison of the shapes between the two triangles may increase. The candidates MB and MC combined with the first candidate MA are selected from the group of feature points N. The selection range SR (here, the radii r1 and r2) is experimentally determined in advance so that a figure suitable for the determination is formed by the combinations of the candidates MA, MB, and MC. For example, the first radius r1 may be a value of 1% or more and 20% or less of the size of the reference image IMt (for example, the length in the first direction Dx or the second direction Dy). The second radius r2 may be a value of 30% or more and 70% or less of the size of the reference image IMt.

[0068] After S525 (FIG. 10), the processor 210 selects a second candidate MB from the feature point group N (S540) and selects a third candidate MC (S560). Specifically, for each of the plurality of candidates M in the feature point group N, the processor 210 executes a loop process S530 (including S540 - S585) between the start L52s and the end L52e. In S540, the processor 210 selects an unprocessed candidate M from the feature point group N as the second candidate MB. Hereinafter, it is assumed that the second candidate MB is the j-th candidate in the feature point group N (also expressed as the second candidate MB[j]). The index j is selected, for example, from a range of zero or more and less than the total number NN of candidates M in the feature point group N. After S540, for each of the plurality of candidates M in the feature point group N, the processor 210 executes a loop process S550 (including S560 - S585) between the start L53s and the end L53e. In S560, the processor 210 selects an unprocessed candidate M from the feature point group N as the third candidate MC. Hereinafter, it is assumed that the third candidate MC is the k-th candidate in the feature point group N (expressed as the third candidate MC[k]). The index k is selected, for example, from a range of j + 1 or more and less than the total number NN of candidates M in the feature point group N. Thereby, a candidate M different from the second candidate MB is selected as the third candidate MC. In this way, the processor 210 selects a combination of three candidates MA, MB, and MC (that is, a candidate combination MU).

[0069] In S565, the processor 210 determines whether the combination condition CC is satisfied using the candidate combination MU. The combination condition CC is a condition for selecting the candidate combination MU as the target combination MT for determining the shape condition CW1 described later. The determination of the combination condition CC is performed using the triangle formed by the candidates MA, MB, and MC of the candidate combination MU.

[0070] Figures 12(A) and 12(B) are diagrams showing examples of triangles formed by candidate MAs, MBs, and MCs. Each candidate MA, MB, and MC represents a pair of a reference feature point Tt and a read feature point Ts, respectively. Figure 12(A) shows a triangle TRt (referred to as the reference triangle TRt) formed by the reference feature points Tta, Ttb, and Ttc of candidates MA, MB, and MC. Figure 12(B) shows a triangle TRs (referred to as the read triangle TRs) formed by the read feature points Tsa, Tsb, and Tsc of candidates MA, MB, and MC. The positions of one or more feature points may differ between the reference triangle TRt and the read triangle TRs.

[0071] Each figure shows symbols indicating each of the sides of the triangle, the lengths of the sides, and the magnitudes of the interior angles. Symbols starting with S (e.g., Sabt) indicate sides. The two characters following S indicate the two candidates M connected by the side (a, b, and c indicate candidates MA, MB, and MC, respectively). The t or s at the end of the symbol indicates the reference triangle TRt or the read triangle TRs, respectively. For example, side Sabt (Figure 12(A)) is the side connecting the reference feature point Tta and the reference feature point Ttb of the reference triangle TRt. The symbol obtained by replacing the leading S of the symbol indicating the side with L indicates the length of that side. For example, length Labt indicates the length of side Sabt. Symbols starting with A (e.g., Aat) indicate the magnitudes of the interior angles of the triangle. The one character following A indicates the candidate M forming the vertex of the interior angle (a, b, and c indicate candidates MA, MB, and MC, respectively). The t or s at the end of the symbol indicates the reference triangle TRt or the read triangle TRs, respectively. For example, interior angle Aat (Figure 12(A)) indicates the magnitude of the interior angle with the reference feature point Tta of the reference triangle TRt as the vertex. In other figures described later, symbols based on the above rules are also used. Note that direction Oct in Figure 12(A) indicates the direction of feature point Ttc (Figure 6: S320), and direction Ocs in Figure 12(B) indicates the direction of feature point Tsc (Figure 6: S325).

[0072] FIG. 13 is a flowchart showing an example of the process of determining the combination condition CC (FIG. 10: S565). FIG. 13 shows a process of determining whether or not one triangle satisfies the individual combination condition. The processor 210 executes the process of FIG. 13 for each of the reference triangle TRt (FIG. 12(A)) and the read triangle TRs (FIG. 12(B)). When both the reference triangle TRt and the read triangle TRs satisfy the individual combination condition, the processor 210 determines that the combination condition CC is satisfied. Hereinafter, the process of FIG. 13 will be described using the reference triangle TRt as an example. In the description of the process of FIG. 13, symbols in which the trailing letter (t or s) is omitted are used as symbols indicating each of the side, the side length, and the magnitude of the interior angle.

[0073] In S610, the processor 210 determines whether or not the three candidates MA, MB, and MC are different from each other. When two or more candidates are the same candidate (S610: No), in S640, the processor 210 determines that the individual combination condition is not satisfied, and ends the process of FIG. 13.

[0074] When the three candidates MA, MB, and MC are different from each other (S610: Yes), in S615, the processor 210 determines whether or not the first length condition CL1 is satisfied. The first length condition CL1 indicates that the first length Lab is within the allowable length range LR that is greater than the lower limit Lth1 and smaller than the upper limit Lth2 (the allowable length range LR is also simply referred to as the length range LR). When the first length Lab is outside the length range LR (S615: No), the processor 210 proceeds to S640. The length range LR (here, the lower limit Lth1 and the upper limit Lth2) is experimentally determined in advance so that a figure suitable for determination is formed by the combination of the three candidates MA, MB, and MC, similar to the radii r1, r2 of the selection range SR (FIG. 11). For example, the lower limit Lth1 may be the same as the first radius r1, and the upper limit Lth2 may be the same as the second radius r2.

[0075] When the first length Lab is within the length range LR (S615: Yes), in S620, the processor 210 determines whether a second length condition CL2 indicating that the second length Lac is within the length range LR is satisfied. When the second length Lac is outside the length range LR (S620: No), the processor 210 proceeds to S640.

[0076] When the second length Lac is within the length range LR (S620: Yes), in S625, the processor 210 determines whether the first interior angle condition CA is satisfied. The first interior angle condition CA indicates that the interior angle Aa is within an allowable interior angle range AR that is greater than the lower limit Ath1 and less than the upper limit Ath2 (the allowable interior angle range AR is also simply referred to as the interior angle range AR). When the interior angle Aa is outside the interior angle range AR (S625: No), the processor 210 proceeds to S640. The interior angle range AR (here, the lower limit Ath1 and the upper limit Ath2) is experimentally determined in advance so that, similar to the radii r1, r2 of the selection range SR (Fig. 11), a figure suitable for determination is formed by the combination of the three candidates MA, MB, and MC. For example, the lower limit Ath1 may be a value of 10 degrees or more and 60 degrees or less. The upper limit Ath2 may be a value of 90 degrees or more and 160 degrees or less.

[0077] When the interior angle Aa is within the interior angle range AR (S625: Yes), in S630, the processor 210 determines whether the scalene condition CQ is satisfied. The scalene condition CQ indicates that the lengths of the three sides of a triangle (here, the reference triangle TRt) are different from each other. That is, the scalene condition CQ indicates that the shape of the triangle is different from any of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle. For an isosceles triangle and an equilateral triangle, even if two sides of the same length are swapped, the original shape is maintained. When the reference triangle TRt is an isosceles triangle or an equilateral triangle, even if two appropriate read feature points Ts respectively associated with two reference feature points Tt are swapped, the difference between the shape of the reference triangle TRt and the shape of the read triangle TRs becomes small. As a result, candidates MA, MB, and MC including two inappropriate candidates M may be erroneously determined to be those to be left for determining the coordinate correspondence relationship. The same applies when the reference triangle TRt is a triangle similar to an isosceles triangle or a triangle similar to an equilateral triangle. Therefore, in this embodiment, when the scalene condition CQ is not satisfied (S630: No), the processor 210 determines in S640 that the individual combination condition is not satisfied and ends the process of FIG. 13. When the scalene condition CQ is satisfied (S630: Yes), the processor 210 determines in S635 that the individual combination condition is satisfied and ends the process of FIG. 13.

[0078] The specific configuration of the scalene condition CQ may be various configurations. In this embodiment, the scalene condition CQ is that all of the following conditions CQ1 - CQ3 are satisfied. (CQ1) The absolute value of the difference between the lengths Lab and Lac is greater than the length difference threshold dLth. (CQ2) The absolute value of the difference between the lengths Lab and Lbc is greater than the length difference threshold dLth. (CQ3) The absolute value of the difference between the lengths Lbc and Lac is greater than the length difference threshold dLth. Each of the conditions CQ1 - CQ3 indicates that the difference in the lengths of two sides is large. The scalene condition CQ defined by the conditions CQ1 - CQ3 indicates that the triangle does not include two sides having a difference in length less than or equal to the length difference threshold dLth. Similar to the radii r1, r2 of the selection range SR (Fig. 11), the length difference threshold dLth is experimentally determined in advance so that a figure suitable for judgment is formed by the combination of the three candidates MA, MB, and MC. For example, the length difference threshold dLth may be a value of 2% or more and 10% or less of the upper limit Lth2 referred to in S615, S620 (Fig. 13).

[0079] In S565 (Fig. 10), the processor 210 executes the process of Fig. 13 for each of the reference triangle TRt and the read triangle TRs. Then, when both the reference triangle TRt and the read triangle TRs satisfy the individual combination conditions, the processor 210 determines that the candidate combination MU satisfies the combination condition CC. When one or both of the reference triangle TRt and the read triangle TRs do not satisfy the individual combination conditions, the processor 210 determines that the candidate combination MU does not satisfy the combination condition CC.

[0080] In S570, the processor 210 branches the process according to the determination result of S565. When the combination condition CC is satisfied (S570: Yes), in S573, the processor 210 selects the candidate combination MU as the target combination MT. In S575, the processor 210 determines whether the shape condition CW1 is satisfied using the target combination MT.

[0081] FIG. 14 is a flowchart showing an example of a process for determining the shape condition CW1. When each of the three candidates MA, MB, and MC (FIGS. 12(A) and 12(B)) of the target combination MT shows an appropriate pair of feature points Tt and Ts, the shape of the reading triangle TRs is approximately the same as the shape of the reference triangle TRt. Also, when the object rotates within the image, the direction of the feature point (e.g., the direction Oct of the feature point Ttc) rotates together with the object indicated by the feature point. The shape condition CW1 is configured in consideration of the above properties. In this embodiment, it is determined that the shape condition CW1 is satisfied when all of the three conditions CR, CD, and CF described later are satisfied.

[0082] In S710, the processor 210 calculates the ratio of the lengths of two sides of the triangles TRt and TRs. Specifically, the processor 210 calculates the following two ratios Rt and Rs. Rt = Labt / Lact Rs = Labs / Lacs These ratios Rt and Rs are invariant with respect to the scale and rotation of the object in the images IMtg and IMs.

[0083] In S715, the processor 210 determines whether the ratio condition CR indicating that the ratio Rt / Rs is close to 1 is satisfied. Since the ratios Rt and Rs are invariant with respect to the scale and rotation of the object in the images IMt and IMs, the determination result of the ratio condition CR is invariant with respect to the scale and rotation. When the shape of the reading triangle TRs is the same as the shape of the reference triangle TRt, the ratio Rt / Rs = 1. Even when each of the candidates MA, MB, and MC shows an appropriate pair of feature points Tt and Ts, the ratio Rt / Rs may deviate from 1. The ratio condition CR is configured to allow such a deviation. For example, the ratio condition CR may be that the ratio Rt / Rs is greater than the lower limit Rth1 and less than the upper limit Rth2 (where Rth1 < 1 < Rth2).

[0084] When the ratio condition CR is satisfied (S715: Yes), in S720, the processor 210 determines whether a second interior angle condition CD indicating that the interior angle Abs of the reading triangle TRs is close to the interior angle Abt of the reference triangle TRt is satisfied. The interior angles Abt and Abs are invariant with respect to the scale and rotation of the objects in the images IMt and IMs. Therefore, the determination result of the second interior angle condition CD is invariant with respect to scale and rotation.

[0085] When the shape of the reading triangle TRs is the same as the shape of the reference triangle TRt, the interior angle Abs is the same as the interior angle Abt. Even when each of the candidates MA, MB, and MC indicates an appropriate pair of feature points Tt and Ts, the interior angle Abs may deviate from the interior angle Abt. The second interior angle condition CD is configured to allow such a deviation. For example, the second interior angle condition CD may be that the absolute value of the difference between the interior angle Abs and the interior angle Abt is less than the interior angle difference threshold dAth (where dAth > 0).

[0086] When the second interior angle condition CD is satisfied (S720: Yes), in S725, the processor 210 calculates the angle formed by the direction and side of the feature point. Specifically, the processor 210 calculates the following two angles Zt and Zs. Zt = AG(Oct, Sact) Zs = AG(Ocs, Sacs) AG is a function for deriving an angle. The directions Oct and Ocs are the directions Oct and Ocs of the feature points Ttc and Tsc (FIGS. 12(A) and 12(B)). As the directions Oct and Ocs of the feature points Ttc and Tsc, the directions calculated for calculating the feature amounts Ft and Fs (S320, S325 (FIG. 6)) may be adopted. The angle Zt is the angle formed by the side Sact and the direction Oct. The angle Zs is the angle formed by the side Sacs and the direction Ocs.

[0087] In this embodiment, the directions Oct and Ocs of the feature points Ttc and Tsc are gradient directions calculated according to the A-KAZE technique. When the object rotates within the images IMtg and IMs, the directions Oct and Ocs rotate together with the object indicated by the feature points Ttc and Tsc in the images IMtg and IMs. The angles Zt and Zs are invariant with respect to the scale and rotation of the object within the images IMt and IMs.

[0088] In S730, the processor 210 determines whether an angle condition CF indicating that the angle Zs of the reading triangle TRs is close to the angle Zt of the reference triangle TRt is satisfied. Since the angles Zt and Zs are invariant with respect to the scale and rotation of the object within the images IMtg and IMs, the determination result of the angle condition CF is invariant with respect to the scale and rotation. When the shape of the reading triangle TRs is the same as the shape of the reference triangle TRt and the feature points Ttc and Tsc indicate the same part of the same object, the angle Zs is approximately the same as the angle Zt. Even when each of the candidates MA, MB, and MC indicates an appropriate pair of the feature points Tt and Ts, the angle Zs can deviate from the angle Zt. The angle condition CF is configured to allow such a deviation. For example, the angle condition CF may be that the absolute value of the difference between the angles Zt and Zs is less than the angle difference threshold dZth (where dZth > 0).

[0089] When all of the conditions CR, CD, and CF are satisfied (S715: Yes, and S720: Yes, and S730: Yes), all of the candidates MA, MB, and MC are presumed to be appropriate pairs. In this case, in S735, the processor 210 determines that the shape condition CW1 is satisfied and ends the process of FIG. 14.

[0090] When one or more of the conditions CR, CD, and CF are not satisfied, it is presumed that one or more of the candidates M among the candidates MA, MB, and MC are inappropriate pairs. When one or more of S715: No, S720: No, and S730: No are satisfied, in S740, the processor 210 determines that the shape condition CW1 is not satisfied and ends the process of FIG. 14.

[0091] Figs. 15(A) to 15(D) are diagrams showing examples of the results of the process of Fig. 14. Fig. 15(A) shows an example of the reference triangle TRt, and Figs. 15(B) to 15(D) show examples of the read triangles TRs associated with the reference triangle TRt.

[0092] Fig. 15(B) shows a case where the candidates MA, MB, and MC are appropriate, and the scale is the same and the rotation angles are different between the reference image IMt and the read image IMs. In this case, since the rotation-invariant conditions CR, CD, and CF (Fig. 14) are satisfied, the shape condition CW1 is satisfied.

[0093] Fig. 15(C) shows a case where the candidates MA, MB, and MC are appropriate, and the rotation angles are the same and the scales are different between the reference image IMt and the read image IMs. In this case, since the scale-invariant conditions CR, CD, and CF (Fig. 14) are satisfied, the shape condition CW1 is satisfied.

[0094] Although not shown, even when the candidates MA, MB, and MC are appropriate and both the scale and the rotation angle are different between the reference image IMt and the read image IMs, since the conditions CR, CD, and CF (Fig. 14) are satisfied, the shape condition CW1 is satisfied.

[0095] Fig. 15(D) shows a case where the candidates MA and MB are appropriate, but the third candidate MC is inappropriate. Specifically, the feature point Tsc is different from the appropriate feature point Tscr. When the combination of the candidates MA, MB, and MC includes an inappropriate candidate M, the determination result of one or more of the conditions CR, CD, and CF (Fig. 14) can be No (that is, the shape condition CW1 is likely not to be satisfied).

[0096] Note that, the looser the conditions CR, CD, and CF (Fig. 14) are, the greater the total number of appropriate combinations of candidates MA, MB, and MC that satisfy the shape condition CW1 can be. However, the total number of inappropriate combinations of candidates MA, MB, and MC that satisfy the shape condition CW1 can also be large. The greater the total number of appropriate candidates, the lower the error in the coordinate correspondence relationship described later. When the total number of inappropriate candidates is large, the error in the coordinate correspondence relationship can increase. The conditions CR, CD, and CF (in this embodiment, the parameters Rth1, Rth2, dAth, and dZth) may be determined experimentally in advance so that the error in the coordinate correspondence relationship is acceptable. The lower limit Rth1 (S715) may be various values less than 1, for example, set to a value of 0.85 or more and less than 1. The upper limit Rth2 may be various values greater than 1, for example, set to a value greater than 1 and 1.15 or less. The inner angle difference threshold dAth (S720) may be various values greater than zero, for example, set to a value greater than zero and 10 degrees or less. The angle difference threshold dZth (S730) may be various values greater than zero, for example, set to a value greater than zero and 10 degrees or less.

[0097] After the process of Fig. 14, that is, after S575 (Fig. 10), at S580, the processor 210 branches the process according to the determination result of S575. When the shape condition CW1 is satisfied (S580: Yes), at S585, the processor 210 selects the candidates MA, MB, and MC of the target combination MT as the candidates to be retained. After S585, the processor 210 ends the loop process S550 for the current combination of candidates MA, MB, and MC.

[0098] When the combination condition CC is not satisfied (S570: No) and when the shape condition CW1 is not satisfied (S580: No), the processor 210 skips S585 and ends the loop process S550 for the current combination of candidates MA, MB, and MC.

[0099] Thereafter, the processor 210 executes a loop process S550 that uses each of the plurality of third candidate MCs, a loop process S530 that uses each of the plurality of second candidate MBs, and a loop process S510 that uses each of the plurality of first candidate MAs. After the completion of the repetitions of the loop processes S510, S530, and S550, at S590, the processor 210 stores data representing the candidate feature point pairs M to be left in the storage device 215 (e.g., the non-volatile storage device 230). Then, the processor 210 ends the process of FIG. 10, i.e., the process of S240 in FIG. 5.

[0100] FIG. 7(D) shows an example of candidate feature point pairs remaining after the process of FIG. 10. As shown in FIGS. 7(C) and 7(D), pairs of inappropriate feature points Tt and Ts can be excluded without remaining. For example, the line RLb shown in FIG. 7(C) indicates a pair of a reference feature point Tt at the upper right and a reading feature point Ts at the lower right of the fourth object OB4. Such a pair can be excluded from the candidate feature point pairs when the shape condition CW1 cannot be satisfied when combined with other pairs (FIG. 10: S580: No). As described in FIGS. 15(A)-15(D), combinations of candidates MA, MB, and MC that form appropriate reference triangles TRt and reading triangles TRs can remain as candidates. Combinations of candidates MA, MB, and MC that form inappropriate reference triangles TRt and reading triangles TRs are not left as candidates.

[0101] Note that even if a combination of candidates MA, MB, and MC including an appropriate candidate M does not satisfy the shape condition CW1, the appropriate candidate M can satisfy the shape condition CW1 by being combined with other appropriate candidate Ms. Thus, in the process of FIG. 10, even if the determination result of S570 or S580 is No, the candidates MA, MB, and MC are not immediately excluded. The processor 210 selects, as the candidate M to be left, a plurality of candidates M selected by S585 through the repeated loop processes S510, S530, and S550. The processor 210 excludes candidates M that are not selected even once by S585 through the repeated loop processes S510, S530, and S550.

[0102] After the process of FIG. 10, that is, after S240 of FIG. 5, at S250, the processor 210 determines the correspondence between the coordinates on the reference image IMt and the coordinates on the read image IMs. FIG. 16 is a diagram showing an example of the expression form of the coordinate correspondence. In the figure, the coordinates COt on the reference image IMt, the coordinates COs on the read image IMs, and the matrix Mtx that associates these coordinates COt and COs are shown. In this embodiment, the matrix Mtx represents a so-called affine transformation. In the figure, the coordinates COt, COs, and the matrix Mtx are represented in a homogeneous coordinate system. Each of the coordinates COt and COs is represented by a three-dimensional vector having the position Xt, Xs in the first direction Dx on the images IMt, IMs, the position Yt, Ys in the second direction Dy, and 1 as the third component. The matrix Mtx is a 3×3 matrix. As shown in the figure, the matrix Mtx is represented by six parameters a-f in two rows and three columns and three components (0, 0, 1) in the third row. Such a matrix Mtx can represent rotation, enlargement, reduction, translation, and skew. The matrix Mtx can be calculated by using three or more pairs of the coordinates COt and COs (that is, three or more feature point pairs).

[0103] The processor 210 uses a plurality of feature point pairs MP (that is, a plurality of candidate feature point pairs M remaining in S220-S240 of FIG. 5) to determine the matrix Mtx (here, six parameters a-f). The method for determining the matrix Mtx may be various methods. For example, the processor 210 may determine the matrix Mtx according to a method called RANdom SAmple Consensus (RANSAC). In this embodiment, as described in FIGS. 7(A)-7(D) and FIGS. 15(A)-15(D), inappropriate candidates of the feature point pairs MP are likely to have been removed. Therefore, the processor 210 may calculate the matrix Mtx by using all the remaining feature point pairs MP. The calculation method may be various methods (for example, the least squares method). For example, a function of OpenCV (Open Source Computer Vision Library) may be used for the determination of the matrix Mtx.

[0104] Upon the completion of S250 (Fig. 5), the process of Fig. 5, i.e., S120 in Fig. 4, ends. In S130, the processor 210 inspects the printed image. The inspection method may be various methods using the correspondence relationship of coordinates (Fig. 16). In this embodiment, the processor 210 generates data of the difference image using the correspondence relationship of coordinates. Figs. 17(A) and 17(B) are diagrams showing examples of the difference image. Fig. 17(A) shows the case where there is no defect in the printed image, and Fig. 17(B) shows the case where the printed image has a defect (here, omission Err). On the left side of each figure, the read images IMs, IMs2 and the reference image IMt arranged on the read images IMs, IMs2 according to the correspondence relationship of coordinates are shown. The images IMd, IMd2 on the right side of each figure represent examples of the difference image between the read images IMs, IMs2 and the reference image IMt. The processor 210 generates data of the difference images IMd, IMd2 representing the difference in color values (e.g., the absolute value of the difference in luminance values) between the read images IMs, IMs2 and the reference image IMt at positions associated by the correspondence relationship of coordinates. The read image IMs in Fig. 17(A) represents a printed image without defects. Therefore, the difference image IMd does not have a portion showing a large difference. The read image IMs2 in Fig. 17(B) represents a printed image having an omission Err. Therefore, in the difference image IMd2, the portion corresponding to the omission Err represents a larger difference compared to other portions.

[0105] In S140 (FIG. 4), the processor 210 outputs the inspection result. The method of outputting the inspection result may be various methods. In this embodiment, the processor 210 displays the differential image on the display unit 240 (FIG. 1). By observing the display unit 240, the operator can easily recognize the defects of the printed image. Alternatively, the processor 210 may output the data representing the inspection result to a storage device (for example, the non-volatile storage device 230 or an external storage device connected to the data processing device 200). Thereby, the data representing the inspection result is stored in the storage device. The data representing the inspection result can be used for various processes (for example, the overall inspection process of the T-shirt 700). After S140, the processor 210 ends the inspection process.

[0106] As described above, in this embodiment, the processor 210 executes the following processes according to the first program 231. In S355 - S360 of FIG. 6, the processor 210 uses the feature amount Fs of each of the plurality of read feature points Ts and the feature amount Ft of each of the plurality of reference feature points Tt to obtain a plurality of candidate feature point pairs M that are pairs of the feature point Ts and the feature point Tt. The read feature point Ts is a feature point in the gray read image IMsg (FIG. 7), that is, a feature point in the read image IMs. The reference feature point Tt is a feature point in the gray reference image IMtg, that is, a feature point in the reference image IMt.

[0107] In the process of FIG. 5 (including S240), the processor 210 selects a plurality of candidate feature point pairs M that satisfy the selection condition CP from the plurality of candidate feature point pairs M as a plurality of feature point pairs MP. The selection condition CP includes the conditions for selecting the candidate feature point pair M in S240. The condition of S240 includes the shape condition CW1 of S575 in FIG. 10. The shape condition CW1 is an example of the first condition for selecting a target combination MT, which is a combination of three candidates MA, MB, and MC, as three feature point pairs MP (hereinafter, the shape condition CW1 is also referred to as the first condition CW1).

[0108] The first condition CW1 includes the conditions CR, CD, and CF in FIG. 14. The ratio condition CR (S715) is determined using the ratios Rt and Rs of the lengths of two sides of the triangles TRt and TRs formed by the three candidate MAs, MBs, and MCs (FIGS. 12(A) and 12(B)). Specifically, the ratio Rt is the ratio of the lengths Labt and Lact of the two sides Sabt and Sact of the reference triangle TRt. The ratio Rs is the ratio of the lengths Labs and Lacs of the two sides Sabs and Sacs of the reading triangle TRs. The second interior angle condition CD (S720) is determined using the interior angles Abt and Abs of the triangles TRt and TRs. The angle condition CF (S730) is determined using the angles Zt and Zs. The angle Zt (FIG. 12(A)) is the angle formed by the side Sact (i.e., the line segment) connecting the two reference feature points Tta and Ttc and the direction Oct associated with one of the two reference feature points Tta and Ttc, which is Ttc. The angle Zs (FIG. 12(B)) is the angle formed by the side Sacs (i.e., the line segment) connecting the two reading feature points Tsa and Tsc and the direction Ocs associated with one of the two reading feature points Tsa and Tsc, which is Tsc. Thus, in this embodiment, the first condition CW1 is determined using the ratios Rt and Rs, the interior angles Abt and Abs, and the angles Zt and Zs.

[0109] In S250 (FIG. 5), the processor 210 uses a plurality of feature point pairs MP (i.e., a plurality of remaining candidate feature point pairs M) to determine the correspondence (in this embodiment, the matrix Mtx) between the coordinates COs on the reading image IMs and the coordinates COt on the reference image IMt.

[0110] Thus, in this embodiment, a plurality of candidate feature point pairs M that satisfy the selection condition CP are selected as a plurality of feature point pairs MP from the plurality of candidate feature point pairs M. The selection condition CP includes a first condition CW1 for selecting a target combination MT, which is a combination of N (in this embodiment, N = 3) candidates MA, MB, and MC, as N feature point pairs MP. The first condition CW1 is determined using the ratios Rt, Rs, the interior angles Abt, Abs, and the angles Zt, Zs. Therefore, the processor 210 can appropriately perform alignment between the read image IMs and the reference image IMt.

[0111] Also, in this embodiment, as described in S110 (FIG. 4), the read image IMs is represented by image data generated by optically reading a printed image IMpp. As described in FIG. 3(A), the reference image IMt is represented by image data for printing. Thus, the processor 210 can appropriately perform alignment between the read image IMs of the printed image IMpp and the reference image IMt represented by the image data for printing.

[0112] Also, in this embodiment, in S240 of FIG. 5, the processor 210 executes the process of FIG. 10. In the process of FIG. 10, the processor 210 selects a combination of N (in this embodiment, N = 3) candidate feature point pairs M as the target combination MT. Specifically, the process of FIG. 10 includes S565 - S573. In S565 - S573, the processor 210 selects a candidate combination MU, which is a combination of N candidate feature point pairs M, as the target combination MT. In S565 included in S565 - S573, the processor 210 determines a combination condition CC for selecting the candidate combination MU as the target combination MT. The combination condition CC includes the conditions CL1, CL2, and CA of FIG. 13.

[0113] The first length condition CL1 (S615) and the second length condition CL2 (S620) indicate that the lengths Lab and Lac of the line segments connecting the two feature points of the two candidate feature point pairs M included in the candidate combination MU are within the allowable length range LR. Specifically, in S615, the length Labt (Fig. 12(A)) and the length Labs (Fig. 12(B)) are evaluated. The length Labt is the length of the line segment (side Sabt) connecting the two reference feature points Tta and Ttb. The length Labs is the length of the line segment (side Sabs) connecting the two read feature points Tsa and Tsb. In S620, the length Lact (Fig. 12(A)) and the length Lacs (Fig. 12(B)) are evaluated. The length Lact is the length of the line segment (side Sact) connecting the two reference feature points Tta and Ttc. The length Lacs is the length of the line segment (side Sacs) connecting the two read feature points Tsa and Tsc.

[0114] The first interior angle condition CA (S625) indicates that the interior angle Aa of the triangle formed by the three feature points of the three candidates MA, MB, and MC included in the candidate combination MU is within the allowable interior angle range AR. Specifically, the interior angle Aat (Fig. 12(A)) and the interior angle Aas (Fig. 12(B)) are evaluated. The interior angle Aat is the magnitude of the interior angle of the reference triangle TRt formed by the three reference feature points Tta, Ttb, and Ttc. The interior angle Aas is the magnitude of the interior angle of the read triangle TRs formed by the three read feature points Tsa, Tsb, and Tsc.

[0115] In this way, the combination condition CC includes the length conditions CL1 and CL2 of the line segments connecting two feature points and the interior angle condition CA of the triangle formed by three feature points. Therefore, the processor 210 can select an appropriate candidate combination MU as the target combination MT.

[0116] Also, in this embodiment, the combination condition CC (FIG. 10: S565) includes the scalene condition CQ (S630) of FIG. 13. The scalene condition CQ indicates that the triangle formed by the three feature points of the three candidate MAs, MBs, and MCs included in the candidate combination MU does not include two sides having a length difference of less than or equal to the length difference threshold dLth. Specifically, three pairs of side pairs can be obtained from the three sides Sabt, Sact, and Sbct of the reference triangle TRt (FIG. 12(A)) (Sabt - Sact, Sabt - Sbct, Sbct - Sact). When the absolute value of the length difference of each of the three pairs of side pairs is greater than the length difference threshold dLth, the processor 210 determines that the reference triangle TRt satisfies the scalene condition CQ. The scalene condition CQ is satisfied when the shape of the reference triangle TRt is different from any of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle. The same applies to the reading triangle TRs (FIG. 12(B)). Thus, the combination condition CC includes the scalene condition CQ. Therefore, the processor 210 can reduce the possibility of erroneously selecting an inappropriate candidate combination MU as the target combination MT.

[0117] Also, in this embodiment, the selection condition CP (FIG. 5) further includes the condition for selecting the candidate feature point pair M at S230. The condition at S230 includes the conditions CSt, CH, CSs, and CV of S425 - S455 in FIG. 8. The conditions CSt, CH, CSs, and CV are defined using the saturation St, hue Ht, and luminance Vt of the first partial region Pt (FIG. 9(A)) and the saturation Ss, hue Hs, and luminance Vs of the second partial region Ps (FIG. 9(B)). The saturation St, hue Ht, and luminance Vt are examples of the first representative color value CJt of the first partial region Pt including the feature point Tt in the reference image IMt. The saturation Ss, hue Hs, and luminance Vs are examples of the second representative color value CJs of the second partial region Ps including the feature point Ts in the read image IMs. The entirety of the conditions CSt, CH, CSs, and CV is an example of the second condition for selecting the candidate feature point pair M as the feature point pair MP (hereinafter, the entirety of the conditions CSt, CH, CSs, and CV is referred to as the second condition CW2).

[0118] Thus, in this embodiment, the selection condition CP further includes a second condition CW2 for selecting the candidate feature point pair M as the feature point pair MP. The second condition CW2 is determined using a first representative color value CJt (including St, Ht, Vt) of a first partial region Pt including the feature point Tt in the reference image IMt and a second representative color value CJs (including Ss, Hs, Vs) of a second partial region Ps including the feature point Ts in the read image IMs. Therefore, the processor 210 can select an appropriate candidate feature point pair M as the feature point pair MP.

[0119] Also, in this embodiment, the first representative color value CJt (FIG. 9(A)) indicates a first saturation St and a first hue Ht, and the second representative color value CJs (FIG. 9(B)) indicates a second hue Hs. As shown in S440 and S445 in FIG. 8, when the first representative color value CJt indicates a saturation St higher than the saturation threshold Sth (S440: Yes), the processor 210 determines the second condition CW2 using the hue Ht indicated by the first representative color value CJt and the hue Hs indicated by the second representative color value CJs (S445). As described above, when the first saturation St of the first representative color value CJt is higher than the saturation threshold Sth, the error of the second hue Hs of the second representative color value CJs can be a small value. Therefore, the processor 210 can appropriately determine the second condition CW2.

[0120] Also, in this embodiment, the first representative color value CJt (FIG. 9(A)) indicates a first saturation St and a first luminance Vt, and the second representative color value CJs (FIG. 9(B)) indicates a second saturation Ss and a second luminance Vs. As shown in S440, S450, and S455 in FIG. 8, when the saturation St of the first representative color value CJt and the saturation Ss of the second representative color value CJs are equal to or lower than the saturation threshold Sth (S440: No, S450: No), the processor 210 determines the second condition CW2 using the luminance Vt indicated by the first representative color value CJt and the luminance Vs indicated by the second representative color value CJs (S455). As described above, even when the saturations St and Ss are low, the luminances Vt and Vs can appropriately represent the colors of the portions indicated by the feature points Tt and Ts. Therefore, the processor 210 can appropriately determine the second condition CW2.

[0121] Also, in this embodiment, as shown in FIG. 5, after S230, S240 is executed. In S230 (FIG. 8), the processor 210 selects a plurality of candidate feature point pairs M that satisfy the second condition CW2. In S240 (FIG. 10), the processor 210 uses the plurality of candidate feature point pairs M selected in S230 to select a plurality of candidate feature point pairs M that satisfy the first condition CW1. As described with reference to FIG. 10, the processor 210 executes the determination (S575) of the first condition CW1 for each of the plurality of target combinations MT. When selecting a target combination MT (three candidate feature point pairs M) from p (p is an integer of 3 or more) candidate feature point pairs M, the total number of target combinations MT is represented by the combination (pC3). The total number of target combinations MT increases rapidly as the total number p of candidate feature point pairs M increases. If S240 is executed before S230, the total number of target combinations MT may increase, and the computational load of S240 (the determination of the first condition CW1) may increase. In this embodiment, since the total number of candidate feature point pairs M that can be included in the target combination MT is reduced by S230, the computational load can be reduced.

[0122] B. Second Embodiment: FIG. 18 is a flowchart showing an example of a process of acquiring data of a reference image. Different from the above embodiments, in this embodiment, image data generated by optically reading an image printed using reference image data is acquired as the data of the reference image. This acquisition process may be performed in various cases. For example, the printing of the image on the T-shirt 700 may be performed multiple times using the same printing image data. In this case, the data of the reference image may be acquired by reading the image printed on the T-shirt 700. Hereinafter, it is assumed that the data processing device 200, the printing device 900, and the reading device 100 in FIG. 1 are used for the acquisition process. It is assumed that the data of the target image IMp (FIG. 3(A)) is the printing image.

[0123] In this embodiment, an operator inputs an instruction to start the acquisition process into the data processing apparatus 200 by operating the operation unit 250. In response to the start instruction, the processor 210 executes the acquisition process according to the second program 232.

[0124] In S810, the processor 210 causes the reading apparatus 100 to execute printing of an image onto the T-shirt 700 using image data for acquisition of a reference image (here, data of the target image IMp (FIG. 3(A))). As with the T-shirt 700 in FIG. 2, the target image IMp is printed on the T-shirt 700. Hereinafter, it is assumed that the printed image on the T-shirt 700 has no defect. Note that the operator may confirm that the printed image has no defect by observing the printed image on the T-shirt 700. If the printed image has a defect, the operator may cause the data processing apparatus 200 to execute S810 again to obtain a printed image without a defect.

[0125] In S820, the T-shirt 700 is placed on the support unit 140 of the reading apparatus 100 (FIG. 2) so that the printed image is visible. In this embodiment, the operator places the T-shirt 700 on the support unit 140. Alternatively, a machine (for example, a robot arm) may place the T-shirt 700 on the support unit 140. After the placement of the T-shirt 700, the operator inputs a progress instruction by operating the operation unit 250. In response to the progress instruction, the processor 210 supplies a reading instruction to the reading apparatus 100. The control device 110 of the reading apparatus 100 reads the T-shirt 700 in response to the reading instruction. The reading process is performed in the same manner as S110 in FIG. 4. The control device 110 generates data of a read image representing the read T-shirt 700 and supplies the generated data to the data processing apparatus 200. The processor 210 of the data processing apparatus 200 stores the acquired data of the read image as data of a reference image in the storage device 215 (for example, the nonvolatile storage device 230). Although not shown, the reference image represents a portion including the printed image of the T-shirt 700, similar to the read image IMs in FIG. 3(B).

[0126] The data of the reference image obtained by the process of FIG. 18 is used in the inspection process of FIG. 4 in place of the data of the reference image IMt. In this way, the reference image may be represented by image data generated by optically reading a printed image. Also in this case, the processor 210 can appropriately perform alignment between the read image IMs and the reference image, similar to the first embodiment.

[0127] Note that the start instruction for the process of FIG. 18 and the progress instruction for S820 may be input to the data processing apparatus 200 via the communication interface 270 by another apparatus different from the data processing apparatus 200.

[0128] C. Third Embodiment: FIG. 19 is a diagram showing another embodiment of the scalene condition (FIG. 13: S630). In the figure, S630b executed in place of S630 of FIG. 13 is shown. In S630b, the processor 210 determines whether or not the scalene condition CQb is satisfied. The scalene condition CQb indicates that the lengths of the three sides of a triangle are different from each other, similar to the scalene condition CQ (FIG. 13). That is, the scalene condition CQb indicates that the shape of the triangle is different from any of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle.

[0129] In this embodiment, the scalene condition CQb is that all of the following conditions CQb1 - CQb3 are satisfied. (CQb1) The ratio Lab / Lac of the lengths Lab and Lac is outside the ratio range PR. (CQb2) The ratio Lab / Lbc of the lengths Lab and Lbc is outside the ratio range PR. (CQb3) The ratio Lbc / Lac of the lengths Lbc and Lac is outside the ratio range PR Here, the ratio range PR is a range from a lower limit RRth1 or more to an upper limit RRth2 or less (where RRth1 < 1 < RRth2).

[0130] Each of the conditions CQb1 - CQb3 indicates that the ratio of the lengths of two sides is different from 1. The scalene condition CQb indicates that a triangle (e.g., the reference triangle TRt or the reading triangle TRs) does not include two sides having a length ratio within the ratio range PR including 1. The lower limit RRth1 and the upper limit RRth2 are experimentally determined in advance by the combination of the three candidates MA, MB, and MC so that a figure suitable for determination is formed, similar to the radii r1, r2 of the selection range SR (Fig. 11). For example, the lower limit RRth1 may be a value of 0.9 or more and 0.98 or less. The upper limit RRth2 may be a value of 1.02 or more and 1.1 or less.

[0131] When the scalene condition CQb is satisfied (S630b: Yes), the processor 210 proceeds to S635 (Fig. 13) and determines that the individual combination condition is satisfied. When the scalene condition CQb is not satisfied (S630b: No), the processor 210 proceeds to S640 and determines that the individual combination condition is not satisfied.

[0132] As described above, in this embodiment, the combination condition CC (Fig. 10: S565) includes the scalene condition CQb (Fig. 19: S630b). Therefore, the processor 210 can reduce the possibility of erroneously selecting an inappropriate candidate combination MU as the target combination MT, similar to the embodiment of Fig. 13.

[0133] D. Modification example: (1) The first condition CW1 (Fig. 10: S575) is not limited to the conditions described in Fig. 14, and may be various conditions indicating that a target combination MT, which is a combination of N (N is 2 or 3) candidate feature point pairs M, is composed of appropriate N candidate feature point pairs M. For example, the reading device 100 may be configured such that the scale for the object is approximately the same between the reference image IMt and the read image IMs. In this case, the first condition CW1 may further include a third length condition indicating that the length of a specific side of the triangle is within an allowable range. The side lengths may be, for example, the lengths Lbct and Lbcs of the sides Sbct and Sbcs (Fig. 12(A), Fig. 12(B)). When the scale is approximately the same between the reference image IMt and the read image IMs, the third length condition can easily exclude the target combination MT including inappropriate candidate feature point pairs M. The allowable range of the third length condition may be determined experimentally in advance, similar to the allowable length range LR in Fig. 13 (for example, the allowable range of the third length condition may be the same as the allowable length range LR). Thus, the first condition CW1 may be defined using the lengths of the sides of the triangle.

[0134] Also, the reading device 100 may be configured such that the rotation angle of the object is approximately the same between the reference image IMt and the read image IMs. In this case, in S320 and S325 of Fig. 6, non-rotation-invariant feature amounts Ft and Fs may be calculated (for example, BRIEF (Binary Robust Independent Elementary Features)). And the calculation of the direction of the feature points may be omitted. The condition using the direction of the feature points (for example, the angle condition CF) may be omitted.

[0135] The first condition CW1 may be a condition that can be determined using two pairs of candidate feature points M (for example, the third length condition). In this case, the candidate combination MU and the target combination MT may be combinations of two pairs of candidate feature points M. The first condition CW1 is preferably determined using one or more of four types of parameters, namely, the ratio of the lengths of two sides of a triangle formed by three feature points (for example, the ratios Rt, Rs), the lengths of the sides of the triangle (for example, the lengths Lbct, Lbcs), the interior angles of the triangle (for example, the interior angles Abt, Abs), and the angle formed by the line segment connecting two feature points and the direction associated with one of the two feature points (for example, the angles Zt, Zs).

[0136] (2) The combination condition CC (FIG. 10: S565) is not limited to the conditions described in FIGS. 13 and 19, and may be various conditions indicating that a combination of N (N is 2 or 3) pairs of candidate feature points M forms a figure suitable for the determination of the first condition CW1. For example, the combination condition CC may include a condition indicating that the lengths Lbct, Lbcs (FIGS. 12(A) and 12(B)) are within the allowable length range LR. The scalene condition CQ (FIG. 13) may be composed of one or two conditions arbitrarily selected in advance from the conditions CQ1 - CQ3. The scalene condition CQb (FIG. 19) may be composed of one or two conditions arbitrarily selected in advance from the conditions CQb1 - CQb3. The combination condition CC may be composed of one or more conditions arbitrarily selected in advance from the first length condition CL1, the second length condition CL2, the first interior angle condition CA, the scalene condition CQ, and the scalene condition CQb. Note that S565 - S570 in FIG. 10 (that is, the determination of the combination condition CC) may be omitted.

[0137] (3) The selection process of feature point pairs using pornographic information (Fig. 5: S230) is not limited to the process of Fig. 8 and may be various processes. For example, instead of the processes of S425 and S435, the processor 210 may calculate the hue of each pixel in the first partial region Pt, and calculate the representative hue of the first partial region Pt using the plurality of hues. The same applies to saturation and brightness. The same also applies to the representative color value of the second partial region Ps. Further, the second condition CW2 may be various conditions using the first representative color value of the first partial region Pt and the second representative color value of the second partial region Ps. For example, the second condition CW2 may be either one of the hue condition CH and the brightness condition CV. It is preferable that the second condition CW2 indicates that the first representative color value of the first partial region Pt is similar to the second representative color value of the second partial region Ps.

[0138] (4) The alignment process may be various processes instead of the process of Fig. 5. For example, S240 may be executed before S230. Also, S230 (that is, the determination of the second condition CW2) may be omitted.

[0139] (5) The detection method of the feature points Tt and Ts (that is, the key points) may be various methods of detecting points indicating parts of objects in the image instead of the methods described in S310 and S315 (Fig. 6). The detection method may be pre-selected, for example, from the search for extreme values (maximum and minimum values) using DoG (Difference-of-Gaussian), Harris corner detection, FAST (Features from Accelerated Segment Test) corner detection, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF).

[0140] The method for calculating the feature quantities Ft and Fs (i.e., the feature descriptors) may be various methods for calculating information that describes the features of the keypoints, instead of the method described in S320 and S325 (FIG. 6). The algorithm for calculating the feature descriptor may be preselected from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, A-KAZE, etc. Also, the method for calculating the distance dF between the feature quantities Ft and Fs may be various methods suitable for the data configurations of the feature quantities Ft and Fs. When the feature quantities Ft and Fs are represented by binary vectors, the distance dF may be the Hamming distance. Instead of the Hamming distance, the distance dF may be various other distances (e.g., norms such as the L1 norm, the L2 norm (also called the Euclidean distance), etc.). The norm is applicable to various feature descriptors.

[0141] (6) Processes other than the process of determining the coordinate correspondence relationship using a plurality of feature points Tt and Ts may be executed according to another program different from the first program 231. For example, the detection of the feature points Tt and Ts and the calculation of the feature quantities Ft and Ts (FIG. 6: S305 - S325) may be executed according to another program. S130 and S140 in FIG. 4 may be executed according to another program. Also, processes other than the process of determining the coordinate correspondence relationship using a plurality of feature points Tt and Ts may be performed by another device different from the data processing device 200.

[0142] (7) The printing medium is not limited to the T-shirt 700 and may be various types of clothing (e.g., various shirts such as polo shirts, outerwear, slacks, etc.). The printing medium may be various types of cloth such as clothing and bags. The printing medium is not limited to cloth and may be various media such as paper, film, leather, etc. Also, instead of an inkjet printing device, the printing device 900 may be a printing device of another type (e.g., a laser type).

[0143] (8) The correspondence relationship of coordinates (FIG. 16) may represent various transformations such as homography transformation instead of affine transformation. Also, the correspondence relationship of coordinates may be represented in various forms such as a lookup table instead of the matrix Mtx.

[0144] The correspondence relationship of coordinates may be used in various processes, not limited to inspection. For example, in the processing of metal parts, the position and orientation of the metal part with respect to the tool may deviate. Here, by analyzing a captured image of the metal part by a digital camera fixed to the tool, the position and orientation of the metal part with respect to the tool may be determined. For example, the processor 210 may determine the reference position of the metal part on the captured image (and thus the position and orientation of the metal part with respect to the tool) by performing alignment between a reference image representing a portion indicating the reference position of the metal part and the captured image. As the alignment process, the process of the above embodiment (FIG. 5) or the process of the above modification example may be adopted. Thus, the read image may be an image representing an object read by a line sensor such as the read sensor 180 (FIG. 2), or may be an image representing an object read by an area sensor such as a digital camera. Also, the reference image may be various images instead of an image related to printing (for example, a pre-prepared image representing a specific portion of an object). In any case, the image may be a grayscale image instead of a color image.

[0145] (9) The data processing device that determines the correspondence relationship of coordinates may be not limited to a personal computer (for example, the data processing device 200 (FIG. 1)), but may be various other devices (for example, a smartphone, a tablet computer, a control device incorporated in a reading device, etc.). Also, a plurality of devices (for example, computers) that can communicate with each other via a network may share the function of data processing by the data processing device in part, and as a whole, provide the function of data processing (a system including these devices corresponds to the data processing device).

[0146] In each of the above embodiments, part of the configuration implemented by hardware may be replaced with software, and conversely, part or all of the configuration implemented by software may be replaced with hardware. For example, the process of S250 in FIG. 5 may be executed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0147] Also, when part or all of the functions of the present disclosure are implemented by a computer program, the program can be provided in a form stored in a computer-readable recording medium (for example, a non-transitory recording medium). The program can be used in a state stored in the same or a different recording medium (a computer-readable recording medium). The "computer-readable recording medium" includes not only portable recording media such as memory cards and CD-ROMs, but also internal storage devices in a computer such as various ROMs, and external storage devices connected to a computer such as hard disk drives.

[0148] The above embodiments and modifications can be combined as appropriate. Also, the above embodiments and modifications are for facilitating the understanding of the present disclosure and do not limit the present invention. The present invention can be changed and improved without departing from its gist, and equivalents of the present invention are included therein.

Explanation of Reference Numerals

[0149] 100…Reading device, 110…Control device, 120…Conveying device, 122…Position sensor, 130…Table, 140…Support part, 180…Reading sensor, 190…Housing, 200…Data processing device, 210…Processor, 215…Storage device, 220…Volatile memory device, 230…Non-volatile memory device, 231…First program, 232…Second program, 240…Display part, 250…Operation part, 270…Communication interface, 700…T-shirt, 900…Printing device, Abs, Abt…Inner angle, AR…Allowable inner angle range, CC…Combination condition, CJt…First representative color value, CJs…Second representative color value, COs, COt…Coordinates, CP…Selection condition, CW1…First condition, CW2…Second condition, Ocs, Oct…Direction, Fs, Ft…Feature amount, Hs, Ht…Hue, IMs, IMs2…Read image, IMt…Reference image, Labs, Labt, Lacs, Lact, Lbcs, Lbct…Length, Sabs, Sabt, Sacs, Sact, Sbcs, Sbct…Side, LR…Allowable length range, Rs, Rt…Ratio, PR…Ratio range, M…Candidate feature point pair, MP…Feature point pair, MT…Target combination, Pt…First partial area, Ps…Second partial area, Ss, St…Chroma, Sth…Chroma threshold, TRs, TRt…Triangle, Ts, Tt…Feature point, Vs, Vt…Luminance, Zs, Zt…Angle

Claims

1. A program, comprising: a candidate acquisition function that acquires a plurality of candidate feature point pairs that are pairs of feature points between the read image and the reference image by using the feature amounts of the respective feature points of the plurality of feature points in the read image and the feature amounts of the respective feature points of the plurality of feature points in the reference image; a pair selection function that selects a plurality of candidate feature point pairs that satisfy a selection condition from the plurality of candidate feature point pairs as a plurality of feature point pairs, wherein the selection condition includes a first condition for selecting a target combination that is a combination of N (N is 2 or 3) candidate feature point pairs as N feature point pairs, and the first condition is determined by using one or more of four types of parameters, namely, the ratio of the lengths of two sides of a triangle formed by three feature points, the length of the sides of the triangle, the interior angle of the triangle, and the angle formed by a line segment connecting two feature points and the direction associated with one of the two feature points, and the pair selection function; a determination function that determines a correspondence relationship between the coordinates on the read image and the coordinates on the reference image by using the plurality of feature point pairs; A program for causing a computer to implement the above.

2. The program according to claim 1, wherein the read image is represented by image data generated by optically reading a printed image, the reference image is represented by image data generated by optically reading an image printed using printing image data or reference image data. A program.

3. The program according to claim 1 or 2, further comprising: a combination selection function for causing a computer to select a combination of N candidate feature point pairs as the target combination, wherein the combination condition for selecting the combination of the N candidate feature point pairs as the target combination includes one or both of the following: the interior angle of a triangle formed by three feature points included in the N candidate feature point pairs is within an allowable interior angle range, and the length of a line segment connecting two feature points included in the N candidate feature point pairs is within an allowable length range. A program.

4. The program according to claim 1 or 2, further comprising: a combination selection function for causing a computer to select a combination of N candidate feature point pairs as the target combination, The combination conditions for selecting the combination of the N candidate feature point pairs as the target combination are that the triangle formed by three feature points included in the N candidate feature point pairs does not include two sides having a difference in length of less than or equal to a difference threshold, and that the triangle does not include two sides having a ratio of lengths within a ratio range including 1, either one or both of which are included. Program. **Claim 5** A program according to claim 1 or 2, wherein the selection condition further includes a second condition for selecting the candidate feature point pair as the feature point pair, and the second condition is determined using a first representative color value of a first partial region including the feature point in the reference image and a second representative color value of a second partial region including the feature point in the read image. Program. **Claim 6** A program according to claim 5, wherein the first representative color value indicates saturation and hue, the second representative color value indicates hue, and the pair selection function makes a determination of the second condition using the hue indicated by the first representative color value and the hue indicated by the second representative color value when the first representative color value indicates the saturation higher than a saturation threshold. Program. **Claim 7** A program according to claim 5, wherein the first representative color value indicates saturation and luminance, the second representative color value indicates saturation and luminance, and the pair selection function makes a determination of the second condition using the luminance indicated by the first representative color value and the luminance indicated by the second representative color value when the saturation of the first representative color value and the saturation of the second representative color value are less than or equal to a saturation threshold. Program. **Claim 8** A program according to claim 5, wherein the pair selection function selects a plurality of candidate feature point pairs that satisfy the first condition using a plurality of candidate feature point pairs that satisfy the second condition. Program. **Claim 9** A data processing device, comprising a candidate acquisition unit that acquires a plurality of candidate feature point pairs that are pairs of feature points in the read image and feature points in the reference image, using the feature amounts of the respective feature points in the plurality of feature points in the read image and the feature amounts of the respective feature points in the plurality of feature points in the reference image. A pair selection unit that selects a plurality of candidate feature point pairs that satisfy selection conditions from the plurality of candidate feature point pairs as a plurality of feature point pairs, wherein the selection conditions include a first condition for selecting a target combination, which is a combination of N (N is 2 or 3) candidate feature point pairs, as N feature point pairs, and the first condition is determined using one or more of four types of parameters: the ratio of the lengths of two sides of a triangle formed by three feature points, the length of the side of the triangle, the interior angle of the triangle, and the angle formed by a line segment connecting two feature points and a direction associated with one of the two feature points; the pair selection unit; A determination unit that determines a correspondence relationship between coordinates on the captured image and coordinates on the reference image using the plurality of feature point pairs; A data processing device comprising the above.

Images