Program and image processing device

Abstract

To provide a program and an image processing device that perform alignment between a plurality of images.SOLUTION: A plurality of candidate feature point pairs, which are pairs of feature points in a scanned image and a feature point in a reference image, are acquired. A plurality of target feature point pairs are determined from the plurality of candidate feature point pairs. Using the plurality of target feature point pairs, a correspondence relation between coordinates on the scanned image and coordinates on the reference image is determined. Processing of determining the plurality of target feature point pairs includes processing of determining whether or not a set of the plurality of candidate feature point pairs is adopted as a candidate for the plurality of target feature point pairs by using the set.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] This specification relates to registration between multiple images. [Background technology]

[0002] In various processes, alignment between multiple images can be performed. Patent Document 1 discloses a technology for detecting defects in an image formed on paper by an image forming device such as a printer or copier. In this technology, a marker image for position determination is formed on the same paper along with a job image instructed to be printed by a user. An image reading unit reads the paper surface and generates a read image. An inspection unit determines the position of the read image corresponding to the reference image based on each feature point of the job image and marker image extracted from the read image to be inspected and each feature point of the job image and marker image extracted from the reference image. The inspection unit compares the reference image and the read image after alignment and detects image areas where the difference in pixel values ​​is greater than or equal to a threshold as defects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-112440 Summary of the Invention [Problem to be solved by the invention]

[0004] Aligning multiple images is not easy and requires some ingenuity.

[0005] This specification discloses a technique for performing registration between multiple images. [Means for solving the problem]

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] A program comprising: a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in a read image and feature points in a reference image, by using feature amounts of each of the plurality of feature points in the read image and feature amounts of each of the plurality of feature points in the reference image; and a pair determination function that determines a plurality of target feature point pairs from the plurality of candidate feature point pairs, wherein when the total number of candidate feature point pairs at a specific stage of determining the plurality of target feature point pairs is a first number, the function determines the plurality of target feature point pairs by executing a process including a first narrowing down process that narrows down candidates for the plurality of target feature point pairs, The program causes a computer to realize a first pair determination function, which is a process of using a first type set, which is a set of N (N is an integer equal to or greater than 2) candidate feature points pairs, to determine whether or not the first type set is to be a candidate for the plurality of target feature points pairs, and a second pair determination function, which determines the plurality of target feature points pairs without executing the first narrowing down process, in a first specific case in which the total number of candidate feature points pairs is a second number greater than the first number, and a correspondence determination function, which determines correspondence between coordinates on the scanned image and coordinates on the reference image using the plurality of target feature points pairs.

[0008] According to this configuration, the plurality of target feature points pairs are determined in accordance with the total number of candidate feature points pairs, so that the alignment between the read image and the reference image can be performed appropriately.

[0009] [Application Example 2] A program including a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in a read image and feature points in a reference image, using feature amounts of each of the feature points in the read image and the feature amounts of each of the feature points in the reference image; and a pair determination function that determines a plurality of target feature point pairs from the plurality of candidate feature point pairs, the function determining the plurality of target feature point pairs by executing a process that includes a fourth narrowing-down process that narrows down the candidates for the plurality of target feature point pairs when an index value indicating the number of colors in the read image or a second image that is the reference image is a third value. The fourth narrowing down process is a process of using a fourth type set, which is a set of P (P is an integer of 2 or more) candidate feature points pairs, to determine whether or not the fourth type set is to be a candidate for the plurality of target feature points pairs; and a fifth pair determination function of determining the plurality of target feature points pairs without executing the fourth narrowing down process if the index value is a fourth value greater than the third value. The program causes a computer to realize the pair determination function and a correspondence determination function of determining a correspondence relationship between coordinates on the read image and coordinates on the reference image using the plurality of target feature points pairs.

[0010] According to this configuration, since a plurality of target feature point pairs are determined according to the index value indicating the number of colors, it is possible to appropriately align the scanned image with the reference image.

[0011] [Application Example 3] A program including: a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in a read image and feature points in a reference image, by using feature amounts of each of the plurality of feature points in the read image and feature amounts of each of the plurality of feature points in the reference image; and a pair determination function that determines a plurality of target feature point pairs from the plurality of candidate feature point pairs, the pair determination function determining the plurality of target feature point pairs by executing a process including a narrowing-down process that narrows down candidates for the plurality of target feature point pairs, the narrowing-down process using a target set, which is a set of two candidate feature point pairs, to narrow down the target set to the plurality of target feature point pairs. a pair determination function that determines whether or not a target set is a candidate for a feature point pair, and the condition for determining that the target set is a candidate for the plurality of target feature point pairs is determined using either a first angle formed by a line segment connecting a first feature point and a second feature point that are respectively associated with the two candidate feature point pairs and a direction associated with the first feature point, or both a second angle formed by the line segment and a direction associated with the second feature point and the first angle; and a correspondence determination function that determines a correspondence relationship between coordinates on the read image and coordinates on the reference image using the plurality of target feature point pairs.

[0012] According to this configuration, the condition for making a target set a candidate for multiple target feature point pairs is determined using either a first angle formed by a line segment connecting a first feature point and a second feature point that are respectively associated with two candidate feature point pairs and a direction associated with the first feature point, or both the first angle and the second angle formed by the line segment and a direction associated with the second feature point, thereby enabling appropriate alignment between the read image and the reference image.

[0013] [Application Example 4] A program that causes a computer to realize: a candidate acquisition function that uses feature amounts of each of a plurality of feature points in a read image and a plurality of feature points in a reference image to acquire a plurality of candidate feature point pairs, which are pairs of feature points in the read image and feature points in the reference image; and a pair determination function that determines a plurality of target feature point pairs from the plurality of candidate feature point pairs, the pair determination function including a function that determines the plurality of target feature point pairs by executing a process including a narrowing-down process that narrows down candidates for the plurality of target feature point pairs, the narrowing-down process being a process that uses a target set, which is a set of a plurality of candidate feature point pairs, to determine whether or not the target set is to be a candidate for the plurality of target feature point pairs, and the condition for determining the target set as a candidate for the plurality of target feature point pairs is determined using the order of a plurality of feature points, which is the order of brightness of partial areas including the feature points; and a correspondence determination function that uses the plurality of target feature point pairs to determine a correspondence between coordinates on the read image and coordinates on the reference image.

[0014] According to this configuration, the conditions for making a target set a candidate for the plurality of target feature point pairs are determined using the order of the plurality of feature points and the order of brightness of the partial areas containing the feature points, so that alignment between the read image and the reference image can be performed appropriately.

[0015] The technology disclosed in this specification can be realized in various forms, such as an image processing method and an image processing device, a computer program for realizing the functions of those methods or devices, a recording medium (e.g., a non-temporary recording medium) on which that computer program is recorded, and the like. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is an explanatory diagram illustrating a data processing device according to an embodiment; [Figure 2] 1 is a perspective view showing an example of a reading device 100. FIG. [Figure 3]1A is a diagram showing an example of an image represented by image data for printing, and FIG. 1B is a diagram showing an example of a scanned image. [Figure 4] 10 is a flowchart illustrating an example of an inspection process. [Figure 5] FIG. 10 is a diagram illustrating an example of an expression format for a correspondence relationship between coordinates. [Figure 6] 10A and 10B are diagrams showing examples of difference images. [Figure 7] 10 is a flowchart illustrating an example of a registration process. [Figure 8] FIG. 1 is a schematic diagram of a three-dimensional histogram HST. [Figure 9] 10 is a flowchart illustrating an example of a feature point matching process. [Figure 10] 10A-10D are diagrams showing examples of images processed by feature point matching. [Figure 11] 10 is a flowchart illustrating an example of a selection process. [Figure 12] 10A and 10B are diagrams illustrating an example of calculation of a representative color value. [Figure 13] 10 is a flowchart illustrating an example of a process for selecting candidate feature points pairs M. [Figure 14] FIG. 10 is a diagram illustrating an example of a feature point group N2. [Figure 15] 10 is a flowchart illustrating an example of a process for determining shape conditions CWz. [Figure 16] 10A to 10D are diagrams showing examples of line segments formed by candidates MA2 and MB2 and the directions of each feature point. [Figure 17] 10 is a flowchart illustrating an example of a process for selecting candidate feature points pairs M. [Figure 18] 10A and 10B are diagrams showing an example of a triangle formed by candidates MA, MB, and MC. [Figure 19] 10 is a flowchart illustrating an example of a process for determining a combination condition CC. [Figure 20] 10 is a flowchart illustrating an example of a process for determining a shape condition CW1. [Figure 21]21A to 21D are diagrams showing examples of the results of the processing of FIG. 20. [Figure 22] 10 is a flowchart illustrating another embodiment of the alignment process. [Figure 23] 10 is a flowchart illustrating another embodiment of the alignment process. [Figure 24] 10 is a flowchart illustrating another embodiment of the alignment process. [Figure 25] 10 is a flowchart illustrating another embodiment of the alignment process. [Figure 26] 10 is a flowchart illustrating another embodiment of the process of selecting candidate feature points pairs M. DETAILED DESCRIPTION OF THE INVENTION

[0017] A. First Example: A1.Device configuration: FIG. 1 is an explanatory diagram showing a data processing device according to one embodiment. The data processing device 200 is, for example, a personal computer. The data processing device 200 performs inspection processing of a printed image. The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0018] The processor 210 is a device configured to perform data processing, such as a central processing unit (CPU) or a system on a chip (SoC). The volatile storage device 220 is, for example, a dynamic random access memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores data of a program 231.

[0019] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel overlaid on the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250.

[0020] The communication interface 270 is an interface for communicating with other devices (for example, it includes one or more of a USB interface, a wired LAN interface, an IEEE802.11 wireless interface, and an industrial camera interface (for example, CameraLink, CoaXPress, etc.)). In this embodiment, the communication interface 270 connects the reading device 100 and the printing device 900. The printing device 900 is a so-called inkjet printing device that prints an image on a printing medium such as cloth or paper by ejecting ink onto the printing medium. The reading device 100 optically reads the object to be read and generates read image data representing the object. Hereinafter, it is assumed that the printing medium is a T-shirt, and that the reading device 100 reads the T-shirt on which an image is printed.

[0021] 2 is a perspective view showing an example of the reading device 100. In the figure, the first direction Da and the second direction Db indicate horizontal directions, and the third direction Dc indicates a vertically upward direction. The first direction Da and the second direction Db are perpendicular to each other.

[0022] In this embodiment, the reading device 100 includes a housing 190, a table 130, a support 140 fixed to the upper surface of the table 130, a conveying device 120, a reading sensor 180, and a control device 110. The control device 110, the conveying device 120, and the reading sensor 180 are fixed to the housing 190.

[0023] The support unit 140 is a plate-like member that forms a flat upper surface for supporting an object to be read (such a member is also called a platen). In the figure, a T-shirt 700 having a printed image IMpp thereon is placed on the support unit 140.

[0024] The conveying device 120 is configured to convey the table 130 in a direction parallel to the second direction Db. The conveying device 120 may have various configurations. Although not shown, in this embodiment, the conveying device 120 has rails that support the table 130 so that it can slide in a direction parallel to the second direction Db, multiple pulleys, a belt that is wound around the multiple pulleys and has a portion fixed to the table 130, and an electric motor that rotates the pulleys. When the electric motor rotates the pulleys, the table 130 (and therefore the support portion 140) moves in a direction parallel to the second direction Db. The conveying device 120 further has a position sensor 122 (e.g., a rotary encoder) that detects the position of the table 130 on the conveying path.

[0025] The reading sensor 180 is disposed at a position higher than the support unit 140 and midway along the transport path PTh of the support unit 140. The reading sensor 180 includes a line sensor (for example, a contact image sensor (CIS) or a charge coupled device (CCD)) that is composed of a plurality of photoelectric conversion elements arranged in a direction intersecting the transport direction Db (in this embodiment, a direction Da perpendicular to the transport direction Db). The reading sensor 180 faces downward. The reading sensor 180 can read a portion of the object supported by the support unit 140 that is located below the reading sensor 180.

[0026] When reading the T-shirt 700, the reading device 100 transports the table 130 in a direction parallel to the second direction Db. The reading sensor 180 repeatedly reads the T-shirt 700 during transportation. This allows the reading sensor 180 to read almost the entire portion of the T-shirt 700 that is supported by the support portion 140.

[0027] The control device 110 is an electric circuit configured to control the transport device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates data of the read image by controlling the transport device 120 and the reading sensor 180.

[0028] A2.Printing process: In this embodiment, an image is printed on a T-shirt 700 (FIG. 2). The printing of the image is performed, for example, as part of a T-shirt sales service. The T-shirt sales service may include on-demand printing. A customer places an order for on-demand printing with a service provider. In response to the customer's order, the service provider prints the image on the T-shirt 700 using image data provided by the customer.

[0029] FIG. 3A is a diagram showing an example of an image represented by image data for printing (referred to as target image IMp). In this embodiment, the data of target image IMp is bitmap data representing the color values ​​(here, the gradation values ​​(e.g., values ​​greater than or equal to zero and less than or equal to 255) of red R, green G, and blue B) of a plurality of pixels arranged in a matrix along a first direction Dx and a second direction Dy. In the example of FIG. 3A, target image IMp represents a background BG and four rectangular objects OB1-OB4. The objects OB1-OB4 are represented in different colors. For example, the color of the first object OB1 is red, the color of the second object OB2 is green, the color of the third object OB3 is blue, and the color of the fourth object OB4 is gray. In this way, target image IMp may include a plurality of objects having different hues.

[0030] Although not shown in the drawings, the target image IMp is printed using the data processing device 200 and the printing device 900. Alternatively, the target image IMp may be printed using another device.

[0031] Various errors may occur during printing of the target image IMp. The printed image may have various defects due to errors. For example, a portion of the printed image may be missing due to abnormal ink ejection. The data processing device 200 detects defects in the printed image through an inspection process described below. In this embodiment, the data of the target image IMp is used in the inspection process. After printing of the target image IMp, the data of the target image IMp is stored in the storage device 215 (e.g., the non-volatile storage device 230) of the data processing device 200 for the inspection process.

[0032] A3. Inspection process: FIG. 4 is a flowchart illustrating an example of an inspection process. For inspection, the T-shirt 700 is placed on the support unit 140 of the reading device 100 (FIG. 2) so that the printed image IMpp is visible. In this embodiment, an operator places the T-shirt 700 on the support unit 140. Alternatively, a machine (e.g., a robot arm) may place the T-shirt 700 on the support unit 140. After the T-shirt 700 is placed, an instruction to start the inspection process is input to the data processing device 200 (FIG. 1). In this embodiment, the operator inputs the instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the start instruction. Note that the start instruction may be input to the data processing device 200 via the communication interface 270 by a device other than the data processing device 200.

[0033] The processor 210 of the data processing device 200 executes the inspection process in accordance with the program 231. In S110, the T-shirt 700 is photographed. The processor 210 supplies a reading instruction to the reading device 100. The control device 110 of the reading device 100 reads the T-shirt 700 by controlling the reading sensor 180 and the conveying device 120 in accordance with the reading instruction. The control device 110 generates read image data representing the read T-shirt 700. The processor 210 of the data processing device 200 acquires the read image data from the control device 110 of the reading device 100, and stores the acquired read image data in the storage device 215 (e.g., the non-volatile storage device 230).

[0034] FIG. 3(B) is a diagram showing an example of a read image. In this embodiment, the data of the read image IMs is bitmap data representing the color values ​​(here, the gradation values ​​(e.g., values ​​greater than or equal to zero and less than or equal to 255) of red R, green G, and blue B) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. The read image IMs in the figure represents a portion of the T-shirt 700 that includes the printed image IMpp. Here, the printed image IMpp is assumed to be an image obtained by printing the target image IMp (FIG. 3(A)). The printed image IMpp, like the target image IMp, represents the background BG and objects OB1-OB4.

[0035] In S120 (FIG. 4), the processor 210 aligns the reference image and the scanned image. In this embodiment, the target image IMp (FIG. 3A) is used as the reference image (hereinafter, the target image IMp is also referred to as the reference image IMt).

[0036] The orientation of the T-shirt 700 relative to the reading sensor 180 may be in various directions. Therefore, the orientation of the objects OB1-OB4 in the image (i.e., the angle of rotation of the objects) may differ between the reference image IMt and the read image IMs. Furthermore, the pixel density (also called resolution) for the same object may differ between the reference image IMt and the read image IMs. In other words, the scale for the object may differ between the reference image IMt and the read image IMs.

[0037] The processor 210 determines the correspondence between coordinates on the reference image IMt and coordinates on the scanned image IMs by performing a registration process, which will be described later. FIG. 5 is a diagram showing an example of a representation format for the coordinate correspondence. The diagram shows coordinates COt on the reference image IMt, coordinates COs on the scanned image IMs, and a matrix Mtx that associates these coordinates COt and COs. In this embodiment, the matrix Mtx represents a so-called affine transformation. In the diagram, the coordinates COt and COs and the matrix Mtx are expressed in a homogeneous coordinate system. Each of the coordinates COt and COs is represented by a three-dimensional vector having positions Xt and Xs in the first direction Dx on the images IMt and IMs, positions Yt and Ys in the second direction Dy, and a third component of 1. The matrix Mtx is a 3-row, 3-column matrix. As shown in the diagram, the matrix Mtx is represented by six parameters af arranged in two rows and three columns, and three components (0, 0, 1) in the third row. Such a matrix Mtx can represent rotation, enlargement, reduction, translation, and skew. The matrix Mtx can be calculated using three or more pairs of coordinates COt, COs.

[0038] In S130 (FIG. 4), the processor 210 inspects the printed image. The inspection method may be various methods using the coordinate correspondence (FIG. 5). In this embodiment, the processor 210 generates difference image data using the coordinate correspondence. FIGS. 6(A) and 6(B) are diagrams showing examples of difference images. FIG. 6(A) shows a case where the printed image has no defects, and FIG. 6(B) shows a case where the printed image has defects (here, missing part Err). The left side of each diagram shows scanned images IMs and IMs2, and a reference image IMt placed on the scanned images IMs and IMs2 according to the coordinate correspondence. Images IMd and IMd2 on the right side of each diagram show examples of difference images between the scanned images IMs and IMs2 and the reference image IMt. The processor 210 generates data for difference images IMd and IMd2 that represent the difference in color values ​​(e.g., the absolute value of the difference in brightness values) between the scanned images IMs and IMs2 and the reference image IMt at positions that correspond to each other based on the coordinate correspondence. The scanned image IMs in FIG. 6A represents a printed image without defects. Therefore, the difference image IMd does not have any parts that show large differences. The scanned image IMs2 in FIG. 6B represents a printed image that has a missing part Err. Therefore, in the difference image IMd2, the part that corresponds to the missing part Err shows a large difference compared to other parts.

[0039] In S140 (FIG. 4), the processor 210 outputs the inspection results. There are various methods for outputting the inspection results. In this embodiment, the processor 210 displays a difference image on the display unit 240 (FIG. 1). By observing the display unit 240, the worker can easily recognize defects in the printed image. Alternatively, the processor 210 may output data representing the inspection results to a storage device (e.g., the non-volatile storage device 230 or an external storage device connected to the data processing device 200). This causes the data representing the inspection results to be stored in the storage device. The data representing the inspection results can be used for various processes (e.g., inspection processing of the entire T-shirt 700). After S140, the processor 210 ends the inspection processing.

[0040] A4. Alignment process: 7 is a flowchart illustrating an example of the alignment process. In this embodiment, the processor 210 obtains a plurality of pairs of feature points by feature point matching, and determines the correspondence between the coordinates on the reference image IMt and the coordinates on the scanned image IMs using the obtained plurality of pairs.

[0041] In S210, the processor 210 generates a color histogram by analyzing the reference image IMt. In this embodiment, the processor 210 generates a three-dimensional RGB histogram.

[0042] FIG. 8 is a schematic diagram of a three-dimensional histogram HST. The figure shows a three-dimensional color solid CSC represented by three gradation values ​​of red (R), green (G), and blue (B). The vertices of the color solid CSC are assigned symbols indicating colors (here, red (R), green (G), blue (B), cyan (Cy), magenta (Mg), yellow (Yl), and white (Wh)). In this embodiment, the ranges of the gradation values ​​of each of red (R), green (G), and blue (B) are divided into J bins (J is an integer equal to or greater than 2, e.g., J=9). Each color value represented by an RGB gradation value is associated with one of the J*J*J bins. The processor 210 calculates the frequency of each bin using multiple color values ​​of multiple pixels in the reference image IMt. This allows the processor 210 to generate a three-dimensional histogram HST, i.e., a color occurrence frequency distribution.

[0043] The reference image IMt may represent various colors. Among the multiple bins in the 3D histogram HST, the frequency of bins corresponding to the colors represented by the reference image IMt increases. The total number of bins with a high frequency is an example of an index value indicating the number of colors represented by the reference image IMt. The 3D histogram HST is used to evaluate the reliability of candidate feature point pairs, as will be described later.

[0044] In S220 (FIG. 7), processor 210 performs feature point matching. FIG. 9 is a flowchart showing an example of the feature point matching process. In S305, processor 210 generates a gray reference image IMtg and a gray read image IMsg by grayscale conversion of reference image IMt and read image IMs. A known relationship can be used as the correspondence between color gradation values ​​and grayscale gradation values ​​(for example, the correspondence between RGB values ​​in an RGB color space and luminance value Y in a YCbCr color space).

[0045] In S310, the processor 210 extracts feature points Tt from the gray reference image IMtg. FIGS. 10A-10D are diagrams showing examples of images processed by feature point matching. FIG. 10A shows examples of feature points detected from images IMtg and IMsg. Each of the multiple black dots on the gray reference image IMtg represents a feature point Tt (also referred to as a reference feature point Tt) detected from the gray reference image IMtg. As shown in the figure, points indicating characteristic portions such as corners and edges of an object are detected as feature points Tt. Such feature points Tt are also called key points. Although not shown in the figure, in practice, many more feature points Tt may be detected (for example, several tens or several hundreds). Note that the reference feature points Tt detected from the gray reference image IMtg correspond to feature points indicating the same portions at the same coordinates on the reference image IMt (FIG. 3A).

[0046] Various methods may be used to detect feature points. In this embodiment, a technique called Accelerated-KAZE (A-KAZE) is used, which detects key points and calculates feature descriptors for each key point. The A-KAZE technique is disclosed in, for example, the following paper: "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013"

[0047] The processor 210 detects a plurality of feature points Tt by analyzing the gray reference image IMtg according to the A-KAZE technique.

[0048] In S315 (FIG. 9), the processor 210 extracts a plurality of feature points Ts from the gray-scanned image IMsg. The plurality of black dots on the gray-scanned image IMsg in FIG. 10(A) each represent a feature point Ts detected from the gray-scanned image IMsg (also referred to as a read feature point Ts). The processor 210 detects a plurality of feature points Ts by analyzing the gray-scanned image IMsg according to the A-KAZE technology. Although not shown, in practice, many more feature points Ts may be detected (for example, several tens or several hundreds). The read feature points Ts detected from the gray-scanned image IMsg correspond to feature points indicating the same parts at the same coordinates on the read image IMs (FIG. 3(B)).

[0049] In S320 (FIG. 9), the processor 210 calculates a feature value Ft for each of the multiple reference feature points Tt. The feature value Ft may be various information describing the features of the feature point Tt. For example, the feature value Ft is calculated so that it changes depending on the distribution of color values ​​of multiple pixels surrounding the feature point Tt. In this embodiment, the processor 210 uses the gray reference image IMtg to calculate the A-KAZE feature descriptor as the feature value Ft. In the A-KAZE technology, the feature descriptor is rotation-invariant. To obtain the rotation-invariant feature descriptor, the orientation of the feature point is also calculated. The orientation of the feature point indicates the direction of the brightness gradient in a neighborhood centered on the feature point (also called the gradient orientation or dominant orientation). The processor 210 calculates the feature value Ft and the orientation of the reference feature point Tt.

[0050] In S325, the processor 210 calculates a feature value Fs for each of the plurality of read feature points Ts. In this embodiment, the processor 210 uses the gray read image IMsg to calculate the feature descriptor (i.e., the feature value Fs) and the direction of the read feature points Ts according to the A-KAZE technique.

[0051] The outline of the processing after S325 is as follows: The processor 210 calculates the distance dF between the feature amounts Ft and Fs for each combination of the reference feature point Tt and the read feature point Ts (S350). Then, it acquires the pair of feature points Tt, Ts showing the distance dF less than the distance threshold dFth as the candidate feature point pair M (S355: Yes, S360).

[0052] Specifically, the process is as follows: The processor 210 executes loop processing S330 (including S335-S360) between start L31s and end L31e for each of the multiple reference feature points Tt. In S335, the processor 210 selects the unprocessed reference feature point Tt as a target feature point Tti, which is the feature point to be processed. The processor 210 executes loop processing S340 (including S345-S360) between start L32s and end L32e for each of the multiple read feature points Ts. In S345, the processor 210 selects the unprocessed read feature point Ts as a target feature point Tsj, which is the feature point to be processed.

[0053] In S350, the processor 210 calculates the distance dF between the two feature amounts Fti and Fsj of the two interest feature points Tti and Tsj. The distance dF is calculated so that a small distance dF indicates a high similarity between the two feature amounts Fti and Fsj. A high similarity (i.e., a small distance dF) indicates that the two interest feature points Tti and Tsj represent similar portions of the two images IMtg and IMsg (e.g., corresponding portions of the same object). In this embodiment, the feature amounts Ft and Fs are A-KAZE feature descriptors and are represented by binary vectors (vectors consisting of one or more binary elements). In this case, the processor 210 may calculate the Hamming distance as the distance dF.

[0054] In S355, the processor 210 determines whether the distance dF is less than the distance threshold dFth. If the distance dF is small, the feature points Tti and Tsj are likely to represent similar portions of the images IMt and IMs (for example, the same portion of the same object). If the distance dF is less than the distance threshold dFth (S355: Yes), in S360, the processor 210 acquires the pair of interest feature points Tti and Tsj as a feature point pair candidate M (also referred to as a candidate feature point pair M). A feature point pair is a pair of a reference feature point Tt and a read feature point Ts used to determine the correspondence between coordinates. The feature points Tt and Ts that correspond to each other are also referred to as a matching pair. After S360, the processor 210 ends the loop processing S340 for the interest feature point Tsj. If the distance dF is equal to or greater than the distance threshold dFth (S355: No), the processor 210 skips S360 and ends the loop processing S340 for the target feature point Tsj. Then, the processor 210 repeats the loop processing S340 and the loop processing S330 to perform the processing of S350-S360 for each of a plurality of combinations of the target feature point Tsj and the target feature point Tti.

[0055] FIG. 10(B) shows an example of a candidate feature point pair M. Each of the multiple lines RL in the figure indicates a candidate feature point pair M. Each line RL connects feature points Tt and Ts that form the candidate feature point pair M. As shown in the figure, a pair of feature points Tt and Ts that indicate the same part may be selected as the candidate feature point pair M. For example, feature point Tt1 that indicates the upper right corner of the first object OB1 in the gray reference image IMtg may be associated with feature point Ts1 that indicates the upper right corner of the first object OB1 in the gray read image IMsg.

[0056] Furthermore, a pair of feature points Tt, Ts indicating mutually different portions may be selected as a candidate feature point pair M. For example, feature point Tt1 indicating the upper right corner of a first object OB1 in the gray reference image IMtg may be associated with feature point Ts2 indicating the upper right corner of a third object OB3 in the gray-scanned image IMsg. Each of the rectangular objects OB1-OB4 has four corners. These corners are locally similar. As a result, feature point Tt indicating one corner of an object in the gray reference image IMtg may be associated with feature point Ts indicating another corner of the same object in the gray-scanned image IMsg, or with feature point Ts indicating a corner of another object. In this way, when each image IMtg and IMsg represents multiple locally similar portions, feature points Tt, Ts indicating mutually different portions may be associated.

[0057] Furthermore, in this embodiment, as described above, the feature quantities Ft and Fs are rotation-invariant. When feature quantities are rotation-invariant, similar feature quantities can be calculated from the same portion of an object regardless of the angle of rotation of the object in the image. By using the distance dF between the rotation-invariant feature quantities Ft and Fs, the processor 210 can match pairs of feature quantities representing the same portion of the object even when the rotation angles of the same object are different between two images IMtg and IMsg. However, if each image IMtg and IMsg represents multiple locally similar portions, feature quantities Tt and Ts representing different portions may be matched. For example, feature quantity Tt1 representing the upper right corner of the first object OB1 in the gray reference image IMtg may be matched with feature quantity Ts3 representing the lower right corner of the first object OB1 in the gray-scanned image IMsg.

[0058] Note that the larger the distance threshold dFth (FIG. 9: S355), the larger the total number of suitable candidate feature point pairs M may be. However, the total number of inappropriate candidates M may also be larger. The larger the total number of suitable candidates M, the smaller the error in the coordinate correspondence, which will be described later. If the total number of inappropriate candidates M is large, the error in the coordinate correspondence may increase. The distance threshold dFth may be determined experimentally in advance so that the error in the coordinate correspondence is tolerable.

[0059] When processing of all combinations of reference feature points Tt and read feature points Ts is completed, in S365, processor 210 stores data representing multiple candidate feature point pairs M in storage device 215 (e.g., non-volatile storage device 230). Then, processor 210 ends the processing of FIG. 9, that is, the processing of S220 in FIG. 7.

[0060] In S230, processor 210 executes a feature point pair selection process using color information. This selection process uses color information to select an appropriate candidate M from a plurality of candidate feature point pairs M (i.e., candidate feature point pairs M are verified). Fig. 11 is a flowchart showing an example of the selection process. Processor 210 executes loop process S410 (including S420-S465) between start L4s and end L4e for each of the plurality of candidate feature point pairs M.

[0061] In S420, the processor 210 selects the unprocessed candidate M as a candidate of interest Mi, which is a candidate to be processed.

[0062] In S425, the processor 210 calculates a representative color value Ct of the first partial region Pt including the reference feature point Tt of the focused candidate Mi (the representative color value Ct is also referred to as the reference representative color value Ct). FIGS. 12A and 12B are diagrams showing an example of calculation of a representative color value. FIG. 12A shows a portion of the reference image IMt including the feature point Tt. The figure also shows the first partial region Pt including the feature point Tt. The processor 210 calculates the reference representative color value Ct using multiple color values ​​of multiple pixels in the first partial region Pt. The method of calculating the representative color value Ct may be any of various methods for calculating a color representative of the first partial region Pt. In this embodiment, the processor 210 calculates the average value of each of the RGB components in the first partial region Pt as the gradation value of each of the RGB components of the representative color value Ct. Note that various summary statistics (e.g., median, mode, etc.) representing the magnitude of the color value may be used instead of the average value. Furthermore, the configuration of the first partial region Pt (specifically, the shape of the first partial region Pt and the relative position of the first partial region Pt with respect to the position of the feature point Tt) may be various configurations that enable calculation of the representative color value Ct representing the color of the portion indicated by the feature point Tt. In this embodiment, the first partial region Pt is a region centered on the feature point Tt, for example, a region of P rows and Q columns and P*Q pixels. In order to reduce the dependency of the representative color value Ct on the angle of rotation of the object, P=Q is preferable (for example, P=Q=5). Instead of a region of P rows and Q columns, the first partial region Pt may be a region whose distance from the feature point Tt is equal to or less than a distance threshold.

[0063] In S430 (FIG. 11), the processor 210 calculates a representative color value Cs of a second partial region Ps including the read feature point Ts of the focused candidate Mi (the representative color value Cs is also referred to as the read representative color value Cs). FIG. 12(B) shows a portion of the read image IMs including the feature point Ts. The figure also shows the second partial region Ps including the feature point Ts. The configuration of the second partial region Ps is the same as the configuration of the first partial region Pt. The second partial region Ps is a region of P rows and Q columns with P*Q pixels centered on the feature point Ts. The calculation method of the read representative color value Cs is the same as the calculation method of the reference representative color value Ct. The processor 210 calculates the average values ​​of red R, green G, and blue B in the second partial region Ps as the read representative color value Cs.

[0064] In S435 (FIG. 11), the processor 210 calculates a first hue Ht, a first saturation St, and a first luminance Vt from the reference representative color value Ct, and calculates a second hue Hs, a second saturation Ss, and a second luminance Vs from the read representative color value Cs. A known relationship can be used as the correspondence between the representative color value, hue, saturation, and luminance (for example, the correspondence between RGB values ​​in an RGB color space and HSV values ​​in an HSV color space). The first hue Ht, the first saturation St, and the first luminance Vt are each an example of a representative color value of the first partial region Pt, similar to the reference representative color value Ct. The second hue Hs, the second saturation Ss, and the second luminance Vs are each an example of a representative color value of the second partial region Ps, similar to the read representative color value Cs.

[0065] In S440, the processor 210 determines whether a first saturation condition CSt is satisfied, which indicates that the first saturation St is higher than the saturation threshold Sth. If the first saturation St is higher than the saturation threshold Sth (S440: Yes), the processor 210 determines in S445 whether a hue condition CH is satisfied, which indicates that the absolute value of the difference between the first hue Ht and the second hue Hs is less than the hue difference threshold dHth. The absolute value of the difference between the hues Ht and Hs (also referred to as the hue difference dH) is the value corresponding to the smaller angle difference between the first hue Ht and the second hue Hs on the hue circle. If the focused candidate Mi is an appropriate pair of feature points Tt and Ts that indicate the same part, the hue difference dH may be a small value. If the hue difference dH is less than the hue difference threshold dHth (S445: Yes), the processor 210 selects the focused candidate Mi as a candidate to be retained in S460. If the hue difference dH is equal to or greater than the hue difference threshold dHth (S445: No), the hues may differ significantly between the first partial region Pt and the second partial region Ps. In other words, the feature points Tt and Ts of the attention candidate Mi are likely to represent different parts. In S465, the processor 210 excludes the attention candidate Mi from the candidates for the feature points pair.

[0066] In this way, when the first saturation St is higher than the saturation threshold Sth (S440: Yes), processor 210 excludes from the candidates a candidate Mi having a hue difference dH equal to or greater than the hue difference threshold dHth. FIG. 10C shows an example of a candidate feature point pair M remaining after the processing of FIG. 11. A pair of feature points Tt, Ts indicating portions having different hues can be excluded. For example, the hues differ between the first object OB1 and the third object OB3. The line RLa shown in FIG. 10B indicates a pair of the reference feature point Tt of the first object OB1 and the read feature point Ts of the third object OB3. As shown in FIG. 10C, this pair is excluded (FIG. 11: S440 (Yes), S445 (No), S465).

[0067] If the first saturation St is equal to or less than the saturation threshold Sth (S440: No), in S450, processor 210 determines whether a second saturation condition CSs is satisfied, indicating that the second saturation Ss is higher than the saturation threshold Sth. If the second saturation Ss is higher than the saturation threshold Sth (S450: Yes), the saturation may differ significantly between the first partial region Pt and the second partial region Ps. In other words, the feature points Tt and Ts of the attention candidate Mi are likely to represent different parts from each other. In S465, processor 210 excludes the attention candidate Mi from the candidates for the feature point pair.

[0068] If the second saturation Ss is equal to or less than the saturation threshold Sth (S450: No), in S455, the processor 210 determines whether a luminance condition CV is satisfied, indicating that the absolute value of the difference between the first luminance Vt and the second luminance Vs is less than the luminance difference threshold dVth. If the target candidate Mi is an appropriate pair of feature points Tt and Ts that represent the same part, the absolute value of the difference between the luminance Vt and Vs (also referred to as the luminance difference dV) may be small. If the target candidate Mi is an inappropriate pair of feature points Tt and Ts that represent different parts, the luminance difference dV may be large. In this way, even if the saturations St and Ss are low, the luminances Vt and Vs may appropriately represent the color of the part represented by the feature points Tt and Ts.

[0069] If the brightness difference dV is less than the brightness difference threshold dVth (S455: Yes), in S460, processor 210 selects the focus candidate Mi as a candidate to be retained. If the brightness difference dV is equal to or greater than the brightness difference threshold dVth (S455: No), in S465, processor 210 excludes the focus candidate Mi from the candidate feature point pair M.

[0070] Note that the more lenient the conditions for retaining the focused candidate Mi, the greater the total number of suitable candidates M that remain without being excluded. However, the total number of inappropriate candidates may also be greater. The greater the total number of suitable candidates, the smaller the error in the coordinate correspondence. If the total number of inappropriate candidates is large, the error in the coordinate correspondence may increase. The parameters Sth, dHth, and dVth used in S440-S455 may be experimentally determined in advance so that the error in the coordinate correspondence is tolerable.

[0071] For example, the larger the hue difference threshold dHth (S445), the more lenient the conditions for retaining the featured candidate Mi. Even if the featured candidate Mi is an appropriate pair of feature points Tt, Ts that represent a highly saturated color portion, the hue difference dH may be a value greater than zero. A hue difference threshold dHth greater than zero allows such a hue difference dH. The hue difference threshold dHth may be set to a value greater than zero and less than the maximum possible value of the hue difference dH.

[0072] Furthermore, the larger the brightness difference threshold dVth (S455), the more lenient the conditions for retaining the target candidate Mi. Even if the target candidate Mi is an appropriate pair of feature points Tt, Ts that represent a low-saturation color portion, the brightness difference dV may be a value greater than zero. A brightness difference threshold dVth greater than zero allows such a brightness difference dV. The brightness difference threshold dVth may be set to a value greater than zero and smaller than the maximum possible value of the brightness difference dV.

[0073] Furthermore, if the saturation threshold value Sth (S440) is small, a focused candidate Mi having a low first saturation St is processed in S445. Even if the reference representative color value Ct (FIG. 12A) is a chromatic color, if the first saturation St is low, the hue of the corresponding region of the scanned image IMs representing the printed image is likely to differ from the first hue Ht. That is, if the first saturation St is low, the error in the second hue Hs and, consequently, the error in the hue difference dH may be large. If the error in the hue difference dH is large, an appropriate focused candidate Mi may be erroneously excluded (S445: No), and an inappropriate focused candidate Mi may be erroneously retained (S445: Yes). This may result in an increase in the error in the coordinate correspondence. The saturation threshold value Sth may be experimentally determined in advance so as to mitigate the influence of an error in the hue difference dH on the error in the coordinate correspondence. The saturation threshold value Sth may be set to a value greater than zero and less than the maximum possible saturation value.

[0074] After S460 or S465, processor 210 proceeds to S420 and executes loop processing S410 for the next candidate of interest Mi. When loop processing S410 for all candidates M has been completed, in S470 processor 210 stores data representing candidate feature points pairs M to be retained in storage device 215 (for example, non-volatile storage device 230). Then, processor 210 ends the processing in FIG. 11, i.e., the processing of S230 in FIG. 7.

[0075] In S240, the processor 210 determines whether the total number of candidate feature points pairs M at the current time is greater than the pair threshold. As will be described later, the processor 210 executes a process to narrow down the plurality of candidate feature points pairs M. In this embodiment, the processor 210 can select a narrowing down process from a high-load narrowing down process and a low-load narrowing down process. A high-load narrowing down process can reduce the possibility of inappropriate candidates M remaining compared to a low-load narrowing down process. The load of each narrowing down process typically increases as the total number of candidate feature points pairs M increases. If the total number of candidate feature points pairs M is greater than the pair threshold (S240: Yes), the processor 210 prioritizes reducing the load and executes a low-load narrowing down process (S245). If the total number of candidate feature points pairs M is equal to or less than the pair threshold (S240: No), the processor 210 prioritizes excluding inappropriate candidates M and executes a high-load narrowing down process (S250). The pair threshold may be experimentally determined in advance so that the load of the alignment process is tolerable when a high-load narrowing-down process is performed. For example, the pair threshold may be about 500.

[0076] The narrowing down process will be described in detail below. If the total number of candidate feature points pairs M is greater than the pair threshold (S240: Yes), in S245, processor 210 executes a process of selecting candidate feature points pairs M using a combination of two candidate feature points pairs M. This selection process selects an appropriate candidate M from the remaining multiple candidates M using the combination of two candidates M (i.e., candidate feature points pairs M are verified).

[0077] 13 is a flowchart showing an example of a process for selecting a candidate feature points pair M using a combination of two candidate feature points pairs M. Processor 210 executes loop processing S810 (including S820-S885) between start L81s and end L81e for each of a plurality of candidate feature points pairs M. In S820, processor 210 selects an unprocessed candidate M as a first candidate MA2.

[0078] In S825, the processor 210 selects a group of feature points N2 within a range between a first radius r1 and a second radius r2, with the feature point Tt of the first candidate MA2 at its center. FIG. 14 is a diagram illustrating an example of the group of feature points N2. The diagram illustrates a reference image IMt and multiple reference feature points Tt. One feature point Tt is selected as the first candidate MA2. The first circle C1 is a circle with a first radius r1 and a center at the feature point Tt of the first candidate MA2. The second circle C2 is a circle with a second radius r2 and a center at the feature point Tt of the first candidate MA2. The selection range SR is the area between the first circle C1 and the second circle C2 (including the portions on the circles C1 and C2). The processor 210 selects the multiple feature points Tt included in the selection range SR as the group of feature points N2.

[0079] After S825 (FIG. 13), processor 210 executes loop processing S830 (including S840-S885) between start L82s and end L82e for each of multiple candidates M in feature point group N2. In S840, processor 210 selects an unprocessed candidate M associated with feature point Tt included in feature point group N2 as a second candidate MB2. The reason for selecting second candidate MB2 from feature point group N2 (i.e., selection range SR (FIG. 14)) will be described later.

[0080] Hereinafter, the second candidate MB2 is assumed to be the u-th candidate M among the candidates M obtained from the feature point group N2 (also expressed as candidate M[u]). The index u is selected, for example, from a range equal to or greater than zero and less than the total number NP of candidates M obtained from the feature point group N2. The combination of the selected candidates MA2 and MB2 is also referred to as candidate combination MU2.

[0081] In S850, if a specific condition is satisfied, the processor 210 selects the candidate combination MU2 as the target combination MT2 for determining the shape condition CWz, which will be described later. In this embodiment, if the candidates MA2 and MB2 are different candidates M, the processor 210 selects the candidate combination MU2 as the target combination MT2. Although not shown, if the second candidate MB2 is the same candidate M as the first candidate MA2, the processor 210 does not select the candidate combination MU2 as the target combination MT2 and ends the loop processing S830. Then, the processor 210 executes the loop processing S830 for the next second candidate MB2. Note that the condition for selecting the candidate combination MU2 as the target combination MT2 may be various conditions.

[0082] In S860, the processor 210 uses the target combination MT2 to determine whether the shape condition CWz is satisfied.

[0083] 15 is a flowchart showing an example of a process for determining whether the shape condition CWz is satisfied. Processor 210 determines whether the shape condition CWz is satisfied by using the line segment connecting the two reference feature points Tt of two candidates MA2 and MB2, the line segment connecting the two read feature points Ts, and the directions of each feature point.

[0084] 16(A) to 16(D) are diagrams showing examples of line segments formed by candidates MA2 and MB2 and the directions of each minutiae. Each candidate MA2 and MB2 represents a pair of a reference minutiae Tt and a read minutiae Ts. FIG. 16(A) shows a line segment Sabt2 connecting the reference minutiae Tta2 and Ttb2 of candidates MA2 and MB2 (referred to as reference line segment Sabt2). The directions Oat2 and Obt2 indicate the directions of the reference minutiae Tta2 and Ttb2, respectively. FIG. 16(B) to 16(D) show a line segment Sabs2 connecting the read minutiae Tsa2 and Tsb2 of candidates MA2 and MB2 (referred to as read line segment Sabs2). The directions Oas2 and Obs2 indicate the directions of the read minutiae Tsa2 and Tsb2, respectively. In this embodiment, the directions Oat2, Obt2, Oas2, and Obs2 are the directions calculated for calculating the feature amounts Ft and Fs (S320 and S325 (FIG. 9)). The differences between FIGS. 16(B) to 16(D) will be described later.

[0085] In Fig. 16(A), angles ZAt and ZBt are angles formed between directions Oat2 and Obt2 and line segment Sabt2. In Fig. 16(B) to Fig. 16(D), angles ZAs and ZBs are angles formed between directions Oas2 and Obs2 and line segment Sabs2. The angles ZAt, ZBt, ZAs, and ZBs are the smaller angles formed between the line segments extending from characteristic points Tta2, Ttb2, Tsa2, and Tsb2 in the directions Oat2, Obt2, Oas2, and Obs2, and the line segments Sabt2 and Sabs2 extending from characteristic points Tta2, Ttb2, Tsa2, and Tsb2. Alternatively, the angles ZAt, ZBt, ZAs, and ZBs may be angles from line segments Sabt2 and Sabs2 to line segments extending in the directions Oat2, Obt2, Oas2, and Obs2 in a clockwise (or counterclockwise) direction around the feature points Tta2, Ttb2, Tsa2, and Tsb2.

[0086] In S910 (FIG. 15), the processor 210 calculates the following two angles ZAt and ZAs associated with the first candidate MA2. ZAt=AG(Oat2, Sabt2) ZAs = AG(Oas2, Sabs2) AG is a function that derives an angle.

[0087] In S915, the processor 210 determines whether a first angle condition CAA is satisfied, which indicates that the first reference angle ZAt is close to the first read angle ZAs. In this embodiment, the first angle condition CAA is that the absolute value of the difference between the angles ZAt and ZAs is less than an angle difference threshold dZth2.

[0088] In this embodiment, the directions Oat2 and Oas2 of the feature points Tta2 and Tsa2 are gradient directions calculated according to the A-KAZE technique. When an object rotates in the images IMtg and IMs, the directions Oat2 and Oas2 rotate together with the object represented by the feature points Tta2 and Tsa2 in the images IMtg and IMs. The angles ZAt and ZAs are invariant to the scale and rotation of the object in the images IMt and IMs. Therefore, the judgment result of the first angle condition CAA is invariant to the scale and rotation.

[0089] If the feature points Tta2 and Tsa2 of the first candidate MA2 indicate the same part of the same object, and the feature points Ttb2 and Tsb2 of the second candidate MB2 indicate the same part of the same object, the angle ZAt will be approximately equal to the angle ZAs. Even if the candidates MA and MB each indicate an appropriate pair of feature points Tt and Ts, the angle ZAt may deviate from the angle ZAs. The first angle condition CAA is configured to allow for such deviation. In this embodiment, the angle difference threshold dZth2 is experimentally determined in advance to allow for deviation between the angles ZAt and ZAs. The angle difference threshold dZth2 may be any value greater than zero.

[0090] In S925, the processor 210 calculates the following two angles ZBt and ZBs associated with the second candidate MB2. ZBt = AG (Obt2, Sabt2) ZBs = AG(Obs2, Sabs2)

[0091] In S930, the processor 210 determines whether a second angle condition CAB is satisfied, which indicates that the second reference angle ZBt is close to the second read angle ZBs. In this embodiment, the second angle condition CAB is defined similarly to the first angle condition CAA. The second angle condition CAB is that the absolute value of the difference between the angles ZBt and ZBs is less than the angle difference threshold dZth2.

[0092] In this embodiment, like the angles ZAt and ZAs referenced in S910, the angles ZBt and ZBs are invariant to the scale and rotation of the objects in the images IMt and IMs. Therefore, the determination result of the second angle condition CAB is invariant to the scale and rotation.

[0093] If the angle conditions CAA and CAB are satisfied (S915: Yes, S930: Yes), it is estimated that all of the candidates MA and MB are suitable pairs. In this case, in S935, the processor 210 determines that the shape condition CWz is satisfied, and ends the processing of FIG. 15, i.e., S860 of FIG. 13.

[0094] If one or more of the angle conditions CAA and CAB are not satisfied, one or more candidates M of the candidates MA and MB are estimated to be an inappropriate pair. If one or more of S915: No and S930: No are satisfied, in S940, processor 210 determines that the shape condition CWz is not satisfied, and terminates the processing of Figure 15, i.e., S860 in Figure 13.

[0095] 16(B) to 16(D) are diagrams showing examples of the results of the processing in FIG. 15. FIG. 16(B) shows a case where the candidates MA and MB are appropriate, and the reference image IMt and the scanned image IMs have the same scale but different rotation angles. In this case, the rotation-invariant angle conditions CAA and CAB (FIG. 15) are satisfied, and therefore the shape condition CWz is also satisfied.

[0096] 16(C) shows the case where the candidates MA and MB are appropriate and the reference image IMt and the scanned image IMs have the same rotation angle but different scales. In this case, the scale-invariant angle conditions CAA and CAB (FIG. 15) are satisfied, and therefore the shape condition CWz is also satisfied.

[0097] Although not shown in the figure, even if the candidates MA and MB are appropriate and both the scale and rotation angle differ between the reference image IMt and the read image IMs, the angle conditions CAA and CAB (Figure 15) are satisfied, and therefore the shape condition CWz is satisfied.

[0098] 16(D) shows a case where the first candidate MA2 is appropriate, but the second candidate MB2 is inappropriate. Specifically, feature point Tsb2 is different from the appropriate feature point Tsbr. If the combination of candidates MA and MB includes an inappropriate candidate M, the determination result for one or more of the angle conditions CAA and CAB (FIG. 15) may be No (i.e., the shape condition CWz is likely not satisfied).

[0099] Note that the more lenient the angle conditions CAA and CAB (FIG. 15), the greater the total number of combinations of appropriate candidates MA and MB that satisfy the shape condition CWz. However, the greater the total number of combinations of inappropriate candidates MA and MB that satisfy the shape condition CWz. The greater the total number of appropriate candidates, the smaller the error in the coordinate correspondence. If the total number of inappropriate candidates is large, the error in the coordinate correspondence may increase. The angle conditions CAA and CAB (in this embodiment, the angle difference threshold dZth2) may be experimentally determined in advance so that the error in the coordinate correspondence is tolerable. The angle difference threshold dZth2 may be set to various values ​​greater than zero, for example, a value greater than zero and equal to or less than 10 degrees.

[0100] 13, the second candidate MB2 is selected from the candidates M associated with the feature point Tt included in the feature point group N2 (i.e., the selection range SR (FIG. 14)). The reason for this is that, as will be explained below, when the lengths of the line segments Sabt2 and Sabs2 (FIGS. 16(A)-16(D)) are significantly long or short, the error in determining the shape condition CWz may increase.

[0101] For example, assume that the second candidate MB2 is a pair of a feature point Ttb2 and an incorrectly read feature point Tsb2 near a correct feature point Ts (e.g., FIG. 16(D)). If the length of the reference line segment Sabt2 is significantly long, the length of the read line segment Sabs2 is also significantly long. In this case, the extension direction of the read line segment Sabs2 is approximately the same as the extension direction of the line segment connecting the two correct feature points Ts (not shown). Therefore, even if the candidate MB2 indicates an incorrect feature point Ts, the difference in angle calculated using the read line segment Sabs2 (e.g., the difference between the angles ZAt and ZAs) may be erroneously determined to be small.

[0102] Furthermore, the positions of the detected feature points may contain errors. If the length of the reference line segment Sabt2 is significantly short, the direction in which the reference line segment Sabt2 extends may change significantly due to an error in the position of the reference feature point Tt. The same is true for the direction in which the read line segment Sabs2 extends. Therefore, even if the candidates MA2 and MB2 each indicate an appropriate pair of feature points Tt and Ts, the difference in the angles calculated using the line segments Sabt2 and Sabs2 (e.g., the difference between the angles ZAt and ZAs) may be erroneously determined to be large.

[0103] The selection range SR (here, radii r1 and r2 (FIG. 14)) is experimentally determined in advance so that a figure suitable for judgment (here, a line segment of an appropriate length) is formed by combining the candidates MA2 and MB2. For example, the first radius r1 may be a value not less than 1% and not more than 20% of the size of the reference image IMt (e.g., the length in the first direction Dx or the second direction Dy). The second radius r2 may be a value not less than 30% and not more than 70% of the size of the reference image IMt.

[0104] After the processing of FIG. 15 , i.e., after S860 ( FIG. 13 ), in S865, processor 210 branches the processing according to the determination result of S860. If shape condition CWz is satisfied (S865: Yes), in S870, processor 210 calculates the brightness of each feature point of target combination MT2. In this embodiment, processor 210 calculates the brightness Vt and Vs described with reference to FIGS. 12(A) and 12(B) as the brightness of each of two reference feature points Tt and two read feature points Ts of candidates MA2 and MB2. Brightness Vt is an example of the brightness of a partial region Pt that includes reference feature point Tt, and brightness Vs is an example of the brightness of a partial region Ps that includes read feature point Ts.

[0105] In S875 (FIG. 13), the processor 210 determines whether the brightness condition CB is satisfied. The brightness condition CB is a condition that indicates that the brightness order of multiple feature points corresponding to the target combination MT2 is the same between the scanned image IMs and the reference image IMt. The processor 210 compares the brightness order of the two reference feature points Tt corresponding to the target combination MT2 (i.e., candidates MA2 and MB2) with the brightness order of the two scanned feature points Ts. The brightness order is represented by the order of the candidates MA2 and MB2 (specifically, the order MA2, MB2, or MB2, MA2). The absolute brightness values ​​may differ between the reference image IMt and the scanned image IMs. However, the brightness order of two portions in the reference image IMt is the same as the brightness order of the corresponding two portions in the scanned image IMs. For example, the brightness order of two objects in the reference image IMt is the same as the brightness order of the same two objects in the scanned image IMs. When the candidates MA2 and MB2 respectively indicate a proper pair of minutiae Tt and Ts, the brightness order of the two reference minutiae Tt is usually the same as the brightness order of the two read minutiae Ts. When the candidates MA2 and MB2 include an improper pair, the brightness order of the two reference minutiae Tt may be different from the brightness order of the two read minutiae Ts.

[0106] If the brightness order of the two reference feature points Tt is the same as the brightness order of the two read feature points Ts (S875: Yes), in S885, the processor 210 selects the candidates MA and MB of the target combination MT2 as candidates to be retained. After S885, the processor 210 ends the loop process S830 for the current combination of candidates MA and MB.

[0107] If the shape condition CWz is not satisfied (S865: No) or if the brightness condition CB is not satisfied (S875: No), the processor 210 skips S885 and terminates the loop processing S830 for the current combination of candidates MA and MB.

[0108] Thereafter, processor 210 executes loop processing S830 using each of the multiple second candidates MB2, and loop processing S810 using each of the multiple first candidates MA2. After completing the repetition of loop processing S810 and S830, in S890 processor 210 stores data representing candidate feature points pairs M to be retained in storage device 215 (e.g., non-volatile storage device 230). Then, processor 210 ends the processing of FIG. 13, i.e., the processing of S245 in FIG. 7.

[0109] Fig. 10(D) shows an example of candidate feature points pairs remaining after the processing of Fig. 13. As shown in Figs. 10(C) and 10(D), pairs of inappropriate feature points Tt, Ts can be excluded. For example, the line RLb shown in Fig. 10(C) indicates a pair of the reference feature point Tt in the upper right and the read feature point Ts in the lower right of the fourth object OB4. When combined with other pairs, such a pair does not satisfy the shape condition CWz or the brightness condition CB, and can be excluded from candidate feature points pairs (Fig. 13: S865: No or S875: No).

[0110] Note that even if a combination of candidates MA and MB including a suitable candidate M does not satisfy the shape condition CWz or the brightness condition CB, the suitable candidate M may satisfy the shape condition CWz and the brightness condition CB when combined with another suitable candidate M. Thus, in the process of FIG. 13, even if the determination result of S865 or S875 is No, the candidates MA and MB are not immediately eliminated. Through repeated loop processes S810 and S830, processor 210 selects multiple candidates M selected by one or more times of S885 as candidates M to be retained. Through repeated loop processes S810 and S830, processor 210 eliminates candidates M that are not selected even once by S885.

[0111] After the processing of FIG. 13, i.e., after S245 of FIG. 7, in S260, the processor 210 determines whether the color number index value is less than the color number threshold. The color number index value is an index value indicating the number of colors represented by the reference image IMt. In this embodiment, the processor 210 calculates, as the color number index value, the total number of bins having a frequency equal to or greater than the frequency threshold among the multiple bins in the three-dimensional histogram HST (FIG. 8) generated in S210. The frequency threshold is experimentally determined in advance so as to minimize the effect of noise in the reference image IMt on the color number index value. The frequency threshold may be any value greater than zero.

[0112] A small color number index value indicates that multiple portions in the reference image IMt have the same color. When the color number index value is small, multiple reference feature points Tt representing different portions in the reference image IMt may exhibit the same color. Multiple read feature points Ts representing different portions in the read image IMs may also exhibit the same color. As a result, inappropriate feature point pairs Tt, Ts that exhibit roughly the same color may be selected as candidates to be retained in S230 (FIG. 7). In other words, the reliability of the candidate feature point pairs M that remain after the narrowing down in S230 and S245 is estimated to be low.

[0113] On the other hand, when the color number index value is large, the multiple reference feature points Tt may exhibit different colors from one another. Similarly, the multiple read feature points Ts may exhibit different colors from one another. Furthermore, the reference feature points Tt and read feature points Ts of an inappropriate candidate feature point pair M may exhibit different colors from one another. As a result, inappropriate candidate feature point pairs M are likely to be excluded in S230 (FIG. 7). In other words, the reliability of the candidate feature point pairs M that remain after the narrowing down in S230 and S245 is estimated to be high.

[0114] The color number threshold may be experimentally determined in advance so that the color number index value is less than the color number threshold when the reliability of the remaining candidate feature point pairs M is low. For example, the color number threshold may be about 15.

[0115] If the color number index value is equal to or greater than the color number threshold (S260: No), the processor 210 ends the narrowing down of the candidate feature point pairs M and proceeds to S290. In S290, the processor 210 determines the coordinate correspondence using the multiple feature point pairs MP. The multiple feature point pairs MP are the multiple candidate feature point pairs M remaining in S290. Hereinafter, the feature point pairs MP used to determine the coordinate correspondence will also be referred to as target feature point pairs MP. In this embodiment, the matrix Mtx ( FIG. 5 ) is determined as the coordinate correspondence. Various methods may be used to determine the matrix Mtx. For example, the processor 210 may determine the matrix Mtx according to a method called RANdom SAmple Consensus (RANSAC). Note that in this embodiment, the total number of inappropriate candidate feature point pairs M may be reduced by narrowing down processes (e.g., S230, S245) performed before S290 in FIG. 7. Therefore, the processor 210 may calculate the matrix Mtx using all the target feature point pairs MP. The calculation method may be various methods (for example, the least squares method). Note that, for example, a function of OpenCV (Open Source Computer Vision Library) may be used to determine the matrix Mtx.

[0116] If the color number index value is less than the color number threshold (S260: Yes), in S265, the processor 210 further narrows down the candidate feature points pairs M. The processor 210 executes a candidate feature points pair M selection process using a combination of three candidate feature points pairs M.

[0117] FIG. 17 is a flowchart illustrating an example of a process for selecting a candidate feature point pair M using a combination of three candidate feature point pairs M. The process in FIG. 17 is outlined as follows: The processor 210 selects three candidate feature point pairs MA, MB, and MC (S520-S560). The second candidate MB and the third candidate MC are selected from an area between a first radius r1 and a second radius r2 centered on the reference feature point Tt of the first candidate MA (details will be described later). Hereinafter, the selected combination of candidates MA, MB, and MC will also be referred to as a candidate combination MU. The processor 210 uses the candidate combination MU to determine whether a combination condition CC is satisfied (S565). If the combination condition CC is satisfied (S570: Yes), the processor 210 selects the candidate combination MU as a target combination MT (S573), and uses the target combination MT to determine whether a shape condition CW1 is satisfied (S575). If the shape condition CW1 is satisfied (S580: Yes), the processor 210 selects the three candidates MA, MBA, and MC of the target combination MT as candidates to be retained (S585).

[0118] Specifically, the process is as follows: Processor 210 executes loop processing S510 (including S520-S585) between start L51s and end L51e for each of a plurality of candidate feature points pairs M. In S520, processor 210 selects an unprocessed candidate M as a first candidate MA.

[0119] In S525, the processor 210 selects a group of feature points N within a selection range SR between a first radius r1 and a second radius r2, with the feature point Tt of the first candidate MA at its center. The selection range SR is the same as the selection range SR described in Fig. 14. The processor 210 selects, as the group of feature points N, multiple feature points Tt included in the selection range SR with the reference feature point Tt of the first candidate MA at its center.

[0120] As will be described later, the processor 210 uses the three reference feature points Tt and the three read feature points Ts of the three candidates MA, MB, and MC to determine whether or not to retain the combination of the candidates MA, MB, and MC. Specifically, when certain conditions are met, including a small difference between the shape of the triangle formed by the three reference feature points Tt and the shape of the triangle formed by the three read feature points Ts, the processor 210 determines that the combination of the three candidates MA, MB, and MC should be retained. If the length of any one of the three sides of the triangle is significantly longer or shorter than the other sides, the error in comparing the shapes of the two triangles may increase. The candidates MB and MC to be combined with the first candidate MA are selected from the candidates M associated with the feature point Tt included in the feature point group N. The selection range SR (here, radii r1 and r2) is experimentally determined in advance so that the combination of the candidates MA, MB, and MC forms a shape suitable for judgment. In this embodiment, the first radii r1 and r2 are the same as the radii r1 and r2 used in S825 (FIG. 13), respectively. However, the first radius used in S525 may be different from the first radius r1 used in S825. Also, the second radius used in S525 may be different from the second radius r2 used in S825.

[0121] After S525 (FIG. 17), processor 210 selects a second candidate MB from candidates M associated with feature point Tt included in feature point group N (S540), and selects a third candidate MC (S560). Specifically, processor 210 executes loop processing S530 (including S540-S585) between start L52s and end L52e for each of multiple candidates M obtained from feature point group N. In S540, processor 210 selects an unprocessed candidate M associated with feature point Tt included in feature point group N as the second candidate MB. Hereinafter, the second candidate MB is assumed to be the j-th candidate among the candidates M obtained from feature point group N (also expressed as candidate M[j]). Index j is selected, for example, from a range of zero or more and less than the total number NN of candidates M obtained from feature point group N. After S540, the processor 210 executes a loop process S550 (including S560-S585) between start L53s and end L53e for each of the multiple candidates M obtained from the feature point group N. In S560, the processor 210 selects an unprocessed candidate M associated with a feature point Tt included in the feature point group N as a third candidate MC. Hereinafter, the third candidate MC is assumed to be the k-th candidate among the candidates M obtained from the feature point group N (also expressed as candidate M[k]). The index k is selected, for example, from a range equal to or greater than j+1 and less than the total number NN of candidates M obtained from the feature point group N. As a result, a candidate M different from the second candidate MB is selected as the third candidate MC. In this way, the processor 210 selects a combination of three candidates MA, MB, and MC (i.e., a candidate combination MU).

[0122] In S565, the processor 210 uses the candidate combination MU to determine whether a combination condition CC is satisfied. The combination condition CC is a condition for selecting the candidate combination MU as a target combination MT for determining the shape condition CW1, which will be described later. The determination of the combination condition CC is performed using the triangle formed by the candidates MA, MB, and MC of the candidate combination MU.

[0123] 18(A) and 18(B) are diagrams showing examples of triangles formed by candidates MA, MB, and MC. Each candidate MA, MB, and MC represents a pair of a reference minutiae point Tt and a read minutiae point Ts. FIG. 18(A) shows a triangle TRt formed by reference minutiae points Tta, Ttb, and Ttc of candidates MA, MB, and MC (referred to as reference triangle TRt). FIG. 18(B) shows a triangle TRs formed by read minutiae points Tsa, Tsb, and Tsc of candidates MA, MB, and MC (referred to as read triangle TRs). The positions of one or more minutiae may differ between the reference triangle TRt and the read triangle TRs.

[0124] Each figure shows symbols indicating the sides of a triangle, their lengths, and the size of the interior angles. Symbols beginning with S (e.g., Sabt) indicate sides. The two letters following S indicate two candidates M connected by the side (a, b, and c indicate candidates MA, MB, and MC, respectively). The suffix t or s of the symbol indicates the reference triangle TRt or the read triangle TRs, respectively. For example, the side Sabt ( FIG. 18(A) ) is the side connecting the reference feature points Tta and Ttb of the reference triangle TRt. The symbol obtained by replacing the leading S of the symbol indicating a side with L indicates the length of that side. For example, the length Labt indicates the length of the side Sabt. Symbols beginning with A (e.g., Aat) indicate the size of the interior angles of a triangle. The single letter following A indicates the candidate M that forms the vertex of the interior angle (a, b, and c indicate candidates MA, MB, and MC, respectively). The suffix t or s of the symbols indicates the reference triangle TRt or the read triangle TRs, respectively. For example, the interior angle Aat (FIG. 18(A)) indicates the size of the interior angle of the reference triangle TRt, with the reference minutiae Tta as its vertex. Symbols based on the above rules will also be used in other figures described later. Note that the direction Oct in FIG. 18(A) indicates the direction of minutiae Ttc (FIG. 9: S320), and the direction Ocs in FIG. 18(B) indicates the direction of minutiae Tsc (FIG. 9: S325).

[0125] FIG. 19 is a flowchart showing an example of the process for determining combination condition CC (FIG. 17: S565). FIG. 19 shows the process for determining whether one triangle satisfies the individual combination condition. The processor 210 executes the process of FIG. 19 for each of the reference triangle TRt (FIG. 18(A)) and the read triangle TRs (FIG. 18(B)). When both the reference triangle TRt and the read triangle TRs satisfy the individual combination condition, the processor 210 determines that the combination condition CC is satisfied. The process of FIG. 19 will be explained below using the reference triangle TRt as an example. In the explanation of the process of FIG. 19, symbols without the final letter (t or s) will be used to indicate the sides, the lengths of the sides, and the sizes of the interior angles.

[0126] In S610, processor 210 determines whether the three candidates MA, MB, and MC are different from one another. If two or more candidates are the same candidate (S610: No), in S640, processor 210 determines that the individual combination condition is not satisfied, and ends the processing of FIG.

[0127] If the three candidates MA, MB, and MC are different from one another (S610: Yes), in S615, processor 210 determines whether a first length condition CL1 is satisfied. The first length condition CL1 indicates that the first length Lab is within an allowable length range LR, which is greater than a lower limit Lth1 and less than an upper limit Lth2 (the allowable length range LR may also be simply referred to as the length range LR). If the first length Lab is outside the length range LR (S615: No), processor 210 proceeds to S640. The length range LR (here, the lower limit Lth1 and the upper limit Lth2), like the radii r1 and r2 of the selection range SR (FIG. 14), is experimentally determined in advance so that a combination of the three candidates MA, MB, and MC forms a figure suitable for judgment. For example, the lower limit Lth1 may be the same as the first radius r1, and the upper limit Lth2 may be the same as the second radius r2.

[0128] If the first length Lab is within the length range LR (S615: Yes), in S620, the processor 210 determines whether a second length condition CL2 is satisfied, which indicates that the second length Lac is within the length range LR. If the second length Lac is outside the length range LR (S620: No), the processor 210 proceeds to S640.

[0129] If the second length Lac is within the length range LR (S620: Yes), in S625, processor 210 determines whether the first interior angle condition CA is satisfied. The first interior angle condition CA indicates that the interior angle Aa is within an allowable interior angle range AR, which is greater than the lower limit Ath1 and less than the upper limit Ath2 (the allowable interior angle range AR is also simply referred to as the interior angle range AR). If the interior angle Aa is outside the interior angle range AR (S625: No), processor 210 proceeds to S640. The interior angle range AR (here, the lower limit Ath1 and the upper limit Ath2) are experimentally determined in advance, similar to the radii r1 and r2 of the selection range SR (FIG. 14), so that a shape suitable for judgment is formed by combining the three candidates MA, MB, and MC. For example, the lower limit Ath1 may be a value between 10 degrees and 60 degrees. The upper limit Ath2 may be a value between 90 degrees and 160 degrees.

[0130] If the interior angle Aa is within the interior angle range AR (S625: Yes), in S630, the processor 210 determines whether the scalene condition CQ is satisfied. The scalene condition CQ indicates that the lengths of the three sides of a triangle (here, the reference triangle TRt) are different from one another. That is, the scalene condition CQ indicates that the shape of the triangle is different from any of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle. An isosceles triangle and an equilateral triangle maintain their original shape even if two sides of the same length are swapped. If the reference triangle TRt is an isosceles triangle or an equilateral triangle, the difference between the shape of the reference triangle TRt and the shape of the read triangle TRs becomes small even if two appropriate read feature points Ts corresponding to two reference feature points Tt are swapped. As a result, the candidates MA, MB, and MC, including two inappropriate candidates M, may be erroneously determined to be retained for determining the coordinate correspondence. The same applies when the reference triangle TRt is a triangle similar to an isosceles triangle or a triangle similar to an equilateral triangle. Therefore, in this embodiment, if the scalene condition CQ is not satisfied (S630: No), the processor 210 determines in S640 that the individual combination condition is not satisfied, and ends the processing of Figure 19. If the scalene condition CQ is satisfied (S630: Yes), the processor 210 determines in S635 that the individual combination condition is satisfied, and ends the processing of Figure 19.

[0131] The specific configuration of the inequality condition CQ may be various. In this embodiment, the inequality condition CQ is that all of the following conditions CQ1 to CQ3 are satisfied. (CQ1) The absolute value of the difference between the lengths Lab and Lac is greater than the length difference threshold dLth. (CQ2) The absolute value of the difference between the lengths Lab and Lbc is greater than the length difference threshold dLth. (CQ3) The absolute value of the difference between the lengths Lbc and Lac is greater than the length difference threshold dLth. Each of the conditions CQ1-CQ3 indicates that the difference in length between the two sides is large. The inequality condition CQ defined by the conditions CQ1-CQ3 indicates that the triangle does not contain two sides whose difference in length is equal to or less than the length difference threshold dLth. The length difference threshold dLth, like the radii r1 and r2 of the selection range SR (FIG. 14), is experimentally determined in advance so that the combination of the three candidates MA, MB, and MC forms a figure suitable for judgment. For example, the length difference threshold dLth may be a value between 2% and 10% of the upper limit Lth2 referenced in S615 and S620 (FIG. 19).

[0132] In S565 (FIG. 17), the processor 210 executes the process of FIG. 19 for each of the reference triangle TRt and the read triangle TRs. Then, if both the reference triangle TRt and the read triangle TRs satisfy the individual combination conditions, the processor 210 determines that the candidate combination MU satisfies the combination condition CC. If one or both of the reference triangle TRt and the read triangle TRs do not satisfy the individual combination conditions, the processor 210 determines that the candidate combination MU does not satisfy the combination condition CC.

[0133] At S570, the processor 210 branches the process according to the determination result at S565. If the combination condition CC is satisfied (S570: Yes), at S573 the processor 210 selects the candidate combination MU as the target combination MT. At S575, the processor 210 uses the target combination MT to determine whether the shape condition CW1 is satisfied.

[0134] FIG. 20 is a flowchart showing an example of the process for determining whether the shape condition CW1 is satisfied. When each of the three candidates MA, MB, and MC (FIGS. 18A and 18B) of the target combination MT indicates an appropriate pair of feature points Tt and Ts, the shape of the read triangle TRs is approximately the same as the shape of the reference triangle TRt. Furthermore, when an object rotates in the image, the orientation of the feature point (e.g., the orientation Oct of the feature point Ttc) rotates along with the object indicated by the feature point. The shape condition CW1 is configured taking the above properties into consideration. In this embodiment, the shape condition CW1 is determined to be satisfied when all three conditions CR, CD, and CF, described below, are satisfied.

[0135] In S710, the processor 210 calculates the ratio between the lengths of the two sides of the triangles TRt and TRs. Specifically, the processor 210 calculates the following two ratios Rt and Rs: Rt=Labt / Lact Rs=Labs / Lacs These ratios Rt, Rs are invariant to the scale and rotation of the objects in the images IMtg, IMs.

[0136] In S715, the processor 210 determines whether a ratio condition CR, which indicates that the ratio Rt / Rs is close to 1, is satisfied. Because the ratios Rt and Rs are invariant to the scale and rotation of the objects in the images IMt and IMs, the determination result of the ratio condition CR is invariant to the scale and rotation. If the shape of the read triangle TRs is the same as the shape of the reference triangle TRt, the ratio Rt / Rs=1. Even if each of the candidates MA, MB, and MC indicates a proper pair of feature points Tt and Ts, the ratio Rt / Rs may deviate from 1. The ratio condition CR is configured to allow such deviation. For example, the ratio condition CR may be that the ratio Rt / Rs is greater than a lower limit Rth1 and less than an upper limit Rth2 (where Rth1<1). <Rth2である)。

[0137] If the ratio condition CR is satisfied (S715: Yes), in S720, the processor 210 determines whether a second interior angle condition CD, which indicates that the interior angle Abs of the read triangle TRs is close to the interior angle Abt of the reference triangle TRt, is satisfied. The interior angles Abt and Abs are invariant to the scale and rotation of the objects in the images IMt and IMs. Therefore, the determination result of the second interior angle condition CD is invariant to the scale and rotation.

[0138] When the shape of the read triangle TRs is the same as the shape of the reference triangle TRt, the interior angle Abs is the same as the interior angle Abt. Even if each of the candidates MA, MB, and MC indicates an appropriate pair of feature points Tt and Ts, the interior angle Abs may deviate from the interior angle Abt. The second interior angle condition CD is configured to allow such deviation. For example, the second interior angle condition CD may be that the absolute value of the difference between the interior angle Abs and the interior angle Abt is less than an interior angle difference threshold dAth (where dAth>0).

[0139] If the second interior angle condition CD is satisfied (S720: Yes), in S725, processor 210 calculates the angle between the direction of the feature point and the side. Specifically, processor 210 calculates the following two angles Zt and Zs: Zt = AG (Oct, Sact) Zs = AG(Ocs, Sacs) AG is a function for deriving angles. The directions Oct and Ocs are the directions Oct and Ocs of the feature points Ttc and Tsc (FIGS. 18(A) and 18(B)). The directions calculated for calculating the feature amounts Ft and Fs (S320, S325 (FIG. 9)) may be used as the directions Oct and Ocs of the feature points Ttc and Tsc. The angle Zt is the angle between the side Sact and the direction Oct. The angle Zs is the angle between the side Sacs and the direction Ocs.

[0140] The angles Zt and Zs are determined by the smaller angle between the line segment extending from the characteristic points Ttc and Tsc in the direction Oct and Ocs and the line segment Sact and Sacs extending from the characteristic points Ttc and Tsc. Alternatively, the angles Zt and Zs may be determined by the angle from the line segment Sact or Sacs to the line segment extending in the direction Oct or Ocs in a clockwise (or counterclockwise) direction around the characteristic points Ttc and Tsc.

[0141] In this embodiment, the directions Oct and Ocs of the feature points Ttc and Tsc are gradient directions calculated according to the A-KAZE technique. When an object rotates in the images IMtg and IMs, the directions Oct and Ocs rotate together with the object represented by the feature points Ttc and Tsc in the images IMtg and IMs. The angles Zt and Zs are invariant to the scale and rotation of the object in the images IMt and IMs.

[0142] In S730, the processor 210 determines whether the angle condition CF, which indicates that the angle Zs of the read triangle TRs is close to the angle Zt of the reference triangle TRt, is satisfied. Because the angles Zt and Zs are invariant to the scale and rotation of the objects in the images IMtg and IMs, the determination result of the angle condition CF is invariant to the scale and rotation. If the shape of the read triangle TRs is the same as the shape of the reference triangle TRt and the feature points Ttc and Tsc indicate the same part of the same object, the angle Zs is approximately the same as the angle Zt. Even if each of the candidates MA, MB, and MC indicates an appropriate pair of feature points Tt and Ts, the angle Zs may deviate from the angle Zt. The angle condition CF is configured to allow such deviation. For example, the angle condition CF may be that the absolute value of the difference between the angles Zt and Zs is less than an angle difference threshold dZth (where dZth>0).

[0143] If all of the conditions CR, CD, and CF are satisfied (S715: Yes, S720: Yes, and S730: Yes), it is estimated that all of the candidates MA, MB, and MC are suitable pairs. In this case, in S735, the processor 210 determines that the shape condition CW1 is satisfied, and ends the processing of FIG. 20.

[0144] If one or more of the conditions CR, CD, and CF are not satisfied, one or more of the candidates MA, MB, and MC, M, are estimated to be an inappropriate pair. If one or more of S715: No, S720: No, and S730: No are satisfied, in S740, processor 210 determines that shape condition CW1 is not satisfied, and ends the processing of FIG. 20.

[0145] Figures 21(A) to 21(D) are diagrams showing examples of the results of the processing in Figure 20. Figure 21(A) shows an example of a reference triangle TRt, and Figures 21(B) to 21(D) show examples of read triangles TRs associated with the reference triangle TRt.

[0146] 21(B) shows the case where the candidates MA, MB, and MC are appropriate, and the scale is the same between the reference image IMt and the scanned image IMs, but the angle of rotation is different. In this case, the rotation invariant conditions CR, CD, and CF (FIG. 20) are satisfied, and therefore the shape condition CW1 is also satisfied.

[0147] 21(C) shows the case where the candidates MA, MB, and MC are appropriate, and the reference image IMt and the scanned image IMs have the same rotation angle but different scales. In this case, the scale-invariant conditions CR, CD, and CF (FIG. 20) are satisfied, and therefore the shape condition CW1 is also satisfied.

[0148] Although not shown in the figure, even if the candidates MA, MB, and MC are appropriate and both the scale and the angle of rotation differ between the reference image IMt and the read image IMs, the conditions CR, CD, and CF (Figure 20) are met, and therefore the shape condition CW1 is met.

[0149] 21(D) shows a case where candidates MA and MB are appropriate, but the third candidate MC is inappropriate. Specifically, feature point Tsc is different from the appropriate feature point Tscr. If the combination of candidates MA, MB, and MC includes an inappropriate candidate M, the determination result of one or more of conditions CR, CD, and CF (FIG. 20) may be No (i.e., the shape condition CW1 is likely not satisfied).

[0150] Note that the more lenient the conditions CR, CD, and CF (FIG. 20), the greater the total number of suitable candidate combinations MA, MB, and MC that satisfy the shape condition CW1. However, the greater the total number of inappropriate candidate combinations MA, MB, and MC that satisfy the shape condition CW1. The greater the total number of suitable candidates, the smaller the error in the coordinate correspondence. If the total number of inappropriate candidates is large, the greater the error in the coordinate correspondence. The conditions CR, CD, and CF (in this embodiment, parameters Rth1, Rth2, dAth, and dZth) may be experimentally determined in advance so that the error in the coordinate correspondence is tolerable. The lower limit Rth1 (S715) may be various values ​​less than 1, for example, 0.85 or more and less than 1. The upper limit Rth2 may be various values ​​greater than 1, for example, greater than 1 and 1.15 or less. The interior angle difference threshold dAth (S720) may be any value greater than zero, for example, greater than zero and less than or equal to 10 degrees. The angle difference threshold dZth (S730) may be any value greater than zero, for example, greater than zero and less than or equal to 10 degrees.

[0151] After the processing of Figure 20, i.e., after S575 (Figure 17), in S580, processor 210 branches the processing according to the determination result of S575. If shape condition CW1 is satisfied (S580: Yes), in S585, processor 210 selects candidates MA, MB, and MC of target combination MT as candidates to be retained. After S585, processor 210 ends loop processing S550 for the current combination of candidates MA, MB, and MC.

[0152] If the combination condition CC is not satisfied (S570: No) or if the shape condition CW1 is not satisfied (S580: No), the processor 210 skips S585 and ends the loop processing S550 for the current combination of candidates MA, MB, and MC.

[0153] Thereafter, processor 210 executes loop processing S550 using each of the multiple third candidates MC, loop processing S530 using each of the multiple second candidates MB, and loop processing S510 using each of the multiple first candidates MA. After completing the repetition of loop processing S510, S530, and S550, processor 210 stores data representing candidate feature point pairs M to be retained in storage device 215 (e.g., non-volatile storage device 230) in S590. Then, processor 210 ends the processing of FIG. 17, i.e., the processing of S265 in FIG. 7. After S265, processor 210 proceeds to S290.

[0154] Note that even if a combination of candidates MA, MB, and MC including a suitable candidate M does not satisfy the shape condition CW1, the suitable candidate M may satisfy the shape condition CW1 by being combined with another suitable candidate M. Thus, in the processing of FIG. 17, even if the determination result of S570 or S580 is No, the candidates MA, MB, and MC are not immediately eliminated. Processor 210 selects multiple candidates M selected by S585 one or more times through repeated loop processing S510, S530, and S550 as candidates M to be retained. Processor 210 eliminates candidates M that are not selected even once by S585 through repeated loop processing S510, S530, and S550.

[0155] The total number (3) of candidate feature points pairs M included in the target combination MT used in S265 (FIG. 17) is greater than the total number (2) of candidate feature points pairs M included in the target combination MT2 used in S245 (FIG. 13). That is, the process of S265 uses more information to determine whether the target combination MT is appropriate compared to the process of S245. Therefore, the process of S265 can eliminate inappropriate candidate feature points pairs M with higher accuracy compared to the process of S245. For example, the process of S265 can eliminate inappropriate candidate feature points pairs M that were overlooked in the process of S245.

[0156] In S240, if the total number of candidate feature points pairs M is equal to or less than the pair threshold value (S240: No), in S250, the processor 210 executes a process of selecting candidate feature points pairs M using a combination of three candidate feature points pairs M. In this embodiment, the process of S250 is the same as the process of S265. After S250, the processor 210 proceeds to S290.

[0157] As described above, in this embodiment, the processor 210 executes the following processes in accordance with the program 231. In S350-S360 of FIG. 9, the processor 210 uses the feature amount Fs of each of the plurality of read feature points Ts and the feature amount Ft of each of the plurality of reference feature points Tt to acquire a plurality of candidate feature point pairs M, which are pairs of feature points Ts and feature points Tt. The read feature points Ts are feature points in the gray read image IMsg ( FIG. 10 ), i.e., feature points in the read image IMs. The reference feature points Tt are feature points in the gray reference image IMtg, i.e., feature points in the reference image IMt.

[0158] In the processing of Fig. 7, the processor 210 determines a plurality of target feature point pairs MP from a plurality of candidate feature point pairs M. The target feature point pairs MP are candidate feature point pairs M used in determining the coordinate correspondence (S290). In S290, the processor 210 uses the plurality of target feature point pairs MP to determine the correspondence between the coordinates COs (Fig. 5) on the scanned image IMs and the coordinates COt on the reference image IMt. In this embodiment, a matrix Mtx is determined as the correspondence.

[0159] In this embodiment, the pair determination process, which is the process of determining multiple target feature points pairs MP, includes various processes. The pair determination process includes, for example, the following processes. If the total number of candidate feature points pairs M in S240 is less than the pair threshold (S240: No), in S250, the processor 210 executes a process of narrowing down the candidates for the multiple target feature points pairs MP. In the narrowing down process, the processor 210 determines whether or not to use the candidate feature points pair M as a candidate for the multiple target feature points pairs MP. In other words, the processor 210 determines whether or not to use the candidate feature points pair M as a candidate for the multiple target feature points pairs MP.

[0160] The narrowing down process of S250 is a process that uses a combination of three candidate feature points pairs M (specifically, a combination of candidates MA, MB, and MC) to determine whether or not to select this combination as a candidate for multiple target feature points pairs MP. Hereinafter, the set of candidate feature points pairs M used in the narrowing down process will be referred to as a pair set, or simply as a set. The narrowing down process of S250 is an example of a first narrowing down process that uses a first type set (here, the set of candidates MA, MB, and MC), which is a set of N (N is an integer greater than or equal to 2) candidate feature points pairs M, to determine whether or not to select this first type set as a candidate for multiple target feature points pairs MP (hereinafter, the narrowing down process of S250 will also be referred to as the first narrowing down process S250). In this way, when the total number of candidate feature points pairs M at a specific stage of the pair determination process is the first number (S240: No), processor 210 executes a process that includes the first narrowing down process S250 to determine multiple target feature points pairs MP. The stage at which S240 is executed is an example of a specific stage of the pair determination process. In this embodiment, the first number is a number equal to or less than the pair threshold value.

[0161] Furthermore, the process of determining multiple target feature points pairs MP includes the following process. In a first specific case where the total number of candidate feature points pairs M at the specific stage is a second number greater than the first number (in this embodiment, S240: Yes and S260: No), processor 210 determines multiple target feature points pairs MP without executing first narrowing-down process S250. In this embodiment, the second number is a number greater than the pair threshold.

[0162] In this way, processor 210 can appropriately determine multiple target feature points pairs MP according to the total number of multiple candidate feature points pairs M. For example, when the total number of candidate feature points pairs M is a first number (S240: No), processor 210 can select appropriate candidate feature points pairs M as target feature points pairs MP by executing first narrowing-down processing S250. Furthermore, in the first specific case where the total number of candidate feature points pairs M is a second number (here, S240: Yes and S260: No), processor 210 does not execute first narrowing-down processing S250, thereby reducing the processing load of determining multiple target feature points pairs MP.

[0163] Furthermore, in this embodiment, the processing executed in the first specific case (in this embodiment, when S240: Yes and S260: No) includes S245, which is processing for narrowing down the candidates for the multiple target feature points pairs MP. The narrowing down processing of S245 is processing for using a combination of two candidate feature points pairs M (specifically, the combination of candidates MA2 and MB2) to determine whether or not to select this combination as a candidate for the multiple target feature points pairs MP. The narrowing down processing of S245 is an example of second narrowing down processing that uses a second type set (here, the set of candidates MA2 and MB2), which is a set of U (U is an integer greater than or equal to 1 and less than N) candidate feature points pairs M, to determine whether or not to select the second type set as a candidate for the multiple target feature points pairs MP (hereinafter, the narrowing down processing of S245 is also referred to as second narrowing down processing S245). In this way, in the first specific case, processor 210 can select appropriate candidate feature points pairs M as target feature points pairs MP by executing processing including second narrowing down processing S245.

[0164] Furthermore, in this embodiment, the process of determining the multiple target feature points pairs MP includes the following processes. If the total number of candidate feature points pairs M at the specific stage is a second number (S240: Yes), then in S260, processor 210 determines whether or not a color number index value, which is an index value indicating the number of colors in reference image IMt, is less than a color number threshold. If the total number of candidate feature points pairs M is the second number and a second specific case occurs in which the color number index value is less than the color number threshold (S240: Yes, S260: Yes), then in S265, processor 210 executes a process of narrowing down the candidates for the multiple target feature points pairs MP. The narrowing down process of S265 is an example of a third narrowing down process that further narrows down the candidates for the multiple target feature points pairs MP from the multiple candidate feature points pairs M that became candidates for the multiple target feature points pairs MP in the second narrowing down process S245 (hereinafter, the narrowing down process of S265 will also be referred to as the third narrowing down process S265). In the second specific case, the processor 210 can select an appropriate candidate feature points pair M as the target feature points pair MP by executing a process including the third narrowing-down process S265.

[0165] Furthermore, in this embodiment, the third narrowing down process S265 is a process that uses a combination of three candidate feature points pairs M (specifically, a combination of candidates MA, MB, MC) to determine whether or not to make that combination a candidate for multiple target feature points pairs MP. In this way, the third narrowing down process S265 includes a process that uses a third type set (here, the set of candidates MA, MB, MC), which is a set of L (L is an integer greater than U) candidate feature points pairs, to determine whether or not to make the third type set a candidate for multiple target feature points pairs. By executing the third narrowing down process S265, the processor 210 can select appropriate candidate feature points pairs M as target feature points pairs MP.

[0166] Furthermore, in this embodiment, the second narrowing-down process S245 uses a second type set, which is a set of two candidate feature points pairs M. The conditions for determining two candidate feature points pairs M as candidates for multiple target feature points pairs MP by the second narrowing-down process S245 include the shape condition CWz described with reference to Figures 13 and 15. In this embodiment, the shape condition CWz (Figure 15) includes a first angle condition CAA and a second angle condition CAB.

[0167] The first angle condition CAA is determined using angles ZAt and ZAs (the angles ZAt and ZAs are referred to as first angles ZAt and ZAs). As shown in FIGS. 16(A) to 16(D), the first angles ZAt and ZAs are calculated using feature points Tta2 and Tsa2 (referred to as first feature points Tta2 and Tsa2) associated with the first candidate MA and feature points Ttb2 and Tsb2 (referred to as second feature points Ttb2 and Tsb2) associated with the second candidate MB. Specifically, the first angles ZAt and ZAs are the angles formed between line segments Sabt2 and Sabs2 and directions Oat2 and Oas2 associated with the first feature points Tta2 and Tsa2. The line segments Sabt2 and Sabs2 are line segments connecting the first feature points Tta2 and Tsa2 and the second feature points Ttb2 and Tsb2.

[0168] The second angle condition CAB is determined using angles ZBt and ZBs (the angles ZBt and ZBs are referred to as second angles ZBt and ZBs). The second angles ZBt and ZBs are the angles between the line segments Sabt2 and Sabs2 and the directions Obt2 and Obs2 associated with the second feature points Ttb2 and Tsb2.

[0169] In this embodiment, the shape condition CWz is determined using both the first angles ZAt, ZAs and the second angles ZBt, ZBs. The first angles ZAt, ZAs may differ between a case where the second type set (here, the set of candidates MA2, MB2) is appropriate and a case where the second type set includes an inappropriate candidate feature point pair M. Similarly, the second angles ZBt, ZBs may differ between a case where the second type set is appropriate and a case where the second type set includes an inappropriate candidate feature point pair M. Therefore, the processor 210 can appropriately determine whether to use the second type set as a candidate for multiple target feature point pairs MP. For example, the processor 210 can reduce the possibility that the set of candidates MA2, MB2 including an inappropriate candidate M will be used as a candidate for multiple target feature point pairs MP.

[0170] Furthermore, in this embodiment, the number N of candidate feature point pairs M in the first type set (here, the set of candidates MA, MB, and MC) used in the first narrowing-down process S250 (FIG. 7) is 3. The conditions for determining the first type set as candidates for multiple target feature point pairs MP by the first narrowing-down process S250 include the shape condition CW1 described in FIG. 17. The shape condition CW1 includes the conditions CR, CD, and CF in FIG. 20. The ratio condition CR (S715) is determined using the ratio Rt and Rs of the lengths of two sides of triangles TRt and TRs formed by three feature points associated with the three candidates MA, MB, and MC, respectively (FIGS. 18(A) and 18(B)). Specifically, the ratio Rt is the ratio of the lengths Labt and Lact of the two sides Sabt and Sac of the reference triangle TRt. The ratio Rs is the ratio of the lengths Labs and Lacs of the two sides Sabs and Sacs of the read triangle TRs. The second interior angle condition CD (S720) is determined using the interior angles Abt and Abs of the triangles TRt and TRs. The angle condition CF (S730) is determined using the angles Zt and Zs. The angle Zt (FIG. 18(A)) is the angle between the side Sact (i.e., the line segment) connecting the two reference minutiae Tta and Ttc and the direction Oct associated with one of the two reference minutiae Tta and Ttc, Ttc. The angle Zs (FIG. 18(B)) is the angle between the side Sacs (i.e., the line segment) connecting the two read minutiae Tsa and Tsc and the direction Ocs associated with one of the two read minutiae Tsa and Tsc, Tsc.

[0171] As described above, in this embodiment, the conditions for determining a first type set (here, the set of candidates MA, MB, and MC) as a candidate for multiple target feature points pairs MP by the first narrowing-down process S250 are determined using the ratios Rt and Rs, the interior angles Abt and Abs, and the angles Zt and Zs. The ratios Rt and Rs, the interior angles Abt and Abs, and the angles Zt and Zs may differ between a case in which the first type set is appropriate and a case in which the first type set includes an inappropriate candidate feature points pair M. Therefore, the processor 210 can appropriately determine whether or not to use the first type set as a candidate for multiple target feature points pairs MP. For example, the processor 210 can reduce the possibility that a set of candidates MA, MB, and MC including an inappropriate candidate M will be used as a candidate for multiple target feature points pairs MP.

[0172] Furthermore, in this embodiment, the conditions for selecting a second type set (here, the set of candidates MA2 and MB2) as a candidate for the plurality of target feature points pairs MP by the second narrowing-down process S245 (FIG. 7) include the brightness condition CB described in FIG. 13. The brightness condition CB is determined using the order of the plurality of feature points in the second type set. As described in S870 and S875, the order of the plurality of feature points is the order of brightness of the partial regions that include the feature points. The brightness order may differ between a case where the second type set is appropriate and a case where the second type set includes an inappropriate candidate feature points pair M. Therefore, the processor 210 can appropriately determine whether or not to select a second type set as a candidate for the plurality of target feature points pairs MP.

[0173] Furthermore, in this embodiment, the process of determining the multiple target feature points pairs MP ( FIG. 7 ) can be considered from another perspective. The pair determination process includes various processes, for example, the following processes. If, in S260 ( FIG. 7 ), the color number index value, which is an index value indicating the number of colors in the reference image IMt, is less than the color number threshold (S260: Yes), in S265, the processor 210 executes a process of narrowing down the candidates for the multiple target feature points pairs MP. The narrowing down process of S265 is a process of using a combination of three candidate feature points pairs M (specifically, the combination of candidates MA, MB, and MC) to determine whether or not to select this combination as a candidate for the multiple target feature points pairs MP. The narrowing down process of S265 is an example of a fourth narrowing down process that uses a fourth type set (here, the set of candidates MA, MB, and MC), which is a set of P (P is an integer greater than or equal to 2) candidate feature points pairs M, to determine whether or not to select this fourth type set as a candidate for the multiple target feature points pairs. In this way, when the index value indicating the number of colors of the reference image IMt is the third value (in this embodiment, S260: Yes), the processor 210 determines multiple target feature points pairs MP by executing processing including the narrowing-down processing of S265. In this embodiment, the third value is a value smaller than the number of colors threshold.

[0174] Furthermore, the process of determining multiple target feature points pairs MP includes the following process. If the color number index value is a fourth value greater than the third value (S260: No in this embodiment), processor 210 determines multiple target feature points pairs MP without executing the narrowing-down process of S265. In this embodiment, the fourth value is a value greater than or equal to the color number threshold.

[0175] In this way, processor 210 can appropriately determine multiple target feature points pairs MP according to the color number index value. For example, if the color number index value is a third value (S260: Yes), processor 210 can select an appropriate candidate feature points pair M as the target feature points pair MP by performing the narrowing-down process of S265. Furthermore, if the color number index value is a fourth value (S260: No), processor 210 does not perform the narrowing-down process of S265, thereby reducing the processing load of determining multiple target feature points pairs MP.

[0176] Furthermore, in this embodiment, the narrowing down process of S265 is the same as the first narrowing down process S250. That is, the number P of candidate feature points pairs M in the pair set (here, the set of candidates MA, MB, and MC) used in the narrowing down process of S265 is 3. The conditions for determining a set of candidate feature points pairs M as a candidate for multiple target feature points pairs MP by the narrowing down process of the third narrowing down process S265 are determined using the ratios Rt and Rs, the interior angles Abt and Abs, and the angles Zt and Zs (FIGS. 17, 18(A), 18(B), and 20). Therefore, the processor 210 can appropriately determine whether or not to determine a pair set as a candidate for multiple target feature points pairs MP. For example, the processor 210 can reduce the possibility that a set of candidates MA, MB, and MC including an inappropriate candidate M will be used as a candidate for multiple target feature points pairs MP.

[0177] Furthermore, in this embodiment, the process of determining the plurality of target feature points pairs MP ( FIG. 7 ) can be considered from another perspective. The pair determination process includes various processes, such as the following processes. In S245 ( FIG. 7 ), the processor 210 executes a narrowing-down process to narrow down candidates for the plurality of target feature points pairs MP. The narrowing-down process of S245 is a process of using a target set (here, the set of candidates MA2 and MB2), which is a set of two candidate feature points pairs, to determine whether or not to make the target set a candidate for the plurality of target feature points pairs MP. The conditions for making the target set a candidate for the plurality of target feature points pairs MP include the shape condition CWz described with reference to FIGS. 13 and 15. In this embodiment, the shape condition CWz ( FIG. 15 ) includes a first angle condition CAA and a second angle condition CAB. The first angle condition CAA is determined using first angles ZAt and ZAs. The second angle condition CAB is determined using second angles ZBt and ZBs. The shape condition CWz is determined using both the first angles ZAt, ZAs and the second angles ZBt, ZBs. Processor 210 determines multiple target feature points pairs MP by executing processing including the narrowing-down processing of S245. Processor 210 can appropriately determine whether or not to select a target set (here, a set of candidates MA, MB) as candidates for multiple target feature points pairs MP.

[0178] Furthermore, in this embodiment, the process of determining the multiple target feature points pairs MP ( FIG. 7 ) can be considered from another perspective. The pair determination process includes various processes, such as the following processes. In S245 ( FIG. 7 ), the processor 210 executes a narrowing-down process to narrow down candidates for the multiple target feature points pairs MP. The narrowing-down process of S245 is a process of using a target set, which is a set of multiple candidate feature points pairs, to determine whether or not to make the target set a candidate for the multiple target feature points pairs MP. The conditions for making the target set a candidate for the multiple target feature points pairs MP include the brightness condition CB of FIG. 13. The brightness condition CB is determined using the order of the multiple feature points in the target set. As described in S870 and S875, the order of the multiple feature points is the order of the brightness of the partial regions that include the feature points. By executing processes including the narrowing-down process of S245, the processor 210 can appropriately determine whether or not to make the target set a candidate for the multiple target feature points pairs MP.

[0179] B. Second Example: 22 is a flowchart showing another embodiment of the alignment process. The figure shows a portion of the alignment process. The only difference from the embodiment of FIG. 7 is that the narrowing-down process of S245 is replaced with the narrowing-down process of S245b, and the narrowing-down process of S250 is replaced with the narrowing-down process of S250b. The remaining portions of the alignment process are the same as the corresponding portions of the process of FIG. 7.

[0180] In S245b, the processor 210 executes a candidate feature points pair M selection process using one candidate feature points pair M. This process may be various processes. For example, the processor 210 may execute a process similar to the process of S230 (FIG. 7) (i.e., the process of FIG. 11) in S245b. Stricter conditions may be applied in S245b compared to S230. For example, a smaller hue difference threshold dHth and a smaller luminance difference threshold dVth may be used in S245b compared to S230. This enables the processor 210 to select a more appropriate candidate feature points pair M as a candidate for the multiple target feature points pairs MP.

[0181] In S250b, processor 210 executes a process for selecting candidate feature points pairs M using a combination of V candidate feature points pairs M. The number V of candidate feature points pairs M may be any number greater than the number of candidate feature points pairs M in the pair set used in the process of S245b (here, 1). In this embodiment, the number V is predetermined to 2 or 3. The selection process when V=2 may be the same as the process of S245 in FIG. 7. The selection process when V=3 may be the same as the process of S250 in FIG. 7.

[0182] In this way, the total number of candidate feature points pairs M of the pair sets used in the narrowing down process (S250b) when the determination result in S240 is No may be a number that is greater than the total number of candidate feature points pairs M of the pair sets used in the narrowing down process (S245b) when the determination result in S240 is Yes. This allows processor 210 to appropriately determine multiple target feature points pairs MP according to the total number of multiple candidate feature points pairs M.

[0183] C. Third Example: Figure 23 is a flowchart showing another embodiment of the alignment process. The figure shows a part of the alignment process. The only differences from the embodiment of Figure 7 are that S245, S260, and S265 are omitted, and the narrowing down process of S250 is replaced with the narrowing down process of S250c. The other parts of the alignment process are the same as the corresponding parts of the process of Figure 7.

[0184] If the determination result in S240 is Yes, the processor 210 proceeds to S290 without performing any further narrowing down process, thereby enabling the processor 210 to reduce the load of the process of determining a plurality of target feature points pairs MP.

[0185] If the determination result in S240 is No, in S250c, the processor 210 executes a process for selecting candidate feature points pairs M using a combination of W candidate feature points pairs M. The number W of candidate feature points pairs M may be various numbers equal to or greater than 1. In this embodiment, the number W is predetermined to 1, 2, or 3. The selection process when W=1 may be the same as the process of S245b (FIG. 22). The selection process when W=2 may be the same as the process of S245 in FIG. 7. The selection process when W=3 may be the same as the process of S250 in FIG. 7.

[0186] In this way, when the determination result in S240 is No, the total number of candidate feature points pairs M of the pair set used in the narrowing down process (S250c) may be various values ​​equal to or greater than 1. When the determination result in S240 is Yes, further narrowing down processes may be omitted. In this embodiment as well, the processor 210 can appropriately determine multiple target feature points pairs MP according to the total number of multiple candidate feature points pairs M.

[0187] D. Fourth Example: 24 is a flowchart showing another embodiment of the alignment process. A part of the alignment process is shown in the figure. In this embodiment, if the determination result of S260 is No, the narrowing down process of S270d is executed. After S270d, the process proceeds to S290. The narrowing down process of S270d may be applied to, for example, the embodiment of FIG. 7 and the embodiment of FIG. 22.

[0188] In S270d, the processor 210 executes a process for selecting candidate feature pairs M using a combination of Z candidate feature pairs M. The number Z of candidate feature pairs M may be any number greater than or equal to 1 and less than the number of candidate feature pairs M in the pair set used in the process of S265 (here, 3). In this embodiment, the number Z is predetermined to 1 or 2. The selection process when Z=1 may be the same as the process of S245b (FIG. 22). The selection process when Z=2 may be the same as the process of S245 in FIG. 7. Here, stricter conditions may be applied in S270d compared to S245. For example, a smaller angle difference threshold dZth2 (FIG. 15) may be used in S270d compared to S245. This enables the processor 210 to select more appropriate candidate feature pairs M as candidates for the multiple target feature pairs MP.

[0189] Thus, in this embodiment, if the color number index value is a third value (S260: Yes in this embodiment), processor 210 determines multiple target feature points pairs MP by executing processing including the narrowing down process of S265. If the color number index value is a fourth value greater than the third value (S260: No in this embodiment), processor 210 determines multiple target feature points pairs MP without executing the narrowing down process of S265. Here, processor 210 determines multiple target feature points pairs MP by executing processing including the narrowing down process of S270d. The narrowing down process of S270d is processing that uses a set of one or two candidate feature points pairs M to determine whether or not to select the set as a candidate for multiple target feature points pairs MP. The narrowing down process of S270d is an example of a fifth narrowing down process that uses a fifth type set, which is a set of Q (Q is an integer greater than or equal to 1 and less than P) candidate feature points pairs, to determine whether or not to select the fifth type set as a candidate for multiple target feature points pairs. The number P is the number of candidate feature point pairs M of the pair set used in the narrowing down process of S265 (in this embodiment, P=3). The load of the narrowing down process of S270d is reduced compared to the load of the narrowing down process of S265.

[0190] As described above, processor 210 can appropriately determine multiple target feature points pairs MP according to the color number index value. For example, if the color number index value is a third value (S260: Yes), processor 210 can select an appropriate candidate feature points pair M as the target feature points pair MP by executing the narrowing-down process of S265. Furthermore, if the color number index value is a fourth value (S260), processor 210 executes the narrowing-down process of S270d, which has a lighter load than S265. This allows processor 210 to select an appropriate candidate feature points pair M as the target feature points pair MP while taking into consideration the reduction in load.

[0191] Furthermore, the narrowing down process of S270d may be similar to the process of S245 in FIG. 7. Stricter conditions may be applied in S270d compared to S245. For example, a smaller angle difference threshold dZth2 (FIG. 15) may be used in S270d compared to S245. This enables processor 210 to select a more appropriate candidate feature points pair M as a candidate for the multiple target feature points pairs MP. In this way, the conditions for determining the set of candidates MA, MB as a candidate for the multiple target feature points pairs MP through the narrowing down process of S270d may include the shape condition CWz described with reference to FIGS. 13 and 15. As described above, the shape condition CWz is determined using both the angles ZAt, ZAs and the angles ZBt, ZBs. Processor 210 can appropriately determine whether to determine whether to determine the set of candidates MA, MB as a candidate for the multiple target feature points pairs MP.

[0192] E. Fifth Example: FIG. 25 is a flowchart showing another embodiment of the alignment process. A portion of the alignment process is shown in the figure. There are two differences from the embodiment of FIG. 7. The first difference is that S240, S245, and S250 are omitted, and the process moves from S230 to S260. The second difference is that the narrowing down process of S265 is replaced by the narrowing down process of S265e. The other parts of the alignment process are the same as the corresponding parts of the process of FIG. 7.

[0193] In S265e, processor 210 executes a process for selecting candidate feature points pairs M using a combination of U candidate feature points pairs M. The number U of candidate feature points pairs M may be various numbers equal to or greater than 1. In this embodiment, the number U is predetermined to 1, 2, or 3. The selection process when U=1 may be the same as the process of S245b (FIG. 22). The selection process when U=2 may be the same as the process of S245 in FIG. 7. The selection process when U=3 may be the same as the process of S250 in FIG. 7.

[0194] 7, in this embodiment, the processor 210 can appropriately determine multiple target feature points pairs MP according to the color number index value. For example, if the color number index value is a third value (S260: Yes), the processor 210 can select an appropriate candidate feature points pair M as the target feature points pair MP by performing the narrowing-down process of S265e. Furthermore, if the color number index value is a fourth value (S260: No), the processor 210 does not perform the narrowing-down process of S265e, thereby reducing the processing load of determining multiple target feature points pairs MP. Note that the number U of candidate feature points pairs M in the pair set used in S265e may be two or more.

[0195] F. Sixth Example: Fig. 26 is a flowchart showing another embodiment of the selection process for candidate feature points pairs M using a combination of three candidate feature points pairs M. The only difference from the embodiment in Fig. 17 is that S582f and S583f are inserted between S580 and S585. The other parts of the selection process are the same as the corresponding parts in Fig. 17. The selection process of this embodiment (i.e., the narrowing-down process using a set of three candidate feature points pairs M) is applicable to each of the embodiments in Figs. 7, 22, 23, 24, and 25.

[0196] If the determination result in S580 is Yes, processor 210 proceeds to S582f. S582f and S583f are the same as S870 and S875, respectively, in FIG. 13. In S582f, processor 210 calculates the brightness of each of the three reference feature points Tt and the three read feature points Ts of the candidates MA, MB, and MC of the target combination MT. The brightness calculation method is the same as the method in S870.

[0197] In S583f, the processor 210 determines whether the brightness condition CBf is satisfied. The brightness condition CBf is a condition indicating that the brightness order of multiple feature points corresponding to the target combination MT is the same between the read image IMs and the reference image IMt. The processor 210 compares the brightness order of the three reference feature points Tt corresponding to the target combination MT (i.e., candidates MA, MB, MC) with the brightness order of the three read feature points Ts. The brightness order is represented by the order of the candidates MA, MB, MC (e.g., MA, MB, MC, MC, MB, MA, etc.). When the candidates MA, MB, MC respectively indicate a proper pair of feature points Tt, Ts, the brightness order of the three reference feature points Tt is usually the same as the brightness order of the three read feature points Ts. When the candidates MA, MB, MC include an inappropriate pair, the brightness order of the three reference feature points Tt may differ from the brightness order of the three read feature points Ts.

[0198] If the brightness order of the three reference feature points Tt is the same as the brightness order of the three read feature points Ts (S583f: Yes), in S585, processor 210 selects candidates MA, MB, and MC of the target combination MT as candidates to be retained. After S585, processor 210 ends the loop processing for the current combination of candidates MA, MB, and MC. If the brightness condition CBf is not satisfied (S583f: No), processor 210 skips S585 and ends the loop processing for the current combination of candidates MA, MB, and MC.

[0199] As described above, the conditions for selecting a set of candidates MA, MB, and MC as candidates for multiple target feature points pairs MP through the narrowing-down process of this embodiment include the brightness condition CBf. The brightness condition CBf is determined using the brightness order of the three feature points. The brightness order may differ between a case where the set of candidates MA, MB, and MC is appropriate and a case where the set of candidates MA, MB, and MC includes an inappropriate candidate feature points pair M. Therefore, the processor 210 can appropriately determine whether or not to select a set of candidates MA, MB, and MC as a candidate for multiple target feature points pairs MP.

[0200] The narrowing-down process of this embodiment may be applied to S250 in Fig. 7, S250b in Fig. 22, and S250c in Fig. 23. S250, S250b, and S250c are narrowing-down processes that are executed when the total number of candidate feature point pairs M in S240 is less than the pair threshold value (S240: No). The conditions used in such narrowing-down processes may be determined using the brightness order of multiple feature points in the set corresponding to the narrowing-down process (here, the set of candidates MA, MB, MC).

[0201] G. Variations: (1) In steps S210 and S260 of FIG. 7, the processor 210 may calculate the color number index value using the scanned image IMs instead of the reference image IMt. However, the scanned image IMs may contain inappropriate portions such as noise. To mitigate the influence of the inappropriate portions, it is preferable that the processor 210 calculates the color number index value using the reference image IMt. Note that the color number index value is not limited to the number of bins in the three-dimensional histogram HST, and may be any of various values ​​that indicate whether the number of colors represented by the image is large.

[0202] (2) In each of the above embodiments and modifications, processor 210 may determine whether the total number of candidate feature points pairs M is large not only in the step (S240) following S230 (FIG. 7), but also in various steps of the pair determination process for determining multiple target feature points pairs MP. For example, S230 may be omitted. Then, processor 210 may execute S240 in the step following S220.

[0203] (3) In each of the above embodiments and modifications, the narrowing-down process using one set of candidate feature points pair M may be various processes instead of the process described in S245b of FIG. 22. For example, S230 may be omitted, and the process may proceed from S220 (FIG. 7) to S240 (FIG. 22). In this case, the process of S245b may be the same as the process of S230. That is, processor 210 may use color information to determine whether or not to select one set of candidate feature points pair M as candidates for multiple target feature points pairs MP. Processor 210 may also count the number of bright pixels having a luminance value greater than the luminance value of the pixel of the feature point from a partial region including the feature point (for example, partial regions Pt, Ps (FIG. 12(A)), FIG. 12(B)). Then, the processor 210 may select the candidate feature point pair M as a candidate for multiple target feature point pairs MP if the difference in the number of bright pixels between the reference feature point Tt and the read feature point Ts of the candidate feature point pair M is less than or equal to a reference value.

[0204] (4) In the above embodiments and modifications, the narrowing-down process using the set of two candidate feature point pairs M may be various processes instead of the processes described with reference to Fig. 13 and Fig. 15. For example, one of the first angle condition CAA and the second angle condition CAB may be omitted from the shape condition CWz (Fig. 15). Also, the brightness condition CB (Fig. 13) may be omitted.

[0205] (5) In the above embodiments and modifications, the narrowing-down process using the set of three candidate feature point pairs M may be various processes instead of the processes described in FIGS. 17, 19, and 20. For example, the reading device 100 may be configured so that the scale of the object is approximately the same between the reference image IMt and the read image IMs. In this case, the shape condition CW1 may further include a third length condition indicating that the length of a specific triangle side is approximately the same between the reference triangle TRt and the read triangle TRs (FIGS. 18(A) and 18(B)). The side length may be, for example, the lengths Lbct and Lbcs of the sides Sbct and Sbcs. The third length condition may be that the absolute value of the difference between the lengths Lbct and Lbcs is less than a length difference threshold (here, the length difference threshold is greater than zero). When the scale is approximately the same between the reference image IMt and the read image IMs, the third length condition can easily eliminate target combinations MT that include inappropriate candidate feature point pairs M. The length difference threshold is determined experimentally in advance so as to eliminate target combinations MT that include inappropriate candidate feature point pairs M. Furthermore, the shape condition CW1 may include a fourth length condition indicating that the length of a specific side of the triangle (e.g., lengths Lbct, Lbcs) is within an acceptable range. The acceptable range may be determined experimentally in advance, similar to the acceptable length range LR in FIG. 19 (e.g., the acceptable range of the third length condition may be the same as the acceptable length range LR). In this way, the shape condition CW1 may be defined using the lengths of the sides of the triangle.

[0206] In each of the above embodiments and modifications, the conditions for determining that a set of three candidate feature points pairs M is a candidate for multiple target feature points pairs MP may be determined using one or more of four parameters: ratios Rt, Rs; interior angles Abt, Abs; angles Zt, Zs; and side lengths Lbct, Lbcs.

[0207] (6) The process of determining multiple target feature points pairs MP may be various processes instead of the above-described embodiments and modifications. For example, the number of candidate feature points pairs M included in the pair set used in S245 ( FIG. 7 , etc.) may be various numbers greater than or equal to one and less than the number of candidate feature points pairs M included in the pair set used in S250. Furthermore, the total number of candidate feature points pairs M included in the pair set used in S265 ( FIG. 7 , etc.) may be two or one. Furthermore, the total number of candidate feature points pairs M included in the pair set used in S265 may be greater than the total number of candidate feature points pairs M included in the pair set used in S245. The pair set of S245 and the pair set of S265 may each be composed of two candidate feature points pairs M. In this case, stricter conditions may be applied in S265 than in S245. For example, a smaller angle difference threshold dZth2 ( FIG. 15 ) may be used in S265 than in S245. The number of candidate feature point pairs M included in the pair set used in S250 may be two.

[0208] (7) The pair set used in the narrowing down process may include four or more candidate feature point pairs M. In this case, various conditions may be used as candidate conditions for determining the pair set as a candidate for the multiple target feature point pairs MP. For example, the processor 210 may compare the length of each side between a polygon formed by multiple reference feature points Tt corresponding to the multiple candidate feature point pairs M and a polygon formed by multiple read feature points Ts. The candidate conditions may include that the lengths of the corresponding sides are approximately the same.

[0209] (8) The reading device 100 may be configured so that the angle of rotation of the object between the reference image IMt and the read image IMs is approximately the same. In this case, in S320 and S325 of FIG. 9, feature quantities Ft and Fs that are not rotation-invariant may be calculated (e.g., BRIEF (Binary Robust Independent Elementary Features)). Then, calculation of the direction of the feature points may be omitted. Conditions using the direction of the feature points (e.g., angle condition CF (FIG. 20)) may be omitted.

[0210] (9) The combination condition CC (FIG. 17: S565) is not limited to the condition described in FIG. 19, and may be various conditions indicating that the set of candidate feature point pairs M forms a figure suitable for determining the shape condition CW1. For example, the combination condition CC may include a condition indicating that the lengths Lbct and Lbcs (FIGS. 18(A) and 18(B)) are within the allowable length range LR. The scalene condition CQ (FIG. 19) may be composed of one or two conditions arbitrarily selected in advance from the conditions CQ1-CQ3. The scalene condition CQ may include a condition that the ratio Lab / Lac of the lengths Lab and Lac is outside a predetermined ratio range. The combination condition CC may be composed of one or more conditions arbitrarily selected in advance from the first length condition CL1, the second length condition CL2, the first interior angle condition CA, and the scalene condition CQ. Note that S565-S570 in FIG. 17 (that is, the determination of the combination condition CC) may be omitted.

[0211] (10) The feature point pair selection process using color information (S230 in FIG. 7) is not limited to the process shown in FIG. 11 and may be various other processes. For example, instead of the processes of S425 and S435, the processor 210 may calculate the hue of each of the multiple pixels in the first partial region Pt and calculate a representative hue for the first partial region Pt using the multiple hues. The same applies to saturation and brightness. The same applies to the representative color value of the second partial region Ps. In the example shown in FIG. 11, the condition CW2 for selecting the candidate feature point pair M as a candidate for multiple target feature point pairs MP is expressed by the entire set of conditions CSt, CH, CSs, and CV. The condition CW2 may be various other conditions that use the first representative color value of the first partial region Pt and the second representative color value of the second partial region Ps. For example, the condition CW2 may be determined using either the hue condition CH or the brightness condition CV. The condition CW2 preferably indicates that the first representative color value of the first partial region Pt is similar to the second representative color value of the second partial region Ps. Note that when the scales of the reference image IMt and the scanned image IMs are different, the first partial region Pt and the second partial region Ps may be configured to represent the same portion on the T-shirt 700.

[0212] (11) The alignment process may be various processes instead of the above-described embodiments and modifications. For example, in the embodiment of FIG. 24, steps S240-S250 (FIG. 7) may be omitted. That is, the process may proceed from S230 (FIG. 7) to S260 (FIG. 24). Also, in the embodiments of FIGS. 7 and 22, steps S260-S265 (FIG. 7) may be omitted.

[0213] (12) The detection method for the feature points Tt and Ts (i.e., key points) may be any method for detecting points representing portions of an object in an image, instead of the method described in S310 and S315 (FIG. 9). The detection method may be preselected from, for example, a search for extrema (maximum and minimum) using Difference-of-Gaussian (DoG), Harris corner detection, Features from Accelerated Segment Test (FAST) corner detection, Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented Fast and Rotated Brief (ORB).

[0214] The calculation method of the feature quantities Ft and Fs (i.e., the feature descriptor) may be various methods for calculating information describing the features of keypoints instead of the methods described in S320 and S325 (FIG. 9). The algorithm for calculating the feature descriptor may be selected in advance from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, and A-KAZE. Furthermore, the calculation method of the distance dF between the feature quantities Ft and Fs may be various methods suitable for the data structure of the feature quantities Ft and Fs. When the feature quantities Ft and Fs are expressed by binary vectors, the distance dF may be the Hamming distance. Instead of the Hamming distance, the distance dF may be various other distances (e.g., norms such as the L1 norm and the L2 norm (also known as the Euclidean distance)). The norm is applicable to various feature descriptors.

[0215] In either case, the direction associated with the feature points Tt and Ts may be the direction used to calculate the feature amounts Ft and Ts, which may be the direction of the gradient of various color values ​​(e.g., luminance).

[0216] (13) Processes other than the process of determining the correspondence between coordinates using a plurality of feature points Tt, Ts may be executed according to a program other than the program 231. For example, S130 and S140 in FIG. 4 may be executed according to another program. Furthermore, processes other than the process of determining the correspondence between coordinates using a plurality of feature points Tt, Ts may be performed by a device other than the data processing device 200. The data processing device 200 is an example of an image processing device that performs image processing for determining the correspondence between coordinates.

[0217] (14) The print medium is not limited to the T-shirt 700 and may be various types of clothing (for example, various shirts such as polo shirts, coats, slacks, etc.). The print medium may be various types of fabric such as clothing or bags. The print medium is not limited to fabric and may be various media such as paper, film, leather, etc. Furthermore, the printing device 900 may be a printing device of another type (for example, a laser type) instead of an inkjet type printing device.

[0218] (15) The coordinate correspondence (FIG. 5) may represent various transformations such as homography transformation instead of affine transformation. Also, the coordinate correspondence may be represented in various formats such as a lookup table instead of the matrix Mtx.

[0219] The coordinate correspondence may be used in various processes, not limited to inspection. For example, in the machining of metal parts, the position and orientation of the metal part relative to the tool may be misaligned. Here, the position and orientation of the metal part relative to the tool may be determined by analyzing an image of the metal part captured by a digital camera fixed to the tool. For example, the processor 210 may determine the reference position of the metal part on the captured image (and thus the position and orientation of the metal part relative to the tool) by aligning the captured image with a reference image representing a portion of the metal part that indicates the reference position. The alignment process may employ the processes of the above-described embodiments or the above-described modifications. In this way, the scanned image may be an image representing an object scanned by a line sensor such as the scanning sensor 180 (FIG. 2), or an image representing an object scanned by an area sensor such as a digital camera. Furthermore, the reference image may be various images (e.g., a prepared image representing a specific portion of an object) instead of an image related to printing. In either case, the image may be a grayscale image instead of a color image.

[0220] (16) The image processing device that determines the correspondence between coordinates is not limited to a personal computer (e.g., data processing device 200 (FIG. 1)), but may be various other devices (e.g., a smartphone, a tablet computer, a control device incorporated in a reading device, etc.). Furthermore, multiple devices (e.g., computers) that can communicate with each other via a network may share part of the data processing functions of the data processing device and collectively provide the data processing functions (a system including these devices corresponds to a data processing device).

[0221] In each of the above embodiments and modifications, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the process of S290 in Fig. 7 may be executed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0222] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used in a state stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0223] The above-described examples and modifications can be combined as appropriate. The above-described examples and modifications are provided to facilitate understanding of the present disclosure and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof. [Explanation of symbols]

[0224] 100...reading device, 110...controller, 120...conveyor device, 122...position sensor, 130...table, 140...support part, 180...reading sensor, 190...casing, 200...data processing device, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 231...program, 240...display unit, 250...operation unit, 270...communication interface, 700...T-shirt, 900...printing device

Claims

1. A program, a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in the read image and feature points in the reference image, by using feature amounts of each of a plurality of feature points in the read image and a plurality of feature points in the reference image; A pair determination function for determining a plurality of target feature point pairs from a plurality of candidate feature point pairs, a first pair determination function that determines the plurality of target feature points pairs by executing a process including a first narrowing-down process that narrows down candidates for the plurality of target feature points pairs when the total number of candidate feature points pairs at a specific stage of determining the plurality of target feature points pairs is a first number, and the first narrowing-down process is a process that uses a first type set that is a set of N (N is an integer of 2 or more) candidate feature points pairs and determines whether or not to make the first type set a candidate for the plurality of target feature points pairs; a second pair determination function that determines the plurality of target feature points pairs without executing the first narrowing-down process in a first specific case where the total number of the candidate feature points pairs is a second number that is greater than the first number; The pair determination function includes: a correspondence determination function that determines a correspondence relationship between coordinates on the scanned image and coordinates on the reference image using the plurality of target feature point pairs; A program that enables a computer to achieve this.

2. 2. The program according to claim 1, The second pair determination function is a function of determining the plurality of target feature points pairs by executing a process including a second narrowing-down process that narrows down candidates for the plurality of target feature points pairs, and the second narrowing-down process is a process of using a second type set that is a set of U (U is an integer equal to or greater than 1 and less than N) candidate feature points pairs and determining whether or not to make the second type set a candidate for the plurality of target feature points pairs. program.

3. 3. The program according to claim 2, The pair determination function further includes: a third pair determination function that, in a second specific case where the total number of the candidate feature points pairs is the second number and an index value indicating the number of colors of the scanned image or the first image that is the reference image is smaller than a reference value, determines the plurality of target feature points pairs by executing a process including a third narrowing-down process that further narrows down the plurality of candidate feature points pairs from the plurality of candidate feature points pairs that have become candidates for the plurality of target feature points pairs by the second narrowing-down process; program.

4. 4. The program according to claim 3, the third narrowing down process includes a process of using a third type set, which is a set of L (L is an integer greater than U) candidate feature points pairs, to determine whether or not the third type set is to be a candidate for the plurality of target feature points pairs. program.

5. 4. The program according to claim 2 or 3, a first specific narrowing-down process that is the first narrowing-down process or the second narrowing-down process uses a set of two candidate feature point pairs as the first type set or the second type set; a condition for determining the two candidate feature points pairs as candidates for the plurality of target feature points pairs by the first identification narrowing-down process is determined using a first angle formed between a line segment connecting a first feature point and a second feature point respectively associated with the two candidate feature points pairs and a direction associated with the first feature point, or using both a second angle formed between the line segment and the direction associated with the second feature point and the first angle. program.

6. 3. The program according to claim 1 or 2, the number N of candidate feature point pairs in the first type set is 3; The condition for determining the first type set as the candidates of the plurality of target feature points pairs by the first narrowing-down process is determined using one or more types of parameters among four types of parameters: a ratio of lengths of two sides of a triangle formed by three feature points respectively associated with three candidate feature points pairs of the first type set; a length of a side of the triangle; an interior angle of the triangle; and an angle formed between a line segment connecting two of the three feature points and a direction associated with one of the two feature points. program.

7. 4. The program according to claim 2 or 3, one or both of the condition for selecting the first type set as the candidates for the plurality of target feature points pairs by the first narrowing-down process and the condition for selecting the second type set as the candidates for the plurality of target feature points pairs by the second narrowing-down process are determined using an order of the plurality of feature points in corresponding sets of the first type set and the second type set, which is an order of brightness of partial regions including the feature points; program.

8. A program, a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in the read image and feature points in the reference image, by using feature amounts of each of a plurality of feature points in the read image and a plurality of feature points in the reference image; A pair determination function for determining a plurality of target feature point pairs from a plurality of candidate feature point pairs, a fourth pair determination function that determines the plurality of target feature points pairs by executing a process including a fourth narrowing down process that narrows down candidates for the plurality of target feature points pairs when an index value indicating the number of colors of the scanned image or the second image that is the reference image is a third value, and the fourth narrowing down process is a process that uses a fourth type set that is a set of P (P is an integer of 2 or more) candidate feature points pairs and determines whether or not the fourth type set is to be a candidate for the plurality of target feature points pairs; a fifth pair determination function that determines the plurality of target feature points pairs without executing the fourth narrowing-down process when the index value is a fourth value greater than the third value; and The pair determination function includes: a correspondence determination function that determines a correspondence relationship between coordinates on the scanned image and coordinates on the reference image using the plurality of target feature point pairs; A program that enables a computer to achieve this.

9. 9. The program according to claim 8, The fifth pair determination function is a function of determining the plurality of target feature points pairs by executing a process including a fifth narrowing down process that narrows down candidates for the plurality of target feature points pairs, and the fifth narrowing down process is a process of using a fifth type set that is a set of Q (Q is an integer equal to or greater than 1 and less than P) candidate feature points pairs, and determining whether or not to make the fifth type set a candidate for the plurality of target feature points pairs. program.

10. 10. The program according to claim 9, the second specific narrowing down process, which is the fourth narrowing down process or the fifth narrowing down process, uses a set of two candidate feature point pairs as the fourth type set or the fifth type set, a condition for determining the two candidate feature points pairs as candidates for the plurality of target feature points pairs by the second identification narrowing-down process is determined using a third angle formed by a line segment connecting a third feature point and a fourth feature point respectively associated with the two candidate feature points pairs and a direction associated with the third feature point, or using both the fourth angle formed by the line segment and the direction associated with the fourth feature point and the third angle. program.

11. 10. The program according to claim 8 or 9, the number P of candidate feature point pairs in the fourth type set is 3; The condition for determining the fourth type set as the candidates of the plurality of target feature points pairs by the fourth narrowing-down process is determined using one or more types of parameters among four types of parameters: a ratio of lengths of two sides of a triangle formed by three feature points respectively associated with three candidate feature points pairs of the fourth type set; a length of a side of the triangle; an interior angle of the triangle; and an angle formed between a line segment connecting two of the three feature points and a direction associated with one of the two feature points. program.

12. A program, a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in the read image and feature points in the reference image, by using feature amounts of each of a plurality of feature points in the read image and a plurality of feature points in the reference image; a pair determination function that determines a plurality of target feature points pairs from a plurality of candidate feature points pairs, the pair determination function including a function that determines the plurality of target feature points pairs by executing a process including a narrowing-down process that narrows down candidates for the plurality of target feature points pairs, the narrowing-down process being a process that uses a target set that is a set of two candidate feature points pairs to determine whether or not the target set is to be a candidate for the plurality of target feature points pairs, and a condition for determining the target set as a candidate for the plurality of target feature points pairs is determined using a first angle formed by a line segment connecting a first feature point and a second feature point that are respectively associated with the two candidate feature points pairs and a direction associated with the first feature point, or using both a second angle formed by the line segment and a direction associated with the second feature point and the first angle; a correspondence determination function that determines a correspondence relationship between coordinates on the scanned image and coordinates on the reference image using the plurality of target feature point pairs; A program that enables a computer to achieve this.

13. A program, a candidate acquisition function that acquires a plurality of candidate feature point pairs, which are pairs of feature points in the read image and feature points in the reference image, by using feature amounts of each of a plurality of feature points in the read image and a plurality of feature points in the reference image; a pair determination function that determines a plurality of target feature points pairs from a plurality of candidate feature points pairs, the pair determination function including a function that determines the plurality of target feature points pairs by executing a process including a narrowing-down process that narrows down candidates for the plurality of target feature points pairs, the narrowing-down process being a process that uses a target set, which is a set of a plurality of candidate feature points pairs, to determine whether or not the target set is to be a candidate for the plurality of target feature points pairs, and a condition for determining the target set as a candidate for the plurality of target feature points pairs is determined using the order of a plurality of feature points, which is the order of brightness of partial regions including the feature points; a correspondence determination function that determines a correspondence relationship between coordinates on the scanned image and coordinates on the reference image using the plurality of target feature point pairs; A program that enables a computer to achieve this.

14. An image processing device, a candidate acquisition unit that acquires a plurality of candidate feature point pairs, each of which is a pair of a feature point in the read image and a feature point in the reference image, by using a feature amount of each of a plurality of feature points in the read image and a feature amount of each of a plurality of feature points in the reference image; a pair determination unit that determines a plurality of target feature point pairs from a plurality of candidate feature point pairs, When the total number of candidate feature points pairs at a specific stage of determining the plurality of target feature points pairs is a first number, the plurality of target feature points pairs are determined by executing a process including a first narrowing-down process that narrows down candidates for the plurality of target feature points pairs, and the first narrowing-down process is a process that uses a first type set that is a set of N (N is an integer of 2 or more) candidate feature points pairs and determines whether or not to make the first type set a candidate for the plurality of target feature points pairs. determining the plurality of target feature points pairs without performing the first narrowing-down process in a first specific case where the total number of the candidate feature points pairs is a second number greater than the first number; the pair determination unit; a correspondence determination unit that determines a correspondence relationship between coordinates on the read image and coordinates on the reference image using the plurality of target feature point pairs; An image processing device comprising:

15. An image processing device, a candidate acquisition unit that acquires a plurality of candidate feature point pairs, each of which is a pair of a feature point in the read image and a feature point in the reference image, by using a feature amount of each of a plurality of feature points in the read image and a feature amount of each of a plurality of feature points in the reference image; a pair determination unit that determines a plurality of target feature point pairs from a plurality of candidate feature point pairs, when an index value indicating the number of colors of the scanned image or the second image that is the reference image is a third value, the plurality of target feature points pairs are determined by executing a process including a fourth narrowing down process that narrows down candidates for the plurality of target feature points pairs, the fourth narrowing down process being a process that uses a fourth type set that is a set of P (P is an integer of 2 or more) candidate feature points pairs and determines whether or not the fourth type set is to be a candidate for the plurality of target feature points pairs; If the index value is a fourth value greater than the third value, the fourth narrowing-down process is not performed and the plurality of target feature point pairs are determined. the pair determination unit; a correspondence determination unit that determines a correspondence relationship between coordinates on the read image and coordinates on the reference image using the plurality of target feature point pairs; An image processing device comprising:

16. An image processing device, a candidate acquisition unit that acquires a plurality of candidate feature point pairs, each of which is a pair of a feature point in the read image and a feature point in the reference image, by using a feature amount of each of a plurality of feature points in the read image and a feature amount of each of a plurality of feature points in the reference image; a pair determination unit that determines a plurality of target feature points pairs from a plurality of candidate feature points pairs, the plurality of target feature points pairs being determined by executing a process including a narrowing-down process that narrows down candidates for the plurality of target feature points pairs, the narrowing-down process being a process that uses a target set, which is a set of two candidate feature points pairs, to determine whether or not the target set is to be a candidate for the plurality of target feature points pairs, and a condition for determining the target set as a candidate for the plurality of target feature points pairs is determined using either a first angle formed by a line segment connecting a first feature point and a second feature point that are respectively associated with the two candidate feature points pairs, and a direction associated with the first feature point, or both a second angle formed by the line segment and a direction associated with the second feature point, and the first angle; a correspondence determination unit that determines a correspondence relationship between coordinates on the read image and coordinates on the reference image using the plurality of target feature point pairs; An image processing device comprising:

17. An image processing device, a candidate acquisition unit that acquires a plurality of candidate feature point pairs, each of which is a pair of a feature point in the read image and a feature point in the reference image, by using a feature amount of each of a plurality of feature points in the read image and a feature amount of each of a plurality of feature points in the reference image; a pair determination unit that determines a plurality of target feature points pairs from a plurality of candidate feature points pairs, and determines the plurality of target feature points pairs by executing a process including a narrowing-down process that narrows down candidates for the plurality of target feature points pairs, the narrowing-down process being a process that uses a target set, which is a set of a plurality of candidate feature points pairs, to determine whether or not the target set is to be a candidate for the plurality of target feature points pairs, and a condition for determining the target set as a candidate for the plurality of target feature points pairs is determined using the order of a plurality of feature points, which is the order of brightness of partial regions including the feature points; a correspondence determination unit that determines a correspondence relationship between coordinates on the read image and coordinates on the reference image using the plurality of target feature point pairs; An image processing device comprising:

Images