Program and data processing device

Abstract

This invention provides a program and data processing device for aligning multiple images. [Solution] The data processing device 200, which processes according to program 231, performs a first orientation calculation process for each of the multiple first type feature points in the first image, and performs a second orientation calculation process for each of the multiple first type feature points. It also performs a first orientation calculation process for each of the multiple second type feature points in the second image, and performs a second orientation calculation process for each of the multiple second type feature points. Then, using the first type orientation and second type orientation of the two first type feature points and two second type feature points that form two feature point pairs, it determines whether or not to select the two feature point pairs as two target pairs, and uses the multiple target pairs to determine the correspondence between coordinates on the first image and coordinates on the second image.

Description

Technical Field

[0001] This specification relates to the alignment between multiple images.

Background Art

[0002] In various processes, alignment between multiple images can be performed. Patent Document 1 discloses a technique for detecting defects in an image formed on a sheet by an image forming apparatus such as a printer or a copier. In this technique, a marker image for positioning is formed on the same sheet together with a job image instructed by the user for printing. An image reading unit reads the sheet surface to generate a read image. An inspection unit determines the position of the read image corresponding to the reference image based on the feature points of the job image and the marker image extracted from the read image to be inspected and the feature points of the job image and the marker image extracted from the reference image. The inspection unit compares the reference image and the read image after alignment, and detects an image area where the difference in pixel values is greater than or equal to a threshold value as a defect.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Alignment between multiple images is not easy and there is room for improvement.

[0005] This specification discloses a technique for aligning multiple images.

Means for Solving the Problems

[0006] The technique disclosed in this specification can be realized as the following application examples.

[0007] [Application Example 1] A program comprising: a function to perform a first orientation calculation process, which is a process to calculate a first type orientation for each of a plurality of first type feature points in a first image including a first object image; a function to perform a second orientation calculation process, which is a process to calculate a second type orientation, which is a different type of orientation from the first type orientation, for each of the plurality of first type feature points in the first image; and a function to perform the first orientation calculation process for each of a plurality of second type feature points in a second image including a second object image, wherein the second object image includes a portion having an appearance shape common to at least a part of the first object image. A program that causes a computer to implement the above function, a function to perform the second orientation calculation process for each of the plurality of second type feature points in the second image, a function to select a plurality of target pairs, each of the plurality of target pairs being a feature point pair consisting of a first type feature point and a second type feature point, and the function to select the plurality of target pairs includes a function to determine whether or not to select the two feature point pairs as two target pairs using the first type orientation and the second type orientation of the two first type feature points and two second type feature points that form two feature point pairs, and a function to determine the correspondence between coordinates on the first image and coordinates on the second image using the plurality of target pairs.

[0008] With this configuration, the system uses the first-type and second-type orientations of the two first-type feature points and two second-type feature points that form two feature point pairs to determine whether or not to select two feature point pairs as two target pairs. This allows for proper alignment between the first and second images using multiple target pairs.

[0009] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, as data processing methods and data processing devices, computer programs for realizing the functions of such methods or devices, and recording media (e.g., non-temporary recording media) on which such computer programs are recorded. [Brief explanation of the drawing]

[0010] [Figure 1] This is an explanatory diagram showing a data processing device as one embodiment. [Figure 2] This is a perspective view showing an example of the reading device 100. [Figure 3] (A) is a diagram showing an example of an image represented by image data for printing. (B) is a diagram showing an example of a readable image. [Figure 4] This is a flowchart illustrating an example of the inspection process. [Figure 5] This diagram illustrates an example of a representation format for coordinate correspondence. [Figure 6] (A) and (B) are diagrams illustrating examples of difference images. [Figure 7] This is a flowchart illustrating an example of the alignment process. [Figure 8] This is a flowchart illustrating an example of feature point matching processing. [Figure 9] Figures (A)-(D) show examples of images processed using feature point matching. [Figure 10] This is a flowchart illustrating an example of a selection process. [Figure 11] (A) and (B) are diagrams illustrating examples of how representative color values ​​are calculated. [Figure 12] This flowchart illustrates an example of the selection process for candidate feature point pairs M. [Figure 13] This is a flowchart illustrating an example of the orientation acquisition process. [Figure 14] Figures (A)-(C) are diagrams illustrating examples of the calculation process for Type 2 orientation. [Figure 15]It is a diagram showing an example of a feature point group N. [Figure 16] It is a flowchart showing an example of the process of determining the shape condition CWz. [Figure 17] (A)-(D) are diagrams showing examples of the line segments formed by candidates MA and MB and the orientations of each feature point. [Figure 18] It is a flowchart showing a second embodiment of the orientation acquisition process. [Figure 19] (A)-(C) are diagrams showing examples of the orientation calculation process. <� [Figure 20] It is a flowchart showing a third embodiment of the orientation acquisition process. [Figure 21] (A), (B) are diagrams showing examples of the orientation calculation process. <�

Mode for Carrying Out the Invention

[0011] A. First Embodiment: A1. Device Configuration: FIG. 1 is an explanatory diagram showing a data processing device as an example. The data processing device 200 is, for example, a personal computer. The data processing device 200 executes an inspection process for a printed image. The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] [[ID=३३]] The processor 210 is a device configured to perform data processing, and is, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile storage device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores the data of the program 231.

[0013] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive user input, such as buttons, levers, or a touch panel superimposed on the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touchscreen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250. The display unit 240 may display elements for operation (e.g., buttons, sliders, etc.), and the displayed elements may be operated through the operation of the operation unit 250.

[0014] The communication interface 270 is an interface for communicating with other devices (for example, including one or more of the following: USB interface, wired LAN interface, IEEE 802.11 wireless interface, industrial camera interface (e.g., CameraLink, CoaXPress, etc.)). In this embodiment, the reading device 100 and the printing device 900 are connected to the communication interface 270. The printing device 900 is a so-called inkjet printer that prints an image on a printing medium such as cloth or paper by ejecting ink onto the medium. The reading device 100 generates data of a read image representing the object by optically reading the object to be read. Hereinafter, the printing medium will be a T-shirt, and the reading device 100 will read a T-shirt on which an image has been printed.

[0015] Figure 2 is a perspective view showing an example of the reading device 100. In the figure, the first direction Da and the second direction Db represent horizontal directions, and the third direction Dc represents the vertically upward direction. The first direction Da and the second direction Db are perpendicular to each other.

[0016] In this embodiment, the reading device 100 comprises a housing 190, a table 130, a support part 140 fixed to the upper surface of the table 130, a transport device 120, a reading sensor 180, and a control device 110. The control device 110, the transport device 120, and the reading sensor 180 are fixed to the housing 190.

[0017] The support portion 140 is a plate-shaped member that forms a flat top surface for supporting the object to be read (such a member is also called a platen). In the figure, a T-shirt 700 with a printed image IMpp is placed on the support portion 140.

[0018] The conveying device 120 is configured to convey the table 130 in a direction parallel to the second direction Db. The configuration of the conveying device 120 may vary. Although not shown in the figures, in this embodiment the conveying device 120 has a rail that supports the table 130 so as to be slidable in a direction parallel to the second direction Db, a plurality of pulleys, a belt that is wrapped around the plurality of pulleys and partly fixed to the table 130, and an electric motor that rotates the pulleys. By rotating the pulleys with the electric motor, the table 130 (and by extension, the support part 140) moves in a direction parallel to the second direction Db. The conveying device 120 further has a position sensor 122 (for example, a rotary encoder) that detects the position of the table 130 on the conveying path.

[0019] The reading sensor 180 is positioned higher than the support unit 140 in the middle of the transport path Pth of the support unit 140. The reading sensor 180 includes a line sensor containing multiple photoelectric conversion elements arranged in a direction intersecting the transport direction Db (in this embodiment, in the direction Da perpendicular to the transport direction Db) (for example, a Contact Image Sensor (CIS) or a Charge Coupled Device (CCD)). The reading sensor 180 faces downward. The reading sensor 180 can read the portion of the object supported by the support unit 140 that is located below the reading sensor 180.

[0020] When reading the T-shirt 700, the reading device 100 transports the table 130 in a direction parallel to the second direction Db. The reading sensor 180 repeatedly reads the T-shirt 700 during transport. As a result, the reading sensor 180 can read approximately the entire portion of the T-shirt 700 that is supported by the support portion 140.

[0021] The control device 110 is an electrical circuit configured to control the transport device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates data of the read image by controlling the transport device 120 and the reading sensor 180.

[0022] A2. Printing process: In this embodiment, an image is printed on a T-shirt 700 (Figure 2). The image printing is performed, for example, as part of a T-shirt sales service. The T-shirt sales service may include on-demand printing. The customer orders on-demand printing from the service provider. The service provider prints the image on the T-shirt 700 using image data provided by the customer, in accordance with the customer's order.

[0023] Figure 3(A) is a diagram showing an example of an image represented by image data for printing (referred to as the target image IMp). In this embodiment, the target image IMp is a two-dimensional image. The data of the target image IMp is bitmap data representing the color values ​​(here, the gradation values ​​of red R, green G, and blue B (for example, values ​​between zero and 255)) of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. In the example of Figure 3(A), the target image IMp represents a background BG and four rectangular objects OB1-OB4. Objects OB1-OB4 are represented by different colors. For example, the color of the first object OB1 is red, the color of the second object OB2 is green, the color of the third object OB3 is blue, and the color of the fourth object OB4 is gray. Thus, the target image IMp may contain multiple objects having different hues.

[0024] Although not shown in the diagram, the target image IMp is printed using the data processing device 200 and the printing device 900. Alternatively, the target image IMp may be printed using other devices.

[0025] Various errors may occur during the printing of the target image IMp. The printed image may have various defects due to these errors. For example, due to an abnormal ink ejection, a portion of the printed image may be missing. The data processing device 200 detects defects in the printed image through an inspection process described later. In this embodiment, the data of the target image IMp is used in the inspection process. After printing the target image IMp, the data of the target image IMp is stored in the storage device 215 (e.g., non-volatile storage device 230) of the data processing device 200 for the inspection process.

[0026] A3. Inspection process: Figure 4 is a flowchart illustrating an example of the inspection process. For inspection, the T-shirt 700 is placed on the support unit 140 of the reader 100 (Figure 2) so that the printed image IMpp is visible. In this embodiment, an operator places the T-shirt 700 on the support unit 140. Alternatively, a machine (e.g., a robotic arm) may place the T-shirt 700 on the support unit 140. After the T-shirt 700 is placed, an instruction to start the inspection process is input to the data processing device 200 (Figure 1). In this embodiment, the operator inputs the instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the instruction to start. The instruction to start the inspection process may be input to the data processing device 200 via the communication interface 270 by a device other than the data processing device 200.

[0027] The processor 210 of the data processing device 200 executes inspection processing according to program 231. In S110, the T-shirt 700 is photographed. The processor 210 supplies a read instruction to the reading device 100. The control device 110 of the reading device 100 reads the T-shirt 700 by controlling the reading sensor 180 and the transport device 120 in response to the read instruction. The control device 110 generates data of a read image representing the read T-shirt 700. The processor 210 of the data processing device 200 acquires the read image data from the control device 110 of the reading device 100 and stores the acquired read image data in the storage device 215 (for example, the non-volatile storage device 230).

[0028] Figure 3(B) is a diagram showing an example of a read image. In this embodiment, the read image IMs is a two-dimensional image. The data of the read image IMs is bitmap data representing the color values ​​(here, the gradation values ​​of red R, green G, and blue B (for example, values ​​between zero and 255)) of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. In the figure, the read image IMs represents the portion of the T-shirt 700 that includes the printed image IMpp. Here, the printed image IMpp is assumed to be the image obtained by printing the target image IMp (Figure 3(A)). The printed image IMpp, like the target image IMp, represents the background BG and objects OB1-OB4.

[0029] In S120 (Figure 4), the processor 210 performs alignment between the reference image and the read image. In this embodiment, the target image IMp (Figure 3(A)) is used as the reference image. Hereafter, the target image IMp will also be referred to as the reference image IMt.

[0030] The orientation of the T-shirt 700 relative to the reading sensor 180 can vary. Therefore, the orientation of objects OB1-OB4 in the images (i.e., the angle of rotation of the objects) may differ between the reference image IMt and the reading image IMs. Also, the pixel density (also called resolution) for the same object may differ between the reference image IMt and the reading image IMs. In other words, the scale of the object may differ between the reference image IMt and the reading image IMs.

[0031] The processor 210 determines the correspondence between coordinates on the reference image IMt and coordinates on the read image IMs by performing the alignment process described later. Figure 5 is a diagram showing an example of the representation format of the coordinate correspondence. The diagram shows the coordinates COt on the reference image IMt, the coordinates COs on the read image IMs, and the matrix Mtx that associates these coordinates COt and COs. In this embodiment, the matrix Mtx represents a so-called affine transformation. In the diagram, the coordinates COt and COs and the matrix Mtx are represented in a homogeneous coordinate system. Each coordinate COt and COs is represented as a three-dimensional vector having positions Xt and Xs in the first direction Dx on the images IMt and IMs, positions Yt and Ys in the second direction Dy, and a third component of 1. The matrix Mtx is a 3x3 matrix. As shown in the diagram, the matrix Mtx is represented by six parameters af in a 2x3 array and three components (0,0,1) in the third row. Such a matrix Mtx can represent rotation, scaling, shrinking, translation, and skew. The matrix Mtx can be calculated using three or more pairs of coordinates COt and COs.

[0032] In S130 (Figure 4), the processor 210 inspects the printed image. The inspection method may be various methods using coordinate correspondence (Figure 5). In this embodiment, the processor 210 generates difference image data using coordinate correspondence. Figures 6(A) and 6(B) show examples of difference images. Figure 6(A) shows the case where the printed image has no defects, and Figure 6(B) shows the case where the printed image has defects (in this case, missing Err). On the left side of each figure are the read images IMs and IMs2, and the reference image IMt placed on the read images IMs and IMs2 according to the coordinate correspondence. The images IMd and IMd2 on the right side of each figure show examples of difference images between the read images IMs and IMs2 and the reference image IMt. The processor 210 generates difference images IMd and IMd2, which represent the difference in color values ​​(e.g., the absolute value of the difference in luminance values) between the read images IMs and IMs2 and the reference image IMt at positions associated by the coordinate correspondence. In Figure 6(A), the read image IMs represents a printed image without defects. Therefore, the difference image IMd does not have any parts that show a large difference. In Figure 6(B), the read image IMs2 represents a printed image with missing Err. Therefore, in the difference image IMd2, the part corresponding to the missing Err represents a larger difference compared to other parts.

[0033] In S140 (Figure 4), the processor 210 outputs the inspection results. There are various methods for outputting the inspection results. In this embodiment, the processor 210 displays the difference image on the display unit 240 (Figure 1). By observing the display unit 240, the operator can easily recognize defects in the printed image. Alternatively, the processor 210 may output data representing the inspection results to a storage device (for example, a non-volatile storage device 230, or an external storage device connected to the data processing device 200). As a result, the data representing the inspection results is stored in the storage device. The data representing the inspection results can be used for various processes (for example, an overall inspection process of the T-shirt 700). After S140, the processor 210 terminates the inspection process.

[0034] A4. Alignment process: Figure 7 is a flowchart illustrating an example of the alignment process. In this embodiment, the processor 210 acquires multiple pairs of feature points by feature point matching and uses the acquired pairs to determine the correspondence between coordinates on the reference image IMt and coordinates on the read image IMs.

[0035] In S220 (Figure 7), the processor 210 performs feature point matching. Figure 8 is a flowchart illustrating an example of the feature point matching process. In S305, the processor 210 generates a grayscale reference image IMtg and a grayscale read image IMsg by performing a grayscale conversion between the reference image IMt and the read image IMs. A known relationship can be used as the correspondence between color gradation values ​​and grayscale gradation values ​​(for example, the correspondence between RGB values ​​in the RGB color space and luminance values ​​Y in the YCbCr color space).

[0036] In S310, processor 210 extracts feature points Tt from the grayscale reference image IMtg. Figures 9(A)-9(D) show examples of images processed by feature point matching. Figure 9(A) shows examples of feature points detected from images IMtg and IMsg. The multiple black dots on the grayscale reference image IMtg each represent a feature point Tt detected from the grayscale reference image IMtg (also called a reference feature point Tt). As shown in the figure, points that indicate characteristic parts of an object, such as corners and edges, are detected as feature points Tt. Such feature points Tt are also called keypoints. Although not shown in the figure, in reality, many more feature points Tt may be detected (for example, tens or hundreds). Note that the reference feature points Tt detected from the grayscale reference image IMtg correspond to feature points that indicate the same part at the same coordinates on the reference image IMt (Figure 3(A)).

[0037] There are various methods for detecting feature points. In this embodiment, a technique called Accelerated-KAZE (A-KAZE) is used, which performs keypoint detection and calculation of feature descriptors for each keypoint. The A-KAZE technique is disclosed, for example, in the following paper. "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013"

[0038] The processor 210 detects multiple feature points Tt by analyzing the gray reference image IMtg according to A-KAZE technology.

[0039] In S315 (Figure 8), the processor 210 extracts multiple feature points Ts from the grayscale image IMsg. The multiple black dots on the grayscale image IMsg in Figure 9(A) each represent a feature point Ts detected from the grayscale image IMsg (also called read feature points Ts). The processor 210 detects multiple feature points Ts by analyzing the grayscale image IMsg according to the A-KAZE technique. Although not shown in the illustration, in reality, many more feature points Ts may be detected (for example, tens or hundreds). Note that the read feature points Ts detected from the grayscale image IMsg correspond to feature points that indicate the same coordinates and the same part on the read image IMs (Figure 3(B)).

[0040] In S320 (Figure 8), the processor 210 calculates the feature quantities Ft for each of the multiple reference feature points Tt. The feature quantities Ft may be various pieces of information that describe the features of the feature point Tt. The feature quantities Ft are calculated such that they change, for example, according to the distribution of color values ​​of multiple pixels surrounding the feature point Tt. In this embodiment, the processor 210 uses a grayscale reference image IMtg to calculate the feature descriptor of A-KAZE as the feature quantity Ft. The reference image IMtg in Figure 9(A) shows a local region Wta containing the reference feature point Tta. The local region Wta is an example of a region whose features are represented by the feature quantity Ft of the reference feature point Tta. The feature quantities Ft are calculated using the image of the local region Wta. Although not shown in the illustration, the feature quantities Ft of other feature points Tt are also calculated using the image of the local region containing the feature point Tt.

[0041] In A-KAZE technology, feature descriptors are rotationally invariant. To obtain rotationally invariant feature descriptors, the orientation of feature points is also calculated. The orientation of a feature point indicates the orientation of the luminance gradient within the neighborhood region centered on the feature point (also called gradient orientation or dominant orientation). Processor 210 calculates the orientation of the feature quantity Ft and the reference feature point Tt.

[0042] In Figure 9(A), the reference image IMtg shows a subregion Ata containing the reference feature point Tta. Subregion Ata is an example of a region used to calculate the orientation of the reference feature point Tta. Subregion Ata is the region centered on the reference feature point Tta. Subregion Ata may differ from the local region Wta. Subregion Ata may include parts not included in the local region Wta. The local region Wta may include parts not included in subregion Ata. Although not shown in the illustration, the orientation of other feature points Tt is also calculated using the image of the subregion containing the feature point Tt.

[0043] In S325 (Figure 8), the processor 210 calculates the feature quantities Fs for each of the multiple read feature points Ts. In this embodiment, the processor 210 uses the grayscale read image IMsg to calculate the feature descriptors (i.e., feature quantities Fs) and the orientation of the read feature points Ts according to the A-KAZE technique. The read image IMsg in Figure 9(A) shows a local region Wsa containing the read feature point Tsa. The feature quantity Fs is calculated using the image of the local region Wsa. The read image IMsg also shows a subregion Asa containing the read feature point Tsa. The subregion Asa is an example of a region used to calculate the orientation of the read feature point Tsa. The local region Wsa and subregion Asa are determined according to the A-KAZE technique, similar to the local region Wta and subregion Ata. Although not shown in the illustration, the feature quantities Fs of the other feature points Ts are also calculated using the images of the local regions containing the feature points Ts. The orientation of other feature points Ts is also calculated using the image of the subregion containing the feature points Ts.

[0044] The following is an overview of the processing after S325. For each combination of reference feature point Tt and read feature point Ts, the processor 210 calculates the distance dF between feature quantities Ft and Fs (S350). Then, it obtains feature point pairs Tt and Ts that have a distance dF less than the distance threshold dFth as candidate feature point pairs M (S355: Yes, S360).

[0045] Specifically, the process is as follows: The processor 210 performs a loop process S330 (including S335-S360) between the start L31s and the end L31e for each of the multiple reference feature points Tt. In S335, the processor 210 selects an unprocessed reference feature point Tt as the feature point of interest Tti, which is the feature point to be processed. The processor 210 performs a loop process S340 (including S345-S360) between the start L32s and the end L32e for each of the multiple read feature points Ts. In S345, the processor 210 selects an unprocessed read feature point Ts as the feature point of interest Tsj, which is the feature point to be processed.

[0046] In S350, the processor 210 calculates the distance dF between two feature quantities Fti and Fsj, which are two feature points Tti and Tsj of interest. The distance dF is calculated such that a small distance dF indicates a high similarity between the two feature quantities Fti and Fsj. A high similarity (i.e., a small distance dF) indicates that the two feature points Tti and Tsj represent similar parts (e.g., corresponding parts of the same object) in two images IMtg and IMsg. In this embodiment, the feature quantities Ft and Fs are A-KAZE feature descriptors and are represented by binary vectors (vectors containing one or more binary elements). In this case, the processor 210 may calculate the Hamming distance as the distance dF.

[0047] In S355, the processor 210 determines whether the distance dF is less than the distance threshold dFth. If the distance dF is small, the feature points Tti and Tsj are likely to represent similar parts of the images IMt and IMs (for example, the same part of the same object). If the distance dF is less than the distance threshold dFth (S355: Yes), in S360, the processor 210 obtains the pair of feature points Tti and Tsj as a candidate feature point pair M (also called candidate feature point pair M). A feature point pair is a pair of reference feature point Tt and read feature point Ts used to determine the coordinate correspondence. The feature points Tt and Ts that are associated with each other are also called a matching pair. After S360, the processor 210 terminates the loop processing S340 for the feature point Tsj. If the distance dF is greater than or equal to the distance threshold dFth (S355: No), the processor 210 skips S360 and terminates the loop processing S340 for the feature point of interest Tsj. Then, the processor 210 repeats the loop processing S340 and loop processing S330 to execute the processing S350-S360 for each of the multiple combinations of the feature point of interest Tsj and the feature point of interest Tti.

[0048] Figure 9(B) shows an example of a candidate feature point pair M. Multiple lines RL in the figure each represent a candidate feature point pair M. Each line RL connects the feature points Tt and Ts that form the candidate feature point pair M. As shown, a pair of feature points Tt and Ts that represent the same part can be selected as a candidate feature point pair M. For example, feature point Tt1, which represents the upper right corner of the first object OB1 in the grayscale reference image IMtg, may be associated with feature point Ts1, which represents the upper right corner of the first object OB1 in the grayscale read image IMsg.

[0049] Furthermore, a pair of feature points Tt and Ts that represent different parts may be selected as a candidate feature point pair M. For example, feature point Tt1, which represents the upper right corner of the first object OB1 in the gray reference image IMtg, may be associated with feature point Ts2, which represents the upper right corner of the third object OB3 in the gray read image IMsg. The rectangular objects OB1-OB4 each have four corners. These corners are locally similar. As a result, feature point Tt, which represents one corner of an object in the gray reference image IMtg, may be associated with feature point Ts, which represents another corner of the same object in the gray read image IMsg, or with feature point Ts, which represents a corner of a different object. Thus, when each image IMtg, IMsg represents multiple locally similar parts, feature points Tt and Ts that represent different parts may be associated.

[0050] Furthermore, in this embodiment, as described above, the feature quantities Ft and Fs are rotationally invariant. When feature quantities are rotationally invariant, similar feature quantities can be calculated from the same part of an object regardless of the rotation angle of the object in the image. By using the distance dF between the rotationally invariant feature quantities Ft and Fs, the processor 210 can associate pairs of feature points that represent the same part of an object, even if the rotation angles of the same object differ between the two images IMtg and IMsg. However, if each image IMtg and IMsg represents multiple locally similar parts, feature points Tt and Ts that represent different parts may be associated. For example, feature point Tt1, which represents the upper right corner of the first object OB1 in the grayscale reference image IMtg, may be associated with feature point Ts3, which represents the lower right corner of the first object OB1 in the grayscale read image IMsg.

[0051] Note that a larger distance threshold dFth (Figure 8: S355) can result in a larger total number of suitable candidate feature point pairs M. However, the total number of inappropriate candidates M can also increase. A larger total number of suitable candidates M reduces the error in the coordinate correspondence, as described later. A larger total number of inappropriate candidates M can increase the error in the coordinate correspondence. The distance threshold dFth may be determined experimentally in advance so that the error in the coordinate correspondence is acceptable.

[0052] When processing all combinations of reference feature points Tt and read feature points Ts is complete, in S365, the processor 210 stores data representing multiple candidate feature point pairs M in the storage device 215 (for example, the non-volatile storage device 230). Then, the processor 210 terminates the process shown in Figure 8, i.e., the process shown in S220 of Figure 7.

[0053] In S230, the processor 210 performs a feature point pair selection process using color information. This selection process selects an appropriate candidate M from multiple candidate feature point pairs M using color information (i.e., the candidate feature point pairs M are validated). Figure 10 is a flowchart illustrating an example of the selection process. For each of the multiple candidate feature point pairs M, the processor 210 performs a loop process S410 (including S420-S465) between a start L4s and an end L4e.

[0054] In S420, the processor 210 selects the unprocessed candidate M as the candidate of interest Mi, which is the candidate to be processed.

[0055] In step S425, the processor 210 calculates the representative color value Ct of the first subregion Pt containing the reference feature point Tt of the candidate Mi of interest (the representative color value Ct is also called the reference representative color value Ct). Figures 11(A) and 11(B) show examples of representative color value calculation. Figure 11(A) shows a portion of the reference image IMt containing the feature point Tt. The figure shows the first subregion Pt containing the feature point Tt. The processor 210 calculates the reference representative color value Ct using multiple color values ​​of multiple pixels in the first subregion Pt. The method for calculating the representative color value Ct may be any of the various methods for calculating a color that represents the first subregion Pt. In this embodiment, the processor 210 calculates the average value of each RGB in the first subregion Pt as the respective gradation values ​​of the RGB of the representative color value Ct. Instead of the average value, various summary statistics that represent the magnitude of the color values ​​(e.g., median, mode, etc.) may be used. Furthermore, the configuration of the first subregion Pt (specifically, the shape of the first subregion Pt and the relative position of the first subregion Pt with respect to the position of the feature point Tt) may be various configurations that enable the calculation of a representative color value Ct representing the color of the portion indicated by the feature point Tt. In this embodiment, the first subregion Pt is a region centered on the feature point Tt, and is, for example, a region of P x Q with P*Q pixels. To reduce the dependence of the representative color value Ct on the rotation angle of the object, P=Q is preferred (for example, P=Q=5). Instead of a region of P x Q, the first subregion Pt may be a region whose distance from the feature point Tt is less than or equal to a distance threshold.

[0056] In S430 (Figure 10), the processor 210 calculates the representative color value Cs of the second subregion Ps containing the read feature point Ts of the candidate Mi of interest (the representative color value Cs is also called the read representative color value Cs). Figure 11(B) shows a portion of the read image IMs containing the feature point Ts. The figure shows the second subregion Ps containing the feature point Ts. The configuration of the second subregion Ps is the same as that of the first subregion Pt. The second subregion Ps is a region of P*Q pixels in a P row and Q column centered on the feature point Ts. The method for calculating the read representative color value Cs is the same as the method for calculating the reference representative color value Ct. The processor 210 calculates the average values ​​of red R, green G, and blue B within the second subregion Ps as the read representative color value Cs.

[0057] In S435 (Figure 10), the processor 210 calculates the first hue Ht, the first chroma St, and the first luminance Vt from the reference representative color value Ct, and calculates the second hue Hs, the second chroma Ss, and the second luminance Vs from the read representative color value Cs. Known relationships can be used as the correspondence between representative color values, hue, chroma, and luminance (for example, the correspondence between RGB values ​​in the RGB color space and HSV values ​​in the HSV color space). The first hue Ht, the first chroma St, and the first luminance Vt are examples of representative color values ​​for the first sub-region Pt, similar to the reference representative color value Ct. The second hue Hs, the second chroma Ss, and the second luminance Vs are examples of representative color values ​​for the second sub-region Ps, similar to the read representative color value Cs.

[0058] In S440, the processor 210 determines whether the first chroma condition CSt is met, which indicates that the first chroma St is higher than the chroma threshold Sth. If the first chroma St is higher than the chroma threshold Sth (S440: Yes), in S445, the processor 210 determines whether the hue condition CH is met, which indicates that the absolute value of the difference between the first hue Ht and the second hue Hs is less than the hue difference threshold dHth. The absolute value of the difference between hues Ht and Hs (also called the hue difference dH) is the value corresponding to the smaller angle difference between the first hue Ht and the second hue Hs on the hue circle. If the candidate of interest Mi is a suitable pair of feature points Tt and Ts that represent the same part, the hue difference dH may be a small value. If the hue difference dH is less than the hue difference threshold dHth (S445: Yes), in S460, the processor 210 selects the candidate of interest Mi as a candidate to keep. If the hue difference dH is greater than or equal to the hue difference threshold dHth (S445: No), the hue may differ significantly between the first subregion Pt and the second subregion Ps. That is, the feature points Tt and Ts of candidate Mi are likely to represent different parts. In S465, processor 210 excludes candidate Mi from the list of candidate feature point pairs.

[0059] Thus, if the first saturation St is higher than the saturation threshold Sth (S440: Yes), the processor 210 excludes candidate Mi of interest that has a hue difference dH greater than or equal to the hue difference threshold dHth. Figure 9(C) shows an example of a candidate feature point pair M that remains after the processing in Figure 10. A pair of feature points Tt and Ts that represent parts with different hues may be excluded. For example, the first object OB1 and the third object OB3 have different hues. The line RLa shown in Figure 9(B) represents a pair of reference feature point Tt of the first object OB1 and read feature point Ts of the third object OB3. As shown in Figure 9(C), this pair is excluded (Figure 10: S440 (Yes), S445 (No), S465).

[0060] If the first saturation St is less than or equal to the saturation threshold Sth (S440: No), in S450, the processor 210 determines whether the second saturation condition CSs is met, which indicates that the second saturation Ss is higher than the saturation threshold Sth. If the second saturation Ss is higher than the saturation threshold Sth (S450: Yes), the saturation may differ significantly between the first subregion Pt and the second subregion Ps. That is, the feature points Tt and Ts of candidate Mi are likely to represent different parts. In S465, the processor 210 removes candidate Mi from the list of candidate feature point pairs.

[0061] If the second saturation Ss is less than or equal to the saturation threshold Sth (S450: No), in S455, the processor 210 determines whether the luminance condition CV is satisfied, which indicates that the absolute value of the difference between the first luminance Vt and the second luminance Vs is less than the luminance difference threshold dVth. If the candidate of interest Mi is an appropriate pair of feature points Tt and Ts that represent the same part, the absolute value of the difference between luminances Vt and Vs (also called the luminance difference dV) may be small. If the candidate of interest Mi is an inappropriate pair of feature points Tt and Ts that represent different parts, the luminance difference dV may be large. Thus, even when the saturation St and Ss are low, the luminances Vt and Vs can appropriately represent the color of the part indicated by the feature points Tt and Ts.

[0062] If the luminance difference dV is less than the luminance difference threshold dVth (S455: Yes), in S460, the processor 210 selects candidate Mi as a candidate to keep. If the luminance difference dV is greater than or equal to the luminance difference threshold dVth (S455: No), in S465, the processor 210 removes candidate Mi from the candidate feature point pair M.

[0063] Furthermore, the looser the conditions for retaining candidate Mi, the greater the total number of suitable candidates M that remain without being excluded. However, the total number of inappropriate candidates may also increase. The greater the total number of suitable candidates, the smaller the error in the coordinate correspondence. If the total number of inappropriate candidates is large, the error in the coordinate correspondence may increase. The parameters Sth, dHth, and dVth used in S440-S455 may be determined experimentally in advance so that the error in the coordinate correspondence is acceptable.

[0064] For example, the larger the hue difference threshold dHth(S445), the looser the conditions for retaining candidate Mi. Even if candidate Mi is a suitable pair of feature points Tt and Ts representing a highly saturated color portion, the hue difference dH may be greater than zero. A hue difference threshold dHth greater than zero allows such a hue difference dH. The hue difference threshold dHth may be set to a value greater than zero and less than the maximum possible value of the hue difference dH.

[0065] Furthermore, the larger the luminance difference threshold dVth(S455), the more lenient the conditions for retaining candidate Mi of interest. Even if candidate Mi of interest is a suitable pair of feature points Tt and Ts representing a low-saturation color region, the luminance difference dV may be greater than zero. A luminance difference threshold dVth greater than zero allows such a luminance difference dV. The luminance difference threshold dVth may be set to a value greater than zero and less than the maximum possible value of the luminance difference dV.

[0066] Furthermore, if the saturation threshold Sth (S440) is small, candidate Mi showing a low first saturation St is processed in S445. Even if the reference representative color value Ct (Figure 11(A)) is chromatic, if the first saturation St is low, the hue of the corresponding area of ​​the read image IMs representing the printed image is likely to differ from the first hue Ht. That is, if the first saturation St is low, the error in the second hue Hs, and consequently the error in the hue difference dH, may become large. If the error in the hue difference dH is large, appropriate candidate Mi may be mistakenly excluded (S445: No), and inappropriate candidate Mi may be mistakenly left in (S445: Yes). As a result, the error in the coordinate correspondence may increase. The saturation threshold Sth may be determined experimentally in advance so as to mitigate the effect of the hue difference dH error on the error in the coordinate correspondence. The saturation threshold Sth may be set to a value greater than zero and less than the maximum possible saturation.

[0067] After S460 or S465, the processor 210 proceeds to S420 and executes the loop processing S410 for the next candidate Mi of interest. When the loop processing S410 for all candidates M is completed, in S470 the processor 210 stores the data representing the candidate feature point pairs M to be retained in the storage device 215 (for example, the non-volatile storage device 230). Then the processor 210 terminates the process in Figure 10, i.e., the process in S230 in Figure 7.

[0068] In S240, the processor 210 performs a candidate feature point pair selection process using combinations of two candidate feature point pairs M. This selection process selects an appropriate candidate M from multiple remaining candidates M using combinations of two candidate M (i.e., the candidate feature point pair M is validated).

[0069] Figure 12 is a flowchart illustrating an example of the selection process for candidate feature point pairs M using a combination of two candidate feature point pairs M. In S805, the processor 210 performs the process of obtaining orientation for each of the multiple feature points Tt and Ts that form multiple candidate feature point pairs M.

[0070] Figure 13 is a flowchart illustrating an example of the orientation acquisition process. The process in Figure 13 represents the acquisition of the orientation of a single feature point. The processor 210 executes the process in Figure 13 for each of the multiple feature points Tt and Ts of multiple candidate feature point pairs M. Hereafter, the feature points targeted by the process in Figure 13 will be referred to as target feature points.

[0071] In S510, the processor 210 obtains the first type orientation of the target feature point. In this embodiment, the processor 210 obtains the orientation of the target feature point obtained in S320 or S325 in Figure 8 as the first type orientation. In S320 and S325, the processor 210 stores data representing the feature quantity and orientation in the storage device 215 (for example, the non-volatile storage device 230). The processor 210 obtains the first type orientation by referring to this data.

[0072] In S520 (Figure 13), the processor 210 calculates a second type orientation of the target feature point. The second type orientation is an orientation that represents the image features of the region of the image close to the target feature point. The second type orientation is a different type of orientation from the first type orientation. The method for calculating the second type orientation may be various methods different from the method for calculating the first type orientation. In this embodiment, S520 includes S525, S530, S535, and S540.

[0073] In S525, the processor 210 defines an off-region and an on-region for a subregion containing the target feature points. The off-region is the region not used for calculating the second type orientation, while the on-region is the region used for calculating the second type orientation.

[0074] Figures 14(A) and 14(C) illustrate examples of the calculation process for the second type of orientation. Figure 14(A) shows a subregion. The subregion may be various regions containing the target feature point PN. In this embodiment, subregion A1 is an Nq x Nq square region centered on the pixel representing the target feature point PN. Subregion A1 has Nq*Nq pixels Px. The target feature point PN is either a reference feature point Tt or a read feature point Ts. Subregion A1 is the region in the image representing the target feature point PN from the grayscale read image IMsg and the grayscale reference image IMtg.

[0075] The figure shows the off-region A1f and the on-region A1n. The off-region A1f is hatched. The off-region A1f is a square region of Mq rows and Mq columns centered on the pixel representing the target feature point PN. The side length Mq of the off-region A1f is shorter than the side length Nq of the subregion A1. The on-region A1n is the remaining region after removing the off-region A1f from the subregion A1. The on-region A1n is a ring-shaped region surrounding the target feature point PN.

[0076] In S530 (Figure 13), the processor 210 calculates the gradient direction Ou and gradient intensity Mu for each pixel Px in the on-region A1n. Figure 14(B) shows an example of the gradient direction Ou and gradient intensity Mu for each pixel Px. In the figure, each pixel Px is labeled with an arrow AR. The direction of the arrow AR indicates the gradient direction Ou, and the length of the arrow AR indicates the gradient intensity Mu. The gradient direction Ou indicates the direction of the gradient of the grayscale value, and the gradient intensity Mu indicates the strength of the gradient of the grayscale value. In this embodiment, the grayscale value of luminance is used as the grayscale value. Various methods may be used to calculate the gradient direction Ou and gradient intensity Mu. In this embodiment, the processor 210 calculates the gradient direction Ou and gradient intensity Mu using a so-called Sobel filter. Note that other filters (e.g., a Prewitt filter) may be used instead of the Sobel filter.

[0077] In S535 (Figure 13), the processor 210 generates a histogram of the gradient direction Ou in the on-region A1n. Figure 14(C) shows an example of a histogram. The histogram H represents the frequency Fw of each of several intervals HB (also called bins) of the gradient direction Ou. In this embodiment, the processor 210 calculates the frequency Fw weighted by the gradient intensity Mu. That is, one pixel Px increases the frequency Fw by the gradient intensity Mu.

[0078] In S540 (Figure 13), the processor 210 determines the second type orientation as the direction associated with the highest frequency. Figure 14(C) shows the highest frequency Fwx and the interval HBx associated with the highest frequency Fwx. The processor 210 adopts the direction Oux, which is included in the interval HBx, as the second type orientation. Direction Oux may be, for example, an intermediate direction within the interval HBx.

[0079] The values ​​Nq and MQ, which determine the size of the on-region A1n, may be various values ​​that can calculate a second-kind orientation representing the image features of the region close to the target feature point PN. The values ​​Nq and MQ may be determined experimentally in advance so that an appropriate second-kind orientation can be calculated. For example, Nq may be 9, and MQ may be 5. The size of the interval HB of the histogram H (i.e., the total number of intervals HB) may be determined experimentally in advance so that an appropriate second-kind orientation can be calculated.

[0080] As described above, the process in S520 is completed, and the process in Figure 13 is finished. In S805 (Figure 12), the processor 210 performs the process in Figure 13 for each of the multiple feature points Tt and Ts that form the multiple candidate feature point pairs M. As a result, the second type orientation of each feature point is calculated.

[0081] After S805, the processor 210 performs a loop process S810 (including S820-S885) between the start L81s and the end L81e for each of the multiple candidate feature point pairs M. In S820, the processor 210 selects the unprocessed candidate M as the first candidate MA.

[0082] In S825, the processor 210 selects a feature point group N within the range between a first radius r1 and a second radius r2, centered on the feature point Tt of the first candidate MA. Figure 15 shows an example of the feature point group N. The figure shows a reference image IMt and multiple reference feature points Tt. One feature point Tt is selected as the first candidate MA. The first circle C1 is a circle with a first radius r1 centered on the feature point Tt of the first candidate MA. The second circle C2 is a circle with a second radius r2 centered on the feature point Tt of the first candidate MA. The selection range SR is the region between the first circle C1 and the second circle C2 (including the parts on circles C1 and C2). The processor 210 selects multiple feature points Tt included in the selection range SR as the feature point group N.

[0083] After S825 (Figure 12), the processor 210 executes a loop process S830 (including S840-S885) between the start L82s and end L82e for each of the multiple candidate M in the feature point cloud N. In S840, the processor 210 selects an unprocessed candidate M associated with a feature point Tt included in the feature point cloud N as the second candidate MB. The reason for selecting the second candidate MB from the feature point cloud N (i.e., the selection range SR (Figure 15)) will be explained later.

[0084] Hereafter, we will assume that the second candidate MB is the u-th candidate M among the candidates M obtained from the feature point cloud N (also expressed as candidate M[u]). The index u is selected, for example, from the range of zero or greater and less than the total number of candidate M obtained from the feature point cloud N, NP. The selected combination of candidate MA and MB is also called the candidate combination MU.

[0085] In S850, the processor 210 selects candidate combination MU as the target combination MT for determining the shape condition CWz, which will be described later, when certain conditions are met. In this embodiment, the processor 210 selects candidate combination MU as the target combination MT when candidate MA and MB are different candidate Ms. Although not shown in the diagram, if the second candidate MB is the same candidate M as the first candidate MA, the processor 210 does not select candidate combination MU as the target combination MT and terminates loop processing S830. Then, the processor 210 executes loop processing S830 for the next second candidate MB. The conditions for selecting candidate combination MU as the target combination MT may be various conditions.

[0086] In the S860, the processor 210 uses the target combination MT to determine whether the shape condition CWz is met.

[0087] Figure 16 is a flowchart illustrating an example of the process for determining the shape condition CWz. The processor 210 uses the line segments connecting the two reference feature points Tt of the two candidate MAs and MBs, the line segments connecting the two read feature points Ts, and the orientation of each feature point to determine whether the shape condition CWz is met.

[0088] Figures 17(A)-17(D) show examples of line segments formed by candidate MAs and MBs and the orientation of each feature point. Each candidate MA and MB represents a pair of reference feature point Tt and read feature point Ts. Figure 17(A) shows the line segment Sabt connecting the reference feature points Tat and Tbt of the candidate MA and MB (referred to as the reference line segment Sabt). Orientations Oat1 and Oat2 are the first and second type orientations associated with the reference feature point Tat. Orientations Obt1 and Obt2 are the first and second type orientations associated with the reference feature point Tbt. Figures 17(B)-17(D) show the line segment Sabs connecting the read feature points Tas and Tbs of the candidate MA and MB (referred to as the read line segment Sabs). Orientations Oas1 and Oas2 are Type 1 and Type 2 orientations corresponding to the read feature point Tas. Orientations Obs1 and Obs2 are Type 1 and Type 2 orientations corresponding to the read feature point Tbs. The differences between Figure 17(B) and Figure 17(D) will be described later.

[0089] In Figure 17(A), angles ZAt1 and ZAt2 are the angles between the orientations Oat1 and Oat2 of feature point Tat and the line segment Sabt. For angles ZAt1 and ZAt2, the smaller of the angles between the line segment extending from feature point Tat to orientations Oat1 and Oat2 and the line segment Sabt extending from feature point Tat is adopted. Angles ZBt1 and ZBt2 are the angles between the orientations Obt1 and Obt2 of feature point Tbt and the line segment Sabt. Angles ZBt1 and ZBt2 are calculated in the same way as angles ZAt1 and ZAt2. That is, for angles ZBt1 and ZBt2, the smaller of the angles between the line segment extending from feature point Tbt to orientations Obt1 and Obt2 and the line segment Sabt extending from feature point Tbt is adopted.

[0090] In Figures 17(B)-17(D), angles ZAs1 and ZAs2 are the angles between the orientations Oas1 and Oas2 of feature point Tas and the line segment Sabs. Angles ZBs1 and ZBs2 are the angles between the orientations OBs1 and OBs2 of feature point Tbs and the line segment Sabs. The method for calculating angles ZAs1, ZAs2, ZBs1, and ZBs2 is the same as the method for calculating angles ZAt1, ZAt2, ZBt1, and ZBt2 in Figure 17(A).

[0091] The angles ZAt1 and ZAt2 (Figure 17) may be angles in a predetermined rotational direction (clockwise or counterclockwise) from the line segment Sabt to the line segments extending to orientations Oat1 and Oat2, with the feature point Tat as the center. The same applies to the other angles ZBt1, ZBt2, ZAs1, ZAs2, ZBs1, and ZBs2.

[0092] In S910 (Figure 16), the processor 210 calculates the following two angles ZAt1 and ZAs1 which are associated with the first candidate MA. ZAt1=AG(Oat1, Sabt) ZAs1 = AG(Oas1, Sabs) AG is a function that derives angles.

[0093] In S915, the processor 210 determines whether condition CA1 is met, which indicates that the angles ZAt1 and ZAs1 of the first type orientations Oat1 and Oas1 of the feature points Tat and Tas of the first candidate MA are close to each other. In this embodiment, condition CA1 is that the absolute value of the difference between angles ZAt1 and ZAs1 is less than the angle difference threshold dZth.

[0094] If the feature points Tat and Tas of the first candidate MA indicate the same part of the same object, and the feature points Tbt and Tbs of the second candidate MB also indicate the same part of the same object, then angle ZAt1 is approximately the same as angle ZAs1. When condition CA1 is satisfied, there is a high probability that each of the candidate MA and MB will show an appropriate pair of feature points Tt and Ts. However, even if each of the candidate MA and MB shows an appropriate pair of feature points Tt and Ts, angle ZAt1 may deviate from angle ZAs1. Condition CA1 is configured to allow for such deviations. In this embodiment, the angle difference threshold dZth is determined experimentally in advance to allow for deviations between angles ZAt1 and ZAs1. The angle difference threshold dZth may be various values ​​greater than zero.

[0095] In this embodiment, the first type of orientation (e.g., orientations Oat1, Oas1) is a gradient orientation calculated according to the A-KAZE technique. When an object rotates within images IMtg and IMs, orientations Oat1 and Oas1 and line segments Sabt and Sabs rotate with the object. The angles ZAt1 and ZAs1 are invariant with respect to the scale and rotation of the object within images IMt and IMs. Therefore, the result of the condition CA1 is invariant with respect to scale and rotation.

[0096] If condition CA1 is met (S915: Yes), in S920, the processor 210 calculates the following two angles ZAt2 and ZAs2 that are associated with the first candidate MA. ZAt2=AG(Oat2, Sabt) ZAs2 = AG(Oas2, Sabs)

[0097] In S925, the processor 210 determines whether condition CA2 is met, which indicates that the angles ZAt2 and ZAs2 of the second type orientations Oat2 and Oas2 of the feature points Tat and Tas of the first candidate MA are close to each other. In this embodiment, condition CA2 is that the absolute value of the difference between angles ZAt2 and ZAs2 is less than the angle difference threshold dZth. The difference between angles ZAt2 and ZAs2 is small, similar to the difference between angles ZAt1 and ZAs1 (S915), when each of the candidate MA and MB shows an appropriate pair of feature points Tt and Ts. Therefore, if condition CA2 is met, there is a high probability that each of the candidate MA and MB shows an appropriate pair of feature points Tt and Ts.

[0098] In this embodiment, the second type of orientation (for example, orientations Oat2 and Oas2) represents the image features of the region of the image close to the target feature point PN, as explained in S520 (Figure 13) and Figures 14(A)-14(C). When an object rotates within images IMtg and IMs, orientations Oat2 and Oas2 and line segments Sabt and Sabs rotate with the object. The angles ZAt2 and ZAs2 are invariant with respect to the scale and rotation of the object within images IMt and IMs. Therefore, the judgment result of condition CA2 is invariant with respect to scale and rotation.

[0099] If condition CA2 is met (S925: Yes), in S930, the processor 210 calculates the first type orientation Obt1 of the feature points Tbt and Tbs of the second candidate MB, and the angles ZBt1 and ZBs1 of Obs1. In S935, the processor 210 determines whether condition CB1 is met, which indicates that angles ZBt1 and ZBs1 are close to each other. S930 and S935 are the same as the processing obtained in S910 and S915 by substituting angles ZAt1 and ZAs1 with angles ZBt1 and ZBs1. If condition CB1 is met, it is highly likely that each of the candidate MA and MB represents an appropriate pair of feature points Tt and Ts. Furthermore, the result of determining condition CB1 is invariant with respect to scale and rotation.

[0100] If condition CB1 is met (S935: Yes), in S940, the processor 210 calculates the angles ZBt2 and ZBs2 of the second type orientation Obt2 and Obs2 of the feature points Tbt and Tbs of the second candidate MB. In S945, the processor 210 determines whether condition CB2 is met, indicating that angles ZBt2 and ZBs2 are close to each other. S940 and S945 are the same as the processing obtained by replacing angles ZAt2 and ZAs2 in the processing of S920 and S925 with angles ZBt2 and ZBs2. If condition CB2 is met, it is highly likely that each of the candidate MA and MB represents an appropriate pair of feature points Tt and Ts. Furthermore, the result of the determination of condition CB2 is invariant with respect to scale and rotation.

[0101] If condition CB2 is met (S945: Yes), in S950, the processor 210 determines the reference luminance relationship BRt and the read luminance relationship BRs. The reference luminance relationship BRt (Figure 17(A)) represents the magnitude relationship between the luminance value Bat of the pixel representing the feature point Tat of the first candidate MA and the luminance value Bbt of the pixel representing the feature point Tbt of the second candidate MB. The read luminance relationship BRs (Figures 17(B)-17(D)) represents the magnitude relationship between the luminance value Bas of the pixel representing the feature point Tas of the first candidate MA and the luminance value Bbs of the pixel representing the feature point Tbs of the second candidate MB.

[0102] As shown in S950 of Figure 16, the luminance relationships BRt and BRs are set to one of three relationships BR1, BR2, and BR3. The first relationship BR1 indicates that the difference obtained by subtracting the luminance value Bb associated with the second candidate MB from the luminance value Ba associated with the first candidate MA is greater than the threshold Bth (where Bth > 0). That is, Ba > Bb. The second relationship BR2 indicates that the absolute value of the above difference is less than or equal to the threshold Bth. That is, the difference between the luminance values ​​Ba and Bb is small. The third relationship BR3 indicates that the above difference is less than -Bth. That is, Bb > Ba.

[0103] If the reference luminance relationship BRt is the same as the read luminance relationship BRs, then candidate MA and MB are likely to represent appropriate pairs of feature points Tt and Ts. If the reference luminance relationship BRt is different from the read luminance relationship BRs, then one or both of candidate MA and MB are likely to represent inappropriate pairs.

[0104] In S955 (Figure 16), the processor 210 determines whether condition CC, which indicates that the reference luminance relation BRt is the same as the read luminance relation BRs, is satisfied. The reference luminance relation BRt and the read luminance relation BRs are independent of the scale and rotation of objects in the images IMt and IMs. That is, condition CC is invariant with respect to the scale and rotation of objects in the images IMt and IMs.

[0105] If condition CC is met (S955: Yes), in S960, the processor 210 determines that the shape condition CWz is met and terminates the process shown in Figure 16, i.e., S860 in Figure 12.

[0106] If one or more of the conditions CA1, CA2, CB1, CB2, and CC are not met, it is estimated that one or more candidate Ms among candidate MA and MB are an inappropriate pair. If one or more of S915:No, S925:No, S935:No, S945:No, and S955:No are true, in S965 the processor 210 determines that the shape condition CWz is not met and terminates the process in Figure 16, i.e., S860 in Figure 12.

[0107] Figures 17(B)-17(D) show examples of the results of the processing shown in Figure 16. Figure 17(B) shows the case where candidate MA and MB are appropriate, and the reference image IMt and the read image IMs have the same scale but different rotation angles. In this case, the rotation-invariant conditions CA1, CA2, CB1, CB2, and CC (Figure 16) are satisfied, and therefore the shape condition CWz is satisfied.

[0108] Figure 17(C) shows the case where candidate MA and MB are appropriate, and the rotation angle is the same but the scale is different between the reference image IMt and the read image IMs. In this case, the scale invariance conditions CA1, CA2, CB1, CB2, and CC (Figure 16) are satisfied, and therefore the shape condition CWz is satisfied.

[0109] Although not shown in the diagram, if candidate MA and MB are appropriate, and both the scale and rotation angle differ between the reference image IMt and the read image IMs, then conditions CA1, CA2, CB1, CB2, and CC (Figure 16) are satisfied, and therefore the shape condition CWz is also satisfied.

[0110] Figure 17(D) shows a case where the first candidate MA is appropriate, but the second candidate MB is inappropriate. Specifically, the feature point Tbs is different from the appropriate feature point Tsbr. If the combination of candidate MA and MB includes an inappropriate candidate M, the result of the judgment of one or more of the conditions CA1, CA2, CB1, CB2, CC (Figure 16) may be No (i.e., the shape condition CWz is likely not met).

[0111] Furthermore, the looser the conditions CA1, CA2, CB1, CB2, CC (Figure 16), the greater the total number of suitable candidate combinations of MA and MB that satisfy the shape condition CWz. However, the greater the total number of inappropriate candidate combinations of MA and MB that satisfy the shape condition CWz. The greater the total number of suitable candidates, the smaller the error in the coordinate correspondence. If the total number of inappropriate candidates is large, the error in the coordinate correspondence may increase. The orientation conditions CA1, CA2, CB1, CB2 (in this embodiment, the angular difference threshold dZth) and the luminance relationship condition CC (in this embodiment, the threshold Bth) may be experimentally determined in advance so that the error in the coordinate correspondence is acceptable. The angular difference threshold dZth may be various values ​​greater than zero. For example, the angular difference threshold dZth may be set to a value greater than zero and less than or equal to 10 degrees. The threshold Bth may be various values ​​greater than zero. The threshold Bth is preferably determined such that appropriate candidate MA and MB combinations satisfy condition CC, while candidate MA and MB combinations containing an inappropriate candidate M do not satisfy condition CC.

[0112] In S840 in Figure 12, the second candidate MB is selected from the candidates M that correspond to the feature point Tt included in the feature point cloud N (i.e., the selected range SR (Figure 15)). The reason for this is that, as explained below, if the length of the line segments Sabt and Sabs (Figures 17(A)-17(D)) is significantly long or significantly short, the error in judging the shape condition CWz may increase.

[0113] For example, suppose the second candidate MB is a pair of feature point Tbt and an incorrect read feature point Tbs in the neighborhood of a suitable read feature point Ts (e.g., Figure 17(D)). If the length of the reference line segment Sabt is significantly long, the length of the read line segment Sabs will also be significantly long. In this case, the direction in which the read line segment Sabs extends is approximately the same as the direction in which the line segment connecting the two suitable read feature points Ts extends (not shown). Therefore, even if the candidate MB indicates an inappropriate read feature point Ts, the difference in angles calculated using the read line segment Sabs (e.g., the difference between angles ZAt1 and ZAs1) may be incorrectly judged to be small.

[0114] Furthermore, the positions of detected feature points may contain errors. If the length of the reference line segment Sabt is significantly short, the direction in which the reference line segment Sabt extends can change considerably due to errors in the position of the reference feature point Tt. The same applies to the direction in which the read line segment Sabs extends. Therefore, even if candidate MA and MB represent appropriate pairs of feature points Tt and Ts, respectively, a large difference in angles calculated using line segments Sabt and Sabs (for example, the difference between angles ZAt1 and ZAs1) may be mistakenly judged.

[0115] The selection range SR (here, radii r1, r2 (Figure 15)) is determined experimentally in advance by the combination of candidate MA and MB so that a shape suitable for judgment (here, a line segment of appropriate length) is formed. For example, the first radius r1 may be a value of 1% or more and 20% or less of the size of the reference image IMt (e.g., the length of the first direction Dx or the second direction Dy). The second radius r2 may be a value of 30% or more and 70% or less of the size of the reference image IMt.

[0116] After the processing shown in Figure 16, that is, after S860 (Figure 12), at S865, the processor 210 branches the process according to the decision result of S860. If the shape condition CWz is met (S865: Yes), at S885, the processor 210 selects candidate MA and MB of the target combination MT as candidates to be kept. After S885, the processor 210 terminates the loop processing S830 for the current combination of candidate MA and MB.

[0117] If the shape condition CWz is not met (S865: No), the processor 210 skips S885 and terminates loop processing S830 for the current combination of candidate MA and MB.

[0118] Subsequently, the processor 210 executes a loop process S830 using each of the multiple second candidate MBs, and a loop process S810 using each of the multiple first candidate MAs. After the completion of the repeated loop processes S810 and S830, in S890, the processor 210 stores data representing the candidate feature point pairs M that should be retained into the storage device 215 (for example, the non-volatile storage device 230). Then, the processor 210 terminates the process shown in Figure 12, that is, the process shown in S240 of Figure 7.

[0119] Figure 9(D) shows an example of a candidate feature point pair remaining after processing in Figure 12. As shown in Figures 9(C) and 9(D), inappropriate feature point pairs Tt and Ts can be excluded. For example, line RLb shown in Figure 9(C) represents a pair of the upper right reference feature point Tt and the lower right read feature point Ts of the fourth object OB4. Such a pair may be excluded from the list of candidate feature point pairs because, when combined with other pairs, the shape condition CWz is not met (Figure 12:S865:No).

[0120] Even if a combination of candidate MA and MB containing a suitable candidate M does not satisfy the shape condition CWz, that suitable candidate M may satisfy the shape condition CWz when combined with other suitable candidate Ms. Thus, in the process shown in Figure 12, even if the judgment result of S865 is No, candidate MA and MB are not immediately excluded. Through the repeated loop processes S810 and S830, the processor 210 selects multiple candidate Ms that have been selected by S885 one or more times as candidate Ms to be kept. Through the repeated loop processes S810 and S830, the processor 210 excludes candidate Ms that have not been selected even once by S885.

[0121] After the processing shown in Figure 12, that is, after S240 in Figure 7, in S250, the processor 210 determines the coordinate correspondence using multiple feature point pairs MP. These multiple feature point pairs MP are the multiple candidate feature point pairs M remaining in S250. Hereinafter, the feature point pairs MP used to determine the coordinate correspondence will also be referred to as target feature point pairs MP, or simply target pairs MP. In this embodiment, the matrix Mtx (Figure 5) is determined as the coordinate correspondence. There may be various methods for determining the matrix Mtx. For example, the processor 210 may determine the matrix Mtx according to a method called RANDOM SAmple Consensus (RANSAC). In this embodiment, the total number of inappropriate candidate feature point pairs M can be reduced by the filtering process performed before S250 in Figure 7 (e.g., S230, S240). Therefore, the processor 210 may calculate the matrix Mtx using all target pairs MP. The calculation method can be varied (for example, the least squares method). Furthermore, functions from OpenCV (Open Source Computer Vision Library), for example, may be used to determine the matrix Mtx.

[0122] After S250, the processor 210 completes the alignment process shown in Figure 7, i.e., S120 in Figure 4.

[0123] As described above, in this embodiment, the processor 210 performs the following processing according to the program 231. In S320 (Figure 8), the processor 210 calculates a first-kind orientation for each of the multiple reference feature points Tt in the grayscale reference image IMtg (Figure 9(A)) which contains images of objects OB1-OB4 (for example, first-kind orientations Oat1, Obt1 (Figure 17(A))). The grayscale reference image IMtg is an image obtained by grayscale conversion of the reference image IMt (Figure 3(A)) which contains images of objects OB1-OB4 (S305 (Figure 8)). Therefore, the reference feature points Tt in the grayscale reference image IMtg are also feature points in the reference image IMt. The reference image IMt is an example of the first image, and the reference feature points Tt are examples of first-kind feature points in the first image. In S805 (Figure 12), the processor 210 performs the processing shown in Figure 13 for each of the multiple feature points Tt, Ts. In S520 (Figure 13), the processor 210 calculates a second type orientation for each of the multiple reference feature points Tt (for example, second type orientations Oat2, Obt2 (Figure 17(A))). The second type orientation is a different type of orientation from the first type orientation. In S325 (Figure 8), the processor 210 calculates a first type orientation for each of the multiple read feature points Ts in the grayscale read image IMsg (Figure 9(A)) containing images of objects OB1-OB4 (for example, first type orientations Oas1, Obs1 (Figure 17(B)-Figure 17(D))). The grayscale read image IMsg is an image obtained by grayscale conversion of the read image IMs (Figure 3(B)) containing images of objects OB1-OB4 (S305 (Figure 8)). Therefore, the read feature points Ts in the grayscale read image IMsg are also feature points in the read image IMs. The read image IMs is an example of the second image, and the read feature points Ts are examples of type 2 feature points in the second image. The objects OB1-OB4 represented by the read image IMs have the same external shape as the objects OB1-OB4 represented by the reference image IMt.In other words, the images of objects OB1-OB4 contained in the read images IMs include portions that have a common external shape with at least a portion of the images of objects OB1-OB4 contained in the reference image IMt. In S520 (Figure 13), the processor 210 calculates a second orientation for each of the multiple read feature points Ts (for example, second type orientations Oas2, Obs2 (Figures 17(B)-17(D))).

[0124] In S240 (Figure 7), the processor 210 selects multiple target pairs MP. Each of the multiple target pairs MP is a pair of a reference feature point Tt and a read feature point Ts, and is the target of processing in S250. The processing in S240 includes S860, S865, and S885 in Figure 12. In S860, S865, and S885, the processor 210 uses the target combination MT, which is a combination of two candidate feature point pairs M, to determine whether or not to select the target combination MT as a candidate to keep. Here, as shown in Figures 17(A)-17(D), the processor 210 uses the first type orientations Oat1, Obt1, Oas1, Obs1 and the second type orientations Oat2, Obt2, Oas2, Obs2 of the two reference feature points Tat, Tbt and the two read feature points Tas, Tbs that form the target combination MT.

[0125] In S250 (Figure 7), the processor 210 uses multiple target pairs MP to determine the correspondence between the coordinates COt (Figure 5) on the reference image IMt and the coordinates COs on the read image IMs.

[0126] With this configuration, the processor 210 can properly align the reference image IMt and the read image IMs. For example, the possibility of selecting an inappropriate target pair MP is reduced compared to when only one of the first or second type orientations is used to select the target pair MP. Furthermore, the computational load for decision-making is reduced compared to when a combination of three or more candidate feature point pairs M is used to determine whether or not to select that combination as a candidate to keep.

[0127] In this embodiment, in S220 and S230 (Figure 7), the processor 210 forms multiple candidate feature point pairs M. The processing in S220 includes S350, S355, and S360 in Figure 8. In S350, S355, and S360, the processor 210 forms multiple candidate feature point pairs M using feature quantities Ft and Fs. As explained in S320, S325, and Figure 9(A), the feature quantities Ft and Fs of feature points Tta and Tsa represent the features of the local regions Wta and Wsa that contain the feature points Tta and Tsa. Similarly, the feature quantities Ft and Fs of other feature points Tt and Ts also represent the features of the local regions that contain the feature points Tt and Ts. Feature quantities Ft and Fs are examples of information that represent the features of the local regions that contain the feature points. The processing in S240, following S230 (Figure 7), includes S805 in Figure 12. In S805, the processor 210 executes the process shown in Figure 13. In S520, the processor 210 calculates a Type II orientation for each of the multiple reference feature points Tt that form multiple candidate feature point pairs M. The processor 210 also calculates a Type II orientation for each of the multiple read feature points Ts that form multiple candidate feature point pairs M. In this way, the process of calculating a Type II orientation is executed for each of the multiple feature points Tt and Ts that form multiple candidate feature point pairs M. Therefore, the possibility of a feature point pair containing an inappropriate feature point that cannot form a candidate feature point pair M being selected as the target pair MP is reduced.

[0128] In this embodiment, in S320 and S325 (Figure 8), the processor 210 calculates a first-type orientation associated with feature points Tta and Tsa (Figure 9(A)) using images of subregions Ata and Asa placed for the feature points Tta and Tsa. Subregions Ata and Asa are examples of first-type subregions used to calculate the first-type orientation. The first-type orientations for other feature points Tt and Ts are calculated similarly. In S520 (Figure 13), the processor 210 calculates a second-type orientation associated with the target feature point PN (Figure 14(A)) using an image of on-region A1n placed for the target feature point PN (S525-S540). The method for calculating the second-type orientation is common to both the reference feature point Tt and the read feature point Ts. On-region A1n is an example of a second-type subregion used to calculate the second-type orientation.

[0129] Here, subregions Ata and Asa (Figure 9(A)) contain feature points Tta and Tsa. Off-region A1f, which contains the target feature point PN, is excluded from on-region A1n (Figure 14(A)). In this way, a specific portion corresponding to a part of subregions Ata and Asa (here, a part containing the corresponding feature points Tta and Tsa) is excluded from on-region A1n. The specific portion is the part of off-region A1f that is included in subregions Ata and Asa. For example, the specific portion could be the entirety of off-region A1f. In this way, since a specific portion corresponding to a part of the first subregion is excluded from the second subregion, the processor 210 can calculate first and second orientations that are associated with the same feature points but are of different types. For example, if one of the Type 1 or Type 2 orientations allows for the selection of an inappropriate target pair MP, the other of the Type 1 or Type 2 orientations can reduce the likelihood of that inappropriate target pair MP being selected.

[0130] Furthermore, in this embodiment, the subregions Ata and Asa (Figure 9(A)), which are examples of the first type subregion, include the above-mentioned portion containing the corresponding feature points Tta and Tsa. The on-region A1n (Figure 14(A)), which is an example of the second type subregion, excludes the above-mentioned specific portion (for example, the entire off-region A1f) containing the corresponding target feature point PN. Therefore, the processor 210 can calculate first and second type orientations that are different from each other but are associated with the same feature points.

[0131] Furthermore, in this embodiment, in the process of calculating the first type orientation (Figure 8: S320, S325), the processor 210 calculates the first type orientation using the orientation calculation algorithm by A-KAZE. The orientation calculation algorithm by A-KAZE is the algorithm described in the aforementioned paper by A-KAZE. The processor 210 can calculate the first type orientation that represents the features of the part of the image that represents the feature points. By using such a first type orientation, the processor 210 can perform an appropriate selection of the target pair MP.

[0132] Furthermore, in this embodiment, as explained in Figure 3(A), the reference image IMt represents the target image IMp, and the target image IMp is an image represented by print image data. Also, as explained in Figure 3(B), the read image IMs represents the portion of the read T-shirt 700 that includes the printed image IMpp. The printed image IMpp is an image obtained by printing the target image IMp (Figure 3(A)). That is, the read image IMs represents an image generated by optically reading the image IMpp, which is printed using print image data. The processor 210 can determine the coordinate correspondence between such a reference image IMt and read image IMs. The determined coordinate correspondence can be used for various processes such as inspection.

[0133] B. Second example: Figure 18 is a flowchart representing a second embodiment of the orientation acquisition process. The process in Figure 18 is executed at S805 (Figure 12) instead of the process in Figure 13. Program 231 (Figure 1) is configured to execute the process in Figure 18 instead of the process in Figure 13. The process in Figure 18 calculates the first and second type orientations of the reference feature point Tt and the read feature point Ts of one candidate feature point pair M. The processor 210 executes the process in Figure 18 for each of the multiple candidate feature point pairs M. Hereinafter, the feature points targeted by the process in Figure 18 will be referred to as target feature points.

[0134] In S610, the processor 210 calculates the variance Sg for each color component using a calculation region centered on the pixel containing the target feature point in the reference image IMt. Figures 19(A) and 19(C) illustrate examples of the orientation calculation process. Figure 19(A) shows an example of a calculation region. The calculation region may be various regions including the area near the target feature point. In this embodiment, calculation region A2 is a square region of Nh rows and Nh columns centered on the pixel representing the target feature point PN2. The target feature point PN2 is the reference feature point Tt of the candidate feature point pair M to be processed. The processor 210 calculates the variance Sg for each color component using the respective gradation values ​​of multiple color components of multiple pixels Px in calculation region A2. In this embodiment, the variance Sg for each of RGB is calculated.

[0135] In S615 (Figure 18), the processor 210 determines the two color components CHa and CHb, which have the largest dispersion Sg. The first color component CHa is the color component that exhibits the largest dispersion Sg, and the second color component CHb is the color component that exhibits the second largest dispersion Sg.

[0136] In S630, the processor 210 calculates the gradient direction Ou2 and gradient intensity Mu2 for each of the multiple pixels Px in the calculation region A2 for each feature point Tt, Ts and each color component CHa, CHb. In this embodiment, the method for calculating the gradient direction Ou2 and gradient intensity Mu2 is the same as the method for calculating the gradient direction Ou and gradient intensity Mu described in S530 (Figure 13). Figure 19(B) is a diagram showing an example of the gradient direction Ou2 and gradient intensity Mu2 for each pixel Px associated with one color component of one feature point. In the figure, each pixel Px is marked with an arrow AR2. The direction of the arrow AR2 indicates the gradient direction Ou2, and the length of the arrow AR2 indicates the gradient intensity Mu2. The gradient direction Ou2 indicates the direction of the gradient of the gradation value, and the gradient intensity Mu2 indicates the strength of the gradient of the gradation value. Furthermore, the reference image IMt is used for processing the reference feature point Tt, and the read image IMs is used for processing the read feature point Ts. The gradation value of the first color component CHa is used for processing the first color component CHa, and the gradation value of the second color component CHb is used for processing the second color component CHb.

[0137] In S635 (Figure 18), the processor 210 generates a histogram in the gradient direction Ou2 for each feature point Tt, Ts and each color component CHa, CHb. Figure 19(C) shows an example of a histogram associated with one color component of one feature point. The histogram H2, like the histogram H in Figure 14(C), represents the frequencies Fw2 of each of the multiple intervals HB2 in the gradient direction Ou2. In this embodiment, the processor 210 calculates the frequencies Fw2 weighted by the gradient intensity Mu2. That is, one pixel Px increases the frequency Fw2 by the gradient intensity Mu2.

[0138] In S640 (Figure 18), the processor 210 determines the first and second type orientations for each feature point Tt and Ts in the direction of the maximum frequency in the histogram of the color components CHa and CHb. Specifically, it is as follows: The processor 210 determines the first type orientation of the reference feature point Tt in the direction corresponding to the maximum frequency in the histogram of the first color component CHa of the reference image IMt. Figure 19(C) shows the maximum frequency Fw2x and the interval HB2x corresponding to the maximum frequency Fw2x. The processor 210 adopts the direction Ou2x included in the interval HB2x as the orientation. The direction Ou2x may be, for example, the middle direction of the interval HB2x.

[0139] The second type orientation of the reference feature point Tt, and the first and second type orientations of the read feature point Ts are determined similarly. The processor 210 determines the second type orientation of the reference feature point Tt using the histogram of the second color component CHb of the reference image IMt. The processor 210 determines the first type orientation of the read feature point Ts using the histogram of the first color component CHa of the read image IMs. The processor 210 determines the second type orientation of the read feature point Ts using the histogram of the second color component CHb of the read image IMs.

[0140] After S640, the processor 210 completes the process shown in Figure 18, i.e., the process of S805 (Figure 12). The other parts of the alignment process (Figure 7) are the same as the corresponding parts of the alignment process in the first embodiment.

[0141] As described above, in this embodiment, at S640 (Figure 18), the processor 210 calculates the first and second type orientations for feature points Tt and Ts, respectively. The processor 210 calculates the first type orientation associated with the reference feature point Tt using the first color component CHa of the reference image IMt containing the reference feature point Tt. The processor 210 calculates the second type orientation associated with the reference feature point Tt using the second color component CHb of the reference image IMt containing the reference feature point Tt. The second color component CHb is a different color component from the first color component CHa. The same applies to the read feature point Ts. The processor 210 calculates the first type orientation associated with the read feature point Ts using the first color component CHa of the read image IMs containing the read feature point Ts. The processor 210 calculates a second type orientation associated with the read feature point Ts using the second color component CHb of the read image IMs containing the read feature point Ts. With this configuration, the processor 210 can calculate first and second type orientations that are associated with the same feature point but are of different types. For example, if one of the first and second type orientations allows for the selection of an inappropriate target pair MP, the other of the first and second type orientations can reduce the possibility of that inappropriate target pair MP being selected.

[0142] In this embodiment, the reference image IMt is represented by three color components of RGB. The processor 210 performs the following processing. In S610 (Figure 18), the processor 210 calculates the variance Sg of each of the three color components (here, RGB) in the calculation region A2 that includes the target feature point PN2 (Figure 19(A)) in the reference image IMt. The target feature point PN2 is an example of the reference feature point Tt of interest. The calculation region A2 is an example of a subregion that includes the target feature point PN2. The variance Sg is an example of an index value that indicates the degree of color variation. In S615, the processor 210 determines the first color component CHa for the target feature point PN2 to be the color component that corresponds to the largest variance Sg. The largest variance Sg indicates the largest degree of variation. Also in S615, the processor 210 determines the second color component CHb for the target feature point PN2 to be the color component that corresponds to the second largest variance Sg. The second largest variance Sg indicates the second largest degree of variation. Color components with a large variance Sg can better represent the image features compared to color components with a small variance Sg. By using the first color component CHa, which exhibits the largest variance Sg, and the second color component CHb, which exhibits the second largest variance Sg, the processor 210 can calculate appropriate first and second orientations for selecting appropriate candidate feature point pairs M(S240 (Figure 7)).

[0143] The only difference between the alignment process of this embodiment and the alignment process of the first embodiment is the method for calculating the first and second type orientations. Therefore, this embodiment can provide the same various advantages as the first embodiment.

[0144] C. Third embodiment: Figure 20 is a flowchart representing a third embodiment of the orientation acquisition process. The process in Figure 20 is executed at S805 (Figure 12) instead of the processes in Figures 13 and 18. Program 231 (Figure 1) is configured to execute the process in Figure 20 instead of the processes in Figures 13 and 18. The process in Figure 20 represents the acquisition of the first and second type orientations for a single feature point. Processor 210 executes the process in Figure 20 for each of the multiple feature points Tt and Ts of multiple candidate feature point pairs M. Hereinafter, the feature points targeted by the process in Figure 20 will be referred to as target feature points.

[0145] In S710, the processor 210 calculates the gradient direction and gradient intensity of each pixel using a first calculation region containing the target feature point. Figures 21(A) and 21(B) are diagrams illustrating examples of the orientation calculation process. Figure 21(A) shows an example of the first calculation region. The first calculation region may be various regions containing the target feature point. In this embodiment, the first calculation region A3a is a square region with N3a rows and N3a columns centered on the pixel representing the target feature point PN3. The target feature point PN3 is a reference feature point Tt or a read feature point Ts. The first calculation region A3a is a region on the image (here, a grayscale read image IMsg or a grayscale read image IMsg) containing the target feature point PN3.

[0146] The processor 210 calculates the gradient direction Ou3 and gradient intensity Mu3 for each pixel Px in the first calculation area A3a. In this embodiment, the method for calculating the gradient direction Ou3 and gradient intensity Mu3 is the same as the method for calculating the gradient direction Ou and gradient intensity Mu described in S530 (Figure 13). Figure 21(B) shows an example of the gradient direction Ou3 and gradient intensity Mu3 for each pixel Px. In the figure, each pixel Px is marked with an arrow AR3. The direction of the arrow AR3 indicates the gradient direction Ou3, and the length of the arrow AR3 indicates the gradient intensity Mu3. In this embodiment, the luminance gradation value is used to calculate the gradient direction Ou3 and gradient intensity Mu3.

[0147] In S720 (Figure 20), the processor 210 generates a histogram of the gradient direction Ou3 in the first calculation region A3a. In this embodiment, the method for generating the histogram is the same as the method described in S535 (Figure 13) and Figure 14(C).

[0148] In S730, the processor 210 determines the first type of orientation in the direction corresponding to the highest frequency in the histogram. In this embodiment, the method for determining the orientation from the histogram is the same as the method described in S540 (Figure 13) and Figure 14(C).

[0149] In S740, S750, and S760, the processor 210 determines a second type of orientation using a second calculation region containing the target feature point. The processing in S740, S750, and S760 is the same as the processing in S710, S720, and S730, except that the second calculation region is used instead of the first calculation region A3a. Figure 21(A) shows an example of the second calculation region. The second calculation region may be various regions containing the target feature point. In this embodiment, the second calculation region A3b is a square region with N3b rows and N3b columns centered on the pixel representing the target feature point PN3. Here, the side length N3b of the second calculation region A3b is longer than the side length N3a of the first calculation region A3a. The second calculation region A3b includes the first calculation region A3a and the surrounding region A3c that encloses the first calculation region A3a.

[0150] In S740, the processor 210 calculates the gradient direction Ou3 and gradient intensity Mu3 for each pixel Px in the second calculation region A3b. The gradient direction Ou3 and gradient intensity Mu3 for pixels Px included in the first calculation region A3a within the second calculation region A3b have already been calculated in S710. Therefore, the processor 210 only needs to calculate the gradient direction Ou3 and gradient intensity Mu3 for each pixel Px in the surrounding region A3c.

[0151] In S750, the processor 210 generates a histogram in the second calculation region A3b with a gradient direction Ou3. In S760, the processor 210 determines the second type of orientation to correspond to the direction with the highest frequency in the histogram generated in S750.

[0152] After S760, the processor 210 completes the process shown in Figure 20, i.e., the process of S805 (Figure 12). The other parts of the alignment process (Figure 7) are the same as the corresponding parts of the alignment process in the first and second embodiments.

[0153] As described above, in this embodiment, in steps S710-S730 (Figure 20), the processor 210 calculates a first type orientation associated with the target feature point PN3 using an image of the first calculation region A3a containing the target feature point PN3. The target feature point PN3 is an example of a first feature point, which is the feature point to be calculated for the first type orientation. The target feature point PN3 is a reference feature point Tt or a read feature point Ts. The first calculation region A3a is an example of a first type subregion containing the target feature point PN3. In steps S740-S760, the processor 210 calculates a second type orientation associated with the target feature point PN3 using an image of the second calculation region A3b containing the target feature point PN3. The target feature point PN3 is an example of a second feature point, which is the feature point to be calculated for the second type orientation. The second calculation region A3b is an example of a second type subregion containing the target feature point PN3. As explained in Figure 21(A), the size of the second calculation region A3b is the same as the size of the region that includes the first calculation region A3a and the surrounding region A3c that encloses the first calculation region A3a. With this configuration, the processor 210 can calculate first and second type orientations that are different from each other but are associated with the same feature points. For example, if one of the first and second type orientations allows for the selection of an inappropriate target pair MP, the other of the first and second type orientations can reduce the possibility of that inappropriate target pair MP being selected.

[0154] The only difference between the alignment process of this embodiment and the alignment process of the first embodiment is the method for calculating the first and second type orientations. Therefore, this embodiment can provide the same various advantages as the first embodiment.

[0155] D. Variations: (1) The methods for calculating the first and second type orientations may vary. For example, in the embodiment shown in Figure 13, the method for calculating the second type orientation is not limited to the method described in Figures 13 and 14(A)-14(C), but may vary. For example, in calculating the frequency Fw of the histogram H (Figure 14(C)), the weight of the gradient intensity Mu may be omitted. Also, the histogram H may be omitted. For example, the processor 210 may adopt the direction represented by the sum of multiple vectors representing multiple arrows AR (Figure 14(B)) as the second type orientation. Here, the length of each of the multiple arrows AR may be set to the same value (e.g., 1), regardless of the gradient intensity Mu. Also, the shapes of the sub-region A1 and the off-region A1f are not limited to a square, but may vary. For example, the sub-region may be a circular region centered on the target feature point PN. The off-region may be smaller than the subregion and may be a circular region centered on the target feature point PN.

[0156] Furthermore, in the embodiment shown in Figure 13, the first type subregion (e.g., subregions Ata, Asa (Figure 9(A))) is used to calculate the first type orientation, and the second type subregion (e.g., on-region A1n (Figure 14(A))) is used to calculate the second type orientation. Here, the specific portion included in the first type subregion and excluded from the second type subregion is not limited to the portion containing the feature point (e.g., off-region A1f), but can be various portions. For example, the specific portion may be a rhombus-shaped region centered on the target feature point PN. This rhombus-shaped region may have a diagonal extending in the first direction Dx and a diagonal extending in the second direction Dy. The specific portion may be determined using the first type orientation. For example, the specific portion may be a region extending parallel to the first type orientation, or a region extending perpendicular to the first type orientation. The specific region may be a rectangular area having two sides parallel to the orientation of the first kind and two sides perpendicular to the orientation of the first kind. The specific region may be a ring-shaped area surrounding a feature point, but without including the feature point itself. In either case, the shape of the subregion of the second kind may be rotationally symmetric about the target feature point PN (for example, K-fold symmetry (where K is an integer greater than or equal to 2)).

[0157] (2) In the embodiment shown in Figure 20, the method for calculating the first and second type orientations is not limited to the method described in Figures 20, 21(A), and 21(B), but may be any of the various methods. For example, in the calculation of the frequencies of the histograms (S720, S750), the weight of the gradient intensity Mu3 may be omitted. Also, the histogram may be omitted. For example, the processor 210 may adopt as the orientation the direction represented by the sum of multiple vectors representing multiple arrows AR3 (Figure 21(B)). Here, the length of each of the multiple arrows AR3 may be set to the same value (e.g., 1) regardless of the gradient intensity Mu3. Also, the shapes of the first calculation region and the second calculation region are not limited to a square, but may be any of the various shapes. For example, the first calculation region may be a circular region centered on the target feature point PN3. The second calculation region may be larger than the first calculation region and also be a circular region centered on the target feature point PN3. The first calculation region may be an annular region surrounding the target feature point PN3, but without including the target feature point PN3 itself. The second calculation region may include the surrounding region that encloses the first calculation region and the first type subregion. The surrounding region may be an annular region having various shapes.

[0158] (3) In the embodiment shown in Figure 18, the method for calculating the first and second type orientations is not limited to the method described in Figures 18 and 19(A)-19(C), but may be any of the various methods. For example, in the calculation of the frequency of the histogram (S635), the weight of the gradient intensity Mu2 may be omitted. Also, the histogram may be omitted. For example, the processor 210 may adopt as the orientation the direction represented by the sum of multiple vectors representing multiple arrows AR2 (Figure 19(B)). Here, the length of each of the multiple arrows AR2 may be set to the same value (e.g., 1), regardless of the gradient intensity Mu2. Also, the shape of the calculation region is not limited to a square, but may be any of the various shapes. For example, the calculation region may be a circular region centered on the target feature point PN2.

[0159] The reference image IMt may be represented by various color components. The total number of color components U in the reference image IMt may be any value greater than or equal to 3. However, the total number U may be 2 or less. In any case, the color components CHa and CHb may be selected from U color components. Furthermore, the color components CHa and CHb may be selected from multiple color components that include U color components and other color components calculated from U color components. For example, if the reference image IMt is represented by red R, green G, and blue B, the color components CHa and CHb may be the two color components with the largest variance Sg among the four color components of luminance and RGB. In addition, various index values ​​indicating the degree of color variation may be used instead of variance Sg (e.g., standard deviation, difference between maximum and minimum values, etc.). Furthermore, the color components CHa and CHb may be predetermined.

[0160] (4) The first type orientation may be various orientations that represent the image features of the first type subregion associated with the feature point. The first type subregion may be a region containing the feature point, or alternatively, a region in the vicinity of the feature point that does not contain the feature point. Similarly, the second type orientation may be various orientations that represent the image features of the second type subregion associated with the feature point. The second type subregion may be a region containing the feature point, or alternatively, a region in the vicinity of the feature point that does not contain the feature point. The methods for calculating the first type orientation and the methods for calculating the second type orientation may be various different methods. For example, the first type subregion may include parts that are not included in the second type subregion. The second type subregion may include parts that are not included in the first type subregion. Also, the first type orientation in Figure 13 and the second type orientation in Figure 18 or Figure 20 may be used. The first type orientation in Figure 18 may be used in conjunction with the second type orientation in Figure 13 or Figure 20. The first type orientation in Figure 20 may be used in conjunction with the second type orientation in Figure 13 or Figure 18. Furthermore, the first and second types of orientations may be calculated using the orientation calculation algorithm by A-KAZE. Here, the calculation conditions may differ between the first and second types of orientations. For example, the size of the region used to calculate the orientation may differ between the first and second types of orientations. In any case, the order in which the first type orientation is calculated and the second type orientation is calculated may be any order.

[0161] (5) The process of forming candidate feature point pairs M is not limited to the process shown in Figure 8, but may be any of the various methods capable of forming appropriate feature point pairs. For example, the method for detecting feature points Tt and Ts (i.e., keypoints) may be any of the various methods for detecting points that indicate parts of an object in the image, instead of the method described in S310 and S315 (Figure 8). The detection method may be selected in advance from, for example, the search for extrema (maximum and minimum values) using DoG (Difference-of-Gaussian), Harris corner detection, FAST (Features from Accelerated Segment Test) corner detection, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF).

[0162] The method for calculating the features Ft and Fs (i.e., feature descriptors) may be any of the various methods used to calculate information describing the features of keypoints, instead of the method described in S320 and S325 (Figure 8). The algorithm for calculating feature descriptors may be pre-selected from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, and A-KAZE. The method for calculating the distance dF between the features Ft and Fs may be any of the various methods suitable for the data structure of the features Ft and Fs. When the features Ft and Fs are represented by binary vectors, the distance dF may be the Hamming distance. The distance dF may be any of the various other distances (for example, L1 norm, L2 norm (also called Euclidean distance), etc.) instead of the Hamming distance. Norms are applicable to various feature descriptors.

[0163] In either case, in the embodiment shown in Figure 13, if the first type orientation is not calculated in S320 and S325 (Figure 8), the processor 210 may recalculate the first type orientation in S510 (Figure 13).

[0164] (6) The process of selecting two candidate feature point pairs M as two target pairs MP is not limited to the process shown in Figures 12 and 16, but may be performed in various ways. For example, in S825 (Figure 12), the processor 210 may select multiple reference feature points Tt as the feature point group N, regardless of their distance from the feature point Tt of the first candidate MA (Figure 15). Also, instead of the conditions CC in S950 and S955 in Figure 16, a condition may be used that indicates that the relative magnitudes of the grayscale values ​​of multiple color components (e.g., RGB) are the same between the pair Tat, Tbt and the pair Tas, Tbs. The condition regarding the relative magnitudes of the grayscale values ​​(e.g., condition CC) may be omitted.

[0165] (7) The process of forming multiple candidate feature point pairs M used in the process of selecting two candidate feature point pairs M as two target pairs MP (Figure 7: S240) is not limited to the processes of S220 and S230, but may be various processes. The process of S230 is not limited to the process in Figure 10, but may be various processes. For example, instead of the processes of S425 and S435, the processor 210 may calculate the hue of each of the multiple pixels in the first subregion Pt (Figure 11(A)) and use the multiple hues to calculate a representative hue of the first subregion Pt. The same applies to saturation and brightness. The same applies to the representative color value of the second subregion Ps (Figure 11(B)). In the embodiment of Figure 10, the condition CW2 for selecting candidate feature point pairs M is expressed by the entirety of the conditions CSt, CH, CSs, and CV. Condition CW2 may be various conditions that use the first representative color value of the first sub-region Pt and the second representative color value of the second sub-region Ps. For example, condition CW2 may be determined using either the hue condition CH or the luminance condition CV. It is preferable that condition CW2 indicates that the first representative color value of the first sub-region Pt is similar to the second representative color value of the second sub-region Ps. If the scales of the reference image IMt and the read image IMs are different, the first sub-region Pt and the second sub-region Ps may be configured to show the same portion on the T-shirt 700.

[0166] Note that S230 may be omitted. That is, in the process of S240 (for example, the process in Figure 12), the processor 210 may form the target combination MT by brute force using the multiple candidate feature point pairs M formed in S220.

[0167] (8) The alignment process may be replaced by various processes instead of the processes in each of the above embodiments and modifications. For example, the coordinate correspondence (Figure 5) may represent various transformations, such as homography transformation, instead of affine transformation. Also, the coordinate correspondence may be represented in various formats, such as lookup tables, instead of the matrix Mtx.

[0168] (9) The printing medium is not limited to T-shirts 700, but may be various types of clothing (for example, various shirts such as polo shirts, coats, slacks, etc.). The printing medium may be various fabrics such as clothes and bags. The printing medium is not limited to fabric, but may be various media such as paper, film, leather, etc. Also, the printing device 900 may be a printing device of another type (for example, a laser printing device) instead of an inkjet printing device.

[0169] (10) Processing other than the process of determining the coordinate correspondence using multiple feature points Tt and Ts may be performed according to a program other than program 231. For example, S130 and S140 in Figure 4 may be performed according to a program other than program. Also, processing other than the process of determining the coordinate correspondence using multiple feature points Tt and Ts may be performed by a device other than data processing device 200.

[0170] (11) The coordinate correspondence may be used in various processes, not just inspection. For example, in the machining of metal parts, the position and orientation of the metal part relative to the tool may be misaligned. Here, the position and orientation of the metal part relative to the tool may be determined by analyzing an image of the metal part taken by a digital camera fixed to the tool. For example, the processor 210 may determine the reference position of the metal part on the captured image (and thus the position and orientation of the metal part relative to the tool) by performing alignment between a reference image representing a portion indicating the reference position of the metal part and the captured image. As the alignment process, the processes of each embodiment described above, or the processes of each modified example described above, may be adopted. Thus, the read image may be an image representing an object read by a line sensor such as the reading sensor 180 (Figure 2), or an image representing an object read by an area sensor such as a digital camera. Also, the reference image may be any image other than the image related to printing (for example, a pre-prepared image representing a specific part of an object).

[0171] Furthermore, the coordinate correspondence may be used in the process of stitching together multiple partially overlapping images to generate a single image (the generated image is also called a panorama). If both the first image and the second image contain portions representing the same object, the processor 210 may determine the correspondence between the coordinates on the first image and the coordinates on the second image so that the objects overlap. By performing the alignment process of each of the above embodiments or modifications, the processor 210 can determine an appropriate coordinate correspondence using multiple target pairs MP obtained from portions representing the same object (i.e., portions having a common external shape).

[0172] Thus, the first and second images used to determine the coordinate correspondence may be various images. The first image may be various images including the first object image. The second image may be various images including the second object image which includes a portion having an appearance shape common to at least a part of the first object image. In either case, the first image may be represented by one or more different color components (e.g., RGB, luminance values, etc.). Similarly, the second image may be represented by one or more different color components.

[0173] (12) The data processing device that determines the coordinate correspondence is not limited to a personal computer (for example, data processing device 200 (Figure 1)), but may be any other device (for example, a smartphone, a tablet computer, a control device incorporated into a reader, etc.). Alternatively, multiple devices (for example, computers) that can communicate with each other via a network may each share a portion of the data processing function performed by the data processing device, and together they may provide the data processing function (a system equipped with these devices corresponds to the data processing device).

[0174] In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, the orientation calculation process (e.g., the process in S520 (Figure 13), the process in Figure 18, or the process in Figure 20) may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0175] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0176] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of Symbols]

[0177] 100...Reading device, 110...Control device, 120...Transport device, 122...Position sensor, 130...Table, 140...Support unit, 180...Reading sensor, 190...Housing, 200...Data processing unit, 210...Processor, 215...Storage device, 220...Volatile storage device, 230...Non-volatile storage device, 231...Program, 240...Display unit, 250...Operation unit, 270...Communication interface, 700...T-shirt, 900...Printing device

Claims

1. It is a program, A function to perform a first orientation calculation process, which is a process to calculate the first orientation for each of the multiple first type feature points, which are multiple feature points in the first image containing the first object image, A function to perform a second orientation calculation process for each of the plurality of first type feature points in the first image, which is a process to calculate a second type orientation that is a different type of orientation from the first type orientation, A function that performs the first orientation calculation process for each of a plurality of second type feature points, which are a plurality of feature points in a second image including a second object image, wherein the second object image includes a portion having an external shape common to at least a part of the first object image. A function to perform the second orientation calculation process for each of the plurality of second type feature points in the second image, A function for selecting multiple target pairs, wherein each of the multiple target pairs is a feature point pair consisting of a first-type feature point and a second-type feature point, and is a feature point pair to be processed, and the function for selecting the multiple target pairs includes a function for determining whether or not to select the two feature point pairs as two target pairs using the first-type orientation and the second-type orientation of the two first-type feature points and two second-type feature points that form the two feature point pairs, A function to determine the correspondence between coordinates on the first image and coordinates on the second image using the aforementioned multiple pairs of objects, A program that enables a computer to realize something.

2. The program according to claim 1, further, The computer will be given the ability to form multiple feature point pairs using information that represents the features of a local region containing feature points. The function that performs the second orientation calculation process for each of the plurality of first type feature points performs the second orientation calculation process for each of the plurality of first type feature points that form the plurality of feature point pairs, The function that performs the second orientation calculation process for each of the plurality of second type feature points performs the second orientation calculation process for each of the plurality of second type feature points that form the plurality of feature point pairs. program.

3. A program according to claim 1 or 2, The first orientation calculation process includes a process of calculating a first type orientation associated with a first feature point using an image of a first type subregion placed with respect to the first feature point, which is a feature point that is the target of the calculation of the first type orientation. The second orientation calculation process includes a process of calculating a second type orientation associated with a second feature point, which is a feature point that is the target of the calculation of the second type orientation, using an image of a second type subregion placed with respect to the second feature point, wherein a specific portion corresponding to a part of the first type subregion is excluded from the second type subregion. program.

4. The program according to claim 3, The first type of subregion includes the portion containing the first feature point, The specific portion containing the second characteristic point is excluded from the second type of subregion. program.

5. A program according to claim 1 or 2, The first orientation calculation process includes a process of calculating a first type orientation associated with a first feature point using an image of a first type subregion that includes a first feature point which is the feature point to be used for calculating the first type orientation, The second orientation calculation process includes a process of calculating a second type orientation associated with a second feature point using an image of a second type subregion containing the second feature point, which is the feature point to be calculated for the second type orientation, wherein the size of the second type subregion is the same as the size of the region including the first type subregion and the surrounding region enclosing the first type subregion. program.

6. A program according to claim 1 or 2, The first orientation calculation process includes a process of calculating a first type orientation associated with a first feature point using the first color component of an image containing the first feature point which is the feature point to be used for calculating the first type orientation, The second orientation calculation process includes a process of calculating a second type of orientation associated with the second feature point using a second color component different from the first color component of an image containing the second feature point, which is the feature point for which the second type of orientation is to be calculated. program.

7. L according to claim 6, The first image described above is represented by U color components (where U is an integer greater than or equal to 3), The aforementioned program, further, A function to calculate an index value indicating the degree of color variation of each of the U color components in a subregion containing a particular first type feature point in the first image, A function to determine the first color component for the aforementioned first type feature point of interest to be the color component that corresponds to the index value showing the greatest degree of variation, The function determines the second color component for the aforementioned first type feature point of interest to be the color component that corresponds to the index value indicating the second largest degree of variation, A program that enables a computer to realize something.

8. A program according to claim 1 or 2, The first orientation calculation process includes a process for calculating the first type of orientation using the orientation calculation algorithm of A-KAZE. program.

9. A program according to claim 1 or 2, The first image above represents an image represented by image data for printing, The second image represents an image generated by optically reading an image to be printed using the printable image data. program.

10. A data processing device, A first calculation unit performs a first orientation calculation process, which is a process of calculating a first type orientation for each of the multiple first type feature points, which are multiple feature points in the first image including the first object image. A second calculation unit performs a second orientation calculation process, which is a process of calculating a second type orientation, which is a different type of orientation from the first type orientation, for each of the plurality of first type feature points in the first image. A third calculation unit performs the first orientation calculation process for each of a plurality of second type feature points which are a plurality of feature points in a second image including a second object image, wherein the second object image includes a portion having an external shape common to at least a part of the first object image, A fourth calculation unit performs the second orientation calculation process for each of the plurality of second type feature points in the second image, A selection unit for selecting multiple target pairs, wherein each of the multiple target pairs is a feature point pair consisting of a first-type feature point and a second-type feature point, and is a feature point pair to be processed, and the selection unit for selecting the multiple target pairs determines whether or not to select the two feature point pairs as two target pairs using the first-type orientation and the second-type orientation of the two first-type feature points and two second-type feature points that form the two feature point pairs, A determination unit that determines the correspondence between coordinates on the first image and coordinates on the second image using the plurality of target pairs, A data processing device equipped with the following features.

Images