Computer programs and image processing devices

Abstract

Identify the positional relationship between multiple images. [Solution] Multiple candidate pairs are determined using the feature quantities of each of the multiple reference feature points in the reference image and the feature quantities of each of the multiple target feature points in the target image. Candidate pairs are candidates for matching pairs, which are pairs of the reference feature points and target feature points that correspond to each other. The reference reference points corresponding to the n reference feature points that make up the selected n candidate pairs and the target reference points corresponding to the n target feature points are determined. If the n candidate pairs satisfy a specific condition, the n candidate pairs are determined as matching pairs. The specific condition is determined based on a first feature quantity calculated using the n reference feature points and the reference reference points, and a second feature quantity calculated using the n target feature points and the target reference points. The correspondence between coordinates on the reference image and coordinates on the target image is determined using the multiple matching pairs.

Description

Technical Field

[0001] This specification relates to a technique for specifying the positional relationship between a plurality of images.

Background Art

[0002] In various processes, the positional relationship between a plurality of images can be specified. Patent Document 1 discloses a technique for detecting defects in an image formed on a sheet by an image forming apparatus such as a printer or a copier. In this technique, together with a job image instructed to be printed by a user, a marker image for positioning is formed on the same sheet. An image reading unit reads the sheet surface and generates a read image. An inspection unit determines the position of the read image corresponding to the reference image based on the feature points of the job image and the marker image extracted from the read image to be inspected and the feature points of the job image and the marker image extracted from the reference image. The inspection unit compares the reference image and the read image after alignment based on the determined position, and detects an image area where the difference in pixel values is greater than or equal to a threshold value as a defect.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Specifying the positional relationship between a plurality of images is not easy and there is room for improvement.

[0005] This specification discloses a new technique for specifying the positional relationship between a plurality of images.

Means for Solving the Problems

[0006] The technologies disclosed herein have been made to solve at least some of the problems described above and can be realized in the following examples of applications.

[0007] [Application Example 1] A computer program comprising a candidate determination function that determines a plurality of candidate pairs using the respective feature quantities of a plurality of reference feature points of a reference image and the respective feature quantities of a plurality of target feature points of a target image, wherein the candidate pair is a candidate matching pair, which is a pair of the reference feature point and the target feature point that corresponds to each other, and the candidate determination function, A selection function that selects n candidate pairs (where n is an integer of 2 or more) from the plurality of candidate pairs; a reference point determination function that determines a reference reference point of the reference image corresponding to the n reference feature points constituting the n candidate pairs, and a target reference point of the target image corresponding to the n target feature points constituting the n candidate pairs, wherein the reference reference point is determined independently of the target image based on at least one of the corresponding n reference feature points, and the target reference point is determined independently of the reference image based on at least one of the corresponding n target feature points; and the n candidate pairs satisfy specific conditions. A computer program that causes a computer to implement a matching pair determination function, which determines the n candidate pairs as matching pairs when the n candidate pairs are added together, wherein the specific condition is determined based on a first feature quantity calculated using n reference feature points and reference reference points corresponding to the n reference feature points, and a second feature quantity calculated using n target feature points and target reference points corresponding to the n target feature points, and a correspondence relationship determination function that determines the correspondence relationship between coordinates on the reference image and coordinates on the target image using the determined plurality of matching pairs.

[0008] According to this configuration, the specific condition for determining n candidate pairs as matching pairs is determined based on a first feature calculated using n reference feature points and reference reference points corresponding to the n reference feature points, and a second feature calculated using n target feature points and target reference points corresponding to the n target feature points. As a result, matching pairs can be determined with higher accuracy than when reference reference points and target reference points are not used, so the positional relationship between the reference image and the target image can be appropriately identified. Furthermore, since the reference reference points are determined independently of the target image based on at least one of the corresponding n reference feature points, and the target reference points are determined independently of the reference image based on at least one of the corresponding n target feature points, the processing load for identifying the positional relationship between the reference image and the target image can be reduced.

[0009] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, as image processing methods, image processing devices, image processing systems, computer programs for realizing the functions of these methods or devices, and recording media (e.g., non-temporary recording media) on which such computer programs are recorded. [Brief explanation of the drawing]

[0010] [Figure 1] An explanatory diagram showing an image processing apparatus as one embodiment. [Figure 2] A perspective view showing an example of the reading device 100. [Figure 3] A diagram showing examples of various images. [Figure 4] A flowchart illustrating an example of the inspection process. [Figure 5] A flowchart for the process of determining candidate pairs. [Figure 6] A diagram showing an example of candidate pairs and matching pairs. [Figure 7] A flowchart for the matching pair determination process. [Figure 8] A diagram illustrating an example of point cloud N. [Figure 9] A diagram illustrating the set of characteristic points and reference points of the first embodiment. [Figure 10] Flowchart of the validity determination process of the first embodiment. [Figure 11] Flowchart of the similarity determination process of the first embodiment. [Figure 12] Explanatory diagram of the affine transformation matrix Mtx. [Figure 13] Diagram showing an example of the difference image IMd. [Figure 14] Explanatory diagram of the feature point set and reference points of the second embodiment. [Figure 15] Flowchart of the validity determination process of the second embodiment. [Figure 16] Flowchart of the similarity determination process of the second embodiment. [Figure 17] Flowchart of the similarity determination process of the modified example.

Mode for Carrying Out the Invention

[0011] A. Embodiment A1. Device Configuration FIG. 1 is an explanatory diagram showing an image processing apparatus as an embodiment. The image processing apparatus 200 is, for example, a personal computer. The image processing apparatus 200 executes an inspection process of a printed image. The image processing apparatus 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to perform data processing, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile memory device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile memory device 230 is, for example, a flash memory. The non-volatile memory device 230 stores data of the computer program PG and the reference image IMt. The processor 210 realizes the inspection process described later by executing the computer program PG. As will be described in detail later, the reference image IMt is used in the inspection process.

[0013] The display unit 240 is a device configured to display an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as buttons, levers, or a touch panel arranged on top of the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the image processing apparatus 200 by operating the operation unit 250.

[0014] The communication interface 270 is an interface for communicating with other devices (including, for example, one or more of a USB interface, a wired LAN interface, a wireless interface of IEEE802.11, an interface of an industrial camera (such as CameraLink, CoaXPress, etc.)). In this embodiment, the reading device 100 and the printing device 900 are connected to the communication interface 270. The printing device 900 is a so-called inkjet printing device, and prints an image on a printing medium such as cloth or paper by ejecting ink. The reading device 100 generates data of a read image (scan data) representing the object by optically reading the object to be read. In this embodiment, the printing medium of the printing device 900 is a T-shirt, and the object to be read by the reading device 100 is a T-shirt with an image printed thereon.

[0015] Figure 2 is a perspective view showing an example of the reading device 100. Directions Da and Db in the figure indicate horizontal directions, and direction Dc indicates the vertically upward direction. Directions Da and Db are perpendicular to each other.

[0016] In this embodiment, the reading device 100 comprises a housing 190, a table 130, a support part 140 fixed to the upper surface of the table 130, a transport device 120, a reading sensor 180, and a control device 110. The control device 110, the transport device 120, and the reading sensor 180 are fixed to the housing 190.

[0017] The support portion 140 is a plate-shaped member that forms a flat top surface for supporting the object to be read (such a member is also called a platen). In the figure, a T-shirt 700 with a printed image IMp is placed on the support portion 140.

[0018] The conveying device 120 is configured to convey the table 130 in the conveying direction (a direction parallel to direction Db in Figure 2). The configuration of the conveying device 120 may vary. Although not shown in the figures, in this embodiment the conveying device 120 has a rail that supports the table 130 so as to be slidable in the conveying direction, a plurality of pulleys, a belt that is wrapped around the plurality of pulleys and partly fixed to the table 130, and an electric motor that rotates the pulleys. The electric motor rotates the pulleys, causing the table 130 (and thus the support part 140) to move in the conveying direction. The conveying device 120 further has a position sensor 122 (for example, a rotary encoder) that detects the position of the table 130 on the conveying path.

[0019] The reading sensor 180 is positioned higher than the support unit 140 in the middle of the transport path Pth of the support unit 140. The reading sensor 180 includes a line sensor composed of multiple photoelectric conversion elements arranged in a direction intersecting the transport direction (in this embodiment, in the direction Da perpendicular to direction Db) (for example, a Contact Image Sensor (CIS) or a Charge Coupled Device (CCD)). The reading sensor 180 faces downward. The reading sensor 180 can read the portion of the object supported by the support unit 140 that is located below the reading sensor 180.

[0020] When reading the T-shirt 700, the reading device 100 transports the table 130 in the transport direction. The reading sensor 180 repeatedly reads the T-shirt 700 during transport. As a result, the reading sensor 180 can read approximately the entire portion of the T-shirt 700 that is supported by the support portion 140.

[0021] The control device 110 is an electrical circuit configured to control the transport device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates scan data by controlling the transport device 120 and the reading sensor 180.

[0022] A2. Printing process In this embodiment, an image is printed on a T-shirt 700 (Figure 2). The image printing is performed, for example, as part of a T-shirt sales service. For example, the service provider prints an image on the T-shirt 700 using a printing device 900 according to the customer's order. The image data for printing may be provided by the customer or prepared by the service provider.

[0023] Figure 3 shows examples of various images. Figure 3(A) shows an image represented by image data for printing (referred to as the reference image IMt). In this embodiment, the data of the reference image IMt is bitmap data representing the color values ​​(here, the gradation values ​​of red R, green G, and blue B (for example, values ​​between zero and 255)) of multiple pixels arranged in a matrix along directions Dx and Dy. In the example of Figure 3(A), the reference image IMt includes a background Bt and four objects Ot (Ot1-Ot4). To distinguish the objects Ot in the reference image IMt from the objects in the target images IMs described later, they are also called reference objects Ot.

[0024] Figure 2 shows a printed image IMp formed by printing a reference image IMt by a printing device 900 on the surface of a T-shirt 700. The printed image IMp may have various defects due to, for example, errors during printing. For example, due to an abnormality in ink ejection, a portion of the image may be missing in the printed image IMp. The image processing device 200 detects defects in the printed image IMp through an inspection process described later. In this embodiment, the data of the reference image IMt is used in the inspection process. The data of the reference image IMt used for printing is stored in the storage device 215 (for example, a non-volatile storage device 230) of the image processing device 200 for the inspection process (Figure 1).

[0025] A3. Inspection process Figure 4 is a flowchart illustrating an example of the inspection process. For inspection, the T-shirt 700 is placed on the support unit 140 of the reader 100 (Figure 2) so that the printed image IMp is visible. In this embodiment, an operator places the T-shirt 700 on the support unit 140. Alternatively, a machine (e.g., a robotic arm) may place the T-shirt 700 on the support unit 140. After the T-shirt 700 is placed, an instruction to start the inspection process is input to the image processing device 200 (Figure 1). In this embodiment, the operator inputs the instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the instruction to start. The instruction to start the inspection process may be input to the image processing device 200 via the communication interface 270 by a device other than the image processing device 200.

[0026] The processor 210 of the image processing device 200 executes inspection processing according to the computer program PG. In S100, the processor 210 acquires data of the reference image IMt (Figure 3(A)) used for printing on the T-shirt 700. In this embodiment, the data of the reference image IMt stored in the non-volatile storage device 230 is acquired.

[0027] In S110, the processor 210 obtains data of target images IMs by causing the reader 100 to read the T-shirt 700. Specifically, the processor 210 supplies a reading instruction to the reader 100. The control device 110 of the reader 100 reads the T-shirt 700 by controlling the reading sensor 180 and the transport device 120 in response to the reading instruction. The control device 110 generates data of a read image (scan data) representing the read T-shirt 700. The processor 210 of the image processing device 200 obtains the data of the read image from the reader 100 as target images IMs.

[0028] Figure 3(B) shows the target image IMs. The target image IMs in Figure 3(B) are images showing a T-shirt 700 containing a printed image IMp. In this embodiment, the data of the target image IMs is bitmap data representing the color values ​​(here, the grayscale values ​​of red R, green G, and blue B, for example, values ​​between zero and 255) of multiple pixels arranged in a matrix along directions Dx and Dy.

[0029] As described above, the printed image IMp is obtained by printing the reference image IMt (Figure 3(A)). For this reason, the target image IMs, like the reference image IMt, includes a background Bs and four target objects Os (Os1-Os4). To distinguish the objects Os in the target image IMs from the reference object Ot in the reference image IMt described above, they are also called target objects Os. As shown in Figure 3, there is a one-to-one correspondence between the reference object Ot in the reference image IMt and the target objects Os in the target image IMs. That is, the reference object Oti (where i is an integer from 1 to 4) in Figure 3(A) corresponds to the target object Osi in Figure 3(B).

[0030] In S120, the processor 210 performs grayscale conversion on both the reference image IMt and the target image IMs. A known relationship can be used as the correspondence between color gradation values ​​and grayscale gradation values. For example, the correspondence between RGB values ​​in the RGB color space and luminance value Y in the YCbCr color space can be used.

[0031] In S130, the processor 210 detects multiple feature points of the reference image IMt (referred to as reference feature points Tt) and multiple feature points of the target image IMs (referred to as target feature points Ts).

[0032] Specifically, the processor 210 detects multiple reference feature points Tt by analyzing the grayscale-converted reference image IMt using known methods. The processor 210 also detects multiple target feature points Ts by analyzing the grayscale-converted target image IMs using known methods.

[0033] In this embodiment, a method called Accelerated-KAZE (A-KAZE) is used for feature point detection and feature quantity calculation (described later). A-KAZE is disclosed, for example, in the following paper. "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013"

[0034] Figure 3(A) shows an example of a reference feature point Tt detected in the reference image IMt. Figure 3(B) shows an example of a target feature point Ts detected in the target image IMs. As shown in the figures, points indicating characteristic parts of objects Ot and Os, such as corners, are detected as feature points Tt and Ts. Although not shown in the figures, in reality, many more feature points Tt and Ts may be detected, including internal edges of objects Ot and Os (for example, tens or hundreds).

[0035] In S140, the processor 210 calculates the feature quantities Ft and Fs for multiple reference feature points Tt and multiple target feature points Ts, respectively. The feature quantity Ft for a reference feature point Tt is a local feature quantity that shows the local characteristics of the reference feature point Tt. That is, the feature quantity Ft for a reference feature point Tt is calculated using the image of the local region containing the reference feature point Tt from the reference image IMt. Similarly, the feature quantity Fs for a target feature point Ts is a local feature quantity that shows the local characteristics of the target feature point Ts. That is, the feature quantity Fs for a target feature point Ts is calculated using the image of the local region containing the target feature point Ts from the target image IMs.

[0036] The feature quantities Ft and Fs may be described using various methods for calculating local features. For example, the feature quantities Ft and Fs may be calculated such that they change depending on the distribution of color values ​​of multiple pixels surrounding the feature points Tt and Ts. In this embodiment, the processor 210 uses the grayscale-converted reference image IMt and target image IMs to calculate the A-KAZE feature descriptor and feature direction as the feature quantities Ft and Fs. In A-KAZE technology, the feature descriptor is rotationally invariant. In order to obtain a rotationally invariant feature descriptor, the feature direction of the feature point is also calculated. The feature direction indicates the direction of the luminance gradient in the neighborhood region centered on the feature point (also called the gradient direction or dominant direction).

[0037] In S150, the processor 210 performs a candidate pair determination process. The candidate pair determination process determines multiple candidate pairs using the feature quantities Fs of multiple reference feature points Tt in the reference image IMt and the feature quantities Ft of multiple target feature points Ts in the target image IMs. Candidate pairs are candidates for matching pairs. A matching pair is a pair of reference feature points Tt and target feature points Ts that correspond to each other. That is, a matching pair is a pair of feature points Tt and Ts that shows the correct correspondence between the reference image IMt and the target image IMs, and is used to determine the coordinate correspondence between the reference image IMt and the target image IMs, as will be described later.

[0038] Figure 5 is a flowchart of the candidate pair determination process. In the candidate pair determination process, the processor 210 repeatedly executes the loop process S31 between the start L31s and the end L31e for all feature point pairs. A feature point pair consists of one point selected from among the multiple detected reference feature points Tt and one point selected from among the multiple detected target feature points Ts. If the number of detected reference feature points Tt is J (J is an integer greater than or equal to 1) and the number of detected target feature points Ts is K (K is an integer greater than or equal to 1), the loop process S31 is executed for each of the (J × K) combinations.

[0039] In S310, the processor 210 selects one unprocessed feature point pair from the configurable feature point pairs as the focus feature point pair. The reference feature point Tt that constitutes the focus feature point pair is called the focus reference feature point, and the target feature point Ts that constitutes the focus feature point pair is called the target feature point.

[0040] In S320, the processor 210 calculates the distance dF between the feature quantity Ft of the reference feature point and the feature quantity Fs of the target feature point. The distance dF is calculated such that a smaller distance dF indicates a higher similarity between the two feature quantities Ft and Fs. When the distance dF is small (i.e., the similarity is high), it is likely that the reference feature point and the target feature point represent similar parts of two images IMt and IMs (for example, corresponding parts within the object of interest). In this embodiment, the feature quantities Ft and Fs include A-KAZE feature descriptors. These feature descriptors are represented by binary vectors (vectors consisting of one or more binary elements). For this reason, in this embodiment, the processor 210 calculates the Hamming distance of the feature descriptors contained in the feature quantities Ft and Fs as the distance dF.

[0041] In S330, the processor 210 determines whether the distance dF is less than the distance threshold dFth. If the distance dF is less than the distance threshold dFth (S330: YES), the processor 210 determines the pair of feature points of interest as a candidate pair in S340. After S340, the processor 210 terminates the loop processing S31 for the current pair of feature points of interest.

[0042] If the distance dF is greater than or equal to the distance threshold dFth (S330: NO), the processor 210 skips S340 and terminates the loop processing S31 for the current pair of feature points of interest. In this case, the current pair of feature points of interest is not determined to be a candidate pair. Once the loop processing L31 has been performed for all feature point pairs, the processor 210 proceeds to processing S350.

[0043] In S350, the processor 210 stores the data of the multiple determined candidate pairs in the storage device 215 (for example, the non-volatile storage device 230) and terminates the candidate pair determination process.

[0044] Figure 6 shows an example of candidate pairs and matching pairs. Figure 6(A) shows an example of a candidate pair determined in the candidate pair determination process. The multiple lines RL in the figure each represent a candidate pair. Each line RL connects the reference feature point Tt and the target feature point Ts that form the candidate pair. As shown in the figure, each candidate pair consists of one reference feature point Tt and one target feature point Ts.

[0045] A pair of feature points Tt and Ts that indicate corresponding regions can be determined as a candidate pair. For example, in the example in Figure 6(A), the reference feature point Ttn, which indicates the upper right corner of the reference object Ot2 in the reference image IMt, is associated with the target feature point Tsn, which indicates the upper right corner of the target object Os2 in the target image IMs. Candidate pairs that show such a correct correspondence are ultimately the pairs that should be determined as matching pairs.

[0046] Furthermore, a pair of feature points Tt and Ts that represent areas that do not correspond to each other may be selected as a candidate pair. For example, in the example in Figure 6(A), the reference feature point Ttn, which represents the upper right corner of the reference object Ot2 in the reference image IMt, is associated with the target feature point Tsm, which represents the lower left corner of the target object Os2 in the target image IMs. Candidate pairs that show such incorrect correspondences should ultimately not be selected as matching pairs.

[0047] For example, rectangular objects Ot2 and Os2 each have four corners. These corners are locally similar. As a result, a reference feature point Ttn representing one corner of an object in reference object Ot2 can be associated with any of the target feature points Ts representing four corners of target object Os2. For this reason, the multiple candidate pairs determined in the candidate pair determination process may include candidate pairs that show a correct correspondence and candidate pairs that show an incorrect correspondence.

[0048] In particular, in this embodiment, as described above, the feature descriptors included in the feature quantities Ft and Fs are rotationally invariant. When feature descriptors are rotationally invariant, it is possible to calculate similar feature descriptors from the same part of an object regardless of the rotation angle of the object in the image. By using the distance dF between rotationally invariant feature descriptors, the processor 210 can associate pairs of feature points that represent corresponding parts of an object, even if the rotation angles of the corresponding objects in the two images IMt and IMs are different. However, if each image IMt and IMs contains multiple locally similar parts, feature points Tt and Ts that represent parts that do not correspond to each other may be incorrectly associated.

[0049] Note that a larger distance threshold dFth (Figure 5: S330) may result in a larger total number of candidate pairs showing correct correspondence. However, the total number of candidate pairs showing incorrect correspondence may also increase. A larger total number of candidate pairs showing correct correspondence reduces the error in the coordinate correspondence relationship, as described later. If the total number of candidate pairs showing incorrect correspondence is large, the error in the coordinate correspondence relationship may increase. The distance threshold dFth may be determined experimentally in advance so that the error in the coordinate correspondence relationship is acceptable.

[0050] Once the candidate pair determination process is complete, in S160 of Figure 4, the processor 210 executes the matching pair determination process. The matching pair determination process narrows down the multiple candidate pairs determined in the candidate pair determination process, reducing the possibility of any candidate pairs showing incorrect correspondence remaining. This determines the final matching pair. In the matching pair determination process, candidate pairs are narrowed down using sets (combinations) of two candidate pairs.

[0051] Figure 7 is a flowchart of the matching pair determination process. Processor 210 executes the loop process S41 between the start L41s and the end L41e, treating each of the candidate pairs as the first pair of interest.

[0052] In S410, the processor 210 selects one candidate pair from the unprocessed candidate pairs as the first pair of interest. The reference feature point Tt that constitutes the first pair of interest is also called the first pair of interest reference feature point Tt1, and the target feature point Ts that constitutes the first pair of interest is also called the first pair of interest target feature point Ts1.

[0053] In S415, the processor 210 selects a point cloud N consisting of reference feature points Tt within a range between a first radius r1 and a second radius r2, centered on the first reference feature point Tt1 of interest. Figure 8 shows an example of point cloud N. The figure shows a reference image IMt and multiple reference feature points Tt. The first circle C1 is a circle with a first radius r1 centered on the first reference feature point Tt1 of interest. The second circle C2 is a circle with a second radius r2 centered on the first reference feature point Tt1 of interest. The selection range SR is the region between the first circle C1 and the second circle C2 (including the parts on circles C1 and C2). The processor 210 selects the multiple feature points Tt included in the selection range SR as the point cloud N.

[0054] The processor 210 takes each of the multiple candidate pairs, each containing a reference feature point Tt belonging to the point cloud N, as a second pair of interest and executes a loop process S42 between the start L42s and the end L42e. Loop process S42 is a loop process nested within loop process S41. Here, the reference feature point Tt that constitutes the second pair of interest is also called the second pair of interest reference feature point Tt2, and the target feature point Ts that constitutes the second pair of interest is also called the second pair of interest target feature point Ts2. As can be seen from the above explanation, the second pair of interest reference feature point Tt2 belongs to the point cloud N.

[0055] As described later, the processor 210 compares a set of first-of-interest reference feature points Tt1 and second-of-interest reference feature points Tt2 (also called the reference feature point set TtS) with a set of first-of-interest target feature points Ts1 and second-of-interest target feature points Ts2 (also called the target feature point set TsS). Based on the comparison, the processor 210 evaluates whether these two feature point sets TtS and TsS are similar to each other, and determines whether the first-of-interest pair and the second-of-interest pair are a matching pair. At this time, if the distance between the first-of-interest reference feature point Tt1 and the second-of-interest reference feature point Tt2 is excessively long or excessively short, the error in comparing the two feature point sets TtS and TsS may increase excessively, making it difficult to accurately evaluate whether they are similar or not. The selection range SR (i.e., radii r1, r2) is experimentally determined in advance to a range that allows for accurate evaluation of whether the two feature point sets TtS and TsS are similar or not. For example, the first radius r1 may be a value between 1% and 20% of the size of the reference image IMt (e.g., the length of direction Dx or direction Dy). The second radius r2 may be a value between 30% and 70% of the size of the reference image IMt.

[0056] In S420, the processor 210 selects one unprocessed candidate pair from among the candidate pairs that include a reference feature point Tt belonging to the point cloud N as the second pair of interest.

[0057] In S425, the processor 210 determines the reference points of the reference image IMt and the reference points of the target images IMs. Reference points are points added for reference when comparing two feature point sets TtS and TsS. The reference points added to the reference image IMt are also called reference reference points Rt, and the reference points added to the target images IMs are also called target reference points Rs.

[0058] Figure 9 is an explanatory diagram of the feature point set and reference point in the first embodiment. Figure 9(A) shows an example of a reference feature point set TtS and a reference reference point Rt. The reference reference point Rt is determined based on the first notable reference feature point Tt1 and the second notable reference feature point Tt2 that constitute the reference feature point set TtS. Specifically, the reference reference point Rt is determined on a line that passes through the first notable reference feature point Tt1 and extends in a first specific direction Drt. In this embodiment, the first specific direction Drt is the direction in which the angle αt formed with the line segment L1t connecting the first notable reference feature point Tt1 and the second notable reference feature point Tt2 is 90 degrees. In this embodiment, the angle αt is a clockwise angle. Furthermore, the reference reference point Rt is determined such that the ratio (B / A) of the length B of the line segment L2t connecting the first reference feature point Tt1 and the reference reference point Rt to the length A of the line segment L1t is a predetermined specific value (in this embodiment, 0.5 (50%)). In this way, the reference reference point Rt is determined based solely on the reference feature point set TtS of the reference image IMt, independently of the target image IMs. Since one reference reference point Rt is determined for each reference feature point set TtS, there is a one-to-one correspondence between the reference reference reference point set TtS and the reference feature point set TtS.

[0059] Figure 9(B) shows an example of a target feature point set TsS and a target reference point Rs. The target reference point Rs is determined based on the first target feature point Ts1 and the second target feature point Ts2 that constitute the target feature point set TsS. Specifically, the target reference point Rs is determined on a line that passes through the first target feature point Ts1 and extends in a second specific direction Drs. In this embodiment, the second specific direction Drs is the direction in which the angle αs formed with the line segment L1s connecting the first target feature point Ts1 and the second target feature point Ts2 is 90 degrees. In this embodiment, the angle αs is a clockwise angle. Furthermore, the target reference point Rs is determined such that the ratio (D / C) of the length D of the line segment L2s connecting the first target feature point Ts1 and the target reference point Rs to the length C of the line segment L1s is the specific value mentioned above (in this embodiment, 0.5 (50%)). Thus, the target reference point Rs is determined solely based on the target feature set TsS of the target image IMs, independently of the reference image IMt. Since one target reference point Rs is determined for each target feature set TsS, there is a one-to-one correspondence between the target reference point Rs and the target feature set TsS.

[0060] As can be seen from the above explanation, in this embodiment, the angle αt for determining the reference point Rt and the angle αs for determining the target reference point Rs are equal (αt = αs). Also, the first ratio (B / A) for determining the reference reference point Rt and the second ratio (D / C) for determining the target reference point Rs are equal ((B / A) = (D / C)). For this reason, the triangle formed by the three points of interest, reference feature points Tt1 and Tt2 and the reference reference point Rt (Figure 9(A), also called the reference triangle TRt), and the triangle formed by the three points of interest, target feature points Ts1 and Ts2 and the target reference point Rs (Figure 9(B), also called the target triangle TRs), are similar.

[0061] In S430, the processor 210 performs a validity determination process. The validity determination process determines the validity of the three points of the reference image IMt (the reference feature points of interest Tt1 and Tt2 and the reference reference point Rt) and the three points of the target image IMs (the target feature points of interest Ts1 and Ts2 and the target reference point Rs). Figure 10 is a flowchart of the validity determination process in the first embodiment.

[0062] In S510, the processor 210 determines whether the three points of the reference image IMt are different points. If at least two of the three points of the reference image IMt (the reference feature points of interest Tt1, Tt2 and the reference reference point Rt) are the same, the three points of the reference image IMt are determined not to be different points. If all three points of the reference image IMt are different from each other, the three points of the reference image IMt are determined to be different points. This step is a confirmation step, and normally, the three points of the reference image IMt are determined to be different points.

[0063] If the three points in the reference image IMt are not different points (S510: NO), the processor 210 determines the result to be invalid in S580. If the three points in the reference image IMt are different points (S510: YES), the processor 210 proceeds to S520.

[0064] In S520, the processor 210 determines whether the three points of the target image IMs (the feature points of interest Ts1 and Ts2 and the reference point Rs) are different points. The determination in S520 is performed in the same way as the determination of whether the three points of the image IMt are different points as described above. This step is also a confirmation step, and normally, the three points of the target image IMs are determined to be different points.

[0065] If the three points in the target image IMs are not different points (S520: NO), the processor 210 determines the result to be invalid in S580. If the three points in the target image IMs are different points (S520: YES), the processor 210 proceeds to S530.

[0066] In S530, the processor 210 determines whether the length of the line segment L1s connecting the two feature points of interest Ts1 and Ts2 is within an acceptable range. As mentioned above, the length of the line segment L1t connecting the two reference feature points Tt1 and Tt2 is within the range of a first radius r1 or greater and a second radius r2 or less, so it is considered that the length of the line segment L1s connecting the two feature points of interest Ts1 and Ts2 will not be excessively short or excessively long. In other words, if the length of the line segment L1s is excessively short or excessively long, it is considered unlikely that this reference feature point set TtS and the target feature point set TsS will form a matching pair, without even needing to determine whether they are similar or not. For this reason, in this step, it is determined whether the length of the line segment L1s connecting the two feature points of interest Ts1 and Ts2 is within an acceptable range. This acceptable range is determined experimentally in advance, taking into account the first radius r1 and the second radius r2.

[0067] If the length of the line segment L1s connecting the feature points Ts1 and Ts2 of interest is outside the acceptable range (S530: NO), the processor 210 determines the result to be invalid in S580. If the length of the line segment L1s connecting the feature points Ts1 and Ts2 of interest is within the acceptable range (S530: YES), the processor 210 proceeds to S540.

[0068] In S540, the processor 210 determines whether the reference point Rt and the target reference point Rs are within the image. If the reference point Rt is located within the reference image IMt and the target reference point Rs is located within the target image IMs, then the reference points Rt and Rs are determined to be within the image. If the reference point Rt is located outside the reference image IMt, or if the target reference point Rs is located outside the target image IMs, then the reference points Rt and Rs are determined to be outside the image. If the reference points Rt and Rs are not within the image, it is not possible to determine whether they are similar or not, as described later, so in this step, it is determined whether both reference points Rt and Rs are within the image.

[0069] If the reference points Rt and Rs are not within the image (S540: NO), the processor 210 determines the result to be invalid in S580. If the reference points Rt and Rs are within the image (S540: YES), the processor 210 determines the result to be valid in S570. Once the result is determined to be either valid or invalid, the valid determination process is terminated.

[0070] Once the validity determination process is complete, in S435 of Figure 7, the processor 210 determines whether the result of the validity determination process is valid or invalid. If the result is invalid (S435: NO), the processor 210 terminates the loop process in S42 without executing S440-S450. Therefore, in this case, the first pair of interest and the second pair of interest are not determined to be a matching pair.

[0071] If the determination result is valid (S435: YES), the processor 210 executes the similarity determination process in S440. The similarity determination process determines whether three points of the reference image IMt (the reference feature points of interest Tt1 and Tt2 and the reference reference point Rt) and three points of the target image IMs (the target feature points of interest Ts1 and Ts2 and the target reference point Rs) are similar. Figure 11 is a flowchart of the similarity determination process in the first embodiment.

[0072] In S610, the processor 210 calculates the feature directions of two reference points, namely the feature direction Dt3 of the reference reference point Rt (Figure 9(A)) and the feature direction Ds3 of the target reference point Rs (Figure 9(B)). The method for calculating the feature directions is the same as the method for calculating the feature directions included in the feature quantities Ft and Fs of feature points Tt and Ts calculated in S140 in Figure 4. That is, the feature direction Dt3 of the reference reference point Rt is calculated using the image of the local region containing the reference reference point Rt from the reference image IMt. The feature direction Ds3 of the target reference point Rs is calculated using the image of the local region containing the target reference point Rs from the target image IMs. Note that the feature directions Dt1 and Dt2 of the reference feature points of interest Tt1 and Tt2, and the feature directions Ds1 and Ds2 of the target feature points of interest Ts1 and Ts2 have already been calculated in S140 in Figure 4.

[0073] In the S620, the processor 210 calculates the feature angles θt1, θt2, and θt3 at each vertex of the reference triangle TRt. These feature angles θt1, θt2, and θt3 can be described as feature quantities that represent the characteristics of the reference feature point set TtS. Feature angle θt1 is the angle formed by the feature direction Dt1 of the first reference feature point Tt1 and the line segment L1t connecting the first reference feature point Tt1 and the second reference feature point Tt2. Feature angle θt2 is the angle formed by the feature direction Dt2 of the second reference feature point Tt2 and the line segment L1t. Feature angle θt3 is the angle formed by the feature direction Dt3 of the reference reference point Rt and the line segment L2t connecting the reference reference point Rt and the reference feature point Tt1.

[0074] In this embodiment, the feature angles θt1 and θt2 are the smaller of the angles formed by line segments Ldt1 and Ldt2, which extend from the reference feature points Tt1 and Tt2 in feature directions Dt1 and Dt2, and line segment L1t (angles of 180 degrees or less) (Figure 9(A)). Similarly, the feature angle θt3 is the smaller of the angles formed by line segment Ldt3, which extends from the reference reference point Rt in feature direction Dt3, and line segment L2t (angles of 180 degrees or less). Alternatively, the feature angles θt1 and θt2 may be the angles from line segment L1t to line segments Ldt1 and Ldt2, in a clockwise (or counterclockwise) direction, centered on the reference feature points Tt1 and Tt2. Similarly, the characteristic angle θt3 may be the angle from line segment L2t to line segment Ldt3, centered on the reference point Rt, in a clockwise (or counterclockwise) direction.

[0075] In the S630, the processor 210 calculates the feature angles θs1, θs2, and θs3 for each vertex of the target triangle TRs. These feature angles θs1, θs2, and θs3 can be described as feature quantities that represent the characteristics of the target feature point set TsS. Feature angle θs1 is the angle formed by the feature direction Ds1 of the first target feature point Ts1 and the line segment L1s connecting the first target feature point Ts1 and the second target feature point Ts2. Feature angle θs2 is the angle formed by the feature direction Ds2 of the second target feature point Ts2 and the line segment L1s. Feature angle θs3 is the angle formed by the feature direction Ds3 of the target reference point Rs and the line segment L2s connecting the target reference point Rs and the first target feature point Ts1.

[0076] In this embodiment, the feature angles θs1 and θs2 are the smaller of the angles (angles of 180 degrees or less) between the line segments Lds1 and Lds2, which extend from the feature points of interest Ts1 and Ts2 in feature directions Ds1 and Ds2, and the line segment L1s (Figure 9(B)). Similarly, the feature angle θt3 is the smaller of the angles (angles of 180 degrees or less) between the line segment Lds3, which extends from the target reference point Rs in feature direction Ds3, and the line segment L2s. Alternatively, the feature angles θs1 and θs2 may be the angles from line segment L1s to line segments Lds1 and Lds2, in a clockwise (or counterclockwise) direction, centered on the feature points of interest Ts1 and Ts2. Similarly, the feature angle θs3 may be the angle from line segment L2s to line segment Lds3, centered on the target reference point Rs, in a clockwise (or counterclockwise) direction.

[0077] In the S640, the processor 210 determines whether the feature angles θt1-θt3 of the reference triangle TRt and the feature angles θs1-θs3 of the target triangle TRs approximate each other. Specifically, if all of the following equations (1) to (3) are satisfied, it is determined that the feature angles θt1-θt3 of the reference triangle TRt and the feature angles θs1-θs3 of the target triangle TRs approximate each other. If at least one of equations (1) to (3) is not satisfied, it is determined that the feature angles θt1-θt3 of the reference triangle TRt and the feature angles θs1-θs3 of the target triangle TRs do not approximate each other.

[0078] |θt1-θs1| <THn …(1) |θt2-θs2| <THn …(2) |θt3-θs3| <THn …(3)

[0079] If the characteristic angles θt1~θt3 of the reference triangle TRt and the characteristic angles θs1~θs3 of the target triangle TRs do not approximate each other (S640: NO), the processor 210 determines the result as dissimilar in S690. If the characteristic angles θt1~θt3 of the reference triangle TRt and the characteristic angles θs1~θs3 of the target triangle TRs approximate each other (S640: YES), the processor 210 proceeds to S650.

[0080] In the S650, the processor 210 obtains the brightness values ​​Vt1 to Vt3 for each vertex of the reference triangle TRt and the brightness values ​​Vs1 to Vs3 for each vertex of the target triangle TRs. Specifically, for example, the values ​​of each pixel in the grayscale-converted reference image IMt and the target image IMs represent brightness values ​​V. Therefore, from the grayscale-converted reference image IMt, the pixel values ​​of each vertex of the reference triangle TRt (the reference feature points of interest Tt1, Tt2, and the reference reference point Rt) are obtained as brightness values ​​Vt1, Vt2, and Vt3. From the grayscale-converted target image IMs, the pixel values ​​of each vertex of the target triangle TRs (the target feature points of interest Ts1, Ts2, and the target reference point Rs) are obtained as brightness values ​​Vs1, Vs2, and Vs3.

[0081] In the S660, the processor 210 determines whether the relative magnitudes of the luminances Vt1 to Vt3 at each vertex of the reference triangle TRt are equal to the relative magnitudes of the luminances Vs1 to Vs3 at each vertex of the target triangle TRs.

[0082] Specifically, in this embodiment, if the relationship between luminance Vt1 and luminance Vt2 is equal to the relationship between luminance Vs1 and luminance Vs2, and the relationship between luminance Vt1 and luminance Vt3 is equal to the relationship between luminance Vs1 and luminance Vs3, then the relationship between luminance Vt1 to Vt3 at each vertex of the reference triangle TRt is determined to be equal to the relationship between luminance Vs1 to Vs3 at each vertex of the target triangle TRs. In a modified example, the condition that the relationship between luminance Vt2 and luminance Vt3 is equal to the relationship between luminance Vs2 and luminance Vs3 may be added.

[0083] For example, the relative magnitudes of two luminances V1 and V2 are determined using the difference value ΔV = (V2 - V1) and a positive threshold Th to obtain one of the following three judgment results: If the difference value ΔV is greater than the threshold Th (ΔV > Th), luminance V2 is judged to be sufficiently greater than luminance V1. If the difference value ΔV is greater than or equal to the threshold (-Th) and less than or equal to the threshold Th (-Th ≤ ΔV ≤ Th), luminance V2 is judged to be approximately equal to luminance V1. If the difference value ΔV is less than the threshold (-Th) (-Th > ΔV), luminance V2 is judged to be sufficiently smaller than luminance V1. As for the judgment results for the relative magnitudes, instead of the above three types of judgment results, two types of judgment results, such as satisfying ΔV > 0 or ΔV ≤ 0, may be adopted, or four or more types of judgment results that are further classified may be adopted.

[0084] If the magnitude relationship of the luminances Vt1 to Vt3 at each vertex of the reference triangle TRt is not equal to the magnitude relationship of the luminances Vs1 to Vs3 at each vertex of the target triangle TRs (S660: NO), the processor 210 determines the result as dissimilar in S690. If the magnitude relationship of the luminances Vt1 to Vt3 at each vertex of the reference triangle TRt is equal to the magnitude relationship of the luminances Vs1 to Vs3 at each vertex of the target triangle TRs (S660: YES), the processor 210 determines the result as similar in S680. Once the result is determined to be either similar or dissimilar, the similarity determination process is terminated. As can be seen from the above explanation, the conditions in S640 and S660 are specific conditions for determining whether the reference feature point set TtS and the target feature point set TsS are similar or not.

[0085] Once the similarity determination process is complete, in S445 of Figure 7, the processor 210 determines whether the similarity determination result is similar or dissimilar. If the determination result is dissimilar (S445: NO), the processor 210 terminates the loop process in S42 without executing S450. Therefore, in this case, the first pair of interest and the second pair of interest are not determined to be a matching pair, and the loop process S42 is terminated. This is because, if the determination result is dissimilar, it is considered that the first pair of interest and the second pair of interest do not show a correct correspondence between images IMt and IMs, respectively.

[0086] If the determination results are similar (S445: YES), the processor 210 determines the first pair of interest (the pair of the first reference feature point Tt1 and the first target feature point Ts1) and the second pair of interest (the pair of the second reference feature point Tt2 and the second target feature point Ts2) as the final matching pairs in S450, and terminates the loop process S42. This is because, if the determination results are similar, the first pair of interest and the second pair of interest are considered to represent the correct correspondence between images IMt and IMs, respectively.

[0087] Thereafter, the processor 210 repeatedly performs loop processing S42, which is executed while sequentially changing the second pair of interest. After loop processing S42 has been executed for all candidate pairs that include the reference feature point Tt belonging to the point cloud N, the processor 210 changes the first pair of interest and executes the next loop processing S41. After loop processing S41 has been executed with all candidate pairs as the first pair of interest, the processor 210 proceeds to S460.

[0088] In S460, the processor 210 stores the determined matching pair data in the storage device 215 (for example, the non-volatile storage device 230) and terminates the matching pair determination process.

[0089] Figure 6(B) shows an example of matching pairs determined by the matching pair determination process. Multiple lines ML in the figure each represent a matching pair. Compared to Figure 6(A), feature point pairs showing incorrect correspondences have been removed.

[0090] Returning to Figure 4, let's continue the explanation. Once the matching pair determination process is complete, in S170, the processor 210 determines the correspondence between coordinates on the reference image IMt and coordinates on the target image IMs based on the multiple matching pairs determined by the matching pair determination process. Specifically, the processor 210 determines the affine transformation matrix Mtx based on the multiple matching pairs.

[0091] Figure 12 is an explanatory diagram of the affine transformation matrix Mtx. It shows the coordinates COt on the reference image IMt, the coordinates COs on the target image IMs, and the affine transformation matrix Mtx that associates these coordinates COt and COs. In the figure, the coordinates COt and COs and the matrix Mtx are represented in a homogeneous coordinate system. Each coordinate COt and COs is represented as a three-dimensional vector having positions Xt and Xs in direction Dx on images IMt and IMs, positions Yt and Ys in direction Dy, and a third component of 1. The affine transformation matrix Mtx is a 3x3 matrix. As shown in the figure, the affine transformation matrix Mtx is represented by six parameters a~f in a 2x3 grid and three components (0,0,1) in the third row. Such an affine transformation matrix Mtx can represent rotation, scaling, reduction, translation, and skew. The affine transformation matrix Mtx can be calculated according to known methods by using three or more pairs of coordinates COt and COs, i.e., three or more matching pairs.

[0092] In S180 of Figure 4, the processor 210 inspects the printed image IMp. The inspection method may be one of various methods that use the coordinate correspondence determined in S180. In this embodiment, the processor 210 generates a difference image IMd between the reference image IMt and the target images IMs.

[0093] The orientation of the T-shirt 700 relative to the reading sensor 180 (Figure 2) can vary. Therefore, the orientation of objects in the images (i.e., the angle of rotation of objects) may differ between the reference image IMt and the target image IMs. Also, the size of objects in the images may differ between the reference image IMt and the target image IMs. For this reason, when generating the difference image IMd between the reference image IMt and the target image IMs, it is necessary to adjust the position and size of at least one of the reference image IMt and the target image IMs so that they overlap precisely. To this end, the processor 210 uses the affine transformation matrix Mtx in Figure 12 to transform the coordinate system of the reference image IMt to the coordinate system of the target image IMs, thereby adjusting the orientation and size of the reference image IMt before generating the difference image IMd.

[0094] Figure 13 shows an example of a difference image IMd. Figure 13(A) shows difference image IMd1, which is generated when the printed image IMp does not contain defects, and Figure 13(B) shows difference image IMd2, which is generated when the printed image IMp contains defects. The difference image IMd represents the difference in color values ​​(e.g., the absolute difference in luminance values) between the reference image IMt in Figure 3(A) and the target image IMs in Figure 3(B), pixel by pixel. If the printed image IMp does not contain defects, difference image IMd1 does not include the portion showing a large difference (Figure 13(A)). If the printed image IMp contains defects, difference image IMd2 includes the portion Err (the portion showing a large difference) corresponding to the defect (Figure 13(B)).

[0095] In step S190 of Figure 4, the processor 210 outputs the inspection results. There are various methods for outputting the inspection results. In this embodiment, the processor 210 displays the difference image IMd on the display unit 240 (Figure 1). By observing the difference image IMd displayed on the display unit 240, the operator can easily recognize defects in the printed image IMp. Alternatively, the processor 210 may output data representing the inspection results to a storage device (for example, a non-volatile storage device 230, or an external storage device connected to the image processing device 200). After S190, the processor 210 terminates the inspection process.

[0096] According to the embodiment described above, the processor 210 determines multiple candidate pairs using the feature quantities Ft of each of the multiple reference feature points Tt of the reference image IMt and the feature quantities Fs of each of the multiple target feature points Ts of the target image IMs (S150 in Figure 4, Figure 5). From the multiple candidate pairs determined, the processor 210 selects two candidate pairs as focus pairs (first focus pair and second focus pair) (S410 in Figure 7, S420). The processor 210 determines the reference reference point Rt corresponding to the two focus reference feature points Tt1 and Tt2 (i.e., the reference feature point set TtS) that constitute the two candidate pairs (first focus pair and second focus pair), and the target reference point Rs corresponding to the two focus target feature points Ts1 and Ts2 (i.e., the target feature point set TsS) that constitute the two candidate pairs (first focus pair and second focus pair) (S425 in Figure 7, Figure 9). The reference point Rt is determined independently of the target image IMs based on two corresponding reference feature points Tt1 and Tt2. The target reference point Rs is determined independently of the reference image IMt based on two corresponding target feature points Ts1 and Ts2. The processor 210 determines two candidate pairs (first focus pair and second focus pair) as matching pairs if they satisfy certain conditions (YES at S640 and S660 in Figure 11) (YES at S680 in Figure 11, and S445 and S450 in Figure 4). The specific conditions are determined based on the feature angles θt1~θt3 and Vt1~Vt3, which are feature quantities calculated using the reference triangle TRt (two notable reference feature points Tt1 and Tt2 and a reference reference point Rt), and the feature angles θt1~θt3 and Vt1~Vt3, which are feature quantities calculated using the target triangle TRs (two notable target feature points Ts1 and Ts2 and a target reference point Rs) (S630, S660 in Figure 11). The processor 210 uses the determined matching pairs to determine the correspondence between coordinates on the reference image IMt and coordinates on the target image IMs (S170 in Figure 4). As a result, matching pairs can be determined with higher accuracy than when the reference reference point Rt and target reference point Rs are not used, so the positional relationship between the reference image IMt and the target image IMs can be appropriately identified.Furthermore, since the reference point Rt is determined independently of the target image IMs based only on two corresponding reference feature points Tt1 and Tt2, and the target reference point Rs is determined independently of the reference image IMt based only on two corresponding target feature points Ts1 and Ts2, the processing load required to determine the positional relationship between the reference image IMt and the target image IMs can be reduced.

[0097] For example, in the matching pair determination process, the number of candidate pairs to be judged in one loop process S42 (hereinafter also referred to as the number of candidate pairs of interest n) is 2 (the first pair of interest and the second pair of interest) in the above embodiment (n=2). If the number of candidate pairs of interest n is increased to 3 or more, the number of feature points Tt and Ts included in the reference feature point set TtS and the target feature point set TsS can be increased. This may allow for a more precise determination of whether or not a candidate pair is a matching pair. However, if the number of candidate pairs of interest n is increased, the number of combinations of candidate pairs to be judged increases, which may lead to an excessively high processing load. On the other hand, if the number of candidate pairs of interest n is relatively small, as in the above embodiment, it may be possible to determine whether or not a candidate pair is a matching pair using only the reference reference feature points Tt1 and Tt2 and the target feature points Ts1 and Ts2, without determining the reference reference point Rt and the target reference point Rs. In this case, for example, the accuracy of determining whether or not to classify a pair as a matching pair may decrease, increasing the possibility of incorrectly classifying a candidate pair as a matching pair. According to this embodiment, a reference reference point Rt and a target reference point Rs are determined based on only two feature points, which can be easily determined with relatively low processing load. Then, in addition to the reference feature points of interest Tt1 and Tt2 and the target feature point Ts1, the reference reference point Rt and the target reference point Rs are used to determine whether or not to classify a candidate pair as a matching pair. Therefore, it is possible to suppress the decrease in accuracy of determining whether or not to classify a pair as a matching pair while also suppressing the increase in processing load. Consequently, as described above, the positional relationship between the reference image IMt and the target image IMs can be appropriately identified, and the processing load for identifying the positional relationship between the reference image IMt and the target image IMs can be reduced.

[0098] Furthermore, according to this embodiment, the processor 210 determines a reference reference point Rt on a line passing through the first reference feature point Tt1 and extending in the first specific direction Drt (Figure 9(A)). The processor 210 determines a target reference point Rs on a line passing through the target feature point Ts1 and extending in the second specific direction Drs (Figure 9(B)). As a result, the reference reference point Rt can be easily determined based on the first reference feature point Tt1, and the target reference point Rs can be easily determined based on the first target feature point Ts1.

[0099] More specifically, the first specific direction Drt is the direction in which the angle αt formed with the line segment L2t connecting the first reference feature point Tt1 and the second reference feature point Tt2 is a predetermined specific angle (90 degrees in this embodiment) (Figure 9(A)). The second specific direction Drs is the direction in which the angle αs formed with the line segment L1s connecting the first target feature point Ts1 and the second target feature point Ts2 is the same specific angle (90 degrees in this embodiment) (Figure 9(B)). As a result, the reference reference point Rt can be easily determined based on the two reference feature points Tt1 and Tt2, and the target reference point Rs can be easily determined based on the two target feature points Ts1 and Ts2, thereby further reducing the processing load.

[0100] Furthermore, according to this embodiment, the features used to determine whether a candidate pair is a matching pair include angles θt3 and θs3, which are features based on the feature directions Dt3 and Ds3 of the reference points Rt and Rs (S620 and S630 in Figure 11). That is, the features of the reference triangle TRt include the feature angle θt3 based on the feature direction Dt3 of the reference reference point Rt, which is calculated using the image of the local region containing the reference reference point Rt from the reference image IMt (Figure 9(A)). The features of the target triangle TRs include the feature angle θs3 based on the feature direction Ds3 of the target reference point Rs, which is calculated using the image of the local region containing the target reference point Rs from the target image IMs (Figure 9(B)). As a result, the accuracy of determining whether the reference triangle TRt and the target triangle TRs are similar can be improved using the feature directions Dt3 and Ds3 of the reference points Rt and Rs, so that a candidate pair can be determined with high accuracy.

[0101] Furthermore, according to this embodiment, the features used to determine whether a candidate pair is a matching pair include luminance Vt3 and Vs3, which are features based on the pixel values ​​of the reference points Rt and Rs (S650 in Figure 11). That is, the features of the reference triangle TRt include luminance Vt3, which is calculated based on the pixel value (e.g., RGB value) where the reference reference point Rt is located. The features of the target triangle TRs include luminance Vs3, which is calculated based on the pixel value where the target reference point Rs is located. As a result, the accuracy of determining whether the reference triangle TRt and the target triangle TRs are similar can be improved using the pixel values ​​of the reference points Rt and Rs, so that a candidate pair can be accurately determined as a matching pair.

[0102] Furthermore, according to this embodiment, the processor 210 determines the reference reference point Rt and the target reference point Rs such that the first ratio (B / A) is equal to the second ratio (D / C) (Figure 9). The first ratio (B / A) is the ratio of the length B of the line segment L2t connecting the first reference reference point Tt1 and the reference reference point Rt to the length A of the line segment L1t connecting the first reference reference point Tt1 and the second reference reference point Tt2. The second ratio (D / C) is the ratio of the length D of the line segment L2s connecting the first target feature point Ts1 and the target reference point Rs to the length C of the line segment L1s connecting the first target feature point Ts1 and the second target feature point Ts2. As a result, even if the scales (magnifications) of the size of the target image IMs and the size of the reference image IMt are different, appropriate reference reference point Rt and target reference point Rs can be determined. Therefore, by using an appropriate reference point Rt and target reference point Rs, it is possible to accurately determine whether or not a candidate pair is a matching pair.

[0103] As can be seen from the above explanation, the feature angles θt1~θt3 and luminance Vt1~Vt3 in this embodiment are examples of first features, and the feature angles θs1~θs3 and luminance Vs1~Vs3 are examples of second features. The first specific direction Drt in this embodiment is an example of the first direction, and the second specific direction Drs is an example of the second direction.

[0104] B. Second Example In the second embodiment, the method for determining the reference reference point and the target reference point differs from that of the first embodiment. Furthermore, in the second embodiment, the content of the validity determination process and the similarity determination process differs from that of the first embodiment. Other configurations of the first embodiment are the same as those of the first embodiment.

[0105] Figure 14 is an explanatory diagram of the feature point set and reference point of the second embodiment. Figure 14(A) shows an example of a reference feature point set TtS similar to that of the first embodiment, and a reference reference point Rtb of the second embodiment. The reference reference point Rtb of the second embodiment is determined based on the first notable reference feature point Tt1 and the second notable reference feature point Tt2 that constitute the reference feature point set TtS. Specifically, the reference reference point Rtb is determined on a line that passes through the first notable reference feature point Tt1 and extends in the feature direction Dt1 of the first notable reference feature point Tt1. The feature direction Dt1 has already been calculated in S140 of Figure 4. Furthermore, the reference reference point Rtb of the second embodiment is determined such that the ratio (Bb / Ab) of the length Bb of the line segment L2tb connecting the first notable reference feature point Tt1 and the reference reference point Rtb to the length Ab of the line segment L1tb is a predetermined specific value (0.5 (50%) in this embodiment). Thus, the reference reference point Rtb is determined based solely on the reference feature point set TtS of the reference image IMt, independently of the target image IMs, similar to the reference reference point Rt in the first embodiment.

[0106] Figure 14(B) shows an example of a target feature point set TsS similar to that of the first embodiment, and a target reference point Rsb of the second embodiment. The target reference point Rsb of the second embodiment is determined based on the first target feature point Ts1 and the second target feature point Ts2 that constitute the target feature point set TsS. Specifically, the target reference point Rsb is determined on a line that passes through the first target feature point Ts1 and extends in the feature direction Ds1 of the first target feature point Ts1. In this embodiment, the feature direction Ds1 has already been calculated in S140 of Figure 4. Furthermore, the reference reference point Rtb of the second embodiment is determined such that the ratio (Db / Cb) of the length Db of the line segment L2sb connecting the first target feature point Ts1 and the target reference point Rsb to the length Cb of the line segment L1sb is the specific value described above (0.5 (50%) in this embodiment). Thus, the target reference point Rsb is determined based solely on the target feature point set TsS of the target image IMs, independently of the reference image IMt, similar to the target reference point Rs in the first embodiment.

[0107] As can be seen from the above explanation, in this embodiment, similar to the first embodiment, the first ratio (Bb / Ab) for determining the reference point Rt and the second ratio (Db / Cb) for determining the target reference point Rs are equal ((Bb / Ab)=(Db / Cb)). On the other hand, in this embodiment, the angle θt1 formed by the feature direction Dt1 for determining the reference reference point Rtb and the line segment L1tb, and the angle θs1 formed by the feature direction Ds1 for determining the target reference point Rsb and the line segment L1sb are not necessarily equal. For this reason, the triangle formed by the three points of interest, the reference feature points Tt1 and Tt2 and the reference reference point Rtb (Figure 14(A), also called the reference triangle TRtb), and the triangle formed by the three points of interest, the target feature points Ts1 and Ts2 and the target reference point Rsb (Figure 14(B), also called the target triangle TRsb), are not necessarily similar.

[0108] Next, the validity determination process of the second embodiment will be described. In the validity determination process, the validity of three points in the reference image IMt (the reference feature points of interest Tt1 and Tt2, and the reference reference point Rtb mentioned above) and three points in the target image IMs (the target feature points of interest Ts1 and Ts2, and the target reference point Rsb mentioned above) is determined. Figure 15 is a flowchart of the validity determination process of the second embodiment. In Figure 15, processes that are the same as those in Figure 10 of the first embodiment are denoted by the same reference numerals as in Figure 10, and processes that are different from those in Figure 10 are denoted by the letter "B" at the end of their reference numerals.

[0109] In the validity determination process shown in Figure 15, steps S550B and S560B are added between steps S510-S540 and S570 in Figure 10 of the first embodiment. The other processes in the validity determination process of Figure 15 are the same as those in the validity determination process of Figure 10. The added steps S550B and S560B will be explained below.

[0110] In S540 of Figure 10, if the reference points Rt and Rs are within the image (S540: YES), the processor 210 determines in S550B whether the interior angles of triangles TRtb and TRsb are within an acceptable range. For example, if the interior angle of a specific vertex of the reference triangle TRtb is within a predetermined acceptable range, and the interior angle of the vertex corresponding to that specific vertex of the target triangle TRsb is within a predetermined acceptable range, then it is determined that the interior angles of triangles TRtb and TRsb are within an acceptable range. In this embodiment, it is determined whether the interior angle θt1 of the first reference feature point Tt1 of the three vertices of the reference triangle TRtb (Figure 14(A)) is between 45 degrees and 135 degrees. It is determined whether the interior angle θs1 of the first target feature point Ts1 of the three vertices of triangle TRsb (Figure 14(B)) is between 45 degrees and 135 degrees. Furthermore, if both the interior angle θt1 of the first reference feature point Tt1 and the interior angle θs1 of the first target feature point Ts1 are between 45 degrees and 135 degrees, then the interior angles of triangles TRtb and TRsb are judged to be within the acceptable range. If one interior angle is excessively small (for example, close to 0), even a slight shift in the reference points Rtb and Rsb will cause that one interior angle to change significantly. Also, if one interior angle is excessively large (for example, close to 180 degrees), the other interior angles will become excessively small. Therefore, even a slight shift in the reference points Rtb and Rsb will cause the other interior angles to change significantly. In this case, it may be difficult to accurately determine whether triangles TRtb and TRsb are similar or not.

[0111] If the interior angles of triangles TRtb and TRsb are not within the acceptable range (S550B: NO), processor 210 determines the result to be invalid in S580. If the interior angles of triangles TRtb and TRsb are within the acceptable range (S550B: YES), processor 210 proceeds to S560B.

[0112] In the S560B, the processor 210 determines whether the lengths of the three sides of triangles TRtb and TRsb are different. Specifically, it determines whether the three sides L1tb, L2tb, and L3tb of the reference triangle TRtb are different from each other; in other words, whether the reference triangle TRtb is not an equilateral triangle and is not an isosceles triangle. For each pair of sides L1tb, L2tb, L1tb, L3tb, and L2tb, the processor 210 determines whether the absolute value of the difference between the lengths of the two sides is greater than or equal to a predetermined threshold. If the absolute value of the difference is greater than or equal to the predetermined threshold for all pairs of sides, it is determined that the lengths of the three sides of the reference triangle TRtb are different from each other. If the absolute value of the difference is less than the predetermined threshold for at least one pair of sides, it is determined that the lengths of at least two sides of the reference triangle TRtb are equal. The same determination is made for the target triangle TRsb.

[0113] Equilateral and isosceles triangles have a higher degree of symmetry compared to triangles with sides of different lengths. For this reason, if triangles TRtb and TRsb are equilateral or isosceles, they may be judged as similar even if their vertices show an incorrect correspondence. For example, this erroneous judgment is likely to occur when the reference image IMt and the target image IMs contain objects with symmetry.

[0114] If at least two sides of the reference triangle TRtb or the target triangle TRsb are equal in length (S560B: NO), the processor 210 determines the result to be invalid in S580. If the three sides of triangles TRtb and TRsb are different in length (S560B: YES), the processor 210 determines the result to be valid in S570. Once the result is determined to be either valid or invalid, the valid determination process in Figure 15 is terminated.

[0115] As can be seen from the above explanation, in the second embodiment, when determining the reference points Rtb and Rsb, the feature directions Dt1 and Ds1 of the feature points Tt1 and Ts1 are used. For this reason, unlike the first embodiment, the interior angles of triangles TRtb and TRsb may change, and triangles TRtb and TRsb may have an undesirable shape for the similarity determination process. For this reason, in the second embodiment, S550B and S560B are added in the validity determination process, and if triangles TRtb and TRsb have an undesirable shape for the similarity determination process, triangles TRtb and TRsb are determined to be invalid.

[0116] Next, the similarity determination process of the second embodiment will be described. The similarity determination process determines whether three points in the reference image IMt (the reference feature points of interest Tt1 and Tt2 and the reference reference point Rtb) and three points in the target image IMs (the target feature points of interest Ts1 and Ts2 and the target reference point Rsb) are similar. Figure 16 is a flowchart of the similarity determination process of the second embodiment. In Figure 16, processes that are the same as those in Figure 11 of the first embodiment are denoted by the same reference numerals as in Figure 11, and processes that are different from those in Figure 11 are denoted by the letter "B" at the end of their reference numerals.

[0117] In the similarity determination process shown in Figure 16, step S610 in Figure 11 is not executed. That is, the processor 210 does not calculate the feature directions of the two reference points Rtb and Rsb.

[0118] In S620B, processor 210 calculates the feature angles θt1 and θt2 of the two vertices of the reference triangle TRtb. Feature angle θt1 is the angle formed by the feature direction Dt1 of the first reference feature point Tt1 and the line segment L1t connecting the first reference feature point Tt1 and the second reference feature point Tt2, as explained in the first embodiment (Figure 14(A)). Feature angle θt2 is the angle formed by the feature direction Dt2 of the second reference feature point Tt2 and the line segment L1t (Figure 14(A)). The feature directions Dt1 and Dt2 of the reference feature points Tt1 and Tt2 have already been calculated in S140 in Figure 4. In this embodiment, as mentioned above, the feature direction Dt3b of the reference reference point Rtb is not calculated, and therefore the feature angle θt3b of the reference reference point Rtb is also not calculated.

[0119] In S630B, the processor 210 calculates the characteristic angles θs1 and θs2 of two vertices of the target triangle TRsb. As described in the first embodiment, the characteristic angle θs1 is the angle formed by the characteristic direction Ds1 of the first target characteristic point Ts1 and the line segment L1s connecting the first target characteristic point Ts1 and the second target characteristic point Ts2 (FIG. 14(B)). The characteristic angle θs2 is the angle formed by the characteristic direction Ds2 of the second target characteristic point Ts2 and the line segment L1s (FIG. 14(B)). The characteristic directions Ds1 and Ds2 of the target characteristic points Ts1 and Ts2 have already been calculated in S140 of FIG. 4. In this embodiment, as described above, since the characteristic direction Ds3b of the target reference point Rsb is not calculated, the characteristic angle θs3b of the target reference point Rsb is not calculated either.

[0120] In S640B, the processor 210 determines whether the characteristic angles θt1 and θt2 of the reference triangle TRtb and the characteristic angles θs1 and θs2 of the target triangle TRsb are approximately equal to each other. Specifically, when |θt1 - θs1| < THn and |θt2 - θs2| < THn, it is determined that the characteristic angles θt1 and θt2 of the reference triangle TRtb and the characteristic angles θs1 and θs2 of the target triangle TRsb are approximately equal to each other. When |θt1 - θs1| ≥ THn or |θt2 - θs2| ≥ THn, it is determined that the characteristic angles θt1 and θt2 of the reference triangle TRtb and the characteristic angles θs1 and θs2 of the target triangle TRsb are not approximately equal to each other.

[0121] When the characteristic angles θt1 and θt2 of the reference triangle TRtb and the characteristic angles θs1 and θs2 of the target triangle TRsb are not approximately equal to each other (S640B: NO), the processor 210 determines the determination result as non-similar in S690. When the characteristic angles θt1 and θt2 of the reference triangle TRtb and the characteristic angles θs1 and θs2 of the target triangle TRsb are approximately equal to each other (S640B: YES), the processor 210 proceeds to S650.

[0122] The S650 and S660 in Figure 16 represent the same processing as the S650 and S660 in Figure 11. Specifically, in S650, the processor 210 obtains the brightness Vt1 to Vt3 of each vertex of the reference triangle TRtb and the brightness Vs1 to Vs3 of each vertex of the target triangle TRs. In S660, the processor 210 determines whether the magnitude relationship of the brightness Vt1 to Vt3 of each vertex of the reference triangle TRtb is equal to the magnitude relationship of the brightness Vs1 to Vs3 of each vertex of the target triangle TRsb.

[0123] If the relative magnitudes of the luminances Vt1 to Vt3 at each vertex of the reference triangle TRtb are not equal to the relative magnitudes of the luminances Vs1 to Vs3 at each vertex of the target triangle TRsb (S660: NO), the processor 210 determines the result as dissimilar in S690. If the relative magnitudes of the luminances Vt1 to Vt3 at each vertex of the reference triangle TRtb are equal to the relative magnitudes of the luminances Vs1 to Vs3 at each vertex of the target triangle TRsb (S660: YES), the processor 210 proceeds to S665B.

[0124] In the S665B, processor 210 calculates the interior angles Zt2 and Zt3 of the reference triangle TRtb, and the interior angles Zs2 and Zs3 of the target triangle TRsb. Interior angle Zt2 is the interior angle with the second reference feature point Tt2 as its vertex, and interior angle Zt3 is the interior angle with the reference reference point Rtb as its vertex. The interior angle Zt1 with the first reference feature point Tt1 as its vertex is not calculated because it is equal to the feature angle θt1.

[0125] In S670B, the processor 210 determines whether the interior angles Zt2 and Zt3 of the reference triangle TRtb and the interior angles Zs2 and Zs3 of the target triangle TRsb are approximately equal to each other. Specifically, when |Zt2 - Zs2| < THn and |Zt3 - Zs3| < THn, it is determined that the interior angles Zt2 and Zt3 of the reference triangle TRtb and the interior angles Zs2 and Zs3 of the target triangle TRsb are approximately equal to each other. When |Zt2 - Zs2| ≥ THn or |Zt3 - Zs3| ≥ THn, it is determined that the interior angles Zt2 and Zt3 of the reference triangle TRtb and the interior angles Zs2 and Zs3 of the target triangle TRsb are not approximately equal to each other. The threshold THn is determined experimentally while considering, for example, calculation errors and the like.

[0126] When the interior angles Zt2 and Zt3 of the reference triangle TRtb and the interior angles Zs2 and Zs3 of the target triangle TRsb are not approximately equal to each other (S670B: NO), the processor 210 determines the determination result as non - similar in S690. When the interior angles Zt2 and Zt3 of the reference triangle TRtb and the interior angles Zs2 and Zs3 of the target triangle TRsb are approximately equal to each other (S670B: YES), the processor 210 determines the determination result as similar in S680. When the determination result is determined to be either similar or non - similar, the similarity determination process ends. As can be understood from the above description, the conditions of S640B, S660, and S670B can be said to be specific conditions for determining whether the reference feature point set TtSb and the target feature point set TsSb are similar.

[0127] According to the second embodiment described above, the processor 210 determines a reference reference point Rtb on a line that passes through the first reference feature point Tt1 and extends in the feature direction Dt1 of the reference feature point Tt1 (Figure 14(A)). The processor 210 determines a target reference point Rsb on a line that passes through the target feature point Ts1 and extends in the feature direction Ds1 of the first target feature point Ts1 (Figure 14(B)). As a result, an appropriate reference reference point Rtb can be determined based on the feature direction Dt1, which is a local feature of the first reference feature point Tt1, and an appropriate target reference point Rsb can be determined based on the feature direction Ds1, which is a local feature of the first target feature point Ts1. Furthermore, since the feature directions Dt1 and Ds1 have already been calculated in S140 in Figure 4, the reference points Rtb and Rsb can be easily determined, thus further reducing the processing load.

[0128] Furthermore, according to this embodiment, the three points of interest Tt1, Tt2, and the reference point Rtb are determined to be valid only if the lengths of the three sides of the reference triangle TRtb formed by these three points are different, and are used in the similarity determination process (S560B in Figure 15). In other words, a valid reference point Rtb is determined such that the lengths of the three sides of the reference triangle TRtb are different. Similarly, a valid target reference point Rsb is determined such that the lengths of the three sides of the target triangle TRsb are different (S560B in Figure 15). As a result, as described above, since triangles TRtb and TRsb do not have symmetry, it is possible to suppress the incorrect determination of candidate pairs showing an incorrect correspondence as matching pairs.

[0129] Furthermore, according to this embodiment, the features used to determine whether a candidate pair is a matching pair include two of the angles of the three vertices of the reference triangle TRtb (interior angles Zt2, Zt3) and two of the angles of the three vertices of the target triangle TRsb (interior angles Zs2, Zs3). If the first and second pairs of interest show a correct correspondence, the reference triangle TRtb and the target triangle TRsb are considered to be similar. For this reason, if the first and second pairs of interest show a correct correspondence, the angles of the three vertices of the two triangles TRtb and TRsb are considered to be equal to each other. For this reason, by using at least one of the angles of the three vertices of the two triangles TRtb and TRsb as a feature, it is possible to appropriately determine whether the first and second pairs of interest are determined to be matching pairs.

[0130] Furthermore, according to this embodiment, unlike the first embodiment, the features used to determine whether a candidate pair is a matching pair do not include features based on the feature direction Dt3b of the reference reference point Rtb (e.g., feature angle θt3b (Figure 14(A))) and do not include features based on the feature direction Ds3b of the target reference point Rsb (e.g., feature angle θs3b (Figure 14(B))). As a result, the processor 210 does not need to calculate the feature direction Dt3b of the reference reference point Rtb and the feature direction Ds3b of the target reference point Rsb, thus further reducing the processing load. Also, since the reference points Rt and Rs are not points set according to the feature point detection algorithm, nor are they candidate pairs matched in the candidate pair determination process, the feature directions Dt3b and Ds3b may not be stable. In this embodiment, even if the feature directions Dt3b and Ds3b are unstable, an appropriate matching pair can be determined.

[0131] As can be seen from the above explanation, the feature angles θt1, θt2, luminances Vt1~Vt3, and interior angles Zt2, Zt3 in this embodiment are examples of first features, the feature angles θs1, θs2, luminances Vs1~Vs3, and interior angles Zs2, Zs3 are examples of second features. The feature direction Dt1 of the first reference feature point Tt1 in this embodiment is an example of the first direction, and the feature direction Ds1 of the first target feature point Ts1 is an example of the second direction.

[0132] C. Variations (1) The similarity determination process in each of the above embodiments is an example and can be modified as appropriate. Figure 17 is a flowchart of the similarity determination process of a modified example. This modified example of the similarity determination process is performed in place of the similarity determination process in Figure 16 of the second embodiment. Other processes in this modified example are the same as those in the second embodiment. In Figure 17, processes that are the same as those in Figure 16 of the second embodiment are denoted by the same reference numerals as in Figure 16, and processes that are different from those in Figure 16 are denoted by the letter "C" at the end of their reference numerals.

[0133] In the similarity determination process shown in Figure 17, steps S620B to S640B in Figure 16 are not executed. In other words, in the modified example, similarity determination using feature angles θt1, θt2, θs1, and θs2 is not performed.

[0134] The S650 and S660 in Figure 17 represent the same processing as the S650 and S660 in Figure 16. That is, in S650, the processor 210 obtains the brightness Vt1 to Vt3 of each vertex of the reference triangle TRtb and the brightness Vs1 to Vs3 of each vertex of the target triangle TRs. In S660, the processor 210 determines whether the magnitude relationship of the brightness Vt1 to Vt3 of each vertex of the reference triangle TRtb is equal to the magnitude relationship of the brightness Vs1 to Vs3 of each vertex of the target triangle TRsb.

[0135] If the relative magnitudes of the luminances Vt1 to Vt3 at each vertex of the reference triangle TRtb are not equal to the relative magnitudes of the luminances Vs1 to Vs3 at each vertex of the target triangle TRsb (S660: NO), the processor 210 determines the result as dissimilar in S690. If the relative magnitudes of the luminances Vt1 to Vt3 at each vertex of the reference triangle TRtb are equal to the relative magnitudes of the luminances Vs1 to Vs3 at each vertex of the target triangle TRsb (S660: YES), the processor 210 proceeds to S665C.

[0136] In S665C, the processor 210 calculates the ratio LRt of two sides L1tb and L3tb of the reference triangle TRtb and the ratio LRs of two sides L1sb and L3sb of the target triangle TRsb. The ratio LRt is, for example, the value obtained by dividing the length Eb of side L3tb of the reference triangle TRtb by the length Ab of side L1tb (LRt = (Eb / Ab), Fig. 14(A)). The ratio LRs is, for example, the value obtained by dividing the length Fb of side L3sb of the target triangle TRsb by the length Cb of side L1sb (LRs = (Fb / Cb), Fig. 14(B)).

[0137] In a modified example, the ratio LRt may be the ratio (Eb / Bb) of two other sides L2tb and L3tb of the reference triangle TRtb, or may be the ratio (Fb / Db) of two other sides L2sb and L3sb of the reference triangle TRtb.

[0138] In S670C, the processor 210 determines whether the ratio LRt of two sides L1tb and L3tb of the reference triangle TRtb and the ratio LRs of two sides L1sb and L3sb of the target triangle TRsb are approximate to each other. Specifically, when the absolute value of the difference between the two ratios LRt and LRs is less than a predetermined threshold THr (|LRt − LRs| < THr), it is determined that the two ratios LRt and LRs are approximate to each other. When the absolute value of the difference between the two ratios LRt and LRs is greater than or equal to the predetermined threshold THr (|LRt − LRs| ≧ THr), it is determined that the two ratios LRt and LRs are not approximate to each other. The threshold THr is determined experimentally, for example, while considering calculation errors and the like.

[0139] When the two ratios LRt and LRs are not approximate to each other (S670C: NO), the processor 210 determines the determination result as non - similar in S690. When the two ratios LRt and LRs are approximate to each other (S670C: YES), the processor 210 determines the determination result as similar in S680. When the determination result is determined to be either similar or non - similar, the similarity determination process ends. As can be understood from the above description, it can be said that the conditions of S660 and S670C are specific conditions for determining whether the reference feature point set TtSb and the target feature point set TsSb are similar.

[0140] According to this modified version, the features used to determine whether a candidate pair is a matching pair include the ratio LRt of two of the three sides of the reference triangle TRtb and the ratio LRs of two of the three sides of the target triangle TRsb. As mentioned above, if the first and second pairs of interest show a correct correspondence, the reference triangle TRtb and the target triangle TRsb are considered to be similar. For this reason, if the first and second pairs of interest show a correct correspondence, the ratios of two sides of the two triangles TRtb and TRsb are considered to be equal. For this reason, by using the ratios of the three sides of the two triangles TRtb and TRsb as features, it is possible to appropriately determine whether the first and second pairs of interest are determined to be a matching pair.

[0141] (2) In the first embodiment, the similarity determination process in Figure 11 performs two types of determination: determination using feature angles θt1~θt3 and θs1~θs3 (S610~S640 in Figure 11) and determination using brightness Vt1~Vt3 and Vs1~Vs3 (S650, S660 in Figure 11). Alternatively, the processor 210 may perform only the determination using feature angles θt1~θt3 and θs1~θs3 (S610~S640 in Figure 11) and omit the determination using brightness Vt1~Vt3 and Vs1~Vs3 (S650, S660 in Figure 11). Alternatively, the processor 210 may omit only the determination using feature angles θt1~θt3 and θs1~θs3 (S610~S640 in Figure 11) and perform only the determination using brightness Vt1~Vt3 and Vs1~Vs3 (S650, S660 in Figure 11).

[0142] Furthermore, the determination of S640 in Figure 11 may be performed using the feature angles θt1, θt2, θs1, θs2 of two vertices of triangles TRt and RTs (reference feature points Tt1, Tt2 and target feature points Ts1, Ts2), without using the feature angles θt3, θs3 of one vertex (reference points Rs, Rt). In this case, a determination may be added to check whether the brightness Vt3 of the reference reference point Rt and the brightness Vt3 of the target reference point Rs are approximate.

[0143] (3) In the second embodiment, the similarity determination process in Figure 16 performs three types of determination: determination using feature angles θt1, θt2, θs1, and θs2 (S620B to S640B in Figure 16), determination using brightness Vt1 to Vt3 and Vs1 to Vs3 (S650 and S660 in Figure 16), and determination using the interior angles Zt2, Zt3, Zs2, and Zs3 of triangles TRtb and TRsb (665B and S670B in Figure 16). In the modified example, one of these three types of determination may be omitted.

[0144] (4) In the second embodiment, the similarity determination process in Figure 16 performs determination using feature angles θt1, θt2, θs1, and θs2 (S620B to S640B in Figure 16). In this process, the feature angles θt3b and θs3b at the reference points Rtb and Rsb (Figure 14) are not used. In a modified example, the processor 210 may calculate the feature directions Dt3b and Ds3b at the reference points Rtb and Rsb (Figure 14), and use these feature directions Dt3b and Ds3b to calculate the feature angles θt3b and θs3b at the reference points Rtb and Rsb. Then, in S640B, a condition may be added to check whether the feature angles θt3b and θs3b are similar to each other. In this case, for example, the determination using luminance values ​​Vt1-Vt3 and Vs1-Vs3 (S650, S660 in Figure 16) and the determination using interior angles Zt2, Zt3, Zs2, Zs3 of triangles TRtb and TRsb (665B, S670B in Figure 16), or both or one of them, may be omitted.

[0145] (5) In the first embodiment described above, the specific angles αt and αs (Figure 9) for determining the reference points Rt and Rs are 90 degrees. However, the specific angles αt and αs are not limited to this and may be other angles. Preferably, the specific angles αt and αs are angles between 45 degrees and 135 degrees. If the specific angles αt and αs are excessively small (for example, close to 0) or excessively large (for example, close to 180 degrees), even a slight shift in the reference points Rt and Rs can easily cause the specific angles αt and αs to fluctuate significantly, making it difficult to determine the reference points Rt and Rs according to the specific angles αt and αs.

[0146] (6) In the above embodiment, luminance Vt3 and Vs3 are used as feature quantities based on the pixel values ​​of the reference points Rt and Rs. However, other values, such as the saturation and hue of the reference points Rt and Rs, may be used as feature quantities based on the pixel values ​​of the reference points Rt and Rs. For example, in S660 in Figure 11, it may be determined whether the relative saturation of each vertex of the reference triangle TRt is equal to the relative saturation of each vertex of the target triangle TRs, or whether the hue of each vertex of the reference triangle TRt approximates the hue of each vertex of the target triangle TRs.

[0147] (7) In the first embodiment described above, the first ratio (B / A) and the second ratio (D / C) of the sides of triangles TRt and TRs used to determine the reference points Rt and Rs are 0.5, or 50%. However, the first and second ratios (D / C) may be other ratios. Preferably, the first and second ratios are between 10% and 200%. If the first and second ratios are excessively small (for example, close to 0%), the reference reference point Rt will be excessively close to the first reference feature point Tt1, and the target reference point Rs will be excessively close to the target feature point Ts1. For this reason, if the first and second ratios are excessively small, the meaning of setting up a reference reference point Rt and a target reference point Rs may diminish. If the ratio of the first to the second ratio is excessively large (for example, exceeding 200%), the reference reference point Rt will be excessively far from the first reference feature point Tt1, and the target reference point Rs will be excessively far from the target feature point Ts1. As a result, when the ratio of the first to the second ratio is excessively large, the reference reference point Rt and the target reference point Rs may fall outside the image, increasing the likelihood of the formation of invalid triangles TRt and TRs.

[0148] In the second embodiment described above, the first ratio (Bb / Ab) and the second ratio (Db / Cb) of the sides of triangles TRtb and TRsb used to determine the reference points Rtb and Rsb are not limited to 50%, but may be other ratios. In the second embodiment as well, it is preferable that the first ratio and the second ratio are between 10% and 200%.

[0149] (8) The number of candidate pairs of interest in the matching pair determination process in each of the above embodiments is 2. However, the number of candidate pairs of interest may be 3 or more, for example, 3 or 4. Even when the number of candidate pairs of interest is 3, the processor 210 may determine the reference points Rs and Rt in the same manner as in the embodiments, for example, based on 2 of the 3 reference feature points of interest.

[0150] In this case, for example, the processor 210 may determine three candidate pairs as matching pairs if the interior angles or feature angles of a triangle consisting of three reference feature points of interest approximate the interior angles or feature angles of a triangle consisting of three target feature points, and the brightness of the reference reference point Rt approximates the brightness of the target reference point Rs. This allows matching pairs to be determined with a lower processing load than when there are four candidate pairs of interest, and with higher accuracy than when there are three candidate pairs of interest and no reference points Rs and Rt are provided.

[0151] (9) In the matching pair determination process of the first embodiment described above, the number of reference reference points Rt determined for one reference feature point set TtS is 1, and the number of target reference points Rs determined for one target feature point set TsS is 1. However, the processor 210 may determine two or more reference points (for example, two each) for both the reference feature point set TtS and the target feature point set TsS. For example, the processor 210 determines the first reference reference point Rt in the reference feature point set TtS, passing through the first reference feature point Tt1 of interest and on a line that forms a 90-degree angle with line segment L1t, similar to the first embodiment. The processor 210 further determines the second reference reference point Rt in the reference feature point set TtS, passing through the second reference feature point Tt2 of interest and on a line that forms a 90-degree angle with line segment L1t. The processor 210 determines the first target reference point Rs in the same manner as in the first embodiment, on a line that passes through the first target feature point Ts1 of the target feature point set TsS and makes a 90-degree angle with line segment L1s. The processor 210 further determines the second target reference point Rs in the target feature point set TsS, on a line that passes through the second target feature point Ts2 of the target feature point set TsS and makes a 90-degree angle with line segment L1t.

[0152] In this case, for example, in addition to the conditions of the first embodiment, conditions can be added for determining candidate pairs as matching pairs, such as a condition to compare the brightness of two reference points Rt with the brightness of two target reference points Rs, or a condition to compare the feature angles of two reference points Rt with the feature angles of two target reference points Rs.

[0153] (10) The methods for determining the reference reference points Rt, Rtb and the target reference points Rs, Rsb in each of the above embodiments are examples only and are not limited thereto. For example, the reference reference point Rt may be determined on a line that passes through a predetermined point (e.g., the midpoint) on the line segment L1t connecting two reference feature points of interest Tt1 and Tt2, and intersects the line segment L1t at a predetermined angle (e.g., 90 degrees). In this case, the target reference point Rs is also determined on a line that passes through a predetermined point on the line segment L1s connecting two target feature points of interest Ts1 and Ts2, and intersects the line segment L1s at a predetermined angle.

[0154] Furthermore, in the first embodiment described above, the reference point Rt is determined such that the ratio of the length B of line segment L2t to the length A of line segment L1t (B / A) is 50% (Figure 9(A)), and the target reference point Rs is determined such that the ratio of the length D of line segment L2s to the length C of line segment L1s (D / C) is 50% (Figure 9(B)). However, this is not limited to this, and for example, if the scale of the reference image IMt and the target image IMs are the same, the reference points Rt and Rs may be determined so that the lengths of line segment L2t and line segment L2s are predetermined fixed values.

[0155] (11) The validity determination process in each of the above embodiments (Figures 10 and 15) is an example and may be omitted or modified as appropriate. For example, steps S510 and S520 in Figures 10 and 15 are confirmation processes for confirmation purposes and may be omitted. Also, all or part of steps S530 in Figure 10 and S550B and S560B in Figure 15 may be omitted as appropriate.

[0156] (12) The method for detecting feature points Tt and Ts in S130 of Figure 4 may be any of the various methods for detecting points that indicate parts of an object in the image, instead of the method described in the above embodiment. The detection method may be selected in advance from, for example, the search for extrema (maximum and minimum values) using DoG (Difference-of-Gaussian), Harris corner detection, FAST (Features from Accelerated Segment Test) corner detection, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF).

[0157] The method for calculating the feature descriptors of the feature quantities Ft and Fs may be any of the various methods used to calculate information describing the features of the feature points, instead of the method described in the above embodiment. The algorithm for calculating the feature descriptor may be selected in advance from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, and A-KAZE. Furthermore, the method for calculating the distance dF between the feature quantities Ft and Fs may be any of the various methods suitable for the data structure of the feature quantities Ft and Fs. When the feature quantities Ft and Fs are represented by binary vectors, the distance dF may be the Hamming distance. The distance dF may be any of the various other distances (for example, L1 norm, L2 norm (also called Euclidean distance), etc.) instead of the Hamming distance. Norms are applicable to various feature descriptors.

[0158] In either case, the direction associated with the feature points Tt and Ts may be the direction used to calculate the feature descriptor. Such a direction may be the direction of the gradient of various color values ​​(e.g., luminance).

[0159] (13) The printing medium is not limited to T-shirts 700, but may be various types of clothing (for example, various shirts such as polo shirts, coats, slacks, etc.). The printing medium may be various fabrics such as clothes and bags. The printing medium is not limited to fabric, but may be various media such as paper, film, leather, etc. Also, the printing device 900 may be a printing device of another type (for example, a laser printing device) instead of an inkjet printing device.

[0160] (14) When determining the coordinate correspondence between images IMt and IMs based on matching pairs, an affine transformation matrix Mtx is calculated (Figure 12). Alternatively, various transformation matrices, such as homography transformation matrices, may be calculated. Furthermore, the coordinate correspondence may be represented in various formats, such as lookup tables, instead of the matrix Mtx.

[0161] The coordinate correspondence determined based on matching pairs may be used in various processes, not just inspection processes. For example, in the machining of metal parts, the position and orientation of the metal part relative to the tool may be misaligned. Here, the position and orientation of the metal part relative to the tool may be determined by analyzing images of the metal part taken by a digital camera fixed to the tool. For example, the processor 210 may use an image representing the reference position of the metal part as a reference image IMt, and images obtained by photographing the metal part as target images IMs, and determine the reference position of the metal part on the captured images (and thus the position and orientation of the metal part relative to the tool) by identifying the positional relationship between images IMt and IMs. As the process for identifying the positional relationship, the processes of each embodiment described above, or the processes of each modified example described above, may be adopted. Thus, the target images IMs may be images representing objects read by a line sensor such as the reading sensor 180 (Figure 2), or images representing objects read by an area sensor such as a digital camera. Furthermore, the reference image IMt may be any image other than the image related to printing (for example, a pre-prepared image representing a specific part of an object). In any case, the image may be a grayscale image instead of a color image.

[0162] (15) The image processing device 200 is not limited to a personal computer, but may be any other device (for example, a smartphone, a tablet computer, a control device incorporated into a reader, etc.). Alternatively, multiple devices (for example, computers) that can communicate with each other via a network may each share a portion of the data processing function of the image processing device and, as a whole, provide the data processing function (a system equipped with these devices corresponds to the image processing device).

[0163] In each of the above embodiments and modifications, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, the process of S290 in the figure may be executed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0164] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0165] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of Symbols]

[0166] 100…Reading device, 110…Control device, 120…Transport device, 122…Position sensor, 130…Table, 140…Support unit, 180…Reading sensor, 190…Housing, 200…Image processing device, 210…Processor, 215…Storage device, 220…Volatile storage device, 230…Non-volatile storage device, 240…Display unit, 250…Operation unit, 270…Communication interface, 900…Printing device, IMs…Target image, IMt…Reference image, Mtx…Affine transformation matrix, Os…Target object, Ot…Reference object, PG…Computer program, Rs, Rsb…Target reference point, Rt, Rtb…Reference reference point, TRs, TRsb…Target triangle, TRt, TRtb…Reference triangle, Ts…Target feature point, Tt…Reference feature point

Claims

1. It is a computer program, A candidate determination function that determines multiple candidate pairs using the respective feature quantities of multiple reference feature points of a reference image and the respective feature quantities of multiple target feature points of a target image, wherein the candidate pair is a candidate matching pair, which is a pair of the reference feature point and the target feature point that corresponds to each other, and the candidate determination function A selection function that selects n candidate pairs (where n is an integer greater than or equal to 2) from among the aforementioned plurality of candidate pairs, A reference point determination function for determining reference reference points of a reference image corresponding to n reference feature points constituting n candidate pairs, and target reference points of a target image corresponding to n target feature points constituting n candidate pairs, wherein the reference reference points are determined independently of the target image based on at least one of the corresponding n reference feature points, and the target reference points are determined independently of the reference image based on at least one of the corresponding n target feature points. A matching pair determination function that determines the n candidate pairs as matching pairs when the n candidate pairs satisfy a specific condition, wherein the specific condition is a condition determined based on a first feature quantity calculated using n reference feature points and reference reference points corresponding to the n reference feature points, and a second feature quantity calculated using n target feature points and target reference points corresponding to the n target feature points, A correspondence determination function that determines the correspondence between coordinates on the reference image and coordinates on the target image using the determined plurality of matching pairs, A computer program that enables a computer to realize something.

2. A computer program according to claim 1, The aforementioned reference point determination function is, A reference reference point is determined to lie on a first line that passes through a first reference feature point among the n reference feature points and extends in a first direction, A computer program that determines the target reference point on a second line that passes through a first target feature point among the n target feature points and extends in a second direction.

3. A computer program according to claim 2, The first direction is the direction in which the angle formed by the line connecting the first reference feature point and the second reference feature point among the n reference feature points is a predetermined specific angle. A computer program in which the second direction is the direction in which the angle formed by the line connecting the first target feature point and the second target feature point among the n target feature points is the specific angle.

4. A computer program according to claim 3, A computer program in which the aforementioned specific angle is an angle between 45 degrees and 135 degrees.

5. A computer program according to claim 2, The first direction is the characteristic direction of the first reference feature point, calculated using an image of the local region containing the first reference feature point from the reference image. A computer program in which the second direction is the characteristic direction of the first target feature point calculated using an image of a local region of the target image that includes the first target feature point.

6. A computer program according to claim 2, The reference point determination function is a computer program that determines the reference reference point and the target reference point such that the first ratio of the length of the line segment connecting the first reference reference point and the reference reference point to the length of the line segment connecting the first reference reference point and the second reference reference point among the n reference feature points is equal to the second ratio of the length of the line segment connecting the first target feature point and the target reference point to the length of the line segment connecting the first target feature point and the second target feature point among the n target feature points.

7. A computer program according to claim 6, A computer program in which the first ratio and the second ratio are between 10% and 200%.

8. A computer program according to claim 2, The aforementioned reference point determination function is, The reference reference point is determined such that the lengths of the three sides of the triangle formed by the first reference feature point, the second reference feature point, and the reference reference point are all different from each other. A computer program that determines the target reference point such that the lengths of the three sides of the triangle formed by the first target feature point, the second target feature point, and the target reference point are all different from each other.

9. A computer program according to claim 2, The first feature quantity includes a feature quantity based on the feature direction of the reference reference point, calculated using an image of a local region of the reference image that includes the reference reference point. A computer program that includes a second feature quantity, which is calculated using an image of a local region of the target image that includes the target reference point, and is based on the feature direction of the target reference point.

10. A computer program according to claim 2, The first feature includes a feature based on the pixel value of the reference point, A computer program in which the second feature quantity includes a feature quantity based on the pixel value of the target reference point.

11. A computer program according to claim 5, The first feature quantity includes at least one of the angles of the three vertices of the triangle formed by the first and second reference feature points and the reference reference point among the n reference feature points, A computer program wherein the second feature quantity includes at least one of the angles of the three vertices of a triangle formed by the first and second target feature points and the target reference point among the n target feature points.

12. A computer program according to claim 5, The first feature quantity includes the ratio of at least two of the three sides of the triangle formed by the first and second reference feature points and the reference reference point among the n reference feature points, A computer program wherein the second feature quantity includes the ratio of at least two of the three sides of a triangle formed by the first and second target feature points and the target reference point among the n target feature points.

13. A computer program according to any one of claims 10 to 12, The first feature quantity does not include a feature quantity based on the feature direction of the reference point, which is calculated using an image of a local region of the reference image that includes the reference point. A computer program in which the second feature quantity does not include a feature quantity based on the feature direction of the target reference point, which is calculated using an image of a local region of the target image that includes the target reference point.

14. An image processing device, A candidate determination unit that determines a plurality of candidate pairs using the respective feature quantities of a plurality of reference feature points of a reference image and the respective feature quantities of a plurality of target feature points of a target image, wherein the candidate pair is a candidate matching pair which is a pair of the reference feature point and the target feature point that correspond to each other, A selection unit that selects n candidate pairs (where n is an integer of 2 or more) from the plurality of candidate pairs, A reference point determination unit that determines reference reference points of a reference image corresponding to n reference feature points constituting n candidate pairs, and target reference points of a target image corresponding to n target feature points constituting n candidate pairs, wherein the reference reference points are determined independently of the target image based on at least one of the corresponding n reference feature points, and the target reference points are determined independently of the reference image based on at least one of the corresponding n target feature points. A matching pair determination unit that determines the n candidate pairs as matching pairs when the n candidate pairs satisfy a specific condition, wherein the specific condition is a condition determined based on a first feature quantity calculated using n reference feature points and reference reference points corresponding to the n reference feature points, and a second feature quantity calculated using n target feature points and target reference points corresponding to the n target feature points, A correspondence relationship determination unit that determines the correspondence relationship between coordinates on the reference image and coordinates on the target image using the plurality of matching pairs that have been determined, An image processing device equipped with the following features.

Images