Programs and data processing devices

Abstract

Alignment between a plurality of images is performed. A plurality of candidate feature point pairs are acquired, the candidate feature point pairs being pairs of a feature point in a read image and a feature point in a reference image. A plurality of candidate feature point pairs satisfying a selection condition are selected as a plurality of feature point pairs. The selection condition includes a first condition for selecting a combination of N (N is 2 or 3) candidate feature point pairs, i.e., a subject combination, as N feature point pairs. The first condition is determined using one or more of four parameters of a ratio of lengths of two sides, a length of a side, an internal angle, a line segment connecting two feature points, and an angle formed with a direction corresponding to one of the two feature points. A correspondence relationship between coordinates on a read image and coordinates on a reference image is determined using the plurality of feature point pairs.

Description

Technical Field

[0001] This instruction manual relates to the alignment between multiple images. Background Technology

[0002] In various processing methods, alignment between multiple images can be performed. Patent Document 1 discloses a technique for detecting defects in images formed on paper by an image forming apparatus such as a printer or copier. In this technique, a marker image for position determination and a task image for printing, as instructed by the user, are formed on the same paper. An image reading unit reads the paper surface and generates a reading image. An inspection unit determines the position of the reading image corresponding to the reference image based on the feature points of the task image and marker image extracted from the reading image of the object to be inspected, and the feature points of the task image and marker image extracted from the reference image. The inspection unit compares the aligned reference image with the reading image and detects image areas where the difference in pixel values ​​is greater than a threshold as defects.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-112440 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] Alignment between multiple images is not easy and there is room for research.

[0008] This specification discloses a technique for aligning multiple images.

[0009] Technical solutions for solving the problem

[0010] The techniques disclosed in this specification can be implemented as follows:

[0011] [Application Example 1] A program enables a computer to perform the following functions: a candidate acquisition function, which acquires multiple candidate feature point pairs by using the feature values ​​of multiple feature points in a read image and the feature values ​​of multiple feature points in a reference image, wherein the candidate feature point pairs are a pair of feature points in the read image and a feature point in the reference image; a selection function, which selects multiple candidate feature point pairs that satisfy selection conditions from the multiple candidate feature point pairs as multiple feature point pairs, wherein the selection conditions include a first condition for selecting object combinations as N feature point pairs, wherein the object combination is a combination of N (N is 2 or 3) candidate feature point pairs, and the first condition is determined using one or more of the following four parameters: the ratio of the lengths of two sides of the triangle formed by the three feature points, the length of the sides of the triangle, the interior angle of the triangle, and the angle formed by the line segment connecting the two feature points and the direction corresponding to one of the two feature points; and a determination function, which uses the multiple feature point pairs to determine the correspondence between the coordinates on the read image and the coordinates on the reference image.

[0012] According to this structure, multiple candidate feature point pairs that meet the selection criteria are selected as multiple feature point pairs from multiple candidate feature point pairs. The selection criteria include a first condition for selecting object combinations as N feature point pairs. The object combination is a combination of N (N is 2 or 3) candidate feature point pairs. The first condition is determined using one or more of the following four parameters: the ratio of the lengths of the two sides of the triangle formed by the three feature points, the length of the sides of the triangle, the interior angle of the triangle, and the angle formed by the line segment connecting the two feature points and the direction corresponding to the feature quantity of one of the two feature points. Therefore, it is possible to properly align the read image with the reference image.

[0013] Furthermore, the technology disclosed in this specification can be implemented in various ways, such as by data processing methods and data processing apparatus, computer programs for implementing the functions of these methods or apparatus, and recording media (e.g., non-transitory recording media) on which the computer program is recorded. Attached Figure Description

[0014] Figure 1 This is an explanatory diagram showing a data processing apparatus as one embodiment.

[0015] Figure 2 This is a perspective view showing an example of the reading device 100.

[0016] Figure 3 (A) is a diagram showing an example of an image represented by image data for printing. (B) is a diagram showing an example of reading an image.

[0017] Figure 4This is a flowchart illustrating an example of the inspection process.

[0018] Figure 5 This is a flowchart illustrating an example of bitwise processing.

[0019] Figure 6 This is a flowchart illustrating an example of feature point matching processing.

[0020] Figure 7 (A) through (D) are graphs representing examples of images processed by feature point matching.

[0021] Figure 8 This is a flowchart representing an example of a selected process.

[0022] Figure 9 (A) and (B) are diagrams representing examples of color value calculations.

[0023] Figure 10 This is a flowchart illustrating an example of the selection process for candidate feature points M.

[0024] Figure 11 This is a diagram representing an example of a feature point group N.

[0025] Figure 12 (A) and (B) are diagrams representing examples of triangles formed by candidates MA, MB, and MC.

[0026] Figure 13 This is a flowchart illustrating an example of how the combined condition CC is handled.

[0027] Figure 14 This is a flowchart illustrating an example of the processing of the condition CW1 for determining the shape.

[0028] Figure 15 (A) to (D) represent Figure 14 An example of the results of the processing is shown in the figure.

[0029] Figure 16 This is a diagram illustrating an example of how coordinates correspond to each other.

[0030] Figure 17 (A) and (B) are figures representing examples of difference images.

[0031] Figure 18 This is a flowchart illustrating an example of the process of obtaining data from a reference image.

[0032] Figure 19 It represents the unequal edge condition ( Figure 13 Figure 630 is a diagram of another embodiment of the S630. Detailed Implementation

[0033] A. First embodiment: A1. Device Structure: Figure 1 This is an explanatory diagram illustrating a data processing apparatus as one embodiment. The data processing apparatus 200 is, for example, a personal computer. The data processing apparatus 200 performs inspection processing on printed images. The data processing apparatus 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are interconnected via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0034] Processor 210 is a device configured to perform data processing, such as a Central Processing Unit (CPU) or a System on a Chip (SoC). Volatile storage device 220 is, for example, Dynamic Random Access Memory (DRAM), and non-volatile storage device 230 is, for example, Flash memory. Non-volatile storage device 230 stores data for both the first program 231 and the second program 232. The second program 232 is used in other embodiments described later.

[0035] The display unit 240 is a device for displaying images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device for receiving user operations, such as buttons, levers, and a touch panel superimposed on the display unit 240. The display unit 240 and the operation unit 250 can form a so-called touch screen. By operating the operation unit 250, the user can input various requests and instructions into the data processing device 200.

[0036] Communication interface 270 is an interface used for communicating with other devices (e.g., including one or more of the following: USB interface, wired LAN interface, IEEE 802.11 wireless interface, and industrial camera interface (e.g., CameraLink, CoaXPress, etc.)). In this embodiment, reading device 100 and printing device 900 are connected to communication interface 270. Printing device 900 is a so-called inkjet printing device that prints images on printing media such as cloth or paper by spraying liquid ink onto the printing media. Reading device 100 generates data representing a read image of the object by optically reading the object being read. Hereinafter, the printing media is assumed to be a T-shirt, and reading device 100 reads a T-shirt with a printed image.

[0037] Figure 2This is a perspective view showing an example of a reading device 100. In the figure, the first direction Da and the second direction Db represent horizontal directions, and the third direction Dc represents the vertical direction. The first direction Da and the second direction Db are perpendicular to each other.

[0038] In this embodiment, the reading device 100 includes a housing 190, a worktable 130, a support portion 140 fixed to the upper surface of the worktable 130, a conveying device 120, a reading sensor 180, and a control device 110. The control device 110, the conveying device 120, and the reading sensor 180 are fixed to the housing 190.

[0039] The support portion 140 is a plate-shaped component (also called an impression plate) that forms a flat upper surface for supporting the object being read. In the figure, a T-shirt 700 with a printed image 1Mpp is placed on the support portion 140.

[0040] The conveying device 120 is configured to convey the worktable 130 in a direction parallel to the second direction Db. The conveying device 120 can have various structures. Although not shown in the figures, in this embodiment, the conveying device 120 includes: a guide rail supporting the worktable 130 so that it can slide in a direction parallel to the second direction Db; a plurality of pulleys; a belt wound around the plurality of pulleys and partially fixed to the worktable 130; and an electric motor that rotates the pulleys. The electric motor rotates the pulleys, thereby moving the worktable 130 (and consequently, the support portion 140) in a direction parallel to the second direction Db. The conveying device 120 also includes a position sensor 122 (e.g., a rotary encoder) that detects the position of the worktable 130 on the conveying path.

[0041] The readout sensor 180 is positioned midway along the transport path PTh of the support 140, at a position higher than the support 140. The readout sensor 180 comprises a line sensor (e.g., a contact image sensor (CIS) or a charge-coupled device (CCD)) arranged in a direction intersecting the transport direction Db (in this embodiment, a direction Da perpendicular to the transport direction Db). The readout sensor 180 faces downwards. The readout sensor 180 is capable of reading the portion of the object supported by the support 140 located below the readout sensor 180.

[0042] When reading T-shirt 700, the reading device 100 transports the worktable 130 in a direction parallel to the second direction Db. The reading sensor 180 repeatedly reads T-shirt 700 during transport. Thus, the reading sensor 180 can read approximately the entire portion of T-shirt 700 supported by the support portion 140.

[0043] The control device 110 is a circuit configured to control the conveying device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates data for reading the image by controlling the conveying device 120 and the reading sensor 180.

[0044] A2. Printing Process: In this embodiment, in T-shirt 700 ( Figure 2 The image is printed on the T-shirt. Image printing is implemented, for example, as part of the T-shirt sales service. The T-shirt sales service may include on-demand printing. The customer orders on-demand printing from the service provider. Based on the customer's order, the service provider prints the image on the T-shirt 700 using image data provided by the customer.

[0045] Figure 3 (A) is a diagram representing an example of an image represented by image data for printing (referred to as the object image IMp). In this embodiment, the data of the object image IMp is bitmap data representing the color values ​​(here, the grayscale values ​​of red R, green G, and blue B, respectively (e.g., values ​​above zero and below 255)) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. Figure 3 In example (A), the object image IMp represents the background BG and four rectangular objects OB1 to OB4. Objects OB1 to OB4 are represented by different colors. For example, the first object OB1 is red, the second object OB2 is green, the third object OB3 is blue, and the fourth object OB4 is gray. Thus, the object image IMp can include multiple objects with different hues.

[0046] Although the illustration is omitted, the printing of the object image IMp is performed using the data processing unit 200 and the printing unit 900. Alternatively, other devices can be used to print the object image IMp.

[0047] Various errors may occur during the printing of the object image IMp. The printed image may have various defects due to these errors. For example, due to abnormal ink ejection, a portion of the image may be missing from the printed image. The data processing apparatus 200 detects defects in the printed image through an inspection process described later. In this embodiment, the data of the object image IMp is used in the inspection process. After the object image IMp is printed, the data of the object image IMp is stored in the storage device 215 of the data processing apparatus 200 (e.g., non-volatile storage device 230) for the inspection process.

[0048] A3. Inspection and handling: Figure 4 This is a flowchart illustrating an example of the inspection process. For inspection to be performed, in the reading device 100 ( Figure 2 The T-shirt 700 is positioned on the support portion 140 of the data processing unit 200 in such a way that the printed image 1Mpp can be seen. In this embodiment, the operator positions the T-shirt 700 on the support portion 140. Alternatively, a machine (e.g., a robotic arm) can position the T-shirt 700 on the support portion 140. After the T-shirt 700 is positioned, a start instruction for inspection processing is input to the data processing unit 200. Figure 1 In this embodiment, the operator inputs a start instruction for the inspection by operating the operation unit 250. The processor 210 begins the inspection process according to the start instruction. Furthermore, the start instruction can be input to the data processing device 200 via the communication interface 270 from a device other than the data processing device 200.

[0049] The processor 210 of the data processing device 200 executes the inspection process according to the first program 231. In S110, a T-shirt 700 is photographed. The processor 210 provides a reading instruction to the reading device 100. The control unit 110 of the reading device 100 controls the reading sensor 180 and the conveying device 120 according to the reading instruction, thereby reading the T-shirt 700. The control unit 110 generates data representing the read image of the T-shirt 700. The processor 210 of the data processing device 200 obtains the read image data from the control unit 110 of the reading device 100 and stores the obtained read image data in the storage device 215 (e.g., non-volatile storage device 230).

[0050] Figure 3 (B) is a diagram illustrating an example of reading an image. In this embodiment, the data for reading the image IMs is bitmap data representing the color values ​​(here, the grayscale values ​​of red R, green G, and blue B, respectively (e.g., values ​​above zero and below 255)) of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. The image IMs in the diagram represents the portion of the T-shirt 700 containing the printed image IMpp. Here, the printed image IMpp is assumed to be obtained through the object image IMp (…). Figure 3 The image obtained by printing (A). The printed image IMpp is the same as the object image IMp, representing the background BG and objects OB1 to OB4.

[0051] In S120 ( Figure 4 In this embodiment, the processor 210 performs alignment between the reference image and the read image. The object image IMp (…) is used as the reference image. Figure 3(A) (Hereinafter, the object image IMp will also be referred to as the reference image IMt).

[0052] The orientation of the T-shirt 700 relative to the readout sensor 180 can be various. Therefore, the orientation (i.e., the angle of rotation of the objects OB1 to OB4) within the image may differ between the reference image IMt and the readout images IMs. Furthermore, the pixel density (also known as resolution) for the same object may differ between the reference image IMt and the readout images IMs. In other words, the scale relative to the object may differ between the reference image IMt and the readout images IMs.

[0053] Figure 5 This is a flowchart illustrating an example of alignment processing. In this embodiment, the processor 210 obtains multiple pairs of feature points through feature point matching, and uses these multiple pairs to determine the correspondence between the coordinates on the reference image IMt and the coordinates on the read image IMs.

[0054] In S220, processor 210 performs feature point matching. Figure 6 This is a flowchart illustrating an example of feature point matching processing. In S305, the processor 210 generates a grayscale reference image IMtg and a grayscale read image IMsg by performing grayscale conversion on the reference image IMt and the read image IMs. The correspondence between the grayscale values ​​of the color and the grayscale values ​​can be a known relationship (e.g., the correspondence between RGB values ​​in the RGB color space and the luminance value Y in the YCbCr color space).

[0055] In S310, processor 210 extracts feature points Tt from grayscale reference image IMtg. Figure 7 (A) to Figure 7 (D) is a graph representing an example of an image processed by feature point matching. Figure 7 (A) represents an example of feature points detected from images IMtg and IMsg. Multiple black dots on the grayscale reference image IMtg represent feature points Tt (also called reference feature points Tt) detected from the grayscale reference image IMtg. As shown, points representing feature parts such as corners and ends of an object are detected as feature points Tt. Such feature points Tt are also called keypoints. Although the illustration is omitted, in reality, many more feature points Tt can be detected (e.g., around dozens or hundreds). Furthermore, the reference feature points Tt detected from the grayscale reference image IMtg are equivalent to representing the reference image IMt (… Figure 3 Feature points of the same part with the same coordinates on (A).

[0056] Feature point detection methods can be various. In this embodiment, a technique called Accelerated-KAZE (A-KAZE) is used to perform key point detection and calculate feature descriptors for each key point. The A-KAZE technique is disclosed, for example, in the following paper: "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013". F Alcantaria, Jesus Nuevo, Adrian Bartoli, British Conference on Machine Vision (BMVC), Bristol, UK, September 2013.

[0057] The processor 210 uses A-KAZE technology to detect multiple feature points Tt by analyzing the grayscale reference image IMtg.

[0058] In S315 ( Figure 6 In the image IMsg, the processor 210 extracts multiple feature points Ts from the grayscale image IMsg. Figure 7 The multiple black dots on the grayscale readout image IMsg in (A) represent feature points Ts (also called readout feature points Ts) detected from the grayscale readout image IMsg. The processor 210, following the A-KAZE technique, detects multiple feature points Ts by analyzing the grayscale readout image IMsg. Although the illustration is omitted, it is actually possible to detect many more feature points Ts (e.g., around tens or hundreds). Furthermore, the readout feature points Ts detected from the grayscale readout image IMsg are equivalent to representing the readout image IMsg (… Figure 3 Feature points of the same part with the same coordinates on (B).

[0059] In S320 ( Figure 6In this embodiment, processor 210 calculates the feature quantity Ft for each of multiple reference feature points Tt. The feature quantity Ft can be various information describing the characteristics of feature point Tt. For example, the feature quantity Ft is calculated in a manner that varies according to the distribution of color values ​​of multiple pixels surrounding feature point Tt. In this embodiment, processor 210 uses a grayscale reference image IMtg to calculate an A-KAZE feature descriptor as the feature quantity Ft. In the A-KAZE technique, the feature descriptor is rotation-invariant. Furthermore, to obtain a rotation-invariant feature descriptor, the orientation of the feature point is also calculated. The orientation of the feature point represents the direction of the brightness gradient (also called the gradient direction or dominant direction) in the vicinity region centered on the feature point. Processor 210 calculates the feature quantity Ft and the orientation of the reference feature point Tt.

[0060] In S325, processor 210 calculates the feature quantity Fs of each of the multiple read feature points Ts. In this embodiment, processor 210 uses the grayscale read image IMsg and calculates the feature descriptor (i.e., feature quantity Fs) and the orientation of the read feature points Ts according to the A-KAZE technique.

[0061] The following is a summary of the processing after S325. Processor 210 calculates the distance dF between feature quantities Ft and Fs for all combinations of reference feature point Tt and read feature point Ts (S350). Then, the pairs of feature points Tt and Ts that have a distance dF less than the distance threshold dFth are taken as candidate M of feature point pairs (S355: Yes, S360).

[0062] Specifically, as described below. Processor 210 performs a loop processing S330 (including S335 to S360) between the start L31s and end L31e for each of the multiple reference feature points Tt. In S335, processor 210 selects the unprocessed reference feature point Tt as the feature point to be processed, i.e., the feature point of interest Tti. Processor 210 performs a loop processing S340 (including S345 to S360) between the start L32s and end L32e for each of the multiple read feature points Ts. In S345, processor 210 selects the unprocessed read feature point Ts as the feature point to be processed, i.e., the feature point of interest Tsj.

[0063] In S350, processor 210 calculates the distance dF between two feature quantities Fti and Fsj of two feature points Tti and Tsj of interest. The distance dF is calculated such that a smaller distance dF indicates a higher similarity between the two feature quantities Fti and Fsj. High similarity (i.e., a smaller distance dF) means that the two feature points Tti and Tsj represent similar portions (e.g., corresponding portions of the same object) in two images IMtg and IMsg. In this embodiment, the feature quantities Ft and Fs are A-KAZE feature descriptors, represented by binary vectors (vectors consisting of one or more binary elements). In this case, processor 210 can calculate the Hamming distance as the distance dF.

[0064] In S355, processor 210 determines whether the distance dF is less than the distance threshold dFth. If the distance dF is small, feature points Tti and Tsj are more likely to represent similar parts (e.g., similar parts of the same object) in images IMt and IMs. If the distance dF is less than the distance threshold dFth (S355: Yes), in S360, processor 210 obtains the pair of feature points Tti and Tsj of interest as candidate M (also called candidate feature point pair M) for feature point pairs. A feature point pair is a pair of reference feature point Tt and read feature point Ts used to determine the correspondence of coordinates. The mutually corresponding feature points Tt and Ts are also called matching pairs. After S360, processor 210 ends the loop processing S340 for feature point Tsj of interest. If the distance dF is greater than the distance threshold dFth (S355: No), processor 210 skips S360 and ends the loop processing S340 for feature point Tsj of interest. Then, the processor 210 performs S350-S360 processing on multiple combinations of the feature points Tsj and Tti of interest by repeatedly performing loop processing S340 and loop processing S330.

[0065] Figure 7 (B) represents an example of a candidate feature point pair M. The multiple lines RL in the figure represent candidate feature point pairs M respectively. Each line RL connects the feature points Tt and Ts that form the candidate feature point pair M. As shown, it is possible to select a pair of feature points Tt and Ts representing mutually identical parts as a candidate feature point pair M. For example, it is possible to establish a correspondence between the feature point Ts1 representing the upper right corner of the first object OB1 in the grayscale reading image IMsg and the feature point Tt1 representing the upper right corner of the first object OB1 in the grayscale reference image IMtg.

[0066] Furthermore, pairs of feature points Tt and Ts representing mutually different parts can be selected as candidate feature point pairs M. For example, a feature point Ts2 representing the upper right corner of the third object OB3 in the grayscale reading image IMsg can be mapped to a feature point Tt1 representing the upper right corner of the first object OB1 in the grayscale reference image IMtg. The rectangular objects OB1 to OB4 each have four corners. These corners are locally similar. As a result, feature points Ts representing other corners of the same object in the grayscale reading image IMsg, or feature points Ts representing corners of other objects, can be mapped to a feature point Tt representing one corner of an object in the grayscale reference image IMtg. Thus, when each image IMtg and IMsg represents multiple locally similar parts, feature points Tt and Ts representing mutually different parts can be mapped.

[0067] Furthermore, in this embodiment, as described above, the feature quantities Ft and Fs are rotationally invariant. When the feature quantities are rotationally invariant, similar feature quantities can be calculated based on the same parts of the object regardless of the rotation angle of the object within the image. By using the distance dF between the rotationally invariant feature quantities Ft and Fs, the processor 210 can establish a correspondence between feature points representing the same parts of the object, even when the rotation angles of the same object differ between two images IMtg and IMsg. However, when each image IMtg and IMsg represents multiple locally similar parts, feature points Tt and Ts representing mutually different parts can be corresponded. For example, a correspondence can be established between feature point Ts3 representing the lower right corner of the first object OB1 in the grayscale reading image IMsg and feature point Tt1 representing the upper right corner of the first object OB1 in the grayscale reference image IMtg.

[0068] In addition, the distance threshold dFth ( Figure 6 The larger the value of S355, the more likely there will be a total number of suitable candidate feature point pairs M. However, the total number of inappropriate candidate M may also increase. The more suitable candidate M there are, the lower the error in the coordinate correspondence described later. When the total number of inappropriate candidate M is large, the error in the coordinate correspondence may increase. The distance threshold dFth can be determined experimentally in advance in a way that allows for the error in the coordinate correspondence.

[0069] After processing completes the process of all combinations of reference feature point Tt and read feature point Ts, in S365, processor 210 saves the data representing multiple candidate feature point pairs M in storage device 215 (e.g., non-volatile storage device 230). Then, processor 210 ends. Figure 6 The processing, i.e. Figure 5 The processing of S220.

[0070] In S230, processor 210 performs a selection process for feature point pairs using color information. This selection process uses color information to select appropriate candidate M from multiple candidate feature point pairs M (i.e., verify candidate feature point pairs M). Figure 8 This is a flowchart illustrating an example of selection processing. Processor 210 performs loop processing S410 (including S420 to S465) between the start L4s and end L4e for each of the multiple candidate feature point pairs M.

[0071] In S420, the processor 210 selects the unprocessed candidate M as the candidate for processing, i.e., the candidate of interest Mi.

[0072] In S425, processor 210 calculates the representative color value Ct of the first part of the region Pt containing the reference feature point Tt of the candidate of interest Mi (also referred to as the reference representative color value Ct). Figure 9 (A) Figure 9 (B) is a diagram representing an example of color value calculation. Figure 9 (A) shows a portion of the reference image IMt containing feature point Tt. In the figure, a first partial region Pt including feature point Tt is shown. Processor 210 calculates a reference representative color value Ct using multiple color values ​​of multiple pixels in the first partial region Pt. The method for calculating the representative color value Ct can be various methods for calculating the color representing the first partial region Pt. In this embodiment, processor 210 calculates the average value of each RGB within the first partial region Pt as the grayscale value of each RGB of the representative color value Ct. Alternatively, various summary statistics representing the magnitude of color values ​​(e.g., median, most frequent value, etc.) can be used instead of the average value. Furthermore, the composition of the first partial region Pt (specifically, the shape of the first partial region Pt and the relative position of the first partial region Pt with respect to the feature point Tt) can be various compositions capable of calculating the representative color value Ct representing the color of the portion indicated by feature point Tt. In this embodiment, the first partial region Pt is a region centered on feature point Tt, for example, a P-row, Q-column region P. A region of Q pixels. To reduce the dependence of the representative color value Ct on the rotation angle of the object, it is preferable that P = Q (e.g., P = Q = 5). Instead of a region of P rows and Q columns, the first part of the region Pt can be a region whose distance from the feature point Tt is below a distance threshold.

[0073] In S430 ( Figure 8 In the process, processor 210 calculates the representative color value Cs of the second region Ps containing the read feature points Ts of the candidate of interest Mi (also referred to as the read representative color value Cs). Figure 9Figure (B) shows a portion of the read image IMs containing feature point Ts. A second region Ps, also containing feature point Ts, is shown in the figure. The structure of the second region Ps is the same as that of the first region Pt. The second region Ps is a P-row, Q-column region centered on feature point Ts. The region is Q pixels. The method for calculating the representative color value Cs is the same as the method for calculating the reference representative color value Ct. The processor 210 calculates the average value of red (R), green (G), and blue (B) within the second region Ps as the read representative color value Cs.

[0074] In S435 ( Figure 8 In this process, processor 210 calculates a first hue Ht, a first saturation St, and a first brightness Vt based on a reference representative color value Ct, and calculates a second hue Hs, a second saturation Ss, and a second brightness Vs based on a read representative color value Cs. The correspondence between the representative color values, hue, saturation, and brightness can be a known relationship (e.g., the correspondence between RGB values ​​in the RGB color space and HSV values ​​in the HSV color space). Furthermore, the first hue Ht, first saturation St, and first brightness Vt are, respectively, examples of representative color values ​​for a first region Pt, just like the reference representative color value Ct. Similarly, the second hue Hs, second saturation Ss, and second brightness Vs are, respectively, examples of representative color values ​​for a second region Ps, just like the read representative color value Cs.

[0075] In S440, processor 210 determines whether the first saturation condition CSt, indicating that the first saturation St is higher than the saturation threshold Sth, is satisfied. If the first saturation St is higher than the saturation threshold Sth (S440: Yes), in S445, processor 210 determines whether the hue condition CH, indicating that the absolute value of the difference between the first hue Ht and the second hue Hs is less than the hue difference threshold dHth, is satisfied. The absolute value of the difference between hues Ht and Hs (also called the hue difference dH) is taken as the value corresponding to the smaller of the angle differences between the first hue Ht and the second hue Hs on the hue ring. The hue difference dH can be a small value if the candidate of interest Mi is a suitable pair of feature points Tt and Ts representing mutually identical parts. If the hue difference dH is less than the hue difference threshold dHth (S445: Yes), in S460, processor 210 selects the candidate of interest Mi as the candidate to be retained. If the hue difference dH is greater than or equal to the hue difference threshold dHth (S445: No), the hue may differ significantly between the first region Pt and the second region Ps. That is, the feature points Tt and Ts of the candidate Mi are likely to represent distinct regions. In S465, the processor 210 excludes the candidate Mi from the candidate feature point pairs.

[0076] Thus, if the first saturation St is higher than the saturation threshold Sth (S440: Yes), the processor 210 will exclude the candidate of interest Mi with a hue difference dH greater than the hue difference threshold dHth from the candidates. Figure 7 (C) indicates through Figure 8 The remaining candidate feature point pairs M are processed. Pairs of feature points Tt and Ts that have distinct hues can be excluded. For example, the hues differ between the first object OB1 and the third object OB3. Figure 7 Line RLa, shown in (B), represents the pair of reference feature point Tt of the first object OB1 and read feature point Ts of the third object OB3. For example... Figure 7 As shown in (C), the pair was excluded ( Figure 8 S440 (Yes), S445 (No), S465.

[0077] If the first saturation St is below the saturation threshold Sth (S440: No), in S450, the processor 210 determines whether the second saturation condition CSs, indicating that the second saturation Ss is higher than the saturation threshold Sth, is satisfied. If the second saturation Ss is higher than the saturation threshold Sth (S450: Yes), the saturation may differ significantly between the first region Pt and the second region Ps. That is, there is a high probability that the feature points Tt and Ts of the candidate Mi represent distinct regions. In S465, the processor 210 excludes the candidate Mi from the candidate feature point pairs.

[0078] If the second saturation Ss is below the saturation threshold Sth (S450: No), in S455, the processor 210 determines whether the luminance condition CV, which represents the absolute value of the difference between the first luminance Vt and the second luminance Vs, is satisfied and is less than the luminance difference threshold dVth. When the candidate of interest Mi is a suitable pair of feature points Tt and Ts representing mutually identical parts, the absolute value of the difference between luminances Vt and Vs (also called the luminance difference dV) can be a small value. When the candidate of interest Mi is an inappropriate pair of feature points Tt and Ts representing mutually different parts, the luminance difference dV can be a large value. Thus, even when the saturations St and Ss are low, the luminances Vt and Vs can appropriately represent the color of the part indicated by the feature points Tt and Ts.

[0079] When the luminance difference dV is less than the luminance difference threshold dVth (S455: Yes), in S460, the processor 210 selects the candidate of interest Mi as the candidate to be retained. When the luminance difference dV is greater than or equal to the luminance difference threshold dVth (S455: No), in S465, the processor 210 excludes the candidate of interest Mi from the candidate feature point pair M.

[0080] Furthermore, the more lenient the conditions for retaining candidate Mi of interest, the greater the total number of suitable candidates M that are retained without being excluded. However, the total number of unsuitable candidates may also increase. The greater the total number of suitable candidates, the lower the error in the coordinate correspondence described later. With a large number of unsuitable candidates, the error in the coordinate correspondence may increase. The parameters Sth, dHth, and dVth used in S440 to S455 can be determined experimentally beforehand to allow for errors in the coordinate correspondence.

[0081] For example, the larger the hue difference threshold dHth (S445), the more lenient the conditions for retaining interest candidates Mi. Even if the interest candidate Mi is a suitable pair of feature points Tt, Ts representing a portion of a highly saturated color, the hue difference dH can be a value greater than zero. A hue difference threshold dHth greater than zero allows for such a hue difference dH. The hue difference threshold dHth can be set to a value greater than zero and less than the maximum possible value of the hue difference dH.

[0082] Furthermore, the larger the luminance difference threshold dVth (S455), the more lenient the conditions for retaining interest candidates Mi. Even if the interest candidate Mi is a suitable pair of feature points Tt, Ts representing a portion of a low-saturation color, the luminance difference dV can still be a value greater than zero. A luminance difference threshold dVth greater than zero allows for such luminance difference dV. The luminance difference threshold dVth can be set to a value greater than zero and less than the maximum possible value of the luminance difference dV.

[0083] Furthermore, when the saturation threshold Sth (S440) is small, the candidate Mi representing low first saturation St is processed in S445. Even when referring to the representative color value Ct ( Figure 9 (A) is colored. When the first saturation St is low, the hue of the corresponding area of ​​the read image IMs representing the printed image is also likely to differ from the first hue Ht. That is, when the first saturation St is low, the error of the second hue Hs, and consequently the error of the hue difference dH, may increase. When the error of the hue difference dH is large, appropriate attention candidates Mi (S445: No) may be incorrectly excluded, and inappropriate attention candidates Mi (S445: Yes) may be incorrectly retained. As a result, the error in the correspondence of coordinates may increase. The saturation threshold Sth can be determined in advance through experiments to mitigate the influence of the error of the hue difference dH on the error of the correspondence of coordinates. The saturation threshold Sth can be set to a value greater than zero and less than the possible maximum value of saturation.

[0084] After S460 or S465, processor 210 moves to S420 to execute the loop processing S410 for the next candidate Mi of interest. If the loop processing S410 for all candidates M has ended, in S470, processor 210 saves the data indicating the candidate feature point pairs M that should be retained to storage device 215 (e.g., non-volatile storage device 230). Then, processor 210 terminates. Figure 8 The processing, i.e. Figure 5 The processing of S230.

[0085] In S240, processor 210 performs a selection process for candidate feature point pairs M using a combination of multiple candidate M. This selection process uses a combination of multiple candidate M to select an appropriate candidate M from the remaining multiple candidate M (i.e., verify the candidate feature point pairs M).

[0086] Figure 10 This is a flowchart illustrating an example of the selection process for candidate feature points M. Figure 10 The processing is summarized as follows. Processor 210 selects three candidate MA, MB, and MC for the feature point pair (S520 to S560). The second candidate MB and the third candidate MC are selected from the region between the first radius r1 and the second radius r2 centered on the reference feature point Tt of the first candidate MA (details will be described later). Hereinafter, the combination of the selected candidate MA, MB, and MC will also be referred to as candidate combination MU. Processor 210 uses candidate combination MU to determine whether the combination condition CC is satisfied (S565). If the combination condition CC is satisfied (S570: Yes), processor 210 selects candidate combination MU as object combination MT (S573), and uses object combination MT to determine whether the shape condition CW1 is satisfied (S575). If the shape condition CW1 is satisfied (S580: Yes), processor 210 selects three candidate MA, MBA, and MC of object combination MT as candidates to be retained (S585). In this embodiment, through Figure 10 The multiple candidate feature point pairs M retained after processing are used to determine the correspondence of coordinates.

[0087] Specifically, as described below. Processor 210 performs loop processing S510 (including S520 to S585) between the start L51s and end L51e for multiple candidate feature point pairs M respectively. In S520, processor 210 selects the unprocessed candidate M as the first candidate MA.

[0088] In S525, the processor 210 selects a group of feature points N within a range between a first radius r1 and a second radius r2, centered on the feature point Tt of the first candidate MA. Figure 11This is a diagram illustrating an example of a feature point group N. The diagram shows a reference image IMt and multiple reference feature points Tt. A feature point Tt is selected as a first candidate MA. A first circle C1 is a circle with a first radius r1 centered on the feature point Tt of the first candidate MA. A second circle C2 is a circle with a second radius r2 centered on the feature point Tt of the first candidate MA. The selection range SR is the region between the first circle C1 and the second circle C2 (including portions on circles C1 and C2). The processor 210 selects multiple feature points Tt contained within the selection range SR as the feature point group N.

[0089] As described below, processor 210 uses three reference feature points Tt and three read feature points Ts for three candidate MA, MB, and MC to determine whether the combination of candidate MA, MB, and MC should be retained. Specifically, processor 210 determines that the combination of three candidate MA, MB, and MC should be retained if the difference between the shape of the triangle formed by the three reference feature points Tt and the shape of the triangle formed by the three read feature points Ts is small. If the length of any side of a triangle is significantly longer or shorter than the length of the other sides, the error in comparing the shapes of the two triangles may increase. Candidate MB and MC to be combined with the first candidate MA are selected from feature point group N. The range SR (here, radii r1 and r2) is selected in advance through experimentation in a way that forms a suitable shape for judgment through the combination of candidate MA, MB, and MC. For example, the first radius r1 can be a value of more than 1% and less than 20% of the size of the image IMt (e.g., the length of the first direction Dx or the second direction Dy). The second radius r2 can be a value of more than 30% and less than 70% of the size of the image IMt.

[0090] In S525 ( Figure 10After that, processor 210 selects a second candidate MB from feature point group N (S540) and a third candidate MC (S560). Specifically, processor 210 performs loop processing S530 (including S540 to S585) between start L52s and end L52e for each of the multiple candidates M in feature point group N. In S540, processor 210 selects an unprocessed candidate M from feature point group N as the second candidate MB. Hereinafter, the second candidate MB is assumed to be the j-th candidate in feature point group N (also represented as second candidate MB[j]). The index j is selected, for example, from the range of zero and less than the total number NN of candidate M in feature point group N. After S540, processor 210 performs loop processing S550 (including S560 to S585) between start L53s and end L53e for each of the multiple candidates M in feature point group N. In S560, processor 210 selects an unprocessed candidate M from feature point group N as the third candidate MC. Hereinafter, let the third candidate MC be the k-th candidate in the feature point group N (represented as the third candidate MC[k]). The index k is selected, for example, from the range NN of the total number of candidates M above j+1 and less than the feature point group N. Thus, the candidate M, which is different from the second candidate MB, is selected as the third candidate MC. In this way, the processor 210 selects a combination of three candidates MA, MB, and MC (i.e., candidate combination MU).

[0091] In S565, processor 210 uses candidate combinations MU to determine whether the combination condition CC is satisfied. The combination condition CC is the condition used to select candidate combinations MU as the object combination MT for the judgment of shape condition CW1 described later. The judgment of combination condition CC is performed using a triangle formed by candidate MA, MB, and MC of candidate combinations MU.

[0092] Figure 12 (A) Figure 12 (B) is a diagram showing an example of a triangle formed by candidates MA, MB, and MC. Each candidate MA, MB, and MC represents a pair of reference feature point Tt and read feature point Ts. Figure 12 (A) represents the triangle TRt (called the reference triangle TRt) formed by the reference feature points Tta, Ttb, and Ttc of the candidates MA, MB, and MC. Figure 12 (B) represents triangle TRs (called read triangle TRs) formed by the read feature points Tsa, Tsb, and Tsc of candidates MA, MB, and MC. The positions of more than one feature point may differ between the reference triangle TRt and the read triangle TRs.

[0093] In each diagram, symbols are shown to represent the sides, side lengths, and interior angles of a triangle. Symbols beginning with "S" (e.g., "Sabt") represent sides. The two characters following "S" indicate two candidate triangles M connected by the side (a, b, c represent candidate triangles MA, MB, and MC, respectively). The "t" or "s" at the end of the symbol indicates either the reference triangle TRt or the read triangle TRs, respectively. For example, side Sabt ( Figure 12 (A) is the edge connecting reference feature points Tta and Ttb of reference triangle TRt. The symbol representing the edge is obtained by replacing the initial S with L. For example, length Labt represents the length of edge Sabt. Symbols beginning with A (e.g., Aat) represent the size of the interior angles of the triangle. A character following A indicates the candidate M that forms the interior angle (a, b, c represent candidates MA, MB, MC, respectively). The t or s at the end of the symbol represents either the reference triangle TRt or the reading triangle TRs, respectively. For example, interior angle Aat ( Figure 12 (A) represents the size of the interior angle of the reference triangle TRt with the reference feature point Tta as its vertex. In other figures described later, notation based on the above rules is also used. Additionally, Figure 12 In (A), the direction Oct represents the direction of feature point Ttc. Figure 6 (S320) Figure 12 In (B), the direction Ocs represents the direction of the feature point Tsc. Figure 6 (S325).

[0094] Figure 13 This represents the processing of the judgment of the combined condition CC. Figure 10 The flowchart is an example of S565. Figure 13 The process for determining whether a triangle satisfies the individual combination condition is illustrated. Processor 210 considers the reference triangle TRt ( Figure 12 (A) and reading triangle TRs ( Figure 12 (B) will be executed separately. Figure 13 The processing involves determining whether the reference triangle TRt and the read triangle TRs satisfy the individual combination conditions. The processor 210 then determines that the combination condition CC is satisfied. The following explanation uses the reference triangle TRt as an example. Figure 13 The processing. In Figure 13 In the instructions for processing, symbols with the trailing 't' or 's' omitted are used as symbols to represent the sides, the length of the sides, and the size of the interior angles, respectively.

[0095] In S610, processor 210 determines whether the three candidate MA, MB, and MC are different from each other. If two or more candidates are the same (S610: No), in S640, processor 210 determines that the individual combination condition is not met and ends. Figure 13 The processing.

[0096] If the three candidates MA, MB, and MC are different (S610: Yes), in S615, processor 210 determines whether the first length condition CL1 is satisfied. The first length condition CL1 means that the first length Lab is within the allowable length range LR, which is greater than the lower limit Lth1 and less than the upper limit Lth2 (the allowable length range LR is also simply referred to as the length range LR). If the first length Lab is outside the length range LR (S615: No), processor 210 proceeds to S640. The length range LR (here, the lower limit Lth1 and the upper limit Lth2) is related to the radius r1, r2 of the selected range SR. Figure 11 Similarly, the decision is made experimentally in advance by forming a suitable graph for judgment through the combination of three candidate MA, MB, and MC. For example, the lower limit Lth1 can be the same as the first radius r1, and the upper limit Lth2 can be the same as the second radius r2.

[0097] If the first length Lac is within the length range LR (S615: Yes), in S620, the processor 210 determines whether the second length condition CL2, which indicates that the second length Lac is within the length range LR, is satisfied. If the second length Lac is outside the length range LR (S620: No), the processor 210 proceeds to S640.

[0098] If the second length Lac is within the length range LR (S620: Yes), in S625, processor 210 determines whether the first interior angle condition CA is satisfied. The first interior angle condition CA means that interior angle Aa is within the allowable interior angle range AR, which is greater than the lower limit Ath1 and less than the upper limit Ath2 (also referred to as the interior angle range AR). If interior angle Aa is outside the interior angle range AR (S625: No), processor 210 moves to S640. The interior angle range AR (here, the lower limit Ath1 and the upper limit Ath2) and the radius r1, r2 of the selected range SR ( Figure 11 Similarly, the decision is made in advance through experiments by forming a suitable graph for judgment using combinations of three candidate MA, MB, and MC. For example, the lower limit Ath1 can be a value above 10 degrees and below 60 degrees. The upper limit Ath2 can be a value above 90 degrees and below 160 degrees.

[0099] If interior angle Aa is within the interior angle range AR (S625: Yes), in S630, processor 210 determines whether the scalene condition CQ is satisfied. The scalene condition CQ indicates that the lengths of the three sides of the triangle (here, refer to triangle TRt) are all different. That is, the scalene condition CQ indicates that the shape of the triangle is different from that of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle. Isosceles triangles and equilateral triangles maintain their original shape even if two sides of the same length are replaced. In the case where the reference triangle TRt is an isosceles triangle or an equilateral triangle, even if the two appropriate reading feature points Ts corresponding to the two reference feature points Tt are replaced, the difference between the shape of the reference triangle TRt and the shape of the reading triangle TRs becomes smaller. As a result, candidates MA, MB, and MC containing two inappropriate candidate M may be mistakenly judged as something that should be retained to determine the correspondence of coordinates. The same applies when the reference triangle TRt is a triangle similar to an isosceles triangle or a triangle similar to an equilateral triangle. Therefore, in this embodiment, if the unequal side condition CQ is not satisfied (S630: No), the processor 210 determines in S640 that the individual combination condition is not satisfied and ends the process. Figure 13 The processing is as follows: If the unequal side condition CQ is satisfied (S630: Yes), processor 210 determines in S635 that the individual combination condition is satisfied, and ends the process. Figure 13 The processing.

[0100] The specific structure of the unequal edge condition CQ can be various. In this embodiment, the unequal edge condition CQ is all of the following conditions CQ1 to CQ3.

[0101] (CQ1) The absolute value of the difference between lengths Lab and Lac is greater than the length difference threshold dLth. (CQ2) The absolute value of the difference between lengths Lab and Lbc is greater than the length difference threshold dLth. (CQ3) The absolute value of the difference between lengths Lbc and Lac is greater than the length difference threshold dLth.

[0102] Conditions CQ1 to CQ3 indicate that the length difference between the two sides is large. The unequal side condition CQ, determined by conditions CQ1 to CQ3, means that the triangle does not contain two sides with a length difference below the length difference threshold dLth. The length difference threshold dLth is related to the radii r1 and r2 of the selected range SR. Figure 11 The same applies, and is determined experimentally in advance by forming a suitable graph for judgment through the combination of three candidate MA, MB, and MC. For example, the length difference threshold dLth can be in S615, S620 ( Figure 13 The value that is above 2% and below 10% of the upper limit Lth2 referenced in the reference.

[0103] In S565 ( Figure 10In the process, processor 210 executes the following steps on reference triangle TRt and read triangle TRs respectively. Figure 13 The processor 210 then determines that candidate combination MU satisfies combination condition CC if both reference triangle TRt and read triangle TRs satisfy the individual combination conditions. If one or both of reference triangle TRt and read triangle TRs do not satisfy the individual combination conditions, the processor 210 determines that candidate combination MU does not satisfy combination condition CC.

[0104] In S570, processor 210 branches its processing based on the judgment result of S565. If the combination condition CC is met (S570: Yes), in S573, processor 210 selects candidate combination MU as object combination MT. In S575, processor 210 uses object combination MT to determine whether the shape condition CW1 is met.

[0105] Figure 14 This is a flowchart illustrating an example of the processing of the condition CW1 for shape determination. In the object combination MT, there are three candidates MA, MB, MC ( Figure 12 (A) Figure 12 When (B) represents an appropriate pair of feature points Tt and Ts, the shape of the reading triangle TRs is approximately the same as the shape of the reference triangle TRt. When the object rotates within the image, the direction of the feature points (e.g., the direction Oct of feature point Ttc) rotates along with the object represented by the feature points. The shape condition CW1 is constructed taking into account the above-described properties. In this embodiment, if all three conditions CR, CD, and CF described later are satisfied, it is determined that the shape condition CW1 is satisfied.

[0106] In the S710, processor 210 calculates the ratio of the lengths of the two sides of triangles TRt and TRs. Specifically, processor 210 calculates the following two ratios, Rt and Rs: Rt = Labt / Lact; Rs=Labs / Lacs.

[0107] These ratios Rt and Rs are constant relative to the scale and rotation of objects in the images IMtg and IMs.

[0108] In S715, the processor 210 determines whether the ratio condition CR, which indicates that the ratio Rt / Rs is close to 1, is satisfied. Since the ratios Rt and Rs are invariant relative to the scale and rotation of objects within the images IMt and IMs, the result of the ratio condition CR is invariant relative to the scale and rotation. When the shape of the read triangle TRs is the same as the shape of the reference triangle TRt, the ratio Rt / Rs = 1. Even when the candidate MA, MB, and MC represent appropriate pairs of feature points Tt and Ts, respectively, the ratio Rt / Rs may deviate from 1. The ratio condition CR is configured to allow such deviations. For example, the ratio condition CR can be that the ratio Rt / Rs is greater than the lower limit Rth1 and less than the upper limit Rth2 (here, Rth1 < 1 < Rth2).

[0109] If the ratio condition CR is met (S715: Yes), in S720, the processor 210 determines whether the second interior angle condition CD, which indicates that the interior angle Abs of the read triangle TRs is close to the interior angle Abt of the reference triangle TRt, is met. The scale and rotation of the interior angles Abt and Abs relative to the objects in the images IMt and IMs are unchanged. Therefore, the determination result of the second interior angle condition CD is unchanged relative to the scale and rotation.

[0110] When the shape of the reading triangle TRs is the same as the shape of the reference triangle TRt, the interior angle Abs is the same as the interior angle Abt. Even when the candidate MA, MB, and MC represent appropriate pairs of feature points Tt and Ts, respectively, the interior angle Abs may deviate from the interior angle Abt. The second interior angle condition CD is configured to allow such deviations. For example, the second interior angle condition CD can be that the absolute value of the difference between the interior angle Abs and the interior angle Abt is less than the interior angle difference threshold dAth (here, dAth > 0).

[0111] If the second interior angle condition CD is satisfied (S720: Yes), in S725, processor 210 calculates the angle between the direction of the feature point and the edge. Specifically, processor 210 calculates the following two angles Zt and Zs: Zt = AG(Oct, Sact); Zs = AG(Ocs, Sacs).

[0112] AG is a function that derives the angle. The directions Oct and Octs are the directions of the feature points Ttc and Tsc, respectively. Figure 12 (A) Figure 12 (B)). As the directions Oct and Octs of the feature points Ttc and Tsc, they can be used to calculate the feature quantities Ft and Fs (S320, S325). Figure 6The direction is calculated from the angle Zt, which is the angle between edge Sact and direction Oct. The angle Zs is the angle between edge Sacs and direction Oct.

[0113] In this embodiment, the directions Oct and Ocs of feature points Ttc and Tsc are gradient directions calculated using the A-KAZE technique. When objects rotate within images IMtg and IMs, the directions Oct and Ocs rotate together with the objects represented by feature points Ttc and Tsc within images IMtg and IMs. The proportions and rotations of angles Zt and Zs relative to the objects within images IMt and IMs remain unchanged.

[0114] In S730, processor 210 determines whether the angle condition CF, which indicates that the angle Zs of the read triangle TRs is close to the angle Zt of the reference triangle TRt, is satisfied. Since the scale and rotation of angles Zt and Zs relative to objects within images IMtg and IMs are invariant, the determination result of the angle condition CF is invariant relative to the scale and rotation. When the shape of the read triangle TRs is the same as the shape of the reference triangle TRt, and the feature points Ttc and Tsc represent the same part of the same object, angle Zs is approximately the same as angle Zt. Even when candidate MA, MB, and MC represent appropriate pairs of feature points Tt and Ts, respectively, angle Zs may deviate from angle Zt. The angle condition CF is configured to allow such deviations. For example, the angle condition CF can be that the absolute value of the difference between angles Zt and Zs is less than the angle difference threshold dZth (here, dZth > 0).

[0115] If conditions CR, CD, and CF are all satisfied (S715: Yes, S720: Yes, and S730: Yes), it is presumed that all candidates MA, MB, and MC are appropriate pairs. In this case, in S735, processor 210 determines that the shape condition CW1 is satisfied and ends the process. Figure 14 The processing.

[0116] If one or more of the conditions CR, CD, and CF are not met, it is inappropriate to presume that one or more of the candidate MA, MB, and MC is a candidate M. If one or more of S715: No, S720: No, and S730: No are true, in S740, processor 210 determines that the shape condition CW1 is not met and ends the process. Figure 14 The processing.

[0117] Figure 15 (A) to Figure 15 (D) represents Figure 14 An example of the results of the processing is shown in the figure. Figure 15 (A) represents an example referencing triangle TRt. Figure 15 (B) to Figure 15 (D) represents an example of establishing a corresponding reading triangle TRs with reference triangle TRt.

[0118] Figure 15 (B) indicates that candidates MA, MB, and MC are appropriate, and that the scale is the same but the rotation angle is different between the reference image IMt and the read image IMs. In this case, the rotation invariance condition CR, CD, and CF are satisfied. Figure 14 Therefore, the shape condition CW1 is satisfied.

[0119] Figure 15 (C) represents the case where candidates MA, MB, and MC are appropriate, and the rotation angle is the same but the scale is different between the reference image IMt and the read image IMs. In this case, the condition of constant scale is satisfied for CR, CD, and CF. Figure 14 Therefore, the shape condition CW1 is satisfied.

[0120] Although the illustration is omitted, the conditions CR, CD, and CF are also satisfied when the candidate MA, MB, and MC are appropriate, and the scale and rotation angles between the reference image IMt and the read image IMs are different. Figure 14 Therefore, the shape condition CW1 is satisfied.

[0121] Figure 15 (D) represents the case where candidate MA and MB are appropriate, but the third candidate MC is inappropriate. Specifically, feature point Tsc differs from the appropriate feature point Tscr. In the case where the combination of candidate MA, MB, and MC includes an inappropriate candidate M, the conditions CR, CD, and CF ( Figure 14 The result of judging one or more of the conditions can be no (i.e., the probability that the shape condition CW1 is not met is high).

[0122] In addition, conditional CR, CD, CF ( Figure 14The more lenient the condition, the greater the total number of suitable candidate combinations of MA, MB, and MC that satisfy the shape condition CW1. However, the total number of unsuitable candidate combinations of MA, MB, and MC that satisfy the shape condition CW1 may also increase. The greater the total number of suitable candidates, the lower the error in the coordinate correspondence described later. When the total number of unsuitable candidates is large, the error in the coordinate correspondence may increase. The conditions CR, CD, and CF (in this embodiment, parameters Rth1, Rth2, dAth, and dZth) can be determined experimentally in advance in a way that allows for the error in the coordinate correspondence. The lower limit Rth1 (S715) can be any value less than 1, for example, it can be set to a value greater than 0.85 and less than 1. The upper limit Rth2 can be any value greater than 1, for example, it can be set to a value greater than 1 and less than 1.15. The interior angle difference threshold dAth (S720) can be any value greater than zero, for example, it can be set to a value greater than zero and less than 10 degrees. The angle difference threshold dZth (S730) can be any value greater than zero, for example, it can be set to a value greater than zero and less than 10 degrees.

[0123] exist Figure 14 After processing, that is, in S575 ( Figure 10 Following this, in S580, processor 210 branches the processing based on the judgment result of S575. If the shape condition CW1 is satisfied (S580: Yes), in S585, processor 210 selects candidate MA, MB, and MC of the object combination MT as candidates to be retained. After S585, processor 210 ends the loop processing S550 for the current combination of candidate MA, MB, and MC.

[0124] If the combination condition CC is not met (S570: No) and the shape condition CW1 is not met (S580: No), the processor 210 skips S585 and ends the loop processing S550 for the current combination of candidates MA, MB, and MC.

[0125] Subsequently, processor 210 executes respective loop processing S550 using multiple third candidate MCs, respective loop processing S530 using multiple second candidate MBs, and respective loop processing S510 using multiple first candidate MAs. After the loop processing S510, S530, and S550 has been repeated, in S590, processor 210 saves the data indicating the candidate feature point pairs M that should be retained to storage device 215 (e.g., non-volatile storage device 230). Then, processor 210 terminates. Figure 10 The processing, i.e. Figure 5 The processing of S240.

[0126] Figure 7 (D) indicates through Figure 10 Examples of candidate feature point pairs retained after processing. For example... Figure 7 (C) Figure 7 As shown in (D), inappropriate feature point pairs Tt and Ts can be excluded. For example, Figure 7 Line RLb, shown in (C), represents the pair of the upper right reference feature point Tt and the lower right read feature point Ts of the fourth object OB4. Such a pair, when combined with other pairs, cannot satisfy the shape condition CW1 and can be excluded from the candidate feature point pairs. Figure 10 S580: No). (For example, in...) Figure 15 (A) to Figure 15 As explained in (D), combinations of candidate MA, MB, and MC that form an appropriate reference triangle TRt and read triangle TRs can be retained as candidates. Combinations of candidate MA, MB, and MC that form an inappropriate reference triangle TRt and read triangle TRs are not retained as candidates.

[0127] Furthermore, even if the combination of candidate MA, MB, and MC containing a suitable candidate M does not satisfy the shape condition CW1, the suitable candidate M can still satisfy the shape condition CW1 by combining it with other suitable candidate M. Thus, in Figure 10 In the processing, even if the judgment result of S570 or S580 is negative, candidates MA, MB, and MC will not be immediately excluded. Processor 210, through repeated loop processing S510, S530, and S550, selects multiple candidate Ms selected more than once by S585 as the candidate Ms to be retained. Processor 210, through repeated loop processing S510, S530, and S550, excludes candidate Ms that have not been selected even once by S585.

[0128] exist Figure 10 After the processing, that is, in Figure 5 After S240, in S250, the processor 210 determines the correspondence between the coordinates on the reference image IMt and the coordinates on the read image IMs. Figure 16This is a diagram illustrating an example of the correspondence between coordinates. The diagram shows the coordinates COt on the reference image IMt, the coordinates COs on the read image IMs, and the matrix Mtx corresponding to these coordinates COt and COs. In this embodiment, matrix Mtx represents a so-called affine transformation. In the diagram, the coordinates COt, COs, and matrix Mtx are represented by a homogeneous coordinate system. Each coordinate COt and COs is represented by a three-dimensional vector having positions Xt and Xs in the first direction Dx, positions Yt and Ys in the second direction Dy on images IMt and IMs, and 1 as a third component. Matrix Mtx is a 3x3 matrix. As shown, matrix Mtx is represented by six parameters a to f in a 2x3 matrix and three components (0, 0, 1) in the third row. Such a matrix Mtx can represent rotation, scaling, scaling, parallel translation, and skew. Matrix Mtx can be calculated using three or more pairs of coordinates COt and COs (i.e., three or more pairs of feature points).

[0129] Processor 210 uses multiple feature point pairs MP (i.e., in Figure 5 The matrix Mtx (here, six parameters a to f) is determined by selecting multiple candidate feature point pairs M retained in S220 to S240. The matrix Mtx can be determined using various methods. For example, the processor 210 can determine the matrix Mtx using a method called Random Sample Consensus (RANSAC). It should be noted that in this embodiment, if... Figure 7 (A) to Figure 7 (D) and Figure 15 (A) to Figure 15 As illustrated in (D), there is a high probability that inappropriate candidates for feature point pairs MP will be excluded. Therefore, processor 210 can use all remaining feature point pairs MP to compute matrix Mtx. The computation method can be various (e.g., least squares). Alternatively, functions from OpenCV (Open Source Computer Vision Library) can be used, for example, in determining matrix Mtx.

[0130] via S250 ( Figure 5 The end of ) Figure 5 The processing, i.e. Figure 4 S120 ends. In S130, the processor 210 checks the printed image. The checking method can be based on the correspondence of coordinates ( Figure 16 Various methods are used. In this embodiment, the processor 210 uses the correspondence of coordinates to generate the data of the difference image. Figure 17 (A) Figure 17(B) is a diagram representing an example of a difference image. Figure 17 (A) indicates that the printed image has no defects. Figure 17 (B) indicates a case where the printed image has defects (in this case, missing Err). The left side of each figure shows the read images IMs, IMs2 and the reference image IMt arranged on the read images IMs, IMs2 according to the coordinate correspondence. The right side of each figure shows examples of difference images between the read images IMs, IMs2 and the reference image IMt. The processor 210 generates data for difference images IMd, IMd2 representing the difference in color values ​​(e.g., the absolute value of the difference in brightness values) between the read images IMs, IMs2 and the reference image IMt at corresponding positions established by the coordinate correspondence. Figure 17 The read image IMs of (A) represents a printed image without any defects. Therefore, the difference image IMd does not have a portion representing large differences. Figure 17 Image IMs2 (B) represents the printed image with the missing Err. Therefore, within the difference image IMd2, the portion corresponding to the missing Err shows a large difference compared to the other portions.

[0131] In S140 ( Figure 4 In the display unit 240, the processor 210 outputs the inspection results. The method for outputting the inspection results can be various. In this embodiment, the processor 210 displays the differential image on the display unit 240. Figure 1 The operator can easily identify defects in the printed image by observing the display unit 240. Instead, the processor 210 can output data representing the inspection results to a storage device (e.g., a non-volatile storage device 230 or an external storage device connected to the data processing device 200). Thus, the data representing the inspection results is stored in the storage device. The data representing the inspection results can be used for various processes (e.g., overall inspection processing of the T-shirt 700). After S140, the processor 210 ends the inspection processing.

[0132] As described above, in this embodiment, the processor 210 performs the following processing according to the first program 231. Figure 6 In steps S355 to S360, processor 210 uses the feature values ​​Fs of multiple read feature points Ts and the feature values ​​Ft of multiple reference feature points Tt to obtain multiple candidate feature point pairs M. Each candidate feature point pair M is a pair of feature points Ts and Tt. The read feature point Ts is the grayscale read image IMsg (…). Figure 7 The feature points in the image IMs are the feature points read from the image IMs. The reference feature point Tt is the feature point in the grayscale reference image IMtg, that is, the feature point in the reference image IMt.

[0133] exist Figure 5 In the processing (including S240), the processor 210 selects multiple candidate feature point pairs M that satisfy the selection condition CP from multiple candidate feature point pairs M as multiple feature point pairs MP. The selection condition CP includes conditions for selecting candidate feature point pairs M in S240. The conditions in S240 include Figure 10 The shape condition CW1 of S575. The shape condition CW1 is an example of the object combination MT used as the first condition for selecting the three feature points of MP (hereinafter, the shape condition CW1 will also be referred to as the first condition CW1), which is a combination of three candidate MA, MB, MC.

[0134] The first condition CW1 includes Figure 14 The conditions CR, CD, and CF. The ratio condition CR (S715) is determined using the ratio Rt and Rs of the lengths of two sides of the triangles TRt and TRs formed by the three candidate triangles MA, MB, and MC. Figure 12 (A) Figure 12 (B) Specifically, the ratio Rt is the ratio of the lengths Labt and Lact of the two sides Sabt and Sact of triangle TRt. The ratio Rs is the ratio of the lengths Labs and Lact of the two sides Sabs and Sacs of triangle TRs. The second interior angle condition CD (S720) is determined using the interior angles Abt and Abs of triangles TRt and TRs. The angle condition CF (S730) is determined using the angles Zt and Zs. Angle Zt ( Figure 12 (A) is the angle formed by the edge Sact (i.e., the line segment) connecting two reference feature points Tta and Ttc, and the direction Oct corresponding to one of the two reference feature points Tta and Ttc. Angle Zs ( Figure 12 (B) is the angle formed by the edge Sacs (i.e., the line segment) connecting the two reading feature points Tsa and Tsc, and the direction Ocs corresponding to one of the two reading feature points Tsa and Tsc. Thus, in this embodiment, the ratios Rt and Rs, the interior angles Abt and Abs, and the angles Zt and Zs are used to determine the first condition CW1.

[0135] In S250 ( Figure 5 In the process, the processor 210 uses multiple feature point pairs MP (i.e., multiple reserved candidate feature point pairs M) to determine the correspondence between the coordinates COs on the read image IMs and the coordinates COt on the reference image IMt (in this embodiment, the matrix Mtx).

[0136] Thus, in this embodiment, multiple candidate feature point pairs M that satisfy the selection condition CP are selected as multiple feature point pairs MP from multiple candidate feature point pairs M. The selection condition CP includes a first condition CW1 for selecting object combination MT as N feature point pairs MP, where the object combination MT is a combination of N (in this embodiment, N=3) candidate MA, MB, and MC. The first condition CW1 is determined using ratios Rt and Rs, interior angles Abt and Abs, and angles Zt and Zs. Therefore, the processor 210 is able to properly align the read image IMs with the reference image IMt.

[0137] Additionally, in this embodiment, as in S110 ( Figure 4 As explained in [reference needed], the read images IMs are represented by image data generated by optically reading the printed image IMpp. As in [reference needed] Figure 3 As explained in (A), the reference image IMt is represented by image data for printing. Through the above, the processor 210 is able to appropriately align the image IMs of the printed image IMpp with the reference image IMt represented by the image data for printing.

[0138] In addition, in this embodiment, Figure 5 In S240, processor 210 executes Figure 10 The processing. In Figure 10 In the processing, the processor 210 selects a combination of N (in this embodiment, N=3) candidate feature point pairs M as the object combination MT. Specifically, Figure 10 The processing includes steps S565 to S573. In steps S565 to S573, the processor 210 selects a candidate combination MU as the object combination MT, where the candidate combination MU is a combination of N candidate feature point pairs M. In step S565, which is included in steps S565 to S573, the processor 210 determines the combination condition CC used to select the candidate combination MU as the object combination MT. The combination condition CC includes... Figure 13 The conditions are CL1, CL2, and CA.

[0139] The first length condition CL1 (S615) and the second length condition CL2 (S620) indicate that the lengths Lab and Lac of the line segment connecting the two candidate feature points of M contained in the candidate combination MU are within the allowable length range LR. Specifically, in S615, the evaluation length Labt ( Figure 12 (A) and length Labs ( Figure 12 (B)). Length Labt is the length of the line segment (edge ​​Sabt) connecting two reference feature points Tta and Ttb. Length Labs is the length of the line segment (edge ​​Sabs) connecting two read feature points Tsa and Tsb. In S620, the length Lact ( Figure 12 (A) and length Lacs ( Figure 12 (B) Length Lacs is the length of the line segment (edge ​​Sact) connecting two reference feature points Tta and Ttc. Length Lacs is the length of the line segment (edge ​​Sacs) connecting two read feature points Tsa and Tsc.

[0140] The first interior angle condition CA (S625) means that the interior angle Aa of the triangle formed by the three feature points of the three candidate MA, MB, and MC included in the candidate combination MU is within the allowable interior angle range AR. Specifically, the evaluation of interior angle Aat ( Figure 12 (A) and interior angle Aas ( Figure 12 (B) The interior angle Aat is the size of the interior angle of the reference triangle TRt formed by the three reference feature points Tta, Ttb, and Ttc. The interior angle Aas is the size of the interior angle of the reading triangle TRs formed by the three reading feature points Tsa, Tsb, and Tsc.

[0141] As described above, the combination condition CC includes conditions CL1 and CL2 for the lengths of the line segments connecting the two feature points, and condition CA for the interior angles of the triangle formed by the three feature points. Therefore, the processor 210 is able to select an appropriate candidate combination MU as the object combination MT.

[0142] Additionally, in this embodiment, the combination condition CC ( Figure 10 S565) contains Figure 13 The unequal side condition CQ (S630). The unequal side condition CQ states that the triangle formed by the three feature points of the three candidate MA, MB, and MC contained in the candidate combination MU does not contain two sides with a length difference below the length difference threshold dLth. Specifically, from the reference triangle TRt ( Figure 12 The three edges Sabt, Sact, and Sbct of (A) are used to obtain three pairs of edges (Sabt-Sact, Sabt-Sbct, Sbct-Sact). If the absolute value of the difference in length between the three pairs of edges is greater than the length difference threshold dLth, the processor 210 determines that the reference triangle TRt satisfies the unequal side condition CQ. The unequal side condition CQ is satisfied when the shape of the reference triangle TRt is different from that of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle. For reading triangle TRs ( Figure 12 The same applies to (B). Thus, the combination condition CC includes the unequal-edge condition CQ. Therefore, the processor 210 is able to reduce the possibility of erroneously selecting an inappropriate candidate combination MU as the object combination MT.

[0143] In addition, in this embodiment, the selection condition CP ( Figure 5It also includes conditions for selecting candidate feature point pairs M in S230. The conditions in S230 include... Figure 8 Conditions CSt, CH, CSs, and CV from S425 to S455. Conditions CSt, CH, CSs, and CV use the first part of the region Pt ( Figure 9 The saturation St, hue Ht, brightness Vt, and second region Ps of (A) Figure 9 The saturation Ss, hue Hs, and brightness Vs of (B) are used to determine the color. Saturation Ss, hue Hs, and brightness Vt are examples of the first representative color value CJt of the first part region Pt containing feature point Tt in the reference image IMt. Saturation Ss, hue Hs, and brightness Vs are examples of the second representative color value CJs of the second part region Ps containing feature point Ts in the image IMs. The overall condition CSt, CH, CSs, and CV is an example of the second condition used to select candidate feature point pair M as feature point pair MP (hereinafter, the overall condition CSt, CH, CSs, and CV is referred to as the second condition CW2).

[0144] Thus, in this embodiment, the selection condition CP further includes a second condition CW2 for selecting candidate feature point pairs M as feature point pairs MP. The second condition CW2 is determined using the first representative color value CJt (containing St, Ht, and Vt) of a first partial region Pt containing feature point Tt in image IMt, and the second representative color value CJs (containing Ss, Hs, and Vs) of a second partial region Ps containing feature point Ts in image IMs. Therefore, the processor 210 can select a suitable candidate feature point pair M as feature point pair MP.

[0145] Furthermore, in this embodiment, the first representative color value CJt ( Figure 9 (A) represents the first saturation St and the first hue Ht, and the second represents the color value CJs. Figure 9 (B) represents the second hue Hs. For example... Figure 8 As shown in S440 and S445, when the first representative color value CJt represents a saturation St that is higher than the saturation threshold Sth (S440: Yes), the processor 210 uses the hue Ht represented by the first representative color value CJt and the hue Hs represented by the second representative color value CJs to determine the second condition CW2 (S445). As described above, when the first saturation St of the first representative color value CJt is higher than the saturation threshold Sth, the error of the second hue Hs of the second representative color value CJs can be a small value. Therefore, the processor 210 can appropriately determine the second condition CW2.

[0146] In addition, in this embodiment, the first representative color value is CJt ( Figure 9(A) represents the first saturation St and the first brightness Vt, and the second represents the color value CJs. Figure 9 (B) represents the second saturation Ss and the second brightness Vs. For example... Figure 8 As shown in S440, S450, and S455, when the saturation St of the first representative color value CJt and the saturation Ss of the second representative color value CJs are below the saturation threshold Sth (S440: No, S450: No), the processor 210 uses the luminance Vt represented by the first representative color value CJt and the luminance Vs represented by the second representative color value CJs to determine the second condition CW2 (S455). As described above, even when the saturation St and Ss are low, the luminances Vt and Vs can appropriately represent the color of the portion indicated by the feature points Tt and Ts. Therefore, the processor 210 can appropriately determine the second condition CW2.

[0147] Additionally, in this embodiment, as Figure 5 As shown, S240 is executed after S230. In S230 ( Figure 8 In S240, processor 210 selects multiple candidate feature point pairs M that satisfy the second condition CW2. Figure 10 In the process, processor 210 uses the multiple candidate feature point pairs M selected in S230 to select multiple candidate feature point pairs M that satisfy the first condition CW1. For example... Figure 10 As explained, processor 210 performs the first condition CW1 judgment (S575) on each of the multiple object combinations MT. When selecting an object combination MT (three candidate feature point pairs M) from p (p is an integer greater than or equal to 3) candidate feature point pairs M, the total number of object combinations MT is represented by the combination (pC3). The total number of object combinations MT increases sharply as the total number p of candidate feature point pairs M increases. If S240 is executed before S230, the total number of object combinations MT may increase, and the computational load of S240 (the judgment of the first condition CW1) may increase. In this embodiment, the total number of candidate feature point pairs M that can be included in the object combination MT is reduced by S230, thus alleviating the computational load.

[0148] B. Second embodiment: Figure 18 This is a flowchart illustrating an example of a process for acquiring data for a reference image. Unlike the embodiments described above, in this embodiment, image data generated by optically reading an image printed using reference image data is acquired as data for the reference image. This acquisition process can be performed in various situations. For example, the same printing image data can be used multiple times to print an image on the T-shirt 700. In this case, data for the reference image can be acquired by reading the image printed on the T-shirt 700. Hereinafter, it is assumed that in the acquisition process, data for the reference image is acquired using… Figure 1 The data processing unit 200, the printing unit 900, and the reading unit 100. The object image is set as IMp (...). Figure 3 The data in (A) is an image for printing.

[0149] In this embodiment, the operator inputs a start instruction for the acquisition process into the data processing device 200 by operating the operation unit 250. The processor 210 executes the acquisition process according to the start instruction and the second program 232.

[0150] In S810, processor 210 uses image data (here, object image IMp) to obtain a reference image. Figure 3 The data (A) causes the reading device 100 to perform printing of the image onto the T-shirt 700. Figure 2 Similar to T-shirt 700, an image IMp is printed on T-shirt 700. It is assumed that the printed image on T-shirt 700 is free of defects. Furthermore, the operator can confirm the absence of defects by observing the printed image on T-shirt 700. If defects are found in the printed image, the operator can re-execute S810 with data processing device 200 to obtain a printed image free of defects.

[0151] In S820, in the reading device 100 ( Figure 2 The T-shirt 700 is positioned on the support portion 140 of the machine so that the printed image can be seen. In this embodiment, the operator positions the T-shirt 700 on the support portion 140. Alternatively, a machine (e.g., a robotic arm) can position the T-shirt 700 on the support portion 140. After the T-shirt 700 is positioned, the operator inputs a travel instruction by operating the operation unit 250. The processor 210 provides a read instruction to the reading device 100 based on the travel instruction. The control unit 110 of the reading device 100 reads the T-shirt 700 based on the read instruction. The reading process and... Figure 4 The same procedure applies to S110. Control device 110 generates data representing a read image of the T-shirt 700 and provides this data to data processing device 200. Processor 210 of data processing device 200 stores the acquired read image data as reference image data in storage device 215 (e.g., non-volatile storage device 230). Although not illustrated, the reference image and... Figure 3 Similarly, the image read by (B) represents the portion of the T-shirt 700 containing the printed image.

[0152] pass Figure 18 The data obtained from the processing of the reference image is in Figure 4In the inspection process, the data used is replaced by the reference image IMt. Thus, the reference image can be represented by image data generated by optically reading the printed image. In this case, the processor 210, as in the first embodiment, is also capable of appropriately aligning the read image IMs with the reference image.

[0153] also, Figure 18 The start instruction for processing and the execution instruction for S820 can be input to the data processing device 200 via the communication interface 270 from other devices different from the data processing device 200.

[0154] C. Third embodiment: Figure 19 It represents the unequal edge condition ( Figure 13 A diagram of another embodiment of (S630) is shown. In the diagram, an alternative is shown. Figure 13 S630b is executed in conjunction with S630. In S630b, the processor 210 determines whether the inequality condition CQb is satisfied. The inequality condition CQb and the inequality condition CQ ( Figure 13 Similarly, the scalene condition CQb indicates that the lengths of the three sides of the triangle are different from each other. That is, the scalene condition CQb indicates that the shape of the triangle is different from that of an isosceles triangle, a triangle similar to an isosceles triangle, an equilateral triangle, and a triangle similar to an equilateral triangle.

[0155] In this embodiment, the unequal edge condition CQb is all of the following conditions CQb1 to CQb3.

[0156] (CQb1) The ratio of lengths Lab and Lac, Lab / Lac, is outside the ratio range PR. (CQb2) The ratio of lengths Lab and Lbc, Lab / Lbc, is outside the ratio range PR. (CQb3) The ratio of lengths Lbc and Lac, Lbc / Lac, is outside the ratio range PR, where the ratio range PR is the range above the lower limit RRth1 and below the upper limit RRth2 (where RRth1 < 1 < RRth2).

[0157] Conditions CQb1 to CQb3 indicate that the ratio of the lengths of the two sides is different from 1. The unequal side condition CQb indicates that the triangle (e.g., referencing triangle TRt or reading triangle TRs) does not contain two sides with a ratio of lengths within the range PR containing 1. The lower bound RRth1 and upper bound RRth2 are related to the radii r1 and r2 of the selection range SR. Figure 11 Similarly, the decision is made in advance through experiments by forming a suitable graph for judgment using combinations of three candidate MA, MB, and MC. For example, the lower limit RRth1 can be a value above 0.9 and below 0.98. The upper limit RRth2 can be a value above 1.02 and below 1.1.

[0158] If the unequal boundary condition CQb is satisfied (S630b: yes), processor 210 is transferred to S635 ( Figure 13 If the unequal side condition CQb is not met (S630b: No), processor 210 moves to S640, and it is determined that the individual combination condition is not met.

[0159] As described above, in this embodiment, the combination condition CC ( Figure 10 S565) includes the unequal edge condition CQb ( Figure 19 (S630b). Therefore, with Figure 13 Similarly, in other embodiments, processor 210 can reduce the likelihood of erroneously selecting an inappropriate candidate combination MU as the object combination MT.

[0160] D. Variations: (1) First condition CW1 ( Figure 10 (S575) is not limited to Figure 14 The conditions described herein can also refer to various conditions indicating that the object combination MT consists of appropriate N candidate feature point pairs M, where the object combination MT is a combination of N (N is 2 or 3) candidate feature point pairs M. For example, the reading device 100 can be configured such that the scale relative to the object is approximately the same between the reference image IMt and the read image IMs. In this case, the first condition CW1 can also include a third length condition indicating that the length of a specific side of the triangle is within an acceptable range. The length of the side can be, for example, the lengths of the sides Sbct, Sbcs, Lbct, Lbcs (…). Figure 12 (A) Figure 12 (B)). When the scale between the reference image IMt and the read image IMs is approximately the same, the third length condition can easily remove object combinations MT that contain inappropriate candidate feature point pairs M. The allowable range of the third length condition can be... Figure 13 The permissible length range LR is similarly determined beforehand through experimentation (for example, the permissible range of the third length condition can be the same as the permissible length range LR). Thus, the first condition CW1 can be determined using the lengths of the sides of the triangle.

[0161] Additionally, the reading device 100 can be configured such that the rotation angle of the object between the reference image IMt and the read image IMs is approximately the same. In this case, Figure 6 In S320 and S325, non-rotationally invariant feature quantities Ft and Fs can be calculated (e.g., BRIEF (Binary Robust Independent Elementary Features)). Furthermore, the calculation of the feature point orientation can be omitted. Conditions for using the feature point orientation (e.g., angular condition CF) can also be omitted.

[0162] The first condition CW1 can be a condition that allows for the judgment of M using two candidate feature points (e.g., the third length condition). In this case, the candidate combination MU and the object combination MT can be combinations of two candidate feature point pairs M. The first condition CW1 is preferably determined using one or more of the following four parameters: the ratio of the lengths of the two sides of the triangle formed by the three feature points (e.g., ratio Rt, Rs), the length of the sides of the triangle (e.g., length Lbct, Lbcs), the interior angles of the triangle (e.g., interior angles Abt, Abs), and the angle formed by the line segment connecting the two feature points and the direction corresponding to one of the two feature points (e.g., angles Zt, Zs).

[0163] (2) Combination condition CC ( Figure 10 (S565) is not limited to Figure 13 , Figure 19 The conditions described can also be various conditions representing the combination of N (N is 2 or 3) candidate feature point pairs M to form a graph suitable for the judgment of the first condition CW1. For example, the combination condition CC can include lengths Lbct, Lbcs ( Figure 12 (A) Figure 12 (B) is within the allowable length range LR. The unequal side condition CQ ( Figure 13 The condition CQb can be composed of one or two conditions pre-selected from conditions CQ1 to CQ3. Figure 19 The condition CC can be composed of one or two conditions pre-selected from conditions CQb1 to CQb3. The combined condition CC can be composed of one or more conditions pre-selected from the following: first length condition CL1, second length condition CL2, first interior angle condition CA, unequal side condition CQ, and unequal side condition CQb. Furthermore, conditions can be omitted. Figure 10 S565 to S570 (i.e., the judgment of combination condition CC).

[0164] (3) Selection processing of feature point pairs using color information ( Figure 5 (S230) is not limited to Figure 8The processing can be various. For example, processor 210 can replace the processing of S425 and S435 to calculate the hue of each of the multiple pixels within the first partial region Pt, and use the multiple hues to calculate the representative hue of the first partial region Pt. This also applies to saturation and brightness. The same applies to the representative color value of the second partial region Ps. In addition, the second condition CW2 can be various conditions using the first representative color value of the first partial region Pt and the second representative color value of the second partial region Ps. For example, the second condition CW2 can be either the hue condition CH or the brightness condition CV. The second condition CW2 preferably indicates that the first representative color value of the first partial region Pt is similar to the second representative color value of the second partial region Ps.

[0165] (4) Positional processing can be replaced by various other processing methods. Figure 5 The processing can be done in several ways. For example, S240 can be executed before S230. Alternatively, S230 (i.e., the judgment of the second condition CW2) can be omitted.

[0166] (5) The detection methods for feature points Tt and Ts (i.e., key points) can be various methods for detecting points representing parts of an object in an image, replacing those in S310 and S315. Figure 6 The methods described in [the document] can be used to detect corners, such as those using DoG (Difference-of-Gaussian) extremum (maximum and minimum) search, Harris corner detection, FAST (Features from Accelerated Segment Test) corner detection, SIFT (Scale Invariant Feature Transform), SURF (Speeded UpRobust Features), and ORB (Oriented FAST and Rotated BRIEF).

[0167] The calculation methods for feature quantities Ft and Fs (i.e., feature descriptors) can be replaced by those in S320 and S325. Figure 6Various methods for calculating information describing the features of keypoints are described in the document. Algorithms for calculating feature descriptors can be pre-selected from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, and A-KAZE. Furthermore, the distance dF between feature quantities Ft and Fs can be calculated using various methods suitable for the data structures of feature quantities Ft and Fs. When feature quantities Ft and Fs are represented by binary vectors, the distance dF can be the Hamming distance. Instead of the Hamming distance, the distance dF can be various other distances (e.g., L1 norm, L2 norm (also known as Euclidean distance), etc.). Norms can be applied to various feature descriptors.

[0168] (6) Processing other than using multiple feature points Tt and Ts to determine the correspondence of coordinates can be performed according to a different procedure than the first procedure 231. For example, the detection of feature points Tt and Ts and the calculation of feature quantities Ft and Ts ( Figure 6 (S305 to S325) can be executed according to other procedures. Figure 4 S130 and S140 can be executed according to other procedures. In addition, processing other than determining the correspondence of coordinates using multiple feature points Tt and Ts can be performed by other devices different from data processing device 200.

[0169] (7) The printing medium is not limited to T-shirts 700, but can be various garments (e.g., various shirts, jackets, short-sleeved shirts, etc.). The printing medium can be various fabrics such as clothes and bags. The printing medium is not limited to fabric, but can also be various media such as paper, film, and leather. In addition, the printing device 900 can also replace inkjet printing devices with other types of printing devices (e.g., laser printing devices).

[0170] (8) The correspondence between coordinates ( Figure 16 This can represent various transformations, such as homography, to replace affine transformations. Furthermore, the correspondence between coordinates can replace the matrix Mtx and be represented in various forms, such as lookup tables.

[0171] The correspondence of coordinates is not limited to inspection and can be used through various processes. For example, in the processing of metal parts, the position and orientation of the metal part relative to the tool may be offset. Here, the position and orientation of the metal part relative to the tool can be determined by analyzing an image of the metal part taken by a digital camera fixed to the tool. For example, the processor 210 can determine the reference position of the metal part in the captured image (and thus, the position and orientation of the metal part relative to the tool) by performing alignment between a reference image showing the reference position of the metal part and the captured image. As an alignment process, the process described in the above embodiment can be used ( Figure 5 The processing can be the same as described in the above variations. Thus, the image readout can be represented by the reading sensor 180 ( Figure 2 The image can be an image of an object read by a line sensor such as a digital camera, or an image of an object read by a area sensor such as a digital camera. Furthermore, the reference image can be various images instead of images related to printing (e.g., a pre-prepared image representing a specific part of an object). In any case, the image can be a grayscale image instead of a color image.

[0172] (9) The data processing device that determines the correspondence between coordinates is not limited to a personal computer (e.g., data processing device 200). Figure 1 (), or various other devices (e.g., smartphones, tablets, control devices installed on reading devices, etc.). In addition, multiple devices (e.g., computers) that can communicate with each other via a network can share the data processing functions of the data processing device in parts, and provide data processing functions as a whole (including the system of these devices corresponding to the data processing device).

[0173] In the above embodiments, a portion of the hardware-implemented structure can be replaced with software, and conversely, a portion or all of the software-implemented structure can be replaced with hardware. For example, Figure 5 The processing of the S250 can be performed by dedicated hardware circuits such as application-specific integrated circuits (ASICs).

[0174] Furthermore, where some or all of the functions of this disclosure are implemented by a computer program, the program can be provided in the form of a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used while stored on the same or different recording medium (computer-readable recording medium) as it is provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer such as various ROMs, and external storage devices connected to a computer such as hard disk drives.

[0175] The above-described embodiments and modifications can be appropriately combined. Furthermore, the above-described embodiments and modifications are for ease of understanding of this disclosure and do not limit the invention. The invention can be modified and improved without departing from its spirit, and its equivalents are included in the invention.

[0176] Explanation of reference numerals in the attached figures

[0177] 100…Reading device, 110…Control device, 120…Conveying device, 122…Position sensor, 130…Workbench, 140…Support, 180…Reading sensor, 190…Housing, 200…Data processing device, 210…Processor, 215…Storage device, 220…Volatile storage device, 230…Non-volatile storage device, 231…First program, 232…Second program, 240…Display unit, 250…Operating unit, 270…Communication interface, 700…T-shirt, 900…Printing device, Abs, Abt…Interior angle, AR…Permissible interior angle range, CC…Combination condition, CJt…First representative color value, CJs…Second representative color value, COs, COt…Coordinates, CP…Selection condition, CW1…First line Component, CW2…Second condition, Ocs, Oct…Direction, Fs, Ft…Feature quantity, Hs, Ht…Hue, IMs, IMs2…Read image, IMt…Refer to image, Labs, Labt, Lacs, Lact, Lbcs, Lbct…Length, Sabs, Sabt, Sacs, Sact, Sbcs, Sbct…Side, LR…Allowable length range, Rs, Rt…Ratio, PR…Ratio range, M…Candidate feature point pair, MP…Feature point pair, MT…Object combination, Pt…First part region, Ps…Second part region, Ss, St…Saturation, Sth…Saturation threshold, TRs, TRt…Triangle, TS, Tt…Feature point, Vs, Vt…Brightness, Zs, Zt…Angle.

Claims

1. A program that enables a computer to perform the following functions: The candidate acquisition function uses the feature values ​​of multiple feature points in the read image and the feature values ​​of multiple feature points in the reference image to obtain multiple candidate feature point pairs. The candidate feature point pairs are the pair of feature points in the read image and the feature points in the reference image. For the selection function, multiple candidate feature point pairs that meet the selection criteria are selected as multiple feature point pairs from the multiple candidate feature point pairs. The selection criteria include a first condition for selecting a combination of N candidate feature point pairs, i.e., an object combination, as N feature point pairs. The first condition is determined using one or more of the following four parameters: the ratio of the lengths of two sides of the triangle formed by the three feature points, the length of the sides of the triangle, the interior angle of the triangle, and the angle formed by the line segment connecting two feature points and the direction corresponding to one of the two feature points. N is 2 or 3; and The function determines the correspondence between the coordinates on the read image and the coordinates on the reference image using the multiple feature points.

2. The procedure according to claim 1, wherein, The read image is represented by image data generated by optically reading the printed image. The reference image is represented by image data for printing or by image data generated by optically reading an image printed using reference image data.

3. The procedure according to claim 1 or 2, wherein, The program also enables the computer to perform the following combination selection function: selecting a combination of N candidate feature point pairs as the object combination. The combination conditions for selecting the combination of the N candidate feature point pairs as the object combination include one or both of the following conditions: the interior angles of the triangle formed by the three feature points contained in the N candidate feature point pairs are within the allowable interior angle range; and the length of the line segment connecting the two feature points contained in the N candidate feature point pairs is within the allowable length range.

4. The procedure according to claim 1 or 2, wherein, The program also enables the computer to perform the following combination selection function: selecting a combination of N candidate feature point pairs as the object combination. The combination conditions for selecting the combination of the N candidate feature point pairs as the object combination include one or both of the following conditions: the triangle formed by the three feature points contained in the N candidate feature point pairs does not contain two sides whose length difference is below the difference threshold; and the triangle does not contain two sides whose length ratio is in the range of ratios including 1.

5. The procedure according to claim 1 or 2, wherein, The selection criteria also include a second criterion for selecting the candidate feature point pair as the feature point pair, the second criterion being determined using a first representative color value of a first partial region containing the feature points in the reference image and a second representative color value of a second partial region containing the feature points in the read image.

6. The procedure according to claim 5, wherein, The first representative color value represents saturation and hue. The second representative color value indicates hue. When the first representative color value represents a saturation level higher than the saturation threshold, the selection function uses the hue represented by the first representative color value and the hue represented by the second representative color value to determine the second condition.

7. The procedure according to claim 5, wherein, The first representative color value represents saturation and brightness. The second representative color value indicates saturation and brightness. When the saturation of the first representative color value and the saturation of the second representative color value are below a saturation threshold, the selection function uses the brightness represented by the first representative color value and the brightness represented by the second representative color value to determine the second condition.

8. The procedure according to claim 5, wherein, The selection function uses multiple candidate feature point pairs that satisfy the second condition to select multiple candidate feature point pairs that satisfy the first condition.

9. A data processing apparatus comprising: The candidate acquisition unit uses the feature values ​​of each of the multiple feature points in the read image and the feature values ​​of each of the multiple feature points in the reference image to acquire multiple candidate feature point pairs, wherein the candidate feature point pairs are pairs of feature points in the read image and feature points in the reference image. For the selection unit, multiple candidate feature point pairs that satisfy the selection criteria are selected from the multiple candidate feature point pairs as multiple feature point pairs. The selection criteria include a first condition for selecting a combination of N candidate feature point pairs, i.e., an object combination, as N feature point pairs. The first condition is determined using one or more of the following four parameters: the ratio of the lengths of two sides of the triangle formed by the three feature points, the length of the sides of the triangle, the interior angle of the triangle, and the angle formed by the line segment connecting two feature points and the direction corresponding to one of the two feature points. N is 2 or 3; and The decision unit uses the plurality of feature points to determine the correspondence between the coordinates on the read image and the coordinates on the reference image.

Images