Computer program and image processing method

Abstract

This process matches feature points between multiple images. [Solution] The computer program implements the following functions: an object identification function that identifies multiple reference objects in a reference image and multiple target objects in a target image; a correspondence determination function that performs a correspondence process to associate each of the multiple reference objects with a target object; and a matching function that performs a matching process to determine matching pairs, which are pairs of reference feature points and target feature points that correspond to each other, from among the multiple reference feature points of the reference image and the multiple target feature points of the target image. The matching function determines matching pairs such that the reference feature points of the first reference object form a matching pair with the target feature points of the first target object corresponding to the first reference object, and the reference feature points of the second reference object form a matching pair with the target feature points of the second target object corresponding to the second reference object.

Description

Technical Field

[0001] This specification relates to a technique for aligning positions between a plurality of images.

Background Art

[0002] In various processes, alignment between a plurality of images can be performed. Patent Document 1 discloses a technique for detecting defects in an image formed on a sheet by an image forming apparatus such as a printer or a copier. In this technique, together with a job image instructed to be printed by a user, a marker image for positioning is formed on the same sheet. An image reading unit reads the sheet surface and generates a target image. An inspection unit determines the position of a target image corresponding to a reference image based on each feature point of the job image and the marker image extracted from the target image to be inspected and each feature point of the job image and the marker image extracted from the reference image. The inspection unit compares the reference image and the target image after alignment, and detects an image area where the difference in pixel values is greater than or equal to a threshold value as a defect.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Alignment between a plurality of images is not easy and there is room for improvement.

[0005] This specification discloses a technique for performing matching of feature points between a plurality of images.

Means for Solving the Problems

[0006] The technique disclosed in this specification is made to solve at least a part of the above problems and can be realized as the following application examples.

[0007] [Application Example 1] A computer program that causes a computer to execute the following: an object identification function that identifies a plurality of reference objects in a reference image and a plurality of target objects in a target image; a correspondence determination function that performs a correspondence process to associate each of the plurality of reference objects with the target objects; and a matching function that performs a matching process to determine matching pairs, which are pairs of reference feature points and target feature points that correspond to each other, from among a plurality of reference feature points in the reference image and a plurality of target feature points in the target image, wherein the matching function determines the matching pairs such that the reference feature points of the first reference object constitute the matching pair with the target feature points of the first target object corresponding to the first reference object, and the reference feature points of the second reference object constitute the matching pair with the target feature points of the second target object corresponding to the second reference object.

[0008] According to the above configuration, matching pairs are determined such that the reference feature points of the first reference object form a matching pair with the target feature points of the first target object corresponding to the first reference object, and the reference feature points of the second reference object form a matching pair with the target feature points of the second target object corresponding to the second reference object. As a result, it is possible to suppress the determination of matching pairs that show an incorrect correspondence in the matching of feature points. In addition, the computation time required to determine matching pairs can be reduced.

[0009] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, in the form of an image processing method and an image processing apparatus, a computer program for realizing the functions of such methods or apparatuses, a recording medium (e.g., a non-temporary recording medium) on which such computer programs are recorded, and so on. [Brief explanation of the drawing]

[0010] [Figure 1] An explanatory diagram showing an image processing apparatus as one embodiment. [Figure 2] A perspective view showing an example of the reading device 100. [Figure 3] A diagram showing examples of various images. [Figure 4] A flowchart illustrating an example of the inspection process. [Figure 5] A flowchart for object identification processing. [Figure 6] Flowchart for object-based processing. [Figure 7] Flowchart of the first matching process. [Figure 8] Diagram illustrating the first matching process. [Figure 9] Flowchart for the second matching process. [Figure 10] Diagram illustrating the reference feature set TtS and the target feature set TsS. [Figure 11] A diagram illustrating the affine transformation matrix Mtx. [Figure 12] A diagram illustrating an example of a difference image (IMd). [Figure 13] A diagram illustrating the object identification process of the reference image IMt in a modified example. [Figure 14] A flowchart of the object mapping process for modified examples. [Modes for carrying out the invention]

[0011] A. Examples A1. Device Configuration Figure 1 is an explanatory diagram showing an image processing apparatus as one embodiment. The image processing apparatus 200 is, for example, a personal computer. The image processing apparatus 200 performs inspection processing of printed images. The image processing apparatus 200 comprises a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to perform data processing, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile memory device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile memory device 230 is, for example, a flash memory. The non-volatile memory device 230 stores data of a computer program PG, a reference image IMt, and a color conversion profile PF. The processor 210 realizes an inspection process described later by executing the computer program PG. As will be described in detail later, the reference image IMt and the color conversion profile PF are used in the inspection process.

[0013] The display unit 240 is a device configured to display an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as buttons, levers, or a touch panel disposed on top of the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the image processing apparatus 200 by operating the operation unit 250.

[0014] The communication interface 270 is an interface for communicating with other devices (for example, including one or more of a USB interface, a wired LAN interface, a wireless interface of IEEE 802.11, an interface of an industrial camera (for example, CameraLink, CoaXPress, etc.)). In this embodiment, a reading device 100 and a printing device 900 are connected to the communication interface 270. The printing device 900 is a so-called inkjet printing device, and prints an image on a printing medium such as cloth or paper by ejecting ink onto the printing medium. The reading device 100 generates data (scan data) of a reading image representing an object by optically reading the object to be read. In this embodiment, the printing medium of the printing device 900 is a T-shirt, and the object to be read by the reading device 100 is a T-shirt with an image printed thereon.

[0015] FIG. 2 is a perspective view showing an example of the reading device 100. The first direction Da and the second direction Db in the figure indicate horizontal directions, and the third direction Dc indicates the vertically upward direction. The first direction Da and the second direction Db are perpendicular to each other.

[0016] In this embodiment, the reading device 100 includes a housing 190, a table 130, a support portion 140 fixed to the upper surface of the table 130, a conveying device 120, a reading sensor 180, and a control device 110. The control device 110, the conveying device 120, and the reading sensor 180 are fixed to the housing 190.

[0017] The support portion 140 is a plate-like member that forms a flat upper surface for supporting the object to be read (such a member is also called a platen). In the figure, a T-shirt 700 having a printed image IMp is placed on the support portion 140.

[0018] The conveying device 120 is configured to convey the table 130 in a direction parallel to the second direction Db. The configuration of the conveying device 120 may vary. Although not shown in the figures, in this embodiment the conveying device 120 has a rail that supports the table 130 so as to be slidable in a direction parallel to the second direction Db, a plurality of pulleys, a belt that is wrapped around the plurality of pulleys and partly fixed to the table 130, and an electric motor that rotates the pulleys. By rotating the pulleys with the electric motor, the table 130 (and by extension, the support part 140) moves in a direction parallel to the second direction Db. The conveying device 120 further has a position sensor 122 (for example, a rotary encoder) that detects the position of the table 130 on the conveying path.

[0019] The reading sensor 180 is positioned higher than the support unit 140 in the middle of the transport path Pth of the support unit 140. The reading sensor 180 includes a line sensor composed of multiple photoelectric conversion elements arranged in a direction intersecting the transport direction Db (in this embodiment, in a direction Da perpendicular to the transport direction Db) (for example, a Contact Image Sensor (CIS) or a Charge Coupled Device (CCD)). The reading sensor 180 faces downward. The reading sensor 180 can read the portion of the object supported by the support unit 140 that is located below the reading sensor 180.

[0020] When reading the T-shirt 700, the reading device 100 transports the table 130 in a direction parallel to the second direction Db. The reading sensor 180 repeatedly reads the T-shirt 700 during transport. As a result, the reading sensor 180 can read approximately the entire portion of the T-shirt 700 that is supported by the support portion 140.

[0021] The control device 110 is an electrical circuit configured to control the transport device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates scan data by controlling the transport device 120 and the reading sensor 180.

[0022] A2. Printing process In this embodiment, an image is printed on a T-shirt 700 (Figure 2). The image printing is performed, for example, as part of a T-shirt sales service. For example, the service provider prints an image on the T-shirt 700 using a printing device 900 according to the customer's order. The image data for printing may be provided by the customer or prepared by the service provider.

[0023] Figure 3 shows examples of various images. Figure 3(A) shows an image represented by print-ready image data (referred to as the reference image IMt). In this embodiment, the data of the reference image IMt is bitmap data representing the color values ​​(here, the gradation values ​​of red R, green G, and blue B (for example, values ​​between zero and 255)) of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. In the example of Figure 3(A), the reference image IMt includes a background Bt and four objects Ot (Ot1-Ot4). To distinguish the objects Ot in the reference image IMt from the objects in the target images IMs described later, they are also called reference objects Ot.

[0024] Figure 2 shows a printed image IMp formed by printing a reference image IMt by a printing device 900 on the surface of a T-shirt 700. The printed image IMp may have various defects due to, for example, errors during printing. For example, due to an abnormality in ink ejection, a portion of the image may be missing in the printed image IMp. The image processing device 200 detects defects in the printed image IMp through an inspection process described later. In this embodiment, the data of the reference image IMt is used in the inspection process. The data of the reference image IMt used for printing is stored in the storage device 215 (for example, a non-volatile storage device 230) of the image processing device 200 for the inspection process (Figure 1).

[0025] A3. Inspection process Figure 4 is a flowchart illustrating an example of the inspection process. For inspection, the T-shirt 700 is placed on the support unit 140 of the reader 100 (Figure 2) so that the printed image IMp is visible. In this embodiment, an operator places the T-shirt 700 on the support unit 140. Alternatively, a machine (e.g., a robotic arm) may place the T-shirt 700 on the support unit 140. After the T-shirt 700 is placed, an instruction to start the inspection process is input to the image processing device 200 (Figure 1). In this embodiment, the operator inputs the instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the instruction to start. The instruction to start the inspection process may be input to the image processing device 200 via the communication interface 270 by a device other than the image processing device 200.

[0026] The processor 210 of the image processing device 200 executes inspection processing according to the computer program PG. In S100, the processor 210 acquires data of the reference image IMt (Figure 3(A)) used for printing on the T-shirt 700. In this embodiment, data of the reference image IMt stored in the non-volatile storage device 230 is acquired.

[0027] In S105, the processor 210 causes the reader 100 to read the T-shirt 700 and acquires data of the read image IMc. Specifically, the processor 210 supplies a read instruction to the reader 100. The control device 110 of the reader 100 reads the T-shirt 700 by controlling the reading sensor 180 and the transport device 120 in response to the read instruction. The control device 110 generates data (scan data) of the read image IMc representing the read T-shirt 700. The processor 210 of the image processing device 200 acquires the data of the read image IMc from the reader 100.

[0028] Figure 3(B) shows the read image IMc. The read image IMc in Figure 3(B) is an image of T-shirt 700 containing the printed image IMp.

[0029] In S107, the processor 210 extracts a predetermined specific range SA from the read image IMc that corresponds to the chest area of ​​the T-shirt 700, and acquires the target image IMs.

[0030] Figure 3(C) shows the target image IMs. In this embodiment, the data of the target image IMs is bitmap data representing the color values ​​(here, the gradation values ​​of red R, green G, and blue B (for example, values ​​between zero and 255)) of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. The target image IMs represents the portion of the read image IMc in Figure 3(C) that includes the printed image IMp (Figure 3(C)).

[0031] As described above, the printed image IMp is obtained by printing the reference image IMt (Figure 3(A)). For this reason, the target image IMs, like the reference image IMt, includes a background Bs and four objects Os (Os1-Os4). To distinguish the objects Os in the target image IMs from the reference object Ot in the reference image IMt described above, they are also called target objects Os. As shown in Figure 3, there is a one-to-one correspondence between the reference object Ot in the reference image IMt and the target object Os in the target image IMs. That is, the reference object Oti (where i is an integer from 1 to 4) in Figure 3(A) corresponds to the target object Osi in Figure 3(B).

[0032] In S110, the processor 210 performs object identification processing. Object identification processing is the process of identifying multiple reference objects Ot in the reference image IMt and multiple target objects Os in the target image IMs.

[0033] Figure 5 is a flowchart of the object identification process. In S210, the processor 210 performs color conversion processing on the target images IMs using the color conversion profile PF (Figure 1). The color conversion profile PF is a profile for matching the color of the reference object Ot in the reference image IMt with the color of the target object Os corresponding to the reference object Ot. Specifically, in this embodiment, the values ​​of multiple pixels in the target images IMs are color values ​​of the scan color system. The scan color system is an equipment-dependent color system that depends on the characteristics of the reader 100. In this embodiment, the values ​​of multiple pixels in the reference image IMt are color values ​​of the sRGB color system, which is standardly used for displaying images on a monitor. The color conversion profile PF is a profile (e.g., a lookup table) that shows the correspondence between the color values ​​of the scan color system and the color values ​​of the sRGB color system. The processor 210 converts the values ​​of multiple pixels in the target images IMs (color values ​​of the scan color system) to sRGB color values ​​by referring to the color conversion profile PF. This makes it possible to nearly match the color value representing the color of a specific reference object Ot (e.g., Ot1) in the reference image IMt with the color value representing the color of the target object Os (e.g., Os1) that corresponds to the specific reference object Ot in the target image IMs.

[0034] In S220, the processor 210 performs grayscale conversion on the reference image IMt and the target image IMs, respectively, to generate grayscale reference image IMt and target image IMs. A known relationship can be used as the correspondence between color values ​​(e.g., RGB values) and grayscale color values ​​(e.g., the correspondence between RGB values ​​in the sRGB color system and luminance value Y in the YCbCr color system).

[0035] In S230, the processor 210 performs a binarization process on the grayscale reference image IMt and the target image IMs, respectively, classifying multiple pixels within these images into object pixels and background pixels. Object pixels are the pixels that make up the object, and background pixels are the pixels that make up the background. This generates the reference binary image Bit and the target binary image BIs. For the binarization process, a known method such as Otsu's binarization is employed.

[0036] Figure 3(D) shows the reference binary image Bit, and Figure 3(E) shows the target binary image BIs. The hatched areas in these images represent the reference object region At and the target object region As, which are composed of object pixels.

[0037] In S240, the processor 210 performs denoising on the reference binary image Bit and the target binary image BIs, respectively. In this embodiment, the denoising process involves performing a stenosis process followed by a dilation process on the binary images Bit and BIs. This removes, for example, isolated object pixels within the binary image as noise. The stenosis and dilation processes may be various known processes. The stenosis process may, for example, be a process that sets the pixel of interest as a background pixel when multiple surrounding pixels, defined by a filter placed at the position of the pixel of interest, include background pixels. The dilation process may, for example, be a process that sets the pixel of interest as an object pixel when multiple surrounding pixels, defined by a filter (also called a kernel) placed at the position of the pixel of interest, include object pixels.

[0038] In S250, the processor 210 performs a dilation process on both the denoised reference binary image Bit and the target binary image BIs. As a result, the reference object region At in the reference binary image Bit and the target object region As in the target binary image BIs become slightly larger. This dilation process is the same as the dilation process included in the denoising process described above.

[0039] In S260, the processor 210 performs labeling on the dilated reference binary image Bit and the target binary image BIs, respectively. Specifically, the processor 210 assigns one identifier to a contiguous region composed of one or more consecutive object pixels. The processor 210 then assigns different identifiers to multiple contiguous regions that are far apart from each other. A contiguous region to which one identifier is assigned is identified as a single object region.

[0040] Through the labeling process, four reference object regions At(At1-At4) are identified in the reference binary image Bit (Figure 3(D)). Since there is a one-to-one correspondence between multiple pixels in the reference binary image Bit and multiple pixels in the reference image IMt, the identification of four reference object regions At(At1-At4) in the reference binary image Bit is equivalent to the identification of four reference objects Ot(Ot1-Ot4) in the reference image IMt.

[0041] Through the labeling process, four target object regions As(As1-As4) are identified in the target binary image BIs (Figure 3(E)). Since there is a one-to-one correspondence between multiple pixels in the target binary image BIs and multiple pixels in the target image IMs, the identification of four target object regions As(As1-As4) in the target binary image BIs is equivalent to the identification of four target objects Os(Os1-Os4) in the target image IMs.

[0042] Figure 3(A) shows the reference object regions At1-At4, identified using the reference binary image Bit, as dashed lines. Figure 3(C) shows the target object regions As1-As4, identified using the target binary image BIs, as dashed lines. These object regions At1-At4 and As1-As4 are enlarged by the dilation process S250 in Figure 5, ensuring that they reliably encompass the corresponding objects Ot1-Ot4 and Os1-Os4.

[0043] Once the object identification process is complete, at S120 in Figure 4, the processor 210 determines whether the number of reference objects Ot is 2 or greater. In the example in Figure 3(A), 4 reference objects Ot have been identified, so it is determined that the number of reference objects Ot is 2 or greater.

[0044] If the number of reference objects Ot is 1 (S120: NO), the processor 210 sets the entire image as a single object in S130. That is, the entire reference image IMt is set as a single reference object Ot, and the entire target images IMs are set as a single target object Os.

[0045] If the number of reference objects Ot is 2 or more (S120: YES), the processor 210 determines in S125 whether the number of reference objects Ot and the number of target objects Os are the same. As described above, the reference image IMt is the image used when printing an image on the T-shirt 700, and the target image IMs is the image obtained by reading the printed image IMp on the T-shirt 700. For this reason, the number of reference objects Ot and the number of target objects Os are usually the same. If the number of reference objects Ot and the number of target objects Os are different, it is possible that, for example, the object identification process has failed to identify the object. In the example in Figure 3, it is determined that the number of reference objects Ot (4) and the number of target objects Os (4) are the same.

[0046] If the number of reference objects Ot and the number of target objects Os are different (S125: NO), the processor 210 sets the entire image as a single object in S130. That is, as described above, the entire reference image IMt is set as a single reference object Ot, and the entire target image IMs is set as a single target object Os.

[0047] If the number of reference objects Ot and the number of target objects Os are the same (S125: YES), the processor 210 executes object mapping in S135. Object mapping is the process of associating a target object Os with each of the N reference objects Ot (where N is an integer greater than or equal to 2).

[0048] Figure 6 is a flowchart of the object mapping process. In S270, the processor 210 calculates the centroids Ct and Cs of N reference objects Ot and N target objects Os. Specifically, the processor 210 calculates the coordinates of the centroids Ct and Cs as the average of the coordinates of multiple object pixels that make up the object regions At and As in which each object Ot and Os is located. In the example in Figure 3, the number of reference objects Ot and the number of target objects Os are both 4 (N=4). Figure 3(A) shows the centroids Ct1-Ct4 of the four reference objects Ot1-Ot4. Figure 3(C) shows the centroids Cs1-Cs4 of the four target objects Os1-Os4.

[0049] The processor 210 executes the loop process S20 between the start L20s and the end L20e, with each of the (2 × N) objects Ot and Os being the object of interest.

[0050] In S282, the processor 210 selects one unprocessed object from among (2 × N) objects Ot and Os as the object of interest.

[0051] In S284, the processor 210 identifies two objects (also called neighboring objects) that have a short distance between their centroids and the object of interest. The two neighboring objects are the objects in the same image as the object of interest (reference image IMt or target image IMs) that have the shortest distance between their centroids and the object of interest, and the second shortest distance.

[0052] In Figure 3(A), the reference object Ot3 has the shortest distance between its centroid and reference object Ot1 among the three reference objects Ot2-Ot3. In Figure 3(A), the reference object Ot2 has the second shortest distance between its centroid and reference object Ot1 among the three reference objects Ot2-Ot3. For this reason, when the object of interest is the reference object Ot1 of the reference image IMt in Figure 3(A), the two reference objects Ot2 and Ot3 are identified as neighboring objects. Similarly, when the object of interest is the target object Os1 of the target image IMs in Figure 3(B), the two target objects Os2 and Os3 are identified as neighboring objects.

[0053] In S286, the processor 210 calculates the angle α formed by two line segments connecting the centroid of the object of interest and the centroids of two neighboring objects. If the object of interest is the reference object Ot1 of the reference image IMt in Figure 3(A), the angle αt1 is calculated, which is formed by the line segment connecting the centroid Ct1 of reference object Ot1 and the centroid Ct2 of reference object Ot2, and the line segment connecting the centroid Ct1 of reference object Ot1 and the centroid Ct3 of reference object Ot3. Since angle α is a feature quantity of the object of interest, angle α is also called the feature angle α. Figure 3(C) shows the feature angle αs1 of the target object Os1. After S286, the loop processing S20 for the current object of interest is terminated.

[0054] When the loop process S20 is repeated for all objects Ot and Os, that is, when the characteristic angles α of all objects Ot and Os have been calculated, the processor 210 executes the loop process S22 between the start L22s and the end L22e, with each of the N reference objects Ot as the reference object of interest.

[0055] In S292, the processor 210 selects one unprocessed object from among the N reference objects Ot of the reference image IMt as the reference object of interest.

[0056] In S294, the processor 210 identifies the object among the N target objects Os in the target image IMs that has the shortest feature angle α relative to the feature angle α of the reference object of interest. For example, if the reference object of interest is the reference object Ot1 in Figure 3(A), then the target object Os1 in Figure 3(B) is identified.

[0057] In S296, the processor 210 associates the reference object of interest with the target object Os identified in S294. After S296, the loop processing S22 for the current reference object of interest is terminated.

[0058] The object mapping process terminates when the loop process S22 is repeated for all reference objects Ot. The object mapping process creates a one-to-one correspondence between N reference objects Ot and N target objects Os. In the example in Figure 3, the reference object Oti (where i is an integer from 1 to 4) in Figure 3(A) is associated with the target object Osi in Figure 3(B). In the object mapping process, the pair of reference object Ot and target object Os that has been mapped is also called an object pair. When the object mapping process is executed, N object pairs are formed. Note that if the object mapping process is not executed, and in S130 of Figure 4, the entire reference image IMt is treated as one reference object Ot and the entire target image IMs is treated as one target object Os, then the pair of the entire reference image IMt and the entire target image IMs is the only object pair.

[0059] Returning to Figure 4, let's continue the explanation. In S140, the processor 210 performs the first matching process. Figure 7 is a flowchart of the first matching process. In S310, the processor 210 detects multiple feature points of the reference image IMt (referred to as reference feature points Tt) and multiple feature points of the target image IMs (referred to as target feature points Ts).

[0060] Specifically, the processor 210 detects multiple reference feature points Tt by analyzing the grayscale-converted reference image IMt using known methods. The processor 210 also detects multiple target feature points Ts by analyzing the grayscale-converted target image IMs using known methods.

[0061] In this embodiment, a method called Accelerated-KAZE (A-KAZE) is used for feature point detection and feature quantity calculation (described later). A-KAZE is disclosed, for example, in the following paper. "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013"

[0062] Figure 8 is an explanatory diagram of the first matching process. Figure 8(A) shows an example of a reference feature point Tt detected in the reference image IMt and a target feature point Ts detected in the target image IMs. As shown in the figure, points indicating characteristic parts of an object, such as corners and edges, are detected as feature points Tt and Ts. Although not shown in the figure, in reality, many more feature points Tt and Ts may be detected, including internal edges of the object (for example, tens or hundreds).

[0063] In S320, the processor 210 calculates feature quantities Ft and Fs for multiple reference feature points Tt and multiple target feature points Ts. The feature quantity Ft for the reference feature point Tt may be various pieces of information describing the features of the reference feature point Tt, and the feature quantity Fs for the target feature point Ts may be various pieces of information describing the features of the target feature point Ts. The feature quantities Ft and Fs are calculated, for example, to change according to the distribution of color values ​​of multiple pixels surrounding the feature points Tt and Ts. In this embodiment, the processor 210 uses the grayscale-converted reference image IMt and target image IMs to calculate the A-KAZE feature descriptor and direction as feature quantities Ft and Fs. In A-KAZE technology, the feature descriptor is rotationally invariant. The direction of the feature point is also calculated in order to obtain a rotationally invariant feature descriptor. The direction of the feature point indicates the direction of the luminance gradient in the neighborhood region centered on the feature point (also called the gradient direction or dominant direction).

[0064] Once the feature vectors Ft and Fs are calculated, the processor 210 executes a loop process S30 between the start L30s and the end L30e, treating all object pairs as the object pairs of interest. As described above, if object mapping is performed, the object pairs are the N object pairs determined by the object mapping process. If object mapping is not performed, the object pairs are the entire reference image IMt and the entire target image IMs.

[0065] In S330, processor 210 selects an unprocessed object pair as the object pair of interest. The reference object Ot that constitutes the object pair of interest is also called the reference object of interest, and the target object Os that constitutes the object pair of interest is also called the target object of interest.

[0066] When a pair of objects of interest is selected, the processor 210 executes a loop process S31 between the start L31s and the end L31e, treating all feature point pairs belonging to the object of interest as feature point pairs. Loop process S31 is a loop process nested within loop process S30. Here, a feature point pair belonging to the object of interest means a feature point pair consisting of a reference feature point Tt of the reference object of interest and a target feature point Ts of the object of interest. If the number of reference feature points Tt of the reference object of interest is J (J is an integer greater than or equal to 1) and the number of target feature points Ts of the object of interest is K (K is an integer greater than or equal to 1), then loop process S31 is executed for each of the (J × K) combinations. Here, the reference feature point Tt of the reference object of interest is the reference feature point Tt located in the reference object region At that encompasses the reference object of interest. The target feature point Ts of the object of interest is the target feature point Ts located in the target object region As that encompasses the object of interest.

[0067] In S340, the processor 210 selects one unprocessed feature point pair from among the feature point pairs belonging to the object pair of interest as the feature point pair of interest. The reference feature point Tt that constitutes the feature point pair of interest is called the reference feature point of interest, and the target feature point Ts that constitutes the feature point pair of interest is called the target feature point of interest.

[0068] In S350, the processor 210 calculates the distance dF between the feature quantity Ft of the reference feature point of interest and the feature quantity Fs of the feature point of interest. The distance dF is calculated such that a smaller distance dF indicates a higher similarity between the two feature quantities Ft and Fs. When the distance dF is small (i.e., the similarity is high), it is likely that the reference feature point of interest and the feature point of interest represent similar parts of two images IMt and IMs (for example, corresponding parts within the object of interest). In this embodiment, the feature quantities Ft and Fs include A-KAZE feature descriptors. These feature descriptors are represented by binary vectors (vectors consisting of one or more binary elements). For this reason, in this embodiment, the processor 210 calculates the Hamming distance of the feature descriptors contained in the feature quantities Ft and Fs as the distance dF.

[0069] In S360, the processor 210 determines whether the distance dF is less than the distance threshold dFth. If the distance dF is less than the distance threshold dFth (S360: YES), the processor 210 determines the pair of feature points of interest as a candidate feature point pair M in S370. The candidate feature point pair is a candidate for a valid feature point pair (also called a matching pair) used to determine the coordinate correspondence between the reference image IMt and the target image IMs. After S370, the processor 210 terminates the loop processing S31 for the current pair of feature points of interest.

[0070] If the distance dF is greater than or equal to the distance threshold dFth (S360: NO), the processor 210 skips S370 and terminates the loop processing S31 for the current pair of feature points of interest. In this case, the current pair of feature points of interest is not determined to be a candidate pair of feature points M. Once the loop processing L31 has been executed for all feature point pairs belonging to the pair of object points of interest, the processor 210 returns to S330 to execute the next loop processing S30 if there are any unprocessed object pairs, or proceeds to S380 if there are no unprocessed object pairs.

[0071] In S380, the processor 210 stores data representing the determined candidate feature point pairs M in the storage device 215 (for example, the non-volatile storage device 230) and terminates the first matching process.

[0072] Figure 8(A) shows an example of a candidate feature point pair M determined in the first matching process. Multiple lines RL in the figure each represent a candidate feature point pair M. Each line RL connects a reference feature point Tt and a target feature point Ts that form the candidate feature point pair M. As shown in the figure, in this embodiment, all candidate feature point pairs M consist of a reference feature point Tt of one reference object Ot and a target feature point Ts of the target object Os corresponding to that reference object Ot. For example, the reference feature point Tt of the first reference object Ot1 and the target feature point Ts that constitute the candidate feature point pair M are the target feature point Ts of the first target object Os1, and not the target feature point Ts of the second to fourth target objects Os2-Os4. The target feature point Ts that constitutes the candidate feature point pair M with the reference feature point Tt of the third reference object Ot3 is the target feature point Ts of the third target object Os3, and not the target feature point Ts of the first, second, and fourth target objects Os1, Os2, and Os4.

[0073] A pair of feature points Tt and Ts, which indicate corresponding parts located in corresponding objects Ot and Os, can be selected as a candidate feature point pair M. For example, in the example in Figure 8(A), the feature point Ttn, which indicates the upper right corner of the second reference object Ot2 in the reference image IMt, is associated with the feature point Tsn, which indicates the upper right corner of the second target object Os2 in the target image IMs. A candidate feature point pair M that shows such a correct correspondence is ultimately the pair that should be determined as the matching pair.

[0074] Furthermore, a pair of feature points Tt and Ts, which are located in corresponding objects Ot and Os but do not correspond to each other, may be selected as a candidate feature point pair M. For example, in the example in Figure 8(A), the feature point Ttn, which indicates the upper right corner of the second reference object Ot2 in the reference image IMt, is associated with the feature point Tsm, which indicates the lower right corner of the second target object Os2 in the target image IMs. A candidate feature point pair M that shows such an incorrect correspondence should ultimately not be determined as a matching pair.

[0075] For example, rectangular objects Ot2 and Os2 each have four corners. These corners are locally similar. As a result, a reference feature point Ttn representing one corner of an object in the second reference object Ot2 can be associated with any of the target feature points Ts representing the four corners of the second target object Os2. For this reason, the multiple candidate feature point pairs M determined in the first matching process may include candidate feature point pairs M that show a correct correspondence and candidate feature point pairs M that show an incorrect correspondence.

[0076] In particular, in this embodiment, as described above, the feature descriptors included in the feature quantities Ft and Fs are rotationally invariant. When feature descriptors are rotationally invariant, it is possible to calculate similar feature descriptors from the same part of an object regardless of the rotation angle of the object in the image. By using the distance dF between rotationally invariant feature descriptors, the processor 210 can associate pairs of feature points that represent corresponding parts of an object, even if the rotation angles of the corresponding objects in the two images IMt and IMs are different. However, if each image IMt and IMs contains multiple locally similar parts, feature points Tt and Ts that represent parts that do not correspond to each other may be incorrectly associated.

[0077] Note that the larger the distance threshold dFth (Figure 7: S360), the greater the total number of candidate feature point pairs M that show a correct correspondence. However, the greater the total number of candidate feature point pairs M that show an incorrect correspondence, the greater the error in the coordinate correspondence, as described later. If the total number of candidate feature point pairs M that show an incorrect correspondence is large, the error in the coordinate correspondence may increase. The distance threshold dFth may be determined experimentally in advance so that the error in the coordinate correspondence is acceptable.

[0078] Once the first matching process is complete, at S145 in Figure 4, the processor 210 executes the second matching process. The second matching process narrows down the multiple candidate feature point pairs M determined in the first matching process, reducing the possibility of any candidate feature point pairs M showing an incorrect match remaining. The second matching process uses a set (combination) of two candidate feature point pairs M to narrow down the candidate feature point pairs M.

[0079] Figure 9 is a flowchart of the second matching process. Processor 210 executes a loop process S50 between the start L50s and the end L50e, with each of the multiple object sets as the object set of interest. Here, an object set is a combination of two objects selected from the N reference objects Ot of the reference image IMt. In the examples of Figure 3(A) and Figure 8(A), the reference image IMt contains four reference objects Ot1-Ot4. For this reason, there are six multiple object sets: A set of the first reference object Ot1 and the second reference object Ot2. A set of the first reference object Ot1 and the third reference object Ot3. A set of the first reference object Ot1 and the fourth reference object Ot4. A set of the second reference object Ot2 and the third reference object Ot3. A set of the second reference object Ot2 and the fourth reference object Ot4. A set of third criterion object Ot3 and fourth criterion object Ot4 Therefore, in the example shown in Figure 3(A), the loop process S50 is executed 6 times.

[0080] In S510, an unprocessed set of objects is selected as the object set of interest. The two reference objects Ot that make up the object set of interest are also called the first reference object of interest and the second reference object of interest, respectively.

[0081] Once a set of objects of interest is selected, the processor 210 executes a loop process S51 between the start L51s and the end L51e, using the set of all candidate feature point pairs M belonging to the set of objects of interest as the set of pairs of interest. Loop process S51 is a loop process nested within loop process S50. Here, the set of candidate feature point pairs M belonging to the set of objects of interest means a set consisting of a candidate feature point pair M containing the reference feature point Tt of the first reference object of interest, and a candidate feature point pair M containing the reference feature point Tt of the second reference object of interest.

[0082] In S520 and S530, a set of candidate feature point pairs M to be processed (also called a set of focus pairs) is selected. First, in S520, the processor 210 selects one unprocessed candidate feature point pair M containing the reference feature point Tt of the first focus reference object as the first focus pair. For example, in the example in Figure 8(A), if the first focus reference object is the first reference object Ot1, then a candidate feature point pair M consisting of the reference feature point Tta of the first reference object Ot1 in Figure 8(A) and the target feature point Tsa of the first target object Os1 may be selected as the first focus pair.

[0083] In S530, the processor 210 selects one unprocessed candidate feature point pair M containing the reference feature point Tt of the second reference object of interest as the second attention pair. For example, in the example in Figure 8(A), if the second reference object of interest is the second reference object Ot2, then the candidate feature point pair M consisting of the reference feature point Ttb of the second reference object Ot2 in Figure 8(A) and the target feature point Tsb of the second target object Os2 may be selected as the second attention pair. The set of the first attention pair selected in S520 and the second attention pair selected in S530 constitutes the attention pair set.

[0084] In S540, processor 210 calculates the features of the reference feature set TtS. Figure 10 is an explanatory diagram of the reference feature set TtS and the target feature set TsS. Here, the reference feature points Tt that constitute the first pair of focus are called the first focus reference feature points Tt1, and the reference feature points Tt that constitute the second pair of focus are called the second focus reference feature points Tt2. The reference feature set TtS is a set consisting of the first focus reference feature points Tt1 and the second focus reference feature points Tt2.

[0085] Figure 10(A) shows an example of a reference feature point set TtS. The reference line segment LNt is a line segment connecting the first reference feature point Tt1 and the second reference feature point Tt2. The first reference direction Dt1 is the direction included in the feature quantity Ft of the first reference feature point Tt1. The second reference direction Dt2 is the direction included in the feature quantity Ft of the second reference feature point Tt2. The reference directions Dt1 and Dt2 have already been calculated as part of the feature quantity Ft in S320 of Figure 7 described above.

[0086] In Figure 10(A), the first reference angle Zt1 is the angle between the first reference direction Dt1 and the reference line segment LNt. The second reference angle Zt2 is the angle between the second reference direction Dt2 and the reference line segment LNt. For angles Zt1 and Zt2, the smaller of the angles between the line segments extending from feature points Tt1 and Tt2 in reference directions Dt1 and Dt2 and the reference line segment LNt extending from the reference feature points of interest Tt1 and Tt2 is adopted. Alternatively, angles Zt1 and Zt2 may be the angles from the reference line segment LNt to the line segments extending in reference directions Dt1 and Dt2, centered on the reference feature points of interest Tt1 and Tt2, in a clockwise (or counterclockwise) direction.

[0087] The processor 210 calculates the first reference angle Zt1 and the second reference angle Zt2 as feature quantities of the reference feature point set TtS.

[0088] In S545, the processor 210 calculates the features of the target feature set TsS. Here, the target feature points that make up the first pair of focus are called the first target feature points Ts1, and the target feature points Ts that make up the second pair of focus are called the second target feature points Ts2. The target feature set TsS is a set consisting of the first target feature points Ts1 and the second target feature points Ts2.

[0089] Figure 10(B) shows an example of the target feature point set TsS. The target line segment LNs is a line segment connecting the first target feature point Ts1 and the second target feature point Ts2. The first target direction Ds1 is the direction included in the feature quantity Fs of the first target feature point Ts1. The second target direction Ds2 is the direction included in the feature quantity Fs of the second target feature point Ts2. The target directions Ds1 and Ds2 have already been calculated as part of the feature quantity Fs in S320 of Figure 7 described above.

[0090] In Figure 10(B), the first symmetric angle Zs1 is the angle between the first symmetric direction Ds1 and the symmetric line segment LNs. The second symmetric angle Zs2 is the angle between the second symmetric direction Ds2 and the symmetric line segment LNs. Similar to the angles Zt1 and Zt2 described above, angles Zs1 and Zs2 are the smaller of the angles between the line segments extending from the feature points of interest Ts1 and Ts2 in the symmetric directions Ds1 and Ds2, and the symmetric line segment LNs extending from the feature points of interest Ts1 and Ts2.

[0091] The processor 210 calculates the first target angle Zs1 and the second target angle Zs2 as feature quantities for the target feature point set TsS.

[0092] In the S550, the processor 210 determines whether the reference feature set TtS and the target feature set TsS are similar based on the feature quantities (angles Zt1, Zt2) of the reference feature set TtS and the feature quantities (angles Zs1, Zs2) of the target feature set TsS. In this embodiment, the reference feature set TtS and the target feature set TsS are determined to be similar if both of the following conditions (1) and (2) are met, and the reference feature set TtS and the target feature set TsS are determined to be dissimilar if at least one of conditions (1) and (2) is not met.

[0093] Condition (1): The absolute value of (Zt1-Zs1), which shows the difference between the first reference angle Zt1 and the first target angle Zs1, is less than the predetermined judgment threshold THz1. Condition (2): The absolute value of (Zt2-Zs2), which shows the difference between the second reference angle Zt2 and the second target angle Zs2, is less than the predetermined judgment threshold THz2.

[0094] In this embodiment, the reference directions Dt1, Dt2 and the target directions Ds1, Ds2 are gradient directions calculated according to A-KAZE technology. For this reason, the reference angles Zt1, Zt2 and the target angles Zs1, Zs2 are invariant with respect to the overall size and rotation of the images IMt and IMs. Therefore, the judgment results of conditions (1) and (2) are invariant with respect to the overall size and rotation of the images IMt and IMs.

[0095] If the first pair of focus shows a correct correspondence between images IMt and IMs, and the second pair of focus also shows a correct correspondence between images IMt and IMs, then the reference angles Zt1 and Zt2 and the target angles Zs1 and Zs2 are approximately the same. If at least one of the first pair of focus or the second pair of focus shows an incorrect correspondence between images IMt and IMs, then the reference angles Zt1 and Zt2 and the target angles Zs1 and Zs2 are likely to be different from each other. For this reason, if both conditions (1) and (2) are met, the first pair of focus and the second pair of focus are considered to show a correct correspondence between images IMt and IMs, respectively. The judgment thresholds THz1 and THz2 are predetermined, for example, experimentally, to allow for errors in angles Zt1, Zt2, Zs1, and Zs2 that may occur even when the first and second pairs of focus show a correct correspondence.

[0096] If the reference feature point set TtS and the target feature point set TsS are similar (S550: YES), the processor 210 determines in S560 that the two candidate feature point pairs M that constitute the set of attention pairs (i.e., the first attention pair and the second attention pair) are the final matching pairs. After S560, the processor 210 terminates the loop processing S51 for the current set of attention pairs.

[0097] If the reference feature point set TtS and the target feature point set TsS are not similar (S550: NO), the processor 210 skips S560 and terminates the loop processing S51 for the current set of interest pairs. In this case, the first and second interest pairs are not determined to be a matching pair. Once the loop processing L51 has been executed for all sets of candidate feature point pairs M belonging to the interest object set, the processor 210 returns to S510 to execute the next loop processing S50 if there are any unprocessed object sets, or proceeds to S570 if there are no unprocessed object set pairs.

[0098] In S570, the processor 210 stores the data of the determined matching pair in the storage device 215 (for example, the non-volatile storage device 230) and terminates the second matching process.

[0099] Figure 8(B) shows an example of matching pairs determined in the second matching process. Multiple lines ML in the figure each represent a matching pair. Compared to Figure 8(A), feature point pairs showing incorrect correspondences have been removed.

[0100] Returning to Figure 4, let's continue the explanation. Once the second matching process is complete, in S150, the processor 210 determines the correspondence between the coordinates on the reference image IMt and the coordinates on the target image IMs based on the multiple matching pairs determined by the second matching process. Specifically, the processor 210 determines the affine transformation matrix Mtx based on the multiple matching pairs.

[0101] Figure 11 is an explanatory diagram of the affine transformation matrix Mtx. It shows the coordinates COt on the reference image IMt, the coordinates COs on the target image IMs, and the affine transformation matrix Mtx that associates these coordinates COt and COs. In the figure, the coordinates COt and COs and the matrix Mtx are represented in a homogeneous coordinate system. Each coordinate COt and COs is represented as a three-dimensional vector having positions Xt and Xs in the first direction Dx on images IMt and IMs, positions Yt and Ys in the second direction Dy, and a third component of 1. The affine transformation matrix Mtx is a 3x3 matrix. As shown in the figure, the affine transformation matrix Mtx is represented by six parameters af in a 2x3 grid and three components (0,0,1) in the third row. Such an affine transformation matrix Mtx can represent rotation, scaling, shrinking, translation, and skew. The affine transformation matrix Mtx can be calculated according to known methods by using three or more pairs of coordinates COt and COs, i.e., three or more matching pairs.

[0102] In S155 (Figure 4), the processor 210 inspects the printed image IMp. The inspection method may be one of various methods that use the coordinate correspondence determined in S155. In this embodiment, the processor 210 generates a difference image IMd between the reference image IMt and the target images IMs.

[0103] The orientation of the T-shirt 700 relative to the reading sensor 180 (Figure 2) can vary. Therefore, the orientation of objects in the images (i.e., the angle of rotation of objects) may differ between the reference image IMt and the target image IMs. Also, the size of objects in the images may differ between the reference image IMt and the target image IMs. For this reason, when generating the difference image IMd between the reference image IMt and the target image IMs, it is necessary to adjust the position and size of at least one of the reference image IMt and the target image IMs so that they overlap precisely. To this end, the processor 210 uses the affine transformation matrix Mtx in Figure 11 to transform the coordinate system of the reference image IMt to the coordinate system of the target image IMs, thereby adjusting the orientation and size of the reference image IMt before generating the difference image IMd.

[0104] Figure 12 shows an example of a difference image IMd. Figure 12(A) shows difference image IMd1, which is generated when the printed image IMp does not contain defects, and Figure 12(B) shows difference image IMd2, which is generated when the printed image IMp contains defects. The difference image IMd represents the difference in color values ​​(e.g., the absolute value of the difference in luminance values) for each pixel between the reference image IMt in Figure 3(A) and the target image IMs in Figure 3(C). If the printed image IMp does not contain defects, difference image IMd1 does not include the portion showing a large difference (Figure 12(A)). If the printed image IMp contains defects, difference image IMd2 includes the portion Err (the portion showing a large difference) corresponding to the defect (Figure 12(B)).

[0105] In S160 (Figure 4), the processor 210 outputs the inspection results. There are various methods for outputting the inspection results. In this embodiment, the processor 210 displays the difference image IMd on the display unit 240 (Figure 1). By observing the difference image IMd displayed on the display unit 240, the operator can easily recognize defects in the printed image IMp. Alternatively, the processor 210 may output data representing the inspection results to a storage device (for example, a non-volatile storage device 230, or an external storage device connected to the image processing device 200). After S160, the processor 210 terminates the inspection process.

[0106] According to the embodiment described above, the processor 210 identifies multiple reference objects Ot in the reference image IMt and multiple target objects Os in the target image IMs (S110 in Figure 4). The processor 210 performs a mapping process to associate each of the multiple reference objects Ot with a target object (S135 in Figure 4). The processor 210 performs a matching process to determine matching pairs, which are pairs of corresponding reference feature points Tt and target feature points Ts (S140, S145 in Figure 4). In the matching process, matching pairs are determined such that the reference feature point Tt of the first reference object Ot1 forms a matching pair with the target feature point Ts of the first target object Os1 corresponding to the first reference object Ot1, and the reference feature point Tt of the second reference object Ot2 forms a matching pair with the target feature point Ts of the second target object Os2 corresponding to the second reference object Ot2. In feature point matching, it is possible to suppress the determination of matching pairs that show incorrect correspondences. In addition, the computation time required to determine matching pairs can be shortened.

[0107] For example, feature points Tt and Ts of two objects Ot and Os that do not correspond to each other, such as the reference feature point Tt of the first reference object Ot1 and the target feature point Ts of the second target object Os2, do not constitute a matching pair. In this embodiment, objects Ot and Os of two images IMt and IMs are identified, and an association is made between the reference object Ot and the target object Os. Then, matching pairs are determined such that the feature points Tt and Ts of two corresponding objects Ot and Os constitute a matching pair. Therefore, compared to, for example, a case where objects are not considered within the two images IMt and IMs and matching pairs are determined by brute force from all feature point pairs of the two images IMt and IMs, this embodiment reduces the possibility of determining matching pairs that show an incorrect correspondence, and the matching accuracy may be improved. Also, compared to the case where matching pairs are determined by brute force, the number of feature point pairs to be considered is reduced, so the computation time required to determine matching pairs can be shortened.

[0108] Furthermore, in this embodiment, during the object mapping process (Figure 6), the processor 210 calculates the feature quantities (specifically, feature angles α) of multiple reference objects Ot and multiple target objects Os (S284, S286 in Figure 6), and uses these feature quantities to map each of the multiple reference objects Ot to the target objects Os (S294, S296 in Figure 6). As a result, the reference objects Ot and target objects Os can be automatically mapped using the feature angles α.

[0109] Furthermore, in this embodiment, for example, the feature quantity of the reference object Ot1 is the feature angle α formed by two line segments connecting the centroid Ct1 of the reference object Ot1 and the centroids Ct2 and Ct2 of the other two reference objects Ot2 and Ot3 (Figure 3(A), S286 in Figure 6). The feature quantity of the first target object Os1 is the feature angle α formed by two line segments connecting the centroid Cs1 of the first target object Os1 and the centroids Cs2 and Cs3 of the other two target objects Os2 and Os3 (Figure 3(C), S286 in Figure 6). The processor 210 associates the reference object Ot and the target object Os whose feature angles α are similar to each other (S294 and S296 in Figure 6). As a result, the reference object Ot and the target object Os can be appropriately associated using the feature angle α.

[0110] Furthermore, in this embodiment, the other two reference objects Ot2 and Ot3 used to calculate the characteristic angle α of the first reference object Ot1 are the third reference object Ot3, which has the shortest distance from the centroid of the first reference object Ot1, and the second reference object Ot2, which has the second shortest distance from the centroid of the first reference object Ot1. Similarly, the other two target objects Os2 and Os3 used to calculate the characteristic angle α of the first target object Os1 are the third target object Os3, which has the shortest distance from the centroid of the first target object Os1, and the second target object Os2, which has the second shortest distance from the centroid of the target object Os1. As a result, the reference object and the corresponding target object can be determined by considering only the two objects with the shortest distance from their centroids, thus reducing processing time.

[0111] Furthermore, in this embodiment, the processor 210 converts the target images IMs into grayscale images to generate grayscale target images IMs (S220 in Figure 5), and then binarizes the grayscale target images IMs to identify multiple object pixels within the target images IMs (S230 in Figure 5). The processor 210 then uses the identified object pixels to identify multiple target objects Os (S240-S26 in Figure 5). As a result, multiple target objects Os can be identified using the binarization process.

[0112] Furthermore, in this embodiment, the processor 210 performs matching processing on the entirety of images IMt and IMs as a single object when the number of reference objects Ot identified in the object identification process is 1 (NO in S120 of Figure 4), or when the number of reference objects identified and the number of target objects Os are not the same (NO in S125 of Figure 4) (S130, S140 of Figure 4). In other words, in this case, the matching process is performed without using the reference object Ot and target object Os identified in the object identification process. To put it another way, in this case, for example, the matching process is performed by brute-force matching of all reference points between the entirety of images IMt and IMs. This prevents inappropriate matching processing from being performed.

[0113] For example, if the number of reference objects Ot identified in the object identification process is 1, then improvement in matching accuracy and reduction in matching processing time cannot be expected by performing matching between objects. For this reason, in this case, it is preferable to perform the matching process on the entirety of the images IMt and IMs to minimize the occurrence of unconsidered feature points.

[0114] Furthermore, in this embodiment, the target images IMs are obtained by reading an image printed using the reference image IMt; therefore, the target images IMs and the reference image IMt are the same image. For this reason, if the objects in images IMt and IMs are correctly identified, the number of reference objects Ot and the number of target objects Os are considered to be the same. Consequently, if the number of reference objects Ot and the number of target objects Os are not the same, it is considered that the objects are not correctly identified in at least one of the target images IMs and the reference image IMt. When performing matching processing between corresponding objects Ot and Os, it is assumed that objects Ot and Os are correctly identified and correctly associated. For this reason, if objects Ot and Os are not correctly identified, there is a high possibility that objects Ot and Os will not be correctly associated, and improvement in matching accuracy cannot be expected even if matching processing is performed between objects Ot and Os. For this reason, in this case as well, it is considered preferable to perform matching processing between the entirety of images IMt and IMs.

[0115] Furthermore, according to this embodiment, the processor 210 performs color conversion processing on the target images IMs using a color conversion profile PF (S210 in Figure 5). After the color conversion processing, the processor 210 identifies multiple object pixels in the reference image IMt and the target images IMs by binarizing them respectively (S230 in Figure 5), and uses the identified object pixels to identify multiple objects Ot and Os (S240-S260 in Figure 5). With the above configuration, by performing color matching between the reference image IMt and the target images IMs through color conversion processing, multiple reference objects Ot and multiple target objects Os can be identified with high accuracy. As described above, when performing matching processing between objects, it is a prerequisite that objects Ot and Os are correctly identified and correctly associated. By performing color matching using a color conversion profile PF, the accuracy of identifying objects Ot and Os can be improved. As a result, the probability of objects Ot and Os being correctly mapped can be improved, and consequently, the matching accuracy in the matching process can be improved.

[0116] Furthermore, in this embodiment, the matching process includes a second matching process (Figure 9) in which, if a set of two reference feature points Tt and a set of two target feature points Ts are similar (YES in 550 in Figure 9), the pair of the two reference feature points Tt and the two target feature points Ts is determined as a matching pair. In the first matching process, which compares one reference feature point Tt and one target feature point Ts, only the local features of the feature points Tt and Ts are considered. In contrast, the second matching process can perform matching that takes into account more global features, such as angles related to the line segments LNs and LNt connecting the two reference feature points Tt. Therefore, for example, by combining the first matching process and the second matching process, the matching accuracy can be improved. In addition, even in the second matching process that takes into account such global features, performing matching between objects Ot and Os can suppress the determination of incorrect matching pairs and reduce the computation time required for the matching process.

[0117] Furthermore, in the second matching process of this embodiment, the processor 210 selects two different reference objects Ot as the reference objects of interest (S510 in Figure 9), and determines whether the set of reference feature points Tt of the two reference objects of interest is similar to the set of target feature points Ts of the two target objects Os that correspond to the two reference objects of interest (S520-S550 in Figure 9). In other words, in the second matching process, for example, if a set containing the reference feature point Tt of the first reference object Ot1 and the reference feature point Tt of the third reference object Ots is similar to a set containing the target feature point Ts of the first target object Os1 and the target feature point Ts of the third target object Os3 (YES in S550 of Figure 9), then the processor 210 determines the pair of the reference feature point Tt of the first reference object Ot1 and the target feature point Ts of the first target object Os1 as a matching pair (S560 in Figure 9). Thus, in the second matching process, sets of reference feature points Tt located in multiple different reference objects Ot are compared with sets of target feature points Ts located in multiple different target objects Os. As a result, for example, a set of multiple reference feature points Tt that are spaced a certain distance apart can be used, rather than using multiple reference feature points Tt located on a single reference object Ot. If the distance between the multiple reference feature points Tt that make up the set is extremely short, the error in the calculated feature quantities (reference angles Zt1, Zt2) tends to be large. This is because the position of each detected reference feature point Tt contains errors. Also, if the distance between the multiple reference feature points Tt that make up the set is extremely short, it may be possible to capture only local features despite using multiple reference feature points Tt. From this perspective, it is preferable to use a set of multiple reference feature points Tt that are spaced a certain distance apart.

[0118] Furthermore, according to this embodiment, the processor 210 identifies a reference object region At that encompasses each reference object Ot, and a target object region As that encompasses each target object Os (Figures 3(A) and 3(C)). The processor 210 determines one or more matching pairs from among a plurality of reference feature points Tt located in the reference object region At1 (Figure 3(A)) that encompasses the first reference object Ot1, and a plurality of target feature points Ts located in the target object region As1 that encompasses the first target object Os1 (e.g., S340 in Figure 7). Similarly, the processor 210 determines one or more matching pairs from among a plurality of reference feature points Tt located in the reference object region At2 (Figure 3(A)) that encompasses the second reference object Ot2, and a plurality of target feature points Ts located in the target object region As2 that encompasses the second target object Os2 (e.g., S340 in Figure 7). In this way, object regions At and As that encompass each object Ot and Os are identified, and matching pairs are determined from feature points Tt and Ts located in these object regions At and As. This allows for the appropriate determination of matching pairs consisting of feature points Tt and Ts belonging to corresponding objects Ot and Os.

[0119] Furthermore, the processor 210 identifies the region obtained by performing an expansion process on the reference object Ot as the reference object region At, and the region obtained by performing an expansion process on the target object Os as the target object region As (S250, S260 in Figure 5). As a result, object regions At and As that are slightly larger than the actual objects Ot and Os are identified. As a result, feature points Tt and Ts that should belong to each object Ot and Os can be selected as candidates without omission, and matching pairs can be determined. For example, feature points Tt and Ts are easily detected on the outer edges of objects Ot and Os, but it is possible to suppress the omission of feature points Tt and Ts located on the outer edges of objects Ot and Os from being selected as matching pair candidates.

[0120] As can be seen from the above explanation, the target object region As in this embodiment is an example of a target containment region, and the reference object region At is an example of a reference containment region.

[0121] B. Variations (1) The object identification process in the above embodiment (Figure 5) is an example and is not limited thereto. Figure 13 is an explanatory diagram of the object identification process of the reference image IMt in a modified example. Figure 13(A) shows a flowchart of the object identification process of the reference image IMt in a modified example.

[0122] In S610, the processor 210 acquires the α-channel data attached to the data of the reference image IMt. In this embodiment, the α-channel data is stored together with the data of the reference image IMt in the image file that stores the data of the reference image IMt. The α-channel data is, for example, data indicating whether each of the multiple pixels of the reference image IMt is an object pixel or a background pixel.

[0123] In S620, the processor 210 refers to the α-channel data to identify the object image of the reference image IMt. Figure 13(B) is a binary image AIt showing the object pixels identified by referring to the α-channel data. The hatched areas within the binary image AIt indicate the reference object region At, which is composed of object pixels.

[0124] In S630, processor 210 performs a labeling process on the binary image AIt. The specific labeling method is the same as the labeling process in S260 shown in Figure 5. Through the labeling process, four reference object regions At (At1-At4) are identified in the binary image AIt (Figure 13(B)). This identifies four reference objects Ot (Ot1-Ot4) in the reference image IMt.

[0125] Furthermore, the binary image AIt may be subjected to an expansion process to identify a reference object region At that is slightly larger than the reference object Ot, similar to the embodiment. In this modified example, the target object Os of the target image IMs is also identified by the same process as in the embodiment (for example, the process in S210-S260 of Figure 5).

[0126] According to the modified version described above, the processor 210 uses the α channel data as reference data to identify multiple object pixels of the reference image IMt (S620), and uses these multiple object pixels to identify multiple reference objects Ot (S630). Therefore, multiple reference objects Ot can be identified easily and accurately.

[0127] (2) In addition, various other image analysis processes may be used to identify objects in the reference image IMt and target images IMs. For example, the processor 210 divides the reference image IMt into multiple rectangular blocks (for example, 50 blocks vertically x 50 blocks horizontally). The rectangular blocks are arranged in a matrix within the reference image IMt, for example. The processor 210 classifies the multiple blocks into either heterogeneous blocks or homogeneous blocks. For example, the processor 210 calculates the standard deviation SD of the pixels in the block, classifies blocks with a standard deviation SD less than a standard as homogeneous blocks, and classifies blocks with a standard deviation SD greater than or equal to a standard as heterogeneous blocks. The processor 210 labels the multiple blocks to identify one or more consecutive homogeneous blocks as a single reference object, and one or more consecutive heterogeneous blocks as a single reference object. The processor 210 performs the same process on the target images IMs to identify multiple target objects in the target images IMs.

[0128] (3) In addition, various other methods may be used for the binarization process in the object identification process. For example, in the above embodiment, the processor 210 binarizes the grayscale images IMt and IMs. Alternatively, for example, the RGB values ​​of the fabric color of the T-shirt 700 may be stored in the non-volatile storage device 230 in advance. Then, the processor 210 may binarize the images IMt and IMs by classifying pixels that have a different color from the fabric as object pixels and pixels that have the same color as the fabric as background pixels. For example, colors whose distance in the RGB color space from the RGB values ​​of the fabric is greater than or equal to a threshold are judged to be different colors from the fabric, and colors whose distance in the RGB color space from the RGB values ​​of the fabric is less than a threshold are judged to be the same color as the fabric.

[0129] (4) In addition, the color conversion process in the object identification process (S210 in Figure 5) is performed on the target image IMs in the embodiment. Alternatively, the color conversion process may be performed on the reference image IMt. In this case, the processor 210 uses a pre-prepared color conversion profile to convert the values ​​of multiple pixels in the reference image IMt (color values ​​in the sRGB color system) to the color values ​​in the scan color system. Furthermore, the color conversion process may be performed on both the target image IMs and the reference image IMt. In this case, for example, the processor 210 uses a pre-prepared color conversion profile to convert the values ​​of multiple pixels in the reference image IMt (color values ​​in the scan color system) to the color values ​​in the device-independent color system (for example, the color values ​​in the CIELAB color system). Then, the processor 210 uses another pre-prepared color conversion profile to convert the values ​​of multiple pixels in the target image IMs (color values ​​in the sRGB color system) to the color values ​​in the device-independent color system (for example, the color values ​​in the CIELAB color system).

[0130] (5) The object mapping process in the above embodiment (Figure 6) is an example and is not limited thereto. Figure 14 is a flowchart of an object mapping process of a modified example. In S700, the processor 210 calculates the areas St and Ss of N reference objects Ot and N objects Os. Specifically, the processor 210 calculates the number of object pixels that constitute the object regions At and As in which each object Ot and Os is located as the areas St and Ss. It is preferable to use the object regions At and As before the expansion process in S250 of Figure 5 for calculating the areas of the object regions At and As. In the example of Figure 3, the number of reference objects Ot and the number of target objects Os are both 4 (N=4). Therefore, the areas St of the 4 reference objects Ot and the areas Ss of the 4 target objects Os are calculated.

[0131] In S710, the processor 210 associates N reference objects Ot with N target objects Os on a one-to-one basis according to the order of their areas. Specifically, it associates a reference object Ot(n) having the nth largest area St among the N reference objects Ot (where n is an integer between 1 and N), with a target object Os(n) having the nth largest area Ss among the N target objects Os. As a result, in the example in Figure 3, similar to the object association process in the embodiment, the reference object Oti (where i is an integer from 1 to 4) in Figure 3(A) is associated with the target object Osi in Figure 3(B).

[0132] In S720, the processor 210 determines whether there are multiple reference objects Ot among N reference objects Ot whose area difference from each other is less than a predetermined area threshold THs. The area threshold THs is determined experimentally, for example, by considering the possible errors between the areas of object regions At and As identified in the reference image IMt and target image IMs, and the areas of the actual objects Ot and Os in the reference image IMt and target image IMs.

[0133] If there are multiple reference objects Ot whose area difference is less than the area threshold THs (S720: YES), the processor 210, in S730, integrates the multiple reference objects Ot whose area difference is less than the area threshold THs into a single reference object. For example, in the labeling process, the different identifiers assigned to the multiple reference objects Ot are changed to a single common identifier, thereby integrating the multiple reference objects Ot into a single reference object. For example, suppose that among N reference objects Ot, the area Sta of the k-th largest reference object Ot(k) (where k is an integer between 2 and N) and the area Stb of the (k-1)-th largest reference object Ot(k-1) are less than the area threshold THs. In this case, reference object Ot(k) and reference object Ot(k-1) are integrated into a single reference object. As a result, the number of reference objects Ot included in the reference image IMt changes from N to (N-1). In the example in Figure 3, it is assumed that the difference between the area St of the first reference object Ot1 and the area St of the third reference object Ot3 in Figure 3(A) is less than the area threshold THs. In this case, the first reference object Ot1 and the third reference object Ot3 are merged.

[0134] In S740, the processor 210 integrates multiple target objects Os associated with multiple integrated reference objects Ot into a single target object. For example, suppose in S730, a reference object Ot(k) with the k-th largest area St and a reference object Ot(k-1) with the (k-1)-th largest area St are integrated. In this case, the processor 210 integrates the target object Os(k) with the k-th largest area Ss and the target object Os(k-1) with the (k-1)-th largest area Ss. As a result, the number of target objects Os included in the target image IMs changes from N to (N-1). In this way, the number of reference objects Ot and the number of target objects Os are always maintained at the same number. Then, the entire set of the base object Ot(k) and base object Ot(k-1) (i.e., a single base object after integration) is associated with the entire set of the target object Os(k) and target object Os(k-1) (i.e., a single target object after integration).

[0135] In the example in Figure 3, it is assumed that in S730, the first reference object Ot1 and the third reference object Ot3 in Figure 3(A) are merged. In this case, the first target object Os1 and the third target object Os3 in Figure 3(C) are merged. As a result, the entirety of the first reference object Ot1 and the third reference object Ot3 is associated with the entirety of the first target object Os1 and the third target object Os3.

[0136] In the object mapping process of this modified example described above, the feature quantities of objects Ot and Os are the areas St and Ss of objects Ot and Os. The processor 210 maps a reference object Ot that has the nth largest area St among N reference objects Ot (where n is an integer between 1 and N), and a target object Os that has the nth largest area Ss among N target objects Os. As a result, the reference object Ot and the target object Os can be appropriately mapped according to the order of their areas St and Ss.

[0137] Furthermore, according to this modified example, if the difference between the area St of the first reference object Ot1 and the area St of the third reference object Ot3 is less than the area threshold THs (YES in S720 in Figure 14), the processor 210 associates the entirety of the first reference object Ot1 and the third reference object Ot3 with the entirety of the first target object Os1 and the third target object Os3 (S730, S740 in Figure 14). As a result, it is possible to suppress the incorrect association of the reference object Ot and the target object Os. For example, since there is an error in the calculated area St, if the difference between the area St of the first reference object Ot1 and the area St of the third reference object Ot3 is less than the area threshold THs, the actual order of the areas of the first reference object Ot1 and the third reference object Ot3 may be the reverse of the order based on the calculated area St. In that case, in S710, the first reference object Ot1 may be incorrectly associated with the third target object Os3, and the third reference object Ot3 may be incorrectly associated with the first target object Os1. Even if such an error occurs, the association error can be resolved by associating the entire first reference object Ot1 and the third reference object Ot3 with the entire first target object Os1 and the third target object Os3. As mentioned above, if the association between the reference object Ot and the object Os is not performed correctly, it is difficult to determine the matching pair correctly, so resolving the association error can improve the accuracy of determining the matching pair.

[0138] (6) In the modified example of (5) above, it is determined whether there are multiple reference objects Ot among the N reference objects Ot whose area difference is less than a predetermined area threshold THs (S720 in Figure 14). If there are multiple reference objects Ot whose area difference is less than the area threshold THs (S720 in Figure 14: YES), the following are performed: integration of the multiple reference objects whose area difference is less than the area threshold THs (S730 in Figure 14) and integration of the corresponding multiple target objects Os (S740 in Figure 14). Alternatively, the processor 210 may determine whether there are multiple target objects Os among the N target objects Os whose area difference is less than a predetermined area threshold THs. If there are multiple target objects Os whose area difference is less than the area threshold THs, the processor 210 may perform the following: integration of the multiple target objects Os whose area difference is less than the area threshold THs and integration of the corresponding multiple reference objects Ot. In this case, for example, in the example in Figure 3, if the difference between the area Ss of the first target object Os1 and the area Ss of the third target object Os3 is less than the area threshold THs, then the whole of the first reference object Ot1 and the third reference object Ot3 is associated with the whole of the first target object Os1 and the third target object Os3.

[0139] (7) In the object mapping process, the feature quantities of objects Ot and Os used may be other features instead of feature angle α and area Ss and St. Other features may include, for example, the average color value, average brightness, and variance of the pixels that make up the object. Furthermore, the feature quantities of objects Ot and Os do not have to be a single value, but may be multiple values. For example, the feature quantities of objects Ot and Os may be a combination of two or more of the feature angle, area, average color value, average brightness, and variance. For example, the processor 210 may perform the mapping of objects Ot and Os in the order of area St and Ss, and if there are two or more reference objects Ot and two or more target objects Os with no difference in area, it may use the feature angle α to perform the mapping of two or more reference objects Ot and two or more target objects Os.

[0140] (8) Furthermore, even when different features from the areas Ss and St, such as feature angle α and average color value, are used as features of objects Ot and Os, it is preferable to determine whether there are multiple reference objects Ot among the N reference objects Ot whose feature differences are less than a predetermined threshold. If there are multiple reference objects Ot whose feature differences are less than a threshold, it is preferable to perform the integration of the multiple reference objects whose feature differences are less than a threshold, and the integration of the corresponding multiple target objects Os. In other words, regardless of the features used, it is preferable that the whole of the first reference object Ot1 and the third reference object Ot3 be associated with the whole of the first target object Os1 and the third target object Os3 when the difference between the feature of the first target object Os1 and the feature of the third target object Os3 is less than a threshold. This can suppress incorrect associations between objects and improve the accuracy of determining matching pairs.

[0141] (9) In the second matching process of the above embodiment, the reference feature point set TtS is a set of two reference feature points Tt1 and Tt2, and the target feature point set TsS is a set of two target feature points Ts1 and Ts2 (Figure 10). However, the reference feature point set TtS and the target feature point set TsS may be sets of three or more feature points Tt and Ts. For example, the reference feature point set TtS may be a set of three reference feature points Tt, and the target feature point set TsS may be a set of three target feature points Ts. In this case, for example, the feature quantities of the feature point sets TtS and TsS may be the interior angles of the triangle formed by the three feature points Tt and Ts, or the ratio of the three sides.

[0142] (10) The characteristic angle of the reference object Ot1 in the above embodiment is the angle α formed by two line segments connecting the centroid Ct1 of the reference object Ot1 and the centroids Ct2 and Ct3 of the two reference objects Ot2 and Ot3. The other two reference objects Ot2 and Ot3 are the reference object Ot3, which has the shortest distance from the centroid of reference object Ot1, and the reference object Ot2, which has the second shortest distance from the centroid of reference object Ot1. However, the characteristic angle of the reference object Ot1 may also be the angle β formed by two line segments connecting the centroid Ct1 of the reference object Ot1 and the centroids Ct3 and Ct4 of the two reference objects Ot3 and Ot4. Alternatively, the characteristic angle of the reference object Ot1 may also be the angle γ formed by two line segments connecting the centroid Ct1 of the reference object Ot1 and the centroids Ct2 and Ct4 of the two reference objects Ot2 and Ot4. Alternatively, the characteristic angles of the reference object Ot1 may be a combination of two or more of these angles α, β, and γ.

[0143] (11) In the object identification process of the above embodiment, an expansion process (S250 in Figure 5) is performed on the region composed of object pixels identified in the binarization process (i.e., the object itself), and the region that is made slightly larger is identified as object regions At and As (S250, S260 in Figure 5). Alternatively, for example, a bounding rectangle that circumscribes the region composed of object pixels may be identified as object regions At and As. Or, the expansion process of the embodiment may be omitted, and the region composed of object pixels identified in the binarization process may be identified as object regions At and As as is.

[0144] (12) The inspection process in the above embodiment is an example and can be modified as appropriate. For example, in the object mapping process in Figure 6, the reference object Ot and the target object Os are mapped based on the feature quantities of the objects Ot and Os. Alternatively, the processor 210 may map the reference object Ot and the target object Os without using feature quantities, for example, based on user instructions. For example, the processor 210 displays an input screen on the display unit 240 that includes the identification result of the reference object Ot in the reference image IMt (for example, the reference binary image BIt in Figure 3(D)) and the identification result of the target object Os in the target image IMs (for example, the reference binary image BIt in Figure 3(E)). The user inputs instructions via the input screen to specify the target object Os to be mapped to each reference object Ot in the reference image IMt. The processor 210 maps the reference object Ot and the target object Os based on the user instructions obtained via the input screen.

[0145] (13) In the above embodiment, the processor 210 does not perform object mapping processing when the number of identified reference objects Ot is 1 (NO in S120 of Figure 4) or when the number of identified reference objects Ot is not the same as the number of identified target objects Os (NO in S125 of Figure 4), and treats the entirety of images IMt and IMs as a single object. Alternatively, even when the number of reference objects Ot is 1, the processor 210 may associate one reference object Ot with one target object Os and perform matching processing between one reference object Ot and one target object Os. Furthermore, even when the number of identified reference objects Ot is not the same as the number of identified target objects Os, the processor 210 may associate one or more combinations of objects Ot and Os with sufficiently similar feature quantities. In this case, matching is performed between one or more associated objects, and the reference points of objects Ot and Os that are not associated do not need to be used in the matching process.

[0146] (14) Some of the inspection processes in the above embodiment may be omitted as appropriate. For example, in the object identification process in Figure 5, all or part of the color conversion process in S210, the noise reduction process in S240, and the dilation process in S250 may be omitted. Also, the second matching process in S145 in Figure 4 may be omitted, and the candidate feature point pair M determined in the first matching process may be determined as the final matching pair.

[0147] (15) The reading device 100 may be configured such that the rotation angle of the object is approximately the same between the reference image IMt and the target image IMs. In this case, in S320 of Figure 7, rotation-invariant feature quantities Ft and Fs may be calculated (for example, BRIEF (Binary Robust Independent Elementary Features)). The calculation of the direction of the feature points may be omitted.

[0148] (16) The method for detecting feature points Tt and Ts in S310 of Figure 7 may be any of the various methods for detecting points that indicate parts of an object in the image, instead of the method described in the above embodiment. The detection method may be selected in advance from, for example, the search for extrema (maximum and minimum values) using DoG (Difference-of-Gaussian), Harris corner detection, FAST (Features from Accelerated Segment Test) corner detection, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF).

[0149] The method for calculating the feature descriptors of the feature quantities Ft and Fs may be any of the various methods used to calculate information describing the features of the feature points, instead of the method described in the above embodiment. The algorithm for calculating the feature descriptor may be selected in advance from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, and A-KAZE. Furthermore, the method for calculating the distance dF between the feature quantities Ft and Fs may be any of the various methods suitable for the data structure of the feature quantities Ft and Fs. When the feature quantities Ft and Fs are represented by binary vectors, the distance dF may be the Hamming distance. The distance dF may be any of the various other distances (for example, L1 norm, L2 norm (also called Euclidean distance), etc.) instead of the Hamming distance. Norms are applicable to various feature descriptors.

[0150] In either case, the direction associated with the feature points Tt and Ts may be the direction used to calculate the feature descriptor. Such a direction may be the direction of the gradient of various color values ​​(e.g., luminance).

[0151] (17) The printing medium is not limited to T-shirts 700, but may be various types of clothing (for example, various shirts such as polo shirts, coats, slacks, etc.). The printing medium may be various fabrics such as clothes and bags. The printing medium is not limited to fabric, but may be various media such as paper, film, leather, etc. Also, the printing device 900 may be a printing device of another type (for example, a laser printing device) instead of an inkjet printing device.

[0152] (18) When determining the coordinate correspondence between images IMt and IMs based on matching pairs, an affine transformation matrix Mtx is calculated (Figure 11). Alternatively, various transformation matrices, such as homography transformation matrices, may be calculated. Furthermore, the coordinate correspondence may be represented in various formats, such as lookup tables, instead of the matrix Mtx.

[0153] The coordinate correspondence determined based on matching pairs may be used in various processes, not just inspection processes. For example, in the machining of metal parts, the position and orientation of the metal part relative to the tool may be misaligned. Here, the position and orientation of the metal part relative to the tool may be determined by analyzing images of the metal part taken by a digital camera fixed to the tool. For example, the processor 210 may use an image representing the reference position of the metal part as a reference image IMt, and images obtained by photographing the metal part as target images IMs, and determine the reference position of the metal part on the captured images (and thus the position and orientation of the metal part relative to the tool) by performing alignment between images IMt and IMs. As the alignment process, the processes of each embodiment described above, or the processes of each modified example described above, may be adopted. Thus, the target images IMs may be images representing objects read by a line sensor such as the reading sensor 180 (Figure 2), or images representing objects read by an area sensor such as a digital camera. Furthermore, the reference image IMt may be any image other than the image related to printing (for example, a pre-prepared image representing a specific part of an object). In any case, the image may be a grayscale image instead of a color image.

[0154] (19) The image processing device 200 is not limited to a personal computer, but may be any other device (for example, a smartphone, a tablet computer, a control device incorporated into a reader, etc.). Alternatively, multiple devices (for example, computers) that can communicate with each other via a network may each share a portion of the data processing function of the image processing device and, as a whole, provide the data processing function (a system equipped with these devices corresponds to the image processing device).

[0155] In each of the above embodiments and modifications, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, the process of S290 in Figure 7 may be executed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0156] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0157] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of Symbols]

[0158] 100…Reading device, 110…Control device, 120…Transport device, 122…Position sensor, 130…Table, 140…Support unit, 180…Reading sensor, 190…Housing, 200…Image processing device, 210…Processor, 215…Storage device, 220…Volatile storage device, 230…Non-volatile storage device, 240…Display unit, 250…Operation unit, 270…Communication interface, 900…Printing device, IMc…Reading image, IMd…Difference image, IMs…Target image, IMt…Reference image, Os, Ot…Object, PF…Color conversion profile, PG…Computer program, Ts…Target feature point, Tt…Reference feature point

Claims

1. It is a computer program, An object identification function that identifies multiple reference objects in a reference image and multiple target objects in a target image, A correspondence determination function that performs a correspondence process to associate the target object with each of the multiple reference objects, A matching function that performs a matching process to determine matching pairs, which are pairs of corresponding reference feature points and target feature points, from among a plurality of reference feature points of the reference image and a plurality of target feature points of the target image, wherein the matching pair is determined such that the reference feature points of the first reference object form the matching pair with the target feature points of the first target object corresponding to the first reference object, and the reference feature points of the second reference object form the matching pair with the target feature points of the second target object corresponding to the second reference object. A computer program that causes a computer to execute something.

2. A computer program according to claim 1, The aforementioned response determination function is, The feature quantities of each of the multiple reference objects and the multiple target objects are calculated, A computer program that uses the aforementioned feature quantities to associate the target object with each of the multiple reference objects.

3. A computer program according to claim 2, The feature quantity of one of the reference objects includes the angle formed by two line segments connecting the centroid of one of the reference objects and the centroids of the other two reference objects. The feature quantity of one of the target objects includes the angle formed by two line segments connecting the centroid of one of the target objects and the centroids of the other two target objects. The correspondence determination function is a computer program that associates the reference object and the target object whose angles are similar to each other.

4. A computer program according to claim 3, The other two reference objects are the reference object with the shortest distance from its center of gravity to the first reference object, and the reference object with the second shortest distance from its center of gravity to the first reference object. A computer program in which the other two target objects are the target object having the shortest distance from its center of gravity to the first target object, and the target object having the second shortest distance from its center of gravity to the first target object.

5. A computer program according to claim 2, The characteristic quantity of the reference object is the area of ​​the reference object, The characteristic quantity of the target object is the area of ​​the target object, The correspondence determination function is a computer program that associates the reference object having the nth largest area among the N reference objects (where N is an integer of 2 or more) with the target object having the nth largest area among the N target objects.

6. A computer program according to claim 2, If the difference between the feature quantities of the first reference object and the feature quantities of the third reference object is less than a threshold, or if the difference between the feature quantities of the first target object and the feature quantities of the third target object corresponding to the third reference object is less than the threshold, the correspondence determination function is a computer program that, in the correspondence processing, associates the entirety of the first reference object and the third reference object with the entirety of the first target object and the third target object.

7. A computer program according to claim 1, The aforementioned object identification function is The aforementioned target image is converted to a grayscale image to generate a processed target image. By binarizing the processed target image, multiple object pixels within the target image are identified. A computer program that identifies multiple target objects using the multiple object pixels that are identified.

8. A computer program according to claim 1, The data of the reference image includes reference data indicating whether each of the multiple pixels of the reference image is an object pixel or not. The aforementioned object identification function is Using the aforementioned reference data, multiple object pixels of the reference image are identified, A computer program that identifies a plurality of reference objects using the plurality of object pixels that are identified.

9. A computer program according to claim 1, The correspondence determination function is a computer program that, when the number of identified reference objects is 1, and when the number of identified reference objects and the number of target objects are not the same, executes the matching process without using the reference objects and the target objects.

10. A computer program according to claim 1, A color conversion function is implemented in a computer that performs a color conversion process on at least one of the reference image and the target image using a color conversion profile for matching the color of the reference object and the color of the target object corresponding to the reference object. The aforementioned object identification function is After the color conversion process, the reference image and the target image are each binarized to identify multiple object pixels in the reference image and the target image, Using the multiple object pixels of the identified reference image, a plurality of reference objects are identified, A computer program that identifies multiple target objects using the multiple object pixels of the identified target image.

11. A computer program according to claim 1, The matching process includes a specific matching process that determines a pair of the two or more reference feature points and the two or more target feature points as the matching pair when the set of two or more reference feature points and the set of two or more target feature points are similar.

12. A computer program according to claim 11, A computer program that determines the matching pairs of a set including the reference feature points of the first reference object and the reference feature points of the third reference object as matching pairs when the set including the target feature points of the first target object and the target feature points of the third target object corresponding to the third reference object is similar to the set including the target feature points of the first target object and the target feature points of the third target object.

13. A computer program according to claim 1, further, A reference inclusion region encompassing each of the aforementioned reference objects and a target inclusion region encompassing each of the aforementioned target objects are identified. The matching function described above is From among the plurality of reference feature points located in the reference inclusion region that includes the first reference object, and the plurality of target feature points located in the target inclusion region that includes the first target object, one or more matching pairs are determined. A computer program that determines one or more matching pairs from among a plurality of reference feature points located in the reference inclusion region that includes the second reference object, and a plurality of target feature points located in the target inclusion region that includes the second target object.

14. A computer program according to claim 13, further, A computer program in which an area obtained by performing an expansion process on the reference object is identified as the reference inclusion area, and an area obtained by performing an expansion process on the target object is identified as the target inclusion area.

15. An image processing device, An object identification unit that identifies multiple reference objects in a reference image and multiple target objects in a target image, A correspondence determination unit that performs a correspondence process to associate the target object with each of the multiple reference objects, A matching unit that performs a matching process to determine matching pairs, which are pairs of reference feature points and target feature points that correspond to each other, from among a plurality of reference feature points of the reference image and a plurality of target feature points of the target image, wherein the matching unit determines the feature point pairs such that the reference feature points of the first reference object form a feature point pair with the target feature points of the first target object corresponding to the first reference object, and the reference feature points of the second reference object form a feature point pair with the target feature points of the second target object corresponding to the second reference object. An image processing device equipped with the following features.

Images