Program and data processing apparatus

Abstract

To perform feature point matching between a plurality of images.SOLUTION: A generation process for generating a reference image is executed using image data for printing representing an object image. Feature point matching processing is executed using a plurality of feature points in a read image and a plurality of feature points in the reference image. The read image is generated by optically reading an object image to be printed on a medium using image data for printing. Here, the reference image includes a reference transparent portion indicating that no color material is used, and the generation process may include a process of setting the color of the reference transparent portion to a medium color, which is a color representing the medium. Further, the generation process may include an additional process of adding to the reference image an external portion representing a medium color, which is an external portion of the target image, and a color representing the medium.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] This specification relates to feature point matching between multiple images.

Background Art

[0002] In various processes, feature point matching between multiple images can be performed. Patent Document 1 discloses a technique for inspecting a printed matter on which an image is printed on a recording medium. In this technique, an image input to a printing apparatus (also referred to as a reference image or a reference picture) and an image read by a reading apparatus such as a scanner from the printed matter output from the printing apparatus (also referred to as a printed image) are compared. Edge information such as lines and characters in the image is extracted as feature points, and alignment between the reference image and the printed image is performed so that the correlation of the positions between the feature points in the reference image and the printed image becomes the highest.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Feature point matching between multiple images is not easy and there is room for improvement.

[0005] This specification discloses a technique for performing feature point matching between multiple images.

Means for Solving the Problems

[0006] The technique disclosed in this specification can be realized as the following application examples.

[0007] [Applicable Example 1] A program that realizes a function of executing a generation process for generating a reference image using print image data representing a target image, wherein the reference image includes at least a part of a reference transparent part that is a part indicating non-use of a coloring material in the target image, and a colored part that is a part indicating use of a coloring material in the target image, and the generation process includes a process of setting the color of the reference transparent part to a medium color that is a color representing a medium, the function, and a function of executing a feature point matching process using a plurality of feature points in a read image generated by optically reading the target image printed on the medium using the print image data and a plurality of feature points in the reference image. A program that causes a computer to realize this.

[0008] According to this configuration, since the color of the reference transparent part of the reference image is set to the medium color that is the color representing the medium, the color difference between the reference transparent part of the reference image and the corresponding part of the read image is reduced. Therefore, it is possible to appropriately perform feature point matching using a plurality of feature points in the read image and a plurality of feature points in the reference image.

[0009] [Applicable Example 2] A program that realizes a function of executing a generation process for generating a reference image using print image data representing a target image, wherein the generation process includes an addition process of adding an outer part that is an outer part of the target image and indicates a medium color that is a color representing the medium to the reference image, the function, and a function of executing a feature point matching process using a plurality of feature points in a read image generated by optically reading the target image printed on the medium using the print image data and a plurality of feature points in the reference image. A program that causes a computer to realize this.

[0010] According to this configuration, since an outer portion of the target image, which is an outer portion showing a medium color that is the color representing the medium, is added to the reference image, in addition to the feature points located inside the target image, the feature points located at the edges of the target image can be used for feature point matching. Therefore, it is possible to appropriately perform feature point matching using a plurality of feature points in the read image and a plurality of feature points in the reference image.

[0011] Note that the technology disclosed in this specification can be realized in various forms, for example, in the form of a data processing method and a data processing apparatus, a computer program for realizing the functions of those methods or apparatuses, a recording medium (for example, a non-transitory recording medium) recording the computer program, and the like.

Brief Description of Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Mode for Carrying Out the Invention

[0013] A. First Embodiment: A1. Device Configuration: FIG. 1 is an explanatory diagram showing a data processing device as an example. The data processing device 200 is, for example, a personal computer. The data processing device 200 executes an inspection process of a printed image. The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0014] The processor 210 is a device configured to perform data processing, and is, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile storage device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores the data of the program 231.

[0015] The display unit 240 is a device configured to display an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as buttons, levers, or a touch panel disposed on top of the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. By operating the operation unit 250, the user can input various requests and instructions to the data processing device 200.

[0016] The communication interface 270 is an interface for communicating with other devices (for example, including one or more of a USB interface, a wired LAN interface, a wireless interface of IEEE802.11, an interface of an industrial camera (for example, CameraLink, CoaXPress, etc.)). In this embodiment, the reading device 100 and the printing device 900 are connected to the communication interface 270. The printing device 900 is a so-called inkjet printing device, and prints an image on a printing medium by ejecting ink onto the printing medium such as cloth or paper. The reading device 100 generates data of a read image representing an object by optically reading the object to be read. Hereinafter, the printing medium is a T-shirt, and the reading device 100 reads the T-shirt on which an image is printed.

[0017] FIG. 2 is a perspective view showing an example of the reading device 100. The first direction Da and the second direction Db in the figure indicate horizontal directions, and the third direction Dc indicates the vertically upward direction. The first direction Da and the second direction Db are perpendicular to each other.

[0018] In this embodiment, the reading device 100 includes a housing 190, a table 130, a support portion 140 fixed to the upper surface of the table 130, a transport device 120, a reading sensor 180, and a control device 110. The control device 110, the transport device 120, and the reading sensor 180 are fixed to the housing 190.

[0019] The support unit 140 is a plate-like member that forms a flat upper surface for supporting the object to be read (such a member is also called a platen). In the figure, a T-shirt 700 having a printed image IMpp is placed on the support unit 140.

[0020] The conveying device 120 is configured to convey the table 130 in a direction parallel to the second direction Db. The configuration of the conveying device 120 may be various configurations. Although illustration is omitted, in this embodiment, the conveying device 120 includes a rail that slidably supports the table 130 in a direction parallel to the second direction Db, a plurality of pulleys, a belt that is wound around the plurality of pulleys and a part of which is fixed to the table 130, and an electric motor that rotates the pulley. When the electric motor rotates the pulley, the table 130 (and thus the support unit 140) moves in a direction parallel to the second direction Db. The conveying device 120 further includes a position sensor 122 (for example, a rotary encoder) that detects the position of the table 130 on the conveying path.

[0021] The reading sensor 180 is disposed at a position higher than the support unit 140 in the middle of the conveying path PTh of the support unit 140. The reading sensor 180 includes a line sensor composed of a plurality of photoelectric conversion elements arranged in a direction intersecting the conveying direction Db (in this embodiment, a direction Da perpendicular to the conveying direction Db) (for example, a Contact Image Sensor (CIS) or a Charge Coupled Device (CCD)). The reading sensor 180 faces downward. The reading sensor 180 can read a portion of the object supported by the support unit 140 that is located below the reading sensor 180.

[0022] When reading the T-shirt 700, the reading device 100 conveys the table 130 in a direction parallel to the second direction Db. The reading sensor 180 repeatedly reads the T-shirt 700 during conveyance. Thereby, the reading sensor 180 can read approximately the entire portion of the T-shirt 700 that is supported by the support unit 140.

[0023] The control device 110 is an electric circuit configured to control the transport device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates data of a read image by controlling the transport device 120 and the reading sensor 180.

[0024] A2. Printing process: In this embodiment, an image is printed on the T-shirt 700 (FIG. 2). The printing of the image is performed, for example, as part of a T-shirt sales service. The T-shirt sales service may include on-demand printing. A customer places an order for on-demand printing with a service provider. The service provider prints an image on the T-shirt 700 using the image data provided by the customer in response to the customer's order.

[0025] FIG. 3(A) is a diagram showing an example of an image represented by image data for printing (referred to as a target image IMp). In this embodiment, the data of the target image IMp is bitmap data representing the respective color values (here, the respective gradation values of red R, green G, and blue B (for example, values of 0 or more and 255 or less)) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. In the example of FIG. 3(A), the target image IMp represents a background BG and four rectangular objects OB1 - OB4. The objects OB1 - OB4 are represented in different colors. For example, the color of the first object OB1 is red, the color of the second object OB2 is green, the color of the third object OB3 is blue, and the color of the fourth object OB4 is gray. Thus, the target image IMp may include a plurality of objects having different hues.

[0026] In this embodiment, the data of the target image IMp further includes an alpha channel representing the opacity of each of a plurality of pixels. In the background BG, the opacity is set to a value indicating transparency. In this embodiment, the value indicating transparency is used as a value indicating non-use of ink. Hereinafter, the portion indicating non-use of ink in the image is also referred to as a transparent portion. The portion indicating use of ink in the image is also referred to as a colored portion. The background BG is an example of a transparent portion (referred to as a transparent portion P0). The objects OB1-OB4 are examples of colored portions (referred to as colored portions P1).

[0027] Although illustration is omitted, the printing of the target image IMp is performed using the data processing device 200 and the printing device 900. Instead of this, the printing of the target image IMp may be performed using other devices.

[0028] In the printing of the target image IMp, various errors may occur. The printed image may have various defects due to errors. For example, due to abnormal ink ejection, a part of the image may be missing in the printed image. The data processing device 200 detects defects in the printed image by an inspection process described later. In this embodiment, the data of the target image IMp is used in the inspection process. After the printing of the target image IMp, the data of the target image IMp is stored in the storage device 215 (for example, the non-volatile storage device 230) of the data processing device 200 for the inspection process.

[0029] A3. Inspection process: FIG. 4 is a flowchart showing an example of an inspection process. For inspection, the T-shirt 700 is placed on the support portion 140 of the reading device 100 (FIG. 2) so that the printed image IMpp can be seen. In this embodiment, an operator places the T-shirt 700 on the support portion 140. Alternatively, a machine (e.g., a robotic arm) may place the T-shirt 700 on the support portion 140. After the placement of the T-shirt 700, an instruction to start the inspection process is input to the data processing device 200 (FIG. 1). In this embodiment, the operator inputs an instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the start instruction. Note that the start instruction may be input to the data processing device 200 via the communication interface 270 by another device different from the data processing device 200.

[0030] The processor 210 of the data processing device 200 executes an inspection process according to the program 231. In S110, the T-shirt 700 is photographed. The processor 210 supplies a reading instruction to the reading device 100. The control device 110 of the reading device 100 controls the reading sensor 180 and the transport device 120 in response to the reading instruction to read the T-shirt 700. The control device 110 generates data of a reading image representing the read T-shirt 700. The processor 210 of the data processing device 200 acquires the data of the reading image from the control device 110 of the reading device 100 and stores the acquired data of the reading image in the storage device 215 (e.g., the nonvolatile storage device 230).

[0031] FIG. 3(B) is a diagram showing an example of a read image. In the present embodiment, the data of the read image IMs is bitmap data representing the respective color values (here, the respective gradation values of red R, green G, and blue B (for example, values of 0 or more and 255 or less)) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. The read image IMs in the figure represents a portion including the printed image IMpp of the T-shirt 700. Here, it is assumed that the printed image IMpp is an image obtained by printing the target image IMp (FIG. 3(A)). The printed image IMpp represents the objects OB1-OB4 in the same manner as the target image IMp.

[0032] In S120 (FIG. 4), the processor 210 performs alignment between the reference image and the read image. In the present embodiment, an image obtained by processing the target image IMp (FIG. 3(A)) is used as the reference image (details will be described later).

[0033] FIG. 5 is a flowchart showing an example of the alignment process. In the present embodiment, the processor 210 obtains a plurality of pairs of feature points by feature point matching, and uses the obtained plurality of pairs to determine the correspondence between the coordinates on the reference image and the coordinates on the read image IMs.

[0034] In S220, the processor 210 forms candidate pairs that are candidates for feature point pairs. FIG. 6 is a flowchart showing an example of the candidate pair formation process. In S310, the processor 210 reads the data of the target image IMp (FIG. 3(A)) used for printing from the storage device 215. As described above, the data of the target image IMp has been previously stored in the storage device 215 (for example, the non-volatile storage device 230).

[0035] In S315 (Fig. 6), the processor 210 determines a circumscribed rectangle that circumscribes the colored portion P1 of the target image IMp (Fig. 3(A)). Fig. 7 is a diagram showing an example of the circumscribed rectangle. In the figure, the target image IMp and the circumscribed rectangle RA are shown. The circumscribed rectangle RA is a rectangle having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy. The circumscribed rectangle RA is the smallest rectangle that includes the colored portion P1 (here, the objects OB1 - OB4). In the target image IMp of Fig. 7, each of the objects OB1 - OB4 forms a part of the edge of the target image IMp. And the circumscribed contour LRA, which is the contour of the circumscribed rectangle RA, coincides with the target contour LIMp, which is the contour of the target image IMp.

[0036] Although not shown, the circumscribed rectangle RA varies variously according to the position and shape of the colored portion P1 within the target image IMp. For example, when the colored portion P1 is located inside the target contour LIMp, the circumscribed rectangle RA is formed inside the target contour LIMp.

[0037] In S320 (Fig. 6), the processor 210 reads the data of the read image IMs from the storage device 215.

[0038] In S330, the processor 210 generates the data of the reference image using the data of the target image IMp. As will be described later, feature points are detected from the reference image. Points indicating characteristic portions such as the corners and edges of an object are detected as feature points. For example, a point Pw indicating the upper right corner of the first object OB1 (FIG. 7) is suitable as a feature point. For the detection of feature points, the color values of a plurality of pixels in the area around the candidate point (i.e., the partial area including the candidate point) are used. If a feature point is detected from the target image IMp, since the point Pw is located at the edge of the target image IMp, an upper part of the partial area Aw including the point Pw is outside the target contour LIMp. Since the color values of the pixels outside the target image IMp cannot be referred to, it is difficult to detect the point Pw as a feature point. Thus, when an object is located near the edge of an image, the detection of feature points from near the edge of the image is difficult. In the present embodiment, even when an object is located near the edge of the target image IMp, the processor 210 generates a reference image having a peripheral portion outside the circumscribed rectangle RA so that feature points can be detected from near the edge of the target image IMp.

[0039] In the present embodiment, the process of S330 (FIG. 6) includes S335 - S355. In S335, the processor 210 determines the medium color by analyzing the read image IMs. FIG. 8(A) is a flowchart showing an example of the process of determining the medium color. In S910, the processor 210 obtains the frequency distribution of the appearance of colors by analyzing the read image IMs. In the present embodiment, the processor 210 generates a three-dimensional histogram HST of RGB.

[0040] FIG. 8(B) is a schematic diagram of the three-dimensional histogram HST. In the figure, a three-dimensional color solid CC represented by three gradation values of red R, green G, and blue B is shown. Signs indicating colors are attached to the vertices of the color solid CC (here, red R, green G, blue B, cyan Cy, magenta Mg, yellow Yl, white Wh). In this embodiment, the range of each gradation value of red R, green G, and blue B is divided into J bins (J is an integer of 2 or more. For example, J = 9). The color value represented by the gradation values of RGB is associated with any one of the J*J*J bins. The processor 210 calculates the frequency of each bin using the color values of a plurality of pixels of the read image IMs. Thereby, the processor 210 generates a three-dimensional histogram HST, that is, a color appearance frequency distribution.

[0041] In S915 (FIG. 8(A)), the processor 210 determines the representative color Cb using the three-dimensional histogram HST. In this embodiment, the processor 210 selects the bin with the highest frequency among the plurality of bins of the three-dimensional histogram HST. The processor 210 calculates the average value of each RGB of the plurality of color values associated with the bin with the highest frequency in S910 as the RGB gradation value of the representative color Cb. Note that instead of the average value, various summary statistics (for example, median, mode, etc.) representing the magnitude of the color value may be used.

[0042] In S920, the processor 210 determines the medium color Cm as the representative color Cb. Then, the processor 210 ends the process of FIG. 8(A), that is, S335 of FIG. 6. As will be described later, the medium color Cm is used as the color of the printing medium (here, the T-shirt 700). In this embodiment, in the read image IMs (FIG. 3(B)), the area of the blank portion without the printed image of the T-shirt 700 is assumed to be larger than the area of the portion representing the object. In this case, the representative color Cb is a good approximation of the color of the T-shirt 700 in the read image IMs.

[0043] In S340 (FIG. 6), the processor 210 generates data of a temporary reference image by setting the color of the transparent portion P0 of the target image IMp (FIG. 7) to the medium color Cm. FIG. 9 is a diagram showing an example of an image to be processed. The target image IMp is shown in the lower left part of FIG. 9. In this embodiment, the data of the target image IMp is RGB bitmap data having an alpha channel. The processor 210 separates from the target image IMp a color image IMpc representing each gradation value of RGB and a mask image IMpm representing the opacity k of the alpha channel. Here, in the area of the background BG, the opacity k is zero, and in the areas of the objects OB1-OB4, the opacity k is set to 1. The range of the opacity k is from zero or more to 1 or less. K = 0 indicates transparency, and K = 1 indicates opacity. The processor 210 temporarily sets RGB = 255 in the portion where k = 0 in the color image IMpc (that is, the transparent portion P0).

[0044] Note that in at least a part of the target image IMp, the opacity k may indicate semi-transparency (here, 0 < k < 1). In this case, the processor 210 may calculate each gradation value Vo of RGB in the color image IMpc using the opacity k and the original gradation value Vi. For example, the following calculation formula may be adopted. Vo=(k*Vi)+((1-k)*255)

[0045] The processor 210 generates data of a temporary reference image IM11 by setting the color value of the transparent portion P0 of the color image IMpc to the medium color Cm.

[0046] In S345 (FIG. 6), the processor 210 determines a peripheral portion surrounding the circumscribed rectangle RA. An example of the peripheral portion PS is shown above the target image IMp in FIG. 9. In this embodiment, the processor 210 determines an area with a predetermined width wt surrounding the circumscribed rectangle RA as the peripheral portion PS.

[0047] In S350 (Fig. 6), the processor 210 generates data for a new temporary reference image by adding an outer portion of the peripheral portion PS that is located outside the target image IMp to the temporary reference image IM11. The upper right portion in Fig. 9 shows the temporary reference image IM12 to which the outer portion Pu has been added. In the example of Fig. 9, the entire peripheral portion PS is located outside the target image IMp. Accordingly, the entire peripheral portion PS corresponds to the outer portion Pu. The processor 210 sets the color value of the outer portion Pu (here, the same as the peripheral portion PS) to the medium color Cm. In the example of Fig. 9, the circumscribed contour LRA coincides with the contour of the temporary reference image IM11. Accordingly, the peripheral portion PS surrounds the temporary reference image IM11.

[0048] In S355, the processor 210 generates data for a new temporary reference image by deleting a portion outside the peripheral portion PS from the temporary reference image IM12. In the example of Fig. 9, since the temporary reference image IM12 does not have a portion outside the peripheral portion PS, the processed temporary reference image by S355 is the same as the temporary reference image IM12. Although illustration is omitted, when the circumscribed rectangle RA is formed inside the target contour LIMp, the peripheral portion PS can be arranged inside the contour of the temporary reference image IM11. In this case, the portion outside the peripheral portion PS in the temporary reference image IM12 is deleted.

[0049] Through the above S335 - S355, data for the reference image is generated. In the example of Fig. 9, the temporary reference image IM12 is used as the reference image. Hereinafter, the temporary reference image IM12 is also referred to as the reference image IMt.

[0050] In S360, the processor 210 generates a grayscale reference image IMtg and a grayscale read image IMsg by performing grayscale conversion on the reference image IMt and the read image IMs. As the correspondence relationship between the color tone value and the grayscale tone value, a known relationship can be adopted (for example, the correspondence relationship between the RGB values in the RGB color space and the luminance value Y in the YCbCr color space).

[0051] In S365, the processor 210 extracts feature points Tt from the gray reference image IMtg. FIGS. 10(A)-10(C) are diagrams showing examples of images to be processed. FIG. 10(A) shows an example of feature points detected from the images IMtg and IMsg. The plurality of black dots on the gray reference image IMtg each indicate a feature point Tt detected from the gray reference image IMtg (also referred to as a reference feature point Tt). As shown in the figure, points indicating characteristic portions such as corners and ends of an object are detected as the feature points Tt. Such feature points Tt are also called keypoints. Since the surrounding portion PS is added, reference feature points Tt can also be detected from the vicinity of the target contour LIMp. For example, the reference feature point Tt1 corresponds to the point Pw in FIG. 7. Although not shown, actually, more feature points Tt can be detected (for example, about several tens or several hundreds). Note that the reference feature points Tt detected from the gray reference image IMtg correspond to the feature points indicating the same portion at the same coordinates on the reference image IMt (FIG. 9).

[0052] The method for detecting feature points may be various methods. In this embodiment, a technique called Accelerated-KAZE (A-KAZE) that performs detection of keypoints and calculation of feature descriptors for each keypoint is used. The A-KAZE technique is disclosed, for example, in the following paper. “Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013”

[0053] The processor 210 detects a plurality of feature points Tt by analyzing the gray reference image IMtg according to the A-KAZE technique.

[0054] In S370 (FIG. 6), the processor 210 extracts a plurality of feature points Ts from the gray-scale read image IMsg. The plurality of black dots on the gray-scale read image IMsg in FIG. 10(A) each indicate a feature point Ts detected from the gray-scale read image IMsg (also referred to as a read feature point Ts). The processor 210 detects the plurality of feature points Ts by analyzing the gray-scale read image IMsg according to the A-KAZE technique. Although not shown, actually, more feature points Ts can be detected (for example, about several tens or several hundreds). Note that the read feature point Ts detected from the gray-scale read image IMsg corresponds to the feature point indicating the same part at the same coordinates on the read image IMs (FIG. 3(B)).

[0055] In S375, the processor 210 calculates a feature amount Ft for each of the plurality of reference feature points Tt. The feature amount Ft may be various information describing the feature of the feature point Tt. The feature amount Ft is calculated, for example, so as to change according to the distribution of the color values of a plurality of pixels around the feature point Tt. In this embodiment, the processor 210 calculates an A-KAZE feature descriptor as the feature amount Ft using the gray-scale reference image IMtg.

[0056] In S380, the processor 210 calculates a feature amount Fs for each of the plurality of read feature points Ts. In this embodiment, the processor 210 calculates a feature descriptor (that is, the feature amount Fs) according to the A-KAZE technique using the gray-scale read image IMsg.

[0057] In S385, the processor 210 performs matching between a plurality of reference feature points Tt and a plurality of reading feature points Ts. Through the matching, the feature points Tt and Ts indicating similar portions (e.g., the same portion of the same object) in the images IMtg and IMsg are associated with each other. The feature points Tt and Ts associated with each other are also called a matching pair. As will be described later, more appropriate pairs are selected from the plurality of pairs of the feature points Tt and Ts formed in S385. Hereinafter, the finally obtained pair of the feature points Tt and Ts is called a feature point pair. In S385, the processor 210 acquires candidates for the feature point pair (also called candidate feature point pairs).

[0058] FIG. 10(B) shows an example of candidate feature point pairs. A plurality of lines RL in the figure each indicate a candidate feature point pair M. Each line RL connects the feature points Tt and Ts that form the candidate feature point pair M. As shown in the figure, a pair of feature points Tt and Ts indicating the same portion of each other can be selected as the candidate feature point pair M. For example, the feature point Tt1 indicating the upper right corner of the first object OB1 in the gray reference image IMtg can be associated with the feature point Ts1 indicating the upper right corner of the first object OB1 in the gray reading image IMsg.

[0059] Also, a pair of feature points Tt and Ts indicating different portions of each other can be selected as the candidate feature point pair M. For example, the feature point Tt1 indicating the upper right corner of the first object OB1 in the gray reference image IMtg can be associated with the feature point Ts2 indicating the upper right corner of the third object OB3 in the gray reading image IMsg.

[0060] The matching method may be various methods. For example, the processor 210 may associate the feature points Tt and Ts with close distances obtained using the feature amounts Ft and Fs. The distance is calculated such that a small distance indicates a high similarity between the two feature amounts Ft and Fs. Such a distance calculation method may be various methods suitable for the data configuration of the feature amounts Ft and Fs. When the feature amounts Ft and Fs are represented by binary vectors (vectors composed of two-valued elements of 1 or more), such as the A-KAZE feature descriptor, the processor 210 may use the Hamming distance. Instead of the Hamming distance, the processor 210 may use various other distances (for example, norms such as the L1 norm and the L2 norm (also called the Euclidean distance)). Norms are applicable to various feature descriptors.

[0061] The processor 210 associates the closest reading feature point Ts among the plurality of reading feature points Ts with the reference feature point Tt. Instead of this, the processor 210 may associate the closest reference feature point Tt among the plurality of reference feature points Tt with the reading feature point Ts. The method of searching for the closest feature point may be various methods. For example, the processor 210 may form a plurality of candidate feature point pairs M by exhaustive matching. Instead of this, other methods such as FLANN (Fast Library for Approximate Nearest Neighbor)-based matching may be adopted.

[0062] Here, the processor 210 may execute a process of excluding pairs with low reliability. For example, the processor 210 may sort a plurality of pairs of the feature points Tt and Ts in ascending order of distance and select a part of the upper pairs (for example, 50% of the pairs).

[0063] In S385, the processor 210 stores data representing a plurality of candidate feature point pairs M in the storage device 215 (for example, the non-volatile storage device 230). Then, the processor 210 ends the process of FIG. 6, that is, the process of S220 in FIG. 5.

[0064] In S230, the processor 210 executes a selection process for feature point pairs that use color information. This selection process selects an appropriate candidate M from a plurality of candidate feature point pairs M using color information (i.e., the candidate feature point pair M is verified). In this embodiment, the plurality of candidate feature point pairs M selected by the process of S230 are used as the final feature point pairs.

[0065] FIG. 11 is a flowchart showing an example of the selection process. The processor 210 executes loop processing S410 (including S420 - S465) between the start L4s and the end L4e for each of the plurality of candidate feature point pairs M.

[0066] In S420, the processor 210 selects an unprocessed candidate M as a candidate of interest Mi that is the candidate to be processed.

[0067] In S425, the processor 210 calculates a representative color value Ct of the first partial region Pt including the reference feature point Tt of the candidate of interest Mi (the representative color value Ct is also referred to as the reference representative color value Ct). FIGS. 12(A) and 12(B) are diagrams showing examples of calculating the representative color value. FIG. 12(A) shows a part of the reference image IMt including the feature point Tt. In the figure, the first partial region Pt including the feature point Tt is shown. The processor 210 calculates the reference representative color value Ct using the color values of a plurality of pixels in the first partial region Pt. The method of calculating the representative color value Ct may be various methods for calculating the color representing the first partial region Pt. In the present embodiment, the processor 210 calculates the average value of each of red R, green G, and blue B in the first partial region Pt as the representative color value Ct. Note that instead of the average value, various summary statistics representing the magnitude of the color value (for example, median, mode, etc.) may be used. Further, the configuration of the first partial region Pt (specifically, the shape of the first partial region Pt and the relative position of the first partial region Pt with respect to the position of the feature point Tt) may be various configurations capable of calculating the representative color value Ct representing the color of the portion indicated by the feature point Tt. In the present embodiment, the first partial region Pt is a region centered on the feature point Tt, and is, for example, a region of P*Q pixels of P rows and Q columns. In order to reduce the dependency of the representative color value Ct on the rotation angle of the object, P = Q is preferable (for example, P = Q = 5). Instead of the region of P rows and Q columns, the first partial region Pt may be a region where the distance from the feature point Tt is less than or equal to a distance threshold.

[0068] In S430 (FIG. 11), the processor 210 calculates a representative color value Cs of the second partial region Ps including the read feature point Ts of the candidate of interest Mi (the representative color value Cs is also referred to as the read representative color value Cs). FIG. 12(B) shows a part of the read image IMs including the feature point Ts. In the figure, the second partial region Ps including the feature point Ts is shown. The configuration of the second partial region Ps is the same as that of the first partial region Pt. The second partial region Ps is a region of P*Q pixels in P rows and Q columns centered on the feature point Ts. The method for calculating the read representative color value Cs is the same as the method for calculating the reference representative color value Ct. The processor 210 calculates the average value of each of red R, green G, and blue B in the second partial region Ps as the read representative color value Cs.

[0069] In S435 (FIG. 11), the processor 210 calculates a first hue Ht from the reference representative color value Ct and calculates a second hue Hs from the read representative color value Cs. As the correspondence relationship between the representative color value and the hue, a known relationship can be adopted (for example, the correspondence relationship between the RGB value in the RGB color space and the H value in the HSV color space). Note that the first hue Ht is an example of the representative color value CJt of the first partial region Pt, similar to the reference representative color value Ct. The second hue Hs is an example of the representative color value CJs of the second partial region Ps, similar to the read representative color value Cs.

[0070] In S445, the processor 210 determines whether a hue condition CH is satisfied, which indicates that the absolute value of the difference between the first hue Ht and the second hue Hs is less than the hue difference threshold dHth. As the absolute value of the difference between the hues Ht and Hs (also referred to as the hue difference dH), a value corresponding to the smaller angular difference between the first hue Ht and the second hue Hs on the hue circle is adopted. When the candidate of interest Mi is an appropriate pair of feature points Tt and Ts that indicate the same part of each other, the hue difference dH can be a small value. When the hue difference dH is less than the hue difference threshold dHth (S445: Yes), in S460, the processor 210 selects the candidate of interest Mi as a candidate to be retained. When the hue difference dH is greater than or equal to the hue difference threshold dHth (S445: No), between the first partial region Pt and the second partial region Ps, the hues can be significantly different. That is, the feature points Tt and Ts of the candidate of interest Mi are likely to indicate different parts of each other. In S465, the processor 210 excludes the candidate of interest Mi from the candidates of feature point pairs.

[0071] In this way, the processor 210 excludes the candidate of interest Mi having a hue difference dH greater than or equal to the hue difference threshold dHth from the candidates. FIG. 10(C) shows an example of the candidate feature point pairs M remaining by the process of FIG. 11. Pairs of feature points Tt and Ts indicating parts having different hues from each other can be excluded. For example, between the first object OB1 and the third object OB3, the hues are different. The line RLa shown in FIG. 10(B) indicates a pair of the reference feature point Tt of the first object OB1 and the read feature point Ts of the third object OB3. As shown in FIG. 10(C), this pair is excluded (FIG. 11: S445(No), S465).

[0072] Note that, the looser the conditions for leaving the candidate of interest Mi are, the greater the total number of appropriate candidates M that remain without being excluded can be. However, the total number of inappropriate candidates can also be large. The greater the total number of appropriate candidates is, the more reliable the process using a plurality of feature point pairs becomes (for example, the error in the correspondence relationship of coordinates described later is reduced). When the total number of inappropriate candidates is large, the reliability of the process using a plurality of feature point pairs may decrease (for example, the error in the correspondence relationship of coordinates may increase). The color difference threshold dHth (S445) may be determined experimentally in advance so that the reliability of the process using a plurality of feature point pairs is acceptable. For example, the looser the conditions for leaving the candidate of interest Mi are as the color difference threshold dHth is larger. Even if the candidate of interest Mi is an appropriate pair of feature points Tt and Ts, the color difference dH can be a value greater than zero. The color difference threshold dHth greater than zero allows such a color difference dH. The color difference threshold dHth may be set to a value greater than zero and less than the maximum possible value of the color difference dH.

[0073] After S460 or S465, the processor 210 proceeds to S420 and executes the loop process S410 for the next candidate of interest Mi. When the loop process S410 for all candidates M ends, at S470, the processor 210 stores the data representing the candidate feature point pairs M to be left in the storage device 215 (for example, the non-volatile storage device 230). Then, the processor 210 ends the process of FIG. 11, that is, the process of S230 in FIG. 5. The plurality of candidate feature point pairs M selected by the process of S230 are used as the final plurality of feature point pairs (in this embodiment, the plurality of feature point pairs are used for determining the correspondence relationship of coordinates). Thus, in this embodiment, the process of feature point matching includes S385 (FIG. 6) and S230 (FIG. 5).

[0074] In S250, the processor 210 determines the correspondence between the coordinates on the reference image IMt and the coordinates on the captured image IMs. FIG. 13 is a diagram showing an example of the expression form of the coordinate correspondence. In the figure, the coordinates COt on the reference image IMt, the coordinates Cos on the captured image IMs, and the matrix Mtx that associates these coordinates COt and COs are represented. In this embodiment, the matrix Mtx represents a so-called affine transformation. In the figure, the coordinates COt, COs, and the matrix Mtx are represented in a homogeneous coordinate system. Each coordinate COt, COs is represented by a three-dimensional vector having the positions Xt, Xs in the first direction Dx and the positions Yt, Ys in the second direction Dy on the images IMt, IMs, and 1 as the third component. The matrix Mtx is a 3x3 matrix. As shown in the figure, the matrix Mtx is represented by six parameters a-f in two rows and three columns and three components (0, 0, 1) in the third row. Such a matrix Mtx can represent rotation, enlargement, reduction, translation, and skew. The matrix Mtx can be calculated by using three or more pairs of the coordinates COt, COs (that is, three or more feature point pairs).

[0075] The processor 210 uses a plurality of feature point pairs MP (that is, a plurality of candidate feature point pairs M obtained in S220-S230 of FIG. 5) to determine the matrix Mtx (here, six parameters a-f). The method for determining the matrix Mtx may be various methods. For example, the processor 210 may determine the matrix Mtx according to a method called RANdom SAmple Consensus (RANSAC). For example, a function of OpenCV (Open Source Computer Vision Library) may be used to determine the matrix Mtx.

[0076] Upon the completion of S250 (Fig. 5), the process of Fig. 5, i.e., S120 of Fig. 4, ends. In S130, the processor 210 inspects the printed image. The inspection method may be various methods that use the correspondence relationship of coordinates (Fig. 13). In this embodiment, the processor 210 generates data of the difference image using the correspondence relationship of coordinates. Figs. 14(A) and 14(B) are diagrams showing examples of the difference image. Fig. 14(A) shows the case where there is no defect in the printed image, and Fig. 14(B) shows the case where the printed image has a defect (here, a missing Err).

[0077] On the left side of Figs. 14(A) and 14(B), the read images IMs, IMs2 and the target image IMp arranged on the read images IMs, IMs2 are shown. As described with reference to Figs. 6 and 9, the reference image IMt includes at least a part of the target image IMp (here, the part including the colored portion P1). The correspondence relationship between the coordinates COt on the reference image IMt and the coordinates on the target image IMp is represented by a translation that associates the common parts of the images IMt and IMp. The processor 210 determines the correspondence relationship (referred to as the target coordinate relationship) between the coordinates on the target image IMp and the coordinates COs on the read image IMs using the correspondence relationship between the reference image IMt and the target image IMp and the correspondence relationship of Fig. 13. In Figs. 14(A) and 14(B), the target image IMp is arranged on the read images IMs, IMs2 according to the target correspondence relationship.

[0078] The right-side images IMd and IMd2 in FIGS. 14(A) and 14(B) represent examples of difference images between the read images IMs and IMs2 and the reference image IMt. The processor 210 generates data of the difference images IMd and IMd2 that represent the difference in color values (e.g., the absolute value of the difference in luminance values) between the read images IMs and IMs2 and the target image IMp at positions associated by the target coordinate relationship. The read image IMs in FIG. 14(A) represents a printed image without defects. Therefore, the difference image IMd does not have a portion indicating a large difference. The read image IMs2 in FIG. 14(B) represents a printed image having a defect Err. Therefore, within the difference image IMd2, the portion corresponding to the defect Err represents a larger difference compared to other portions.

[0079] Note that the processor 210 may generate data of the difference images IMd and IMd2 using the reference image IMt instead of the target image IMp. In this case, the difference in the non-defective portion of the background BG becomes smaller.

[0080] In S140 (FIG. 4), the processor 210 outputs an inspection result. The method of outputting the inspection result may be various methods. In this embodiment, the processor 210 displays the difference image on the display unit 240 (FIG. 1). By observing the display unit 240, an operator can easily recognize defects in the printed image. Alternatively, the processor 210 may output data representing the inspection result to a storage device (e.g., the non-volatile storage device 230 or an external storage device connected to the data processing device 200). Thereby, the data representing the inspection result is stored in the storage device. The data representing the inspection result can be used for various processes (e.g., the overall inspection process of the T-shirt 700). After S140, the processor 210 ends the inspection process.

[0081] FIG. 15(A) and FIG. 15(B) are diagrams showing another example of the image to be processed. In FIG. 15(A) and FIG. 15(B), color images IMpc2 and IMpc3 of target images IMp2 and IMp3 are shown. The difference from the color image IMpc in FIG. 9 is only that the backgrounds BG2 and BG3 extend outside the circumscribed rectangle RA. In the areas of the backgrounds BG2 and BG3, the opacity k indicates transparency. The objects OB1 - OB4 in the color images IMpc2 and IMpc3 are the same as the objects OB1 - OB4 in the color image IMpc in FIG. 9, respectively. Even when the target images IMp2 and IMp3 are used, a reading image similar to the reading image IMs (FIG. 3(B)) is acquired.

[0082] The background BG2 of the color image IMpc2 in FIG. 15(A) is configured to surround each of the objects OB1 - OB4. In S340 (FIG. 6), the processor 210 generates a temporary reference image IM21 by setting the color value of the transparent portion P02 of the color image IMpc2 to the medium color Cm. In S345, the processor 210 determines the peripheral portion PS2 surrounding the circumscribed rectangle RA. The peripheral portion PS2 is formed inside the contour of the temporary reference image IM21. Therefore, in S350, no new area is added to the temporary reference image IM21. The processed temporary reference image by S350 is the same as the temporary reference image IM21. In S355, the processor 210 generates a temporary reference image IM22 by deleting the portion outside the peripheral portion PS2 from the temporary reference image IM21. The temporary reference image IM22 is used as a reference image (referred to as the reference image IMt2).

[0083] The background BG3 of the color image IMpc3 in FIG. 15(B) includes the portion on the Dx side and the portion on the Dy side of the circumscribed rectangle RA. In S340 (FIG. 6), the processor 210 generates a temporary reference image IM31 by setting the color value of the transparent portion P03 of the color image IMpc2 to the medium color Cm. In S345, the processor 210 determines the peripheral portion PS3 surrounding the circumscribed rectangle RA. The peripheral portion PS3 includes the outer portion Pu3 that is the portion outside the temporary reference image IM31 (specifically, the portion on the -Dx direction side and the portion on the -Dy direction side of the temporary reference image IM31). In S350, the processor 210 generates a temporary reference image IM32 by adding the outer portion Pu3, which is located outside the temporary reference image IM31, to the temporary reference image IM31 among the peripheral portion PS3. The processor 210 sets the color value of the outer portion Pu3 to the medium color Cm. In S355, the processor 210 generates a temporary reference image IM33 by deleting the portion outside the peripheral portion PS3 from the temporary reference image IM32. The temporary reference image IM33 is used as a reference image (referred to as the reference image IMt3). In this way, the outer portion Pu3 is added to the reference image IMt3.

[0084] In this way, similar to the reference image IMt (FIG. 10(A)), the reference images IMt2 and IMt3 include the peripheral portions PS2 and PS3 surrounding the circumscribed rectangle RA. Therefore, similar to the reference image IMt, appropriate candidate feature point pairs M can be formed from the vicinity of the circumscribed contour LRA also from the reference images IMt2 and IMt3.

[0085] Also, when the color images IMpc2 and IMpc3, and thus the target images IMp2 and IMp3, include portions outside the peripheral portions PS2 and PS3, the processor 210 deletes the portions outside the peripheral portions PS2 and PS3 (Fig. 6: S355). As a result, since the sizes of the reference images IMt2 and IMt3 are reduced, the burden of the processes using the reference images IMt2 and IMt3 (for example, the processes of S360 - S385 in Fig. 6) is alleviated. For example, the size of the storage area used for storing the reference images IMt2 and IMt3 is reduced. Hereinafter, among the transparent portions P0, P02, and P03 of the target images IMp, IMp2, and IMp3 (Fig. 10, Fig. 15(A), Fig. 15(B)), the portions included in the reference images IMt, IMt2, and IMt3 are referred to as reference transparent portions P0t, P02t, and P03t.

[0086] As described above, in this embodiment, the processor 210 executes the following processes according to the program 231. In S330 of Fig. 6, the processor 210 executes a generation process of generating the reference image IMt using the print image data representing the target image IMp (Fig. 9) (the process of S330 is also referred to as generation process S330). The reference image IMt includes at least a part of the reference transparent portion P0t, which is a portion indicating not using color materials in the target image IMp, and a colored portion P1, which is a portion indicating using color materials in the target image IMp. The generation process S330 includes S340. In S340, the processor 210 sets the color of the transparent portion P0 (including the reference transparent portion P0t) to the medium color Cm, which is the color representing the T-shirt 700 as an example of the medium. In S385, the processor 210 uses a plurality of feature points Ts in the gray-scale read image IMsg (Fig. 10(A)) (that is, a plurality of feature points Ts in the read image IMs) and a plurality of feature points Tt in the gray-scale reference image IMtg (Fig. 10(A)) (that is, a plurality of feature points Tt in the reference image IMt) to execute a feature point matching process. The read image IMs is an image generated by optically reading the target image IMp printed on the T-shirt 700 using the print image data.

[0087] In this way, since the color of the reference transparent portion P0t included in the reference image IMt (FIG. 9) is set to the medium color Cm, the color of the reference transparent portion P0t becomes close to the color of the corresponding portion of the read image IMs. Therefore, the processor 210 can appropriately perform feature point matching by using the reference feature point Tt and the read feature point Ts. The reference feature point Tt is a feature point in the gray reference image IMtg (FIG. 10), that is, a feature point in the reference image IMt. The read feature point Ts is a feature point in the gray read image IMsg, that is, a feature point in the read image IMs. If the color of the reference transparent portion P0t of the reference image IMt is set to a color unrelated to the medium (here, the T-shirt 700), even if the reference feature point Tt and the read feature point Ts near the reference transparent portion P0t indicate the same portion of the same object, the distance between the feature amounts Ft and Fs may increase. As a result, the total number of appropriate pairs of the feature points Tt and Ts formed by feature point matching may decrease. In this embodiment, the possibility of such a problem is reduced.

[0088] Also, in this embodiment, in S335 (FIG. 6), the processor 210 executes a color determination process for determining the medium color Cm by analyzing the read image IMs (the process of S335 is also referred to as the color determination process S335). The processor 210 can reduce the difference between the color of the reference transparent portion P0t of the reference image IMt and the color of the corresponding portion of the read image IMs. Therefore, the processor 210 can appropriately perform feature point matching by using the reference feature point Tt and the read feature point Ts.

[0089] Also, in this embodiment, the color determination process S335 (FIG. 6) includes S910, S915, and S920 in FIG. 8(A). In S910, the processor 210 obtains a color appearance frequency distribution (here, the three-dimensional histogram HST) by analyzing the read image IMs. In S915, the processor 210 determines a representative color Cb by using the color appearance frequency distribution. In S920, the processor 210 determines the medium color Cm as the representative color Cb. According to this configuration, the processor 210 can determine the medium color Cm suitable for the color of the read image IMs.

[0090] Also, in this embodiment, in S230 (FIG. 5), the processor 210 selects the feature point pair MP using color information. That is, the process of matching the reference feature point Tt and the read feature point Ts includes S230. The process of S230 includes S445 - S465 in FIG. 11. In S445 - S465, the processor 210 uses the hue Ht associated with the feature point Tt and the hue Hs associated with the feature point Ts to select candidates M for the feature points Tt and Ts to be retained. In this way, the process of matching the reference feature point Tt and the read feature point Ts includes a process of determining the correspondence between the feature point Ts in the read image IMs and the feature point Tt in the reference image IMt using the hues Ht and Hs. Between the reference image IMt and the read image IMs, the colors of the same part of the same object may deviate. Here, the hue has a lower possibility of deviation compared to brightness and saturation. By using the hues Ht and Hs, the processor 210 can select an appropriate candidate feature point pair M as the feature point pair MP.

[0091] Also, in this embodiment, the generation process S330 (FIG. 6) further includes S350. As described with reference to FIGS. 9 and 15(B), in S350, the processor 210 adds the outer portions Pu and Pu3, which are the outer portions of the color images IMpc and IMpc3 (i.e., the target images IMp and IMp3) and indicate the medium color Cm, to the temporary reference images IM11 and IM31. In this way, the processor 210 adds the outer portions Pu and Pu3 to the reference images IMt and IMt3 (the process of S350 is also referred to as the addition process S350). Therefore, the processor 210 can form an appropriate candidate feature point pair M from the vicinity of the contours of the target images IMp and IMp3.

[0092] In addition, in this embodiment, the reference images IMt, IMt2, IMt3 (FIGS. 9, 15(A), 15(B)) include a colored portion P1 and peripheral portions PS, PS2, PS3 surrounding an outer circumscribing rectangle RA of the colored portion P1. Further, the reference images IMt, IMt2, IMt3 are images that do not include portions outside the peripheral portions PS, PS2, PS3 of the target images IMp, IMp2, IMp3 (note that the target image IMp does not include a portion outside the peripheral portion PS). The generation process S330 (FIG. 6) further includes S345. In S345, the processor 210 determines the peripheral portions PS, PS2, PS3 using the colored portion P1. As described with reference to FIGS. 9 and 15(B), in the additional process S350, the processor 210 adds outer portions Pu, Pu3 located outside the target images IMp, IMp3 among the peripheral portions PS, PS3 to the reference images IMt, IMt3. According to this configuration, the processor 210 can generate an appropriate reference image regardless of the shape and position of the colored portion within the target image.

[0093] Also, in this embodiment, the processor 210 executes the following processes according to the program 231. In the generation process S330 of FIG. 6, the processor 210 executes a generation process of generating the reference images IMt and IMt3 using the print image data representing the target images IMp and IMp3 (FIGS. 9 and 15(B)). The generation process S330 (FIG. 6) further includes an additional process S350. As described with reference to FIGS. 9 and 15(B), in the additional process S350, the processor 210 adds the outer portions Pu and Pu3 indicating the medium color Cm, which are the outer portions of the color images IMpc and IMpc3 (i.e., the target images IMp and IMp3), to the reference images IMt and IMt3. The reference images IMt and IMt3 are used for forming the candidate feature point pairs M (FIG. 10(A)). A plurality of reference feature points Tt are detected from the gray reference image of the reference image IMt3, similar to the plurality of reference feature points Tt on the gray reference image IMtg of the reference image IMt. In S385 (FIG. 6), the processor 210 executes a feature point matching process using the plurality of feature points Ts in the gray read image IMsg (FIG. 10(A)) (i.e., the plurality of feature points Ts in the read image IMs) and the plurality of feature points Tt in the gray reference images of the reference images IMt and IMt3 (i.e., the plurality of feature points Tt in the reference images IMt and IMt3). The read image IMs is an image generated by optically reading the target image IMp printed on the T-shirt 700 using the print image data. As described above, when the target image IMp3 is used, a read image similar to the read image IMs is obtained. As described above, the processor 210 can form appropriate candidate feature point pairs M from the vicinity of the contours of the target images IMp and IMp3. The larger the total number of appropriate candidate feature point pairs M, the larger the total number of appropriate feature point pairs MP. The larger the total number of appropriate feature point pairs MP, the higher the reliability of the process (e.g., alignment) using the plurality of feature point pairs MP.

[0094] Here, as described above, since the processor 210 determines the medium color Cm by analyzing the read image IMs in the color determination process S335 (FIG. 6), the feature point matching can be appropriately executed.

[0095] Also, as described above, since the color determination process S335 (Fig. 6) includes S910, S915, and S920 in Fig. 8(A), the processor 210 can determine the medium color Cm suitable for the color of the read image IMs.

[0096] Also, as described above, in S230 (Fig. 5), the processor 210 selects the feature point pair MP using color information. The process of S230 includes S445 - S465 in Fig. 11. The process of matching the reference feature point Tt and the read feature point Ts includes the process of determining the correspondence between the feature point Ts in the read image IMs and the feature point Tt in the reference image IMt using the hue Ht and Hs. By using the hue Ht and Hs, the processor 210 can select an appropriate candidate feature point pair M as the feature point pair MP.

[0097] B. Second Embodiment: Fig. 16(A) is a part of the flowchart of another embodiment of the process (S335 (Fig. 6)) for determining the medium color Cm. The difference from the embodiment in Fig. 8(A) is that S905b and S910b are executed instead of S910. In this embodiment, the target area analyzed for determining the medium color Cm is specified by the user.

[0098] In S905b, the processor 210 acquires a user instruction for specifying the target area for analysis. In this embodiment, the processor 210 displays the read image on the display unit 240 (Fig. 1). Fig. 16(B) is a diagram showing an example of the read image IMs displayed by the display unit 240. The user (e.g., an operator) can recognize the part representing the T-shirt 700 (especially the blank part without a printed image) in the read image IMs by observing the displayed read image IMs. The user inputs information indicating the target area ARb representing the blank part of the T-shirt 700 to the data processing device 200 by operating the operation unit 250. In this embodiment, the target area ARb is a rectangular area. The user can specify the target area ARb by specifying the positions of two vertices forming the diagonal of the rectangle on the read image IMs. The processor 210 acquires a user instruction for specifying the target area ARb using the information from the operation unit 250.

[0099] In S910b (FIG. 16(A)), the processor 210 obtains the color appearance frequency distribution by analyzing the target region ARb in the captured image IMs. In this embodiment, the processor 210 calculates the frequency of each bin of the three-dimensional histogram HST (FIG. 8(B)) using the color values of a plurality of pixels in the target region ARb. The portions other than the target region ARb in the captured image IMs are not used.

[0100] After S910b, the processor 210 executes S915 and S920 (FIG. 8(A)). As a result, the medium color Cm is determined as the representative color Cb obtained from the target region ARb. Then, the processor 210 ends the process of determining the medium color Cm, that is, S335 in FIG. 6.

[0101] As described above, in this embodiment, the color determination process S335 (FIG. 6) includes S905b (FIG. 16(A)). In S905b, the processor 210 obtains a user instruction for designating the target region ARb to be analyzed in the captured image IMs. As described in S910b, the target of the analysis of the captured image IMs by the color determination process S335 is the target region ARb in the captured image IMs. According to this configuration, the processor 210 can determine an appropriate medium color Cm representing the color of the T-shirt 700. For example, when the target image IMp (FIG. 3(A)) includes a solid region having a large area, the overall analysis of the captured image IMs can determine the representative color Cb (and thus the medium color Cm) representing the color of the solid region, rather than the color of the blank portion of the T-shirt 700. In this embodiment, the possibility of such a problem is reduced.

[0102] C. Third Embodiment: FIG. 17(A) is a part of a flowchart of another embodiment of the process (S335 (FIG. 6)) for determining the medium color Cm. The difference from the embodiment of FIG. 8(A) is that S910c is executed instead of S910. In this embodiment, the target area in the read image is analyzed to determine the medium color Cm. FIG. 17(B) is a diagram showing an example of the target area. In the figure, the read image IMs is shown. The target area ARc is a specific partial area on the side in the direction corresponding to the downward direction of the T-shirt 700 (here, the second direction Dy). In this embodiment, a predetermined area in the read image IMs is used as the target area ARc.

[0103] In S910c, the processor 210 obtains the color appearance frequency distribution by analyzing the target area ARc in the read image. In this embodiment, the processor 210 uses the color values of a plurality of pixels in the target area ARc to calculate the frequency of each bin of the three-dimensional histogram HST (FIG. 8(B)). The portion other than the target area ARc in the read image IMs is not used.

[0104] After S910c, the processor 210 executes S915 and S920 (FIG. 8(A)). Thereby, the medium color Cm is determined as the representative color Cb obtained from the target area ARc. Then, the processor 210 ends the process of determining the medium color Cm, that is, S335 in FIG. 6.

[0105] As described above, in this embodiment, the medium for printing is the T-shirt 700. The portion of the T-shirt 700 where the image is printed may be a part on the upper side of the T-shirt 700 (for example, the portion corresponding to the chest). A part on the lower side of the T-shirt 700 may be a blank portion where no image is printed. The target region ARc for the analysis of the read image IMs (FIG. 17(B)) by the color determination process S335 (FIG. 6) is a specific partial region on the side corresponding to the downward direction (here, the second direction Dy) of the T-shirt 700 in the read image IMs. Such a target region ARc is likely to represent the blank portion of the T-shirt 700. Therefore, the processor 210 can determine an appropriate medium color Cm representing the color of the T-shirt 700. For example, even when the target image IMp (FIG. 3(A)) includes a solid region having a large area, the processor 210 can determine a representative color Cb (and thus the medium color Cm) representing the color of the blank portion of the T-shirt 700.

[0106] D. Modification example: (1) In the embodiment of FIG. 16(A), the method of specifying the target region in the read image may be various other methods instead of the method of specifying the positions of the vertices of the target region ARb. For example, the processor 210 may adopt, as the target region, a region surrounded by a contour freely drawn by the user (such a selection of the region is also called freehand selection). Further, the processor 210 may use a specified point on the read image specified by the user and select, as the target region, a partial region that includes the specified point and shows a color similar to the color of the specified point (the selection of the region using the specified point is also called the magic wand tool).

[0107] (2) In the embodiments of FIGS. 17(A) and 17(B), the target region ARc in the read image may be various partial regions on the side corresponding to the downward direction (here, the second direction Dy) of the T-shirt 700 in the read image IMs. Hereinafter, the direction corresponding to the downward direction of the T-shirt 700 in the read image IMs is also called the downward direction of the medium. The target region ARc may be, for example, a partial region included in a region IMsL corresponding to half of the length on the downward direction Dy side of the read image IMs.

[0108] (3) Instead of the methods of FIGS. 8(A), 8(B), 16(A), 16(B), 17(A), and 17(B), the method for determining the medium color Cm may be various methods of determining the medium color by analyzing the read image. For example, a reference color may be associated in advance with each of a plurality of bins of the three-dimensional histogram HST (e.g., the average color of the range of the color values of the bin). Then, the processor 210 may adopt the reference color associated with the bin having the highest frequency as the representative color Cb.

[0109] Also, the processor 210 may use a trained machine learning model (e.g., a model for performing region segmentation called instance segmentation (such as Mask R-CNN)) to divide the region representing the medium (here, the T-shirt 700) in the read image into a region representing the printed image and other regions (referred to as blank regions). Then, the processor 210 may determine the medium color Cm by analyzing the blank regions. Here, the processor 210 may calculate a summary statistic (e.g., median, mode, average value, etc.) representing the color values of a plurality of pixels included in the blank regions as the medium color Cm.

[0110] Also, the medium color Cm may be determined without using the read image. For example, data representing the correspondence between the identification information of the medium (e.g., the model number of the medium) and the medium color Cm may be stored in advance in the storage device 215 (e.g., the non-volatile storage device 230). The processor 210 may adopt the medium color Cm associated with the identification information of the medium input by the user. Also, the processor 210 may adopt the color (e.g., the RGB gradation value) specified by the user as the medium color Cm.

[0111] (4) Instead of the method of using the opacity k of the alpha channel, the method for specifying the transparent portion indicating that no colorant is used in the target image may be any other method. For example, the portion used as the transparent portion in the target image may be a portion having a specific color (e.g., white).

[0112] (5) The generation process of generating a reference image using the target image may be various other processes instead of the generation process S330 in FIG. 6. For example, S350 may be executed after S355. The process of deleting the outer part of the surrounding part from the reference image (for example, S355) may be omitted. The process of setting the color of the reference transparent part to the medium color, which is the color representing the medium (for example, S340), may be omitted. The processor 210 may set the color of the transparent part to a predetermined color (for example, white). The additional process of adding the outer part indicating the medium color, which is the outer part of the target image, to the reference image (for example, S350) may be omitted. The process of adding the surrounding part surrounding the circumscribed rectangle circumscribing the colored part to the reference image (for example, S345, S350) may be omitted. The processor 210 may generate a reference image including the target image and the surrounding part surrounding the target image. In any case, the target image may be an image that does not include a transparent part. Also in this case, when the reference image includes the surrounding part surrounding the target image, the processor 210 can form an appropriate pair from the vicinity of the edge of the reference image.

[0113] (6) In S425, S435 (FIG. 11), the processor 210 may calculate the first hue Ht using the target image instead of the reference image. When the target image includes a transparent part, the processor 210 may calculate the first hue Ht using the remaining part of the target image excluding the transparent part. Note that the process of selecting the candidate feature point pair M using color information (for example, S230 (FIG. 5)) may be omitted.

[0114] (7) The method for detecting the feature points Tt and Ts (i.e., the keypoints) may be various methods for detecting points indicating the object portions in the image, instead of the methods described in S365 and S370 (FIG. 6). The detection method may be, for example, selected in advance from searches for extrema (maxima and minima) using DoG (Difference-of-Gaussian), Harris corner detection, FAST (Features from Accelerated Segment Test) corner detection, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc.

[0115] The method for calculating the feature quantities Ft and Fs (i.e., the feature descriptors) may be various methods for calculating information describing the features of the keypoints, instead of the methods described in S375 and S380 (FIG. 6). The algorithm for calculating the feature descriptor may be, for example, selected in advance from BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, A-KAZE, etc. Also, the method for calculating the distance between the feature quantities Ft and Fs may be various methods suitable for the data configurations of the feature quantities Ft and Fs. When the feature quantities Ft and Fs are represented by binary vectors, the distance may be the Hamming distance. Instead of the Hamming distance, the distance may be various other distances (e.g., norms such as the L1 norm, L2 norm (also called the Euclidean distance), etc.). The norm is applicable to various feature descriptors.

[0116] (8) The feature point matching process executed according to program 231 includes, in addition to the process of forming pairs of the read feature points Ts in the read image IMs and the reference feature points Tt in the reference image IMt (for example, S385 (FIG. 6)), the process of detecting a plurality of reference feature points Tt from the reference image IMt (for example, S365), the process of detecting a plurality of read feature points Ts from the read image IMs (for example, S370), the process of calculating the feature amount Ft of each of the plurality of reference feature points Tt (for example, S375), and the process of calculating the feature amount Fs of each of the plurality of read feature points Ts (for example, S380). The generation process of generating a reference image using the image data for printing representing the target image and the feature point matching process using a plurality of feature points in the read image and a plurality of feature points in the reference image may be executed according to other programs different from program 231. For example, S250 (FIG. 5) may be executed according to another program. S130, S140 (FIG. 4) may be executed according to another program. The processes other than the generation process and the feature point matching process may be performed by other devices different from the data processing device 200. Note that the generation process may include one or both of the process of setting the color of the reference transparent portion to the medium color which is the color representing the medium and the additional process of adding an outer portion which is the outer portion of the target image and indicates the medium color which is the color representing the medium to the reference image. In any case, the image to be processed may be a grayscale image instead of a color image.

[0117] (9) The printing medium is not limited to the T-shirt 700 and may be various types of clothing (for example, various shirts such as polo shirts, outerwear, slacks, etc.). The printing medium may be various types of fabrics such as clothing and bags. The printing medium is not limited to fabrics and may be various media such as paper, film, leather, etc. Also, the printing device 900 may be a printing device of another method (for example, laser type) instead of an inkjet type printing device.

[0118] (10) The result of the feature point matching may be used not only for determining the correspondence of coordinates but also for various processes. For example, when a printed image has a defect, the total number of feature point pairs MP formed by the feature point matching may be reduced. The total number of feature point pairs MP may be used for evaluating the quality of the printed image (the higher the total number of feature point pairs MP, the higher the quality).

[0119] (11) The correspondence of coordinates (FIG. 13) may represent various conversions such as homography conversion instead of affine conversion. Also, the correspondence of coordinates may be represented in various forms such as a lookup table instead of the matrix Mtx. Also, the correspondence of coordinates may be used not only for inspection but also for various processes. For example, when one image is generated by synthesizing a target image and a read image, the correspondence of coordinates may be used for aligning the target image and the read image.

[0120] (12) The data processing device that performs the generation process and the feature point matching process is not limited to a personal computer (for example, the data processing device 200 (FIG. 1)) but may be various other devices (for example, a smartphone, a tablet computer, a control device incorporated in a reading device, etc.). Also, a plurality of devices (for example, computers) that can communicate with each other via a network may share the functions of data processing by the data processing device in part and provide the functions of data processing as a whole (a system including these devices corresponds to the data processing device).

[0121] In each of the above embodiments, a part of the configuration realized by hardware may be replaced with software, or conversely, a part or all of the configuration realized by software may be replaced with hardware. For example, the process of S385 in FIG. 6 may be executed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0122] In addition, when some or all of the functions of the present disclosure are implemented by a computer program, the program can be provided in a form stored in a computer-readable recording medium (for example, a non-transitory recording medium). The program can be used while being stored in the same or a different recording medium (computer-readable recording medium) as when it is provided. The "computer-readable recording medium" includes not only portable recording media such as memory cards and CD-ROMs, but also internal storage devices in a computer such as various ROMs, and external storage devices connected to a computer such as hard disk drives.

[0123] The above-described embodiments and modifications can be combined as appropriate. Also, the above-described embodiments and modifications are for facilitating the understanding of the present disclosure and do not limit the present invention. The present invention can be changed and improved without departing from its gist, and equivalents thereof are included in the present invention.

Description of Reference Numerals

[0124] 100... Reading device, 110... Control device, 120... Conveying device, 122... Position sensor, 130... Table, 140... Support portion, 180... Reading sensor, 190... Housing, 200... Data processing device, 210... Processor, 215... Storage device, 220... Volatile memory device, 230... Non-volatile memory device, 231... Program, 240... Display portion, 250... Operation portion, 270... Communication interface, 700... T-shirt, 900... Printing device, ARb, ARc... Target area, Cb... Representative color, Cm... Medium color, Hs, Ht... Hue, IMp, IMp2, IMp3... Target image, IMt, IMt2, IMt3... Reference image, IMs, IMs2... Read image, P0, P02, P03... Transparent portion, P0t... Reference transparent portion, P1... Colored portion, PS, PS2, PS3... Surrounding portion, Pu, Pu3... Outer portion, RA... Outer circumscribed rectangle, S330... Generation process, S335... Color determination process, S350... Addition process, Ts, Tt... Feature point

Claims

1. A program comprising: a function of executing a generation process for generating a reference image using print image data representing a target image, the reference image including at least a part of a reference transparent part which is a part of the target image indicating non-use of a coloring material, i.e., a transparent part, and a colored part which is a part of the target image indicating use of a coloring material, and the generation process including a process of setting the color of the reference transparent part to a medium color which is a color representing a medium; and a function of executing a feature point matching process using a plurality of feature points in a read image generated by optically reading the target image printed on the medium using the print image data and a plurality of feature points in the reference image; The program causes a computer to implement the above functions.

2. The program according to claim 1, further causing a computer to implement: a function of executing a color determination process for determining the medium color by analyzing the read image.

3. The program according to claim 2, wherein the color determination process includes: a process of obtaining a color appearance frequency distribution by analyzing the read image; a process of determining a representative color using the color appearance frequency distribution; and a process of determining the medium color as the representative color. The program includes the above processes.

4. The program according to claim 2 or 3, wherein the color determination process includes a process of obtaining a user instruction for designating an analysis target area in the read image, and the object of analysis of the read image by the color determination process is the target area in the read image. The program is as described above.

5. The program according to claim 2 or 3, wherein the medium is a T-shirt, and the object of analysis of the read image by the color determination process is a specific partial area on the side corresponding to the downward direction of the T-shirt in the read image. The program is as described above.

6. The program according to any one of claims 1 to 3, wherein the feature point matching process includes a process of determining a correspondence relationship between the feature points in the read image and the feature points in the reference image using a hue. The program is as described above.

7. The program according to any one of claims 1 to 3, wherein the generation process further includes an additional process of adding an outer part outside the target image and indicating the medium color to the reference image. The program is as described above.

8. The program according to claim 7, wherein the reference image includes the colored portion and a peripheral portion surrounding an outer circumscribing rectangle of the colored portion, and does not include a portion outside the peripheral portion in the target image; the generation process further includes a process of determining the peripheral portion using the colored portion; the addition process adds the outer portion located outside the target image in the peripheral portion to the reference image; A program comprising: **Claim 9** A program, comprising: a function of executing a generation process of generating a reference image using image data for printing representing a target image, the generation process including an addition process of adding an outer portion indicating a medium color, which is a color representing a medium, outside the target image, to the reference image; a function of executing a feature point matching process using a plurality of feature points in a read image generated by optically reading the target image printed on the medium using the image data for printing and a plurality of feature points in the reference image; A program for causing a computer to implement the functions. **Claim 10** The program according to claim 9, further comprising: a function of causing a computer to implement a color determination process of determining the medium color by analyzing the read image. **Claim 11** The program according to claim 10, wherein the color determination process includes: a process of obtaining a frequency distribution of color appearance by analyzing the read image; a process of determining a representative color using the frequency distribution of color appearance; a process of determining the medium color as the representative color; A program comprising: **Claim 12** The program according to claim 10 or 11, wherein the color determination process includes a process of obtaining a user instruction for designating an analysis target area in the read image; the object of analysis of the read image by the color determination process is the target area in the read image. A program. **Claim 13** The program according to claim 10 or 11, wherein the medium is a T-shirt; the object of analysis of the read image by the color determination process is a specific partial area on the side corresponding to the downward direction of the T-shirt in the read image. A program. **Claim 14** The program according to any one of claims 9 to 11, wherein The process of the feature point matching includes a process of determining the correspondence between the feature points in the read image and the feature points in the reference image using hue. Program. **Claim 15** A data processing apparatus, a generation unit that executes a generation process for generating a reference image using print image data representing a target image, wherein the reference image includes at least a part of a reference transparent part that is a part of the target image indicating non-use of color materials and a colored part that is a part of the target image indicating use of color materials, and the generation process includes a process of setting the color of the reference transparent part to a medium color that is a color representing the medium; the generation unit; a matching unit that executes a feature point matching process using a plurality of feature points in a read image generated by optically reading the target image printed on the medium using the print image data and a plurality of feature points in the reference image; A data processing apparatus comprising the above. **Claim 16** A data processing apparatus, a generation unit that executes a generation process for generating a reference image using print image data representing a target image, wherein the generation process includes an addition process of adding an outer part that is an outer part of the target image and indicates a medium color that is a color representing the medium to the reference image; the generation unit; a matching unit that executes a feature point matching process using a plurality of feature points in a read image generated by optically reading the target image printed on the medium using the print image data and a plurality of feature points in the reference image; A data processing apparatus comprising the above.

Images