Program and data processing device

Abstract

The present invention performs feature point matching between a plurality of images.　In the present invention, a generation process is performed in which a reference image is generated using image data for printing that represents a target image. A feature point matching process is performed using a plurality of feature points in a read image and a plurality of feature points in the reference image. The read image is generated by optically reading the target image, which is printed on a medium using the image data for printing. Here, the reference image may include a reference transparent portion indicating that no color material is used or that there is no color material, and the generation process may include a process for setting the color of the reference transparent portion to a medium color, which is a color representing the medium. The generation process may also include an addition process for adding, to the reference image, an external portion that is a portion outside the target image and that indicates a medium color, which is a color representing the medium.

Description

Program and data processing device

[0001] This specification relates to feature point matching between multiple images.

[0002] Feature point matching between multiple images can be performed in various processes. Patent Document 1 discloses a technique for inspecting a printed matter in which an image is printed on a recording medium. This technique compares an image input to a printing device (also called a reference image) with an image (also called a printed image) obtained by reading the printed matter output from the printing device with a reading device such as a scanner. Edge information such as lines and characters is extracted from the image as feature points, and the reference image and the printed image are aligned so that the correlation between the positions of the feature points in the reference image and the printed image is maximized.

[0003] JP 2013-101015 A

[0004] Matching feature points between multiple images is not easy and requires some ingenuity.

[0005] This specification discloses a technique for matching feature points between multiple images.

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] A program that causes a computer to realize the following functions: a function for executing a generation process to generate a reference image using image data for printing that represents a target image, the reference image including a reference transparent portion that is at least a part of a transparent portion of the target image that indicates that a colorant is not used, and a colored portion of the target image that indicates that a colorant is used, the generation process including a process for setting the color of the reference transparent portion to a medium color that represents a medium; and a function for executing a feature point matching process using a plurality of feature points in a read image that is generated by optically reading the target image that is printed on the medium using the image data for printing, and a plurality of feature points in the reference image.

[0008] According to this configuration, the color of the reference transparent portion of the reference image is set to the medium color, which is the color representing the medium, thereby reducing the color difference between the reference transparent portion of the reference image and the corresponding portion of the scanned image. Therefore, it is possible to appropriately perform feature point matching using multiple feature points in the scanned image and multiple feature points in the reference image.

[0009] [Application Example 2] A program that causes a computer to realize a function of executing a generation process to generate a reference image using image data for printing that represents a target image, the generation process including an additional process of adding an outer portion of the target image that indicates a medium color, which is a color that represents the medium, to the reference image, and a function of executing a feature point matching process using a plurality of feature points in a read image that is generated by optically reading the target image that is printed on the medium using the image data for printing, and a plurality of feature points in the reference image.

[0010] With this configuration, an outer portion of the target image that indicates the medium color, which is the color representing the medium, is added to the reference image, so that feature points located on the edge of the target image, in addition to feature points located inside the target image, can be used for feature point matching. Therefore, feature point matching can be performed appropriately using multiple feature points in the scanned image and multiple feature points in the reference image.

[0011] The technology disclosed in this specification can be realized in various forms, such as a data processing method and a data processing device, a computer program for realizing the functions of the method or device, a recording medium (e.g., a non-temporary recording medium) on which the computer program is recorded, and the like.

[0012] 1 is an explanatory diagram showing a data processing device as an embodiment; FIG. 1 is a perspective view showing an example of a reading device 100; FIG. 1A is a diagram showing an example of an image represented by image data for printing; FIG. 1B is a diagram showing an example of a read image; FIG. 1C is a flowchart showing an example of an inspection process; FIG. 1D is a flowchart showing an example of a registration process; FIG. 1E is a flowchart showing an example of a candidate pair formation process; FIG. 1F is a diagram showing an example of a circumscribing rectangle; FIG. 1A is a flowchart showing an example of a process for determining a medium color; FIG. 1B is a schematic diagram of a three-dimensional histogram HST; FIG. 1D is a diagram showing an example of an image to be processed; FIG. 1C is a diagram showing an example of an image to be processed; FIG. 1D is a flowchart showing an example of a selection process; FIG. 1D is a diagram showing an example of a calculation of a representative color value; FIG. 1E is a diagram showing an example of an expression format for a coordinate correspondence relationship; FIG. 1D is a diagram showing an example of a difference image; FIG. 1D is a diagram showing another example of an image to be processed; FIG. 1E is a part of a flowchart of another embodiment of a process for determining a medium color Cm. 10B is a diagram showing an example of a scanned image IMs displayed by the display unit 240. FIG. 10A is a part of a flowchart of another embodiment of a process for determining a medium color Cm. FIG. 10B is a diagram showing an example of a target area.

[0013] A. First Embodiment: A1. Device Configuration: FIG. 1 is an explanatory diagram showing a data processing device as one embodiment. The data processing device 200 is, for example, a personal computer. The data processing device 200 executes an inspection process for a printed image. The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0014] The processor 210 is a device configured to perform data processing, and may be, for example, a central processing unit (CPU) or a system on a chip (SoC). The volatile storage device 220 may be, for example, a dynamic random access memory (DRAM), and the non-volatile storage device 230 may be, for example, a flash memory. The non-volatile storage device 230 stores data of a program 231.

[0015] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel overlaid on the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250.

[0016] The communication interface 270 is an interface for communicating with other devices (for example, it includes one or more of a USB interface, a wired LAN interface, an IEEE802.11 wireless interface, and an industrial camera interface (for example, CameraLink, CoaXPress, etc.)). In this embodiment, the communication interface 270 connects the reading device 100 and the printing device 900. The printing device 900 is a so-called inkjet printing device that prints an image on a printing medium such as cloth or paper by ejecting ink onto the printing medium. The reading device 100 optically reads the object to be read and generates read image data representing the object. In the following description, the printing medium is a T-shirt, and the reading device 100 reads the T-shirt on which an image is printed.

[0017] 2 is a perspective view showing an example of the reading device 100. In the figure, the first direction Da and the second direction Db indicate horizontal directions, and the third direction Dc indicates a vertically upward direction. The first direction Da and the second direction Db are perpendicular to each other.

[0018] In this embodiment, the reading device 100 includes a housing 190, a table 130, a support portion 140 fixed to the upper surface of the table 130, a conveying device 120, a reading sensor 180, and a control device 110. The control device 110, the conveying device 120, and the reading sensor 180 are fixed to the housing 190.

[0019] The support unit 140 is a plate-like member that forms a flat upper surface for supporting an object to be read (such a member is also called a platen). In the drawing, a T-shirt 700 having a printed image IMpp thereon is placed on the support unit 140.

[0020] The conveying device 120 is configured to convey the table 130 in a direction parallel to the second direction Db. The conveying device 120 may have various configurations. Although not shown, in this embodiment, the conveying device 120 includes a rail that supports the table 130 so that the table 130 is slidable in a direction parallel to the second direction Db, multiple pulleys, a belt that is wound around the multiple pulleys and has a portion fixed to the table 130, and an electric motor that rotates the pulley. As the electric motor rotates the pulley, the table 130 (and thus the support portion 140) moves in a direction parallel to the second direction Db. The conveying device 120 also includes a position sensor 122 (e.g., a rotary encoder) that detects the position of the table 130 on the conveying path.

[0021] The reading sensor 180 is disposed at a position higher than the support unit 140 and midway along the transport path PTh of the support unit 140. The reading sensor 180 includes a line sensor (for example, a contact image sensor (CIS) or a charge coupled device (CCD)) that is composed of multiple photoelectric conversion elements arranged in a direction intersecting the transport direction Db (in this embodiment, a direction Da perpendicular to the transport direction Db). The reading sensor 180 faces downward. The reading sensor 180 can read the portion of the object supported by the support unit 140 that is located below the reading sensor 180.

[0022] When reading T-shirt 700, reading device 100 transports table 130 in a direction parallel to second direction Db. Reading sensor 180 repeatedly reads T-shirt 700 during transportation. This allows reading sensor 180 to read almost the entire portion of T-shirt 700 that is supported by support portion 140.

[0023] The control device 110 is an electric circuit configured to control the conveying device 120 and the reading sensor 180. The control device 110 is configured using, for example, a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). The control device 110 generates read image data by controlling the conveying device 120 and the reading sensor 180.

[0024] A2. Printing Process: In this embodiment, an image is printed on a T-shirt 700 (FIG. 2). The printing of the image is performed, for example, as part of a T-shirt sales service. The T-shirt sales service may include on-demand printing. A customer places an order for on-demand printing with a service provider. In response to the customer's order, the service provider prints the image on the T-shirt 700 using image data provided by the customer.

[0025] FIG. 3A is a diagram showing an example of an image represented by image data for printing (referred to as target image IMp). In this embodiment, the data of target image IMp is bitmap data representing the color values ​​(here, the gradation values ​​(e.g., values ​​greater than or equal to zero and less than or equal to 255) of red R, green G, and blue B) of multiple pixels arranged in a matrix along a first direction Dx and a second direction Dy. In the example of FIG. 3A, target image IMp represents a background BG and four rectangular objects OB1-OB4. Objects OB1-OB4 are represented in different colors. For example, the color of the first object OB1 is red, the color of the second object OB2 is green, the color of the third object OB3 is blue, and the color of the fourth object OB4 is gray. In this way, target image IMp may include multiple objects having different hues.

[0026] In this embodiment, the data of the target image IMp further includes an alpha channel that represents the opacity of each of the multiple pixels. In the background BG, the opacity is set to a value that indicates transparency. In this embodiment, the value that indicates transparency is used as a value that indicates that ink is not used. Hereinafter, the portions of the image that indicate that ink is not used will also be referred to as transparent portions. The portions of the image that indicate that ink is used will also be referred to as colored portions. The background BG is an example of a transparent portion (referred to as transparent portion P0). The objects OB1-OB4 are examples of colored portions (referred to as colored portion P1).

[0027] Although not shown in the drawings, the target image IMp is printed using the data processing device 200 and the printing device 900. Alternatively, the target image IMp may be printed using another device.

[0028] Various errors may occur during printing of the target image IMp. The printed image may have various defects due to errors. For example, a portion of the printed image may be missing due to an ink ejection error. The data processing device 200 detects defects in the printed image through an inspection process described below. In this embodiment, the data of the target image IMp is used in the inspection process. After printing the target image IMp, the data of the target image IMp is stored in the storage device 215 (e.g., the non-volatile storage device 230) of the data processing device 200 for the inspection process.

[0029] A3. Inspection Process: FIG. 4 is a flowchart illustrating an example of the inspection process. For inspection, the T-shirt 700 is placed on the support unit 140 of the reading device 100 (FIG. 2) so that the printed image IMpp is visible. In this embodiment, an operator places the T-shirt 700 on the support unit 140. Alternatively, a machine (e.g., a robot arm) may place the T-shirt 700 on the support unit 140. After the T-shirt 700 is placed, an instruction to start the inspection process is input to the data processing device 200 (FIG. 1). In this embodiment, the operator inputs the instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the start instruction. Note that the start instruction may be input to the data processing device 200 via the communication interface 270 by a device other than the data processing device 200.

[0030] The processor 210 of the data processing device 200 executes the inspection process in accordance with the program 231. In S110, the T-shirt 700 is photographed. The processor 210 supplies a reading instruction to the reading device 100. The control device 110 of the reading device 100 reads the T-shirt 700 by controlling the reading sensor 180 and the conveying device 120 in response to the reading instruction. The control device 110 generates read image data representing the read T-shirt 700. The processor 210 of the data processing device 200 acquires the read image data from the control device 110 of the reading device 100 and stores the acquired read image data in the storage device 215 (e.g., the non-volatile storage device 230).

[0031] FIG. 3B is a diagram showing an example of a scanned image. In this embodiment, the data of the scanned image IMs is bitmap data representing the color values ​​(here, the gradation values ​​(e.g., values ​​greater than or equal to zero and less than or equal to 255) of red R, green G, and blue B) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. The scanned image IMs in the figure represents a portion of the T-shirt 700 that includes the printed image IMpp. Here, the printed image IMpp is assumed to be an image obtained by printing the target image IMp (FIG. 3A). The printed image IMpp, like the target image IMp, represents objects OB1-OB4.

[0032] In S120 (FIG. 4), the processor 210 aligns the reference image with the scanned image. In this embodiment, the reference image is an image obtained by processing the target image IMp (FIG. 3A) (details will be described later).

[0033] 5 is a flowchart illustrating an example of the alignment process. In this example, the processor 210 obtains a plurality of pairs of feature points by feature point matching, and determines the correspondence between the coordinates on the reference image and the coordinates on the scanned image IMs using the obtained plurality of pairs.

[0034] In S220, the processor 210 forms candidate pairs that are candidates for feature points pairs. FIG. 6 is a flowchart showing an example of the candidate pair formation process. In S310, the processor 210 reads data of the target image IMp ( FIG. 3A ) used for printing from the storage device 215. As described above, the data of the target image IMp has been stored in advance in the storage device 215 (e.g., the non-volatile storage device 230).

[0035] In S315 (FIG. 6), the processor 210 determines a circumscribing rectangle that circumscribes the colored portion P1 of the target image IMp (FIG. 3(A)). FIG. 7 is a diagram showing an example of a circumscribing rectangle. The diagram shows the target image IMp and a circumscribing rectangle RA. The circumscribing rectangle RA is a rectangle having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy. The circumscribing rectangle RA is the smallest rectangle that contains the colored portion P1 (here, objects OB1-OB4). In the target image IMp of FIG. 7, each of the objects OB1-OB4 forms a part of the edge of the target image IMp. The circumscribing outline LRA, which is the outline of the circumscribing rectangle RA, coincides with the target outline LIMp, which is the outline of the target image IMp.

[0036] Although not shown in the figures, the circumscribing rectangle RA varies depending on the position and shape of the colored portion P1 in the target image IMp. For example, if the colored portion P1 is located inside the target outline LIMp, the circumscribing rectangle RA is formed inside the target outline LIMp.

[0037] In S320 ( FIG. 6 ), the processor 210 reads the data of the scanned image IMs from the storage device 215 .

[0038] In S330, the processor 210 generates data for the reference image using data for the target image IMp. As described below, feature points are detected from the reference image. Points representing characteristic portions of an object, such as corners or edges, are detected as feature points. For example, point Pw representing the upper right corner of the first object OB1 ( FIG. 7 ) is suitable as a feature point. To detect feature points, color values ​​of multiple pixels in the area surrounding the candidate point (i.e., the partial area including the candidate point) are used. If feature points are detected from the target image IMp, point Pw is located at the edge of the target image IMp, so the upper portion of the partial area Aw including point Pw is outside the target outline LIMp. Since it is not possible to reference the color values ​​of pixels outside the target image IMp, it is difficult to detect point Pw as a feature point. Thus, when an object is located near the edge of an image, it is difficult to detect feature points near the edge of the image. In this embodiment, the processor 210 generates a reference image having a peripheral portion outside the circumscribing rectangle RA so that feature points can be detected near the edges of the target image IMp even if the object is located near the edges of the target image IMp.

[0039] In this embodiment, the process of S330 (FIG. 6) includes S335-S355. In S335, the processor 210 determines the medium color by analyzing the scanned image IMs. FIG. 8A is a flowchart showing an example of the process for determining the medium color. In S910, the processor 210 analyzes the scanned image IMs to obtain a color appearance frequency distribution. In this embodiment, the processor 210 generates an RGB three-dimensional histogram HST.

[0040] FIG. 8B is a schematic diagram of a three-dimensional histogram HST. The figure shows a three-dimensional color solid CC represented by three gradation values: red (R), green (G), and blue (B). The vertices of the color solid CC are labeled with symbols indicating the colors (here, red (R), green (G), blue (B), cyan (Cy), magenta (Mg), yellow (Yl), and white (Wh)). In this embodiment, the ranges of the gradation values ​​of each of red (R), green (G), and blue (B) are divided into J bins (J is an integer equal to or greater than 2, e.g., J=9). Each color value represented by an RGB gradation value is associated with one of the J*J*J bins. The processor 210 calculates the frequency of each bin using multiple color values ​​of multiple pixels in the scanned image IMs. This allows the processor 210 to generate a three-dimensional histogram HST, i.e., a color frequency distribution.

[0041] In S915 (FIG. 8A), the processor 210 determines the representative color Cb using the three-dimensional histogram HST. In this embodiment, the processor 210 selects the bin with the highest frequency from among the multiple bins in the three-dimensional histogram HST. The processor 210 calculates the average values ​​of the RGB of the multiple color values ​​associated with the bin with the highest frequency in S910 as the gradation values ​​of the RGB of the representative color Cb. Note that various summary statistics (e.g., median, mode, etc.) representing the magnitude of the color value may be used instead of the average value.

[0042] In S920, the processor 210 determines the medium color Cm as the representative color Cb. The processor 210 then terminates the process of FIG. 8A, i.e., S335 of FIG. 6. As will be described later, the medium color Cm is used as the color of the print medium (here, the T-shirt 700). In this embodiment, in the scanned image IMs (FIG. 3B), the area of ​​the blank portion of the T-shirt 700 where no printed image is present is assumed to be larger than the area of ​​the portion representing the object. In this case, the representative color Cb is a good approximation of the color of the T-shirt 700 in the scanned image IMs.

[0043] In S340 (FIG. 6), the processor 210 generates temporary reference image data by setting the color of the transparent portion P0 of the target image IMp (FIG. 7) to the medium color Cm. FIG. 9 is a diagram showing an example of an image to be processed. The target image IMp is shown in the lower left portion of FIG. 9. In this embodiment, the data of the target image IMp is RGB bitmap data with an alpha channel. The processor 210 separates from the target image IMp a color image IMpc representing each RGB tone value and a mask image IMpm representing the opacity k of the alpha channel. Here, the opacity k is set to zero in the background BG region and 1 in the objects OB1-OB4 region. The opacity k ranges from zero to 1, with K=0 indicating transparency and K=1 indicating opacity. The processor 210 temporarily sets RGB=255 in the portion of the color image IMpc where k=0 (i.e., the transparent portion P0).

[0044] Note that the opacity k may indicate semi-transparency (here, 0<k<1) in at least a portion of the target image IMp. In this case, the processor 210 may calculate the gradation value Vo of each of the RGB of the color image IMpc using the opacity k and the original gradation value Vi. For example, the following calculation formula may be used: Vo=(k*Vi)+((1-k)*255)

[0045] The processor 210 generates data for a temporary reference image IM11 by setting the color value of the transparent portion P0 of the color image IMpc to the medium color Cm.

[0046] In S345 (FIG. 6), the processor 210 determines a peripheral portion surrounding the circumscribing rectangle RA. An example of the peripheral portion PS is shown above the target image IMp in FIG. 9. In this embodiment, the processor 210 determines an area of ​​a predetermined width wt surrounding the circumscribing rectangle RA as the peripheral portion PS.

[0047] In S350 ( FIG. 6 ), the processor 210 generates new temporary reference image data by adding the outer portion of the surrounding portion PS, which is located outside the target image IMp, to the temporary reference image IM11. The upper right portion of FIG. 9 shows the temporary reference image IM12 to which the outer portion Pu has been added. In the example of FIG. 9 , the entire surrounding portion PS is located outside the target image IMp. Therefore, the entire surrounding portion PS corresponds to the outer portion Pu. The processor 210 sets the color value of the outer portion Pu (here, the same as the surrounding portion PS) to the medium color Cm. In the example of FIG. 9 , the circumscribing contour LRA coincides with the contour of the temporary reference image IM11. Therefore, the surrounding portion PS surrounds the temporary reference image IM11.

[0048] In S355, the processor 210 generates new temporary reference image data by deleting the portion outside the surrounding portion PS from the temporary reference image IM12. In the example of FIG. 9 , the temporary reference image IM12 does not have a portion outside the surrounding portion PS, so the temporary reference image processed in S355 is the same as the temporary reference image IM12. Although not shown, if the circumscribing rectangle RA is formed inside the target contour LIMp, the surrounding portion PS may be positioned inside the contour of the temporary reference image IM11. In this case, the portion of the temporary reference image IM12 outside the surrounding portion PS is deleted.

[0049] Reference image data is generated through the above steps S335 to S355. In the example of Fig. 9, a temporary reference image IM12 is used as the reference image. Hereinafter, the temporary reference image IM12 will also be referred to as a reference image IMt.

[0050] In S360, the processor 210 generates a gray reference image IMtg and a gray scanned image IMsg by grayscale conversion of the reference image IMt and the scanned image IMs. A known relationship can be used as the correspondence between the color gradation values ​​and the grayscale gradation values ​​(for example, the correspondence between the RGB values ​​in the RGB color space and the luminance value Y in the YCbCr color space).

[0051] In S365, the processor 210 extracts feature points Tt from the gray reference image IMtg. FIGS. 10A-10C are diagrams showing examples of images to be processed. FIG. 10A shows examples of feature points detected from images IMtg and IMsg. Each of the multiple black dots on the gray reference image IMtg represents a feature point Tt (also referred to as a reference feature point Tt) detected from the gray reference image IMtg. As shown in the figure, points representing characteristic portions of an object, such as corners or edges, are detected as feature points Tt. Such feature points Tt are also called key points. Because the surrounding portion PS is added, reference feature points Tt can also be detected near the target contour LIMp. For example, reference feature point Tt1 corresponds to point Pw in FIG. 7. Although not shown, in reality, many more feature points Tt can be detected (e.g., several tens or several hundreds). The reference feature points Tt detected from the gray reference image IMtg correspond to feature points that indicate the same parts at the same coordinates on the reference image IMt (FIG. 9).

[0052] Various methods may be used to detect feature points. In this embodiment, a technique called Accelerated-KAZE (A-KAZE) is used, which detects key points and calculates feature descriptors for each key point. The A-KAZE technique is disclosed, for example, in the following paper: "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. Pablo F. Alcantarilla, J. Nuevo and Adrien Bartoli. In British Machine Vision Conference (BMVC), Bristol, UK, September 2013"

[0053] The processor 210 detects a plurality of feature points Tt by analyzing the gray reference image IMtg according to the A-KAZE technique.

[0054] In S370 (FIG. 6), the processor 210 extracts a plurality of feature points Ts from the gray scanned image IMsg. The plurality of black dots on the gray scanned image IMsg in FIG. 10A each represent a feature point Ts detected from the gray scanned image IMsg (also referred to as a read feature point Ts). The processor 210 detects a plurality of feature points Ts by analyzing the gray scanned image IMsg according to the A-KAZE technique. Although not shown, in practice, many more feature points Ts may be detected (for example, several tens or several hundreds). The read feature points Ts detected from the gray scanned image IMsg correspond to feature points indicating the same portions at the same coordinates on the scanned image IMs (FIG. 3B).

[0055] In S375, the processor 210 calculates a feature value Ft for each of the multiple reference feature points Tt. The feature value Ft may be various information describing the features of the feature point Tt. For example, the feature value Ft is calculated so that it changes according to the distribution of color values ​​of multiple pixels surrounding the feature point Tt. In this embodiment, the processor 210 uses the gray reference image IMtg to calculate the A-KAZE feature descriptor as the feature value Ft.

[0056] In S380, the processor 210 calculates a feature value Fs for each of the plurality of read feature points Ts. In this embodiment, the processor 210 calculates the feature descriptor (i.e., the feature value Fs) using the gray read image IMsg according to the A-KAZE technique.

[0057] In S385, the processor 210 performs matching between a plurality of reference feature points Tt and a plurality of read feature points Ts. Through matching, feature points Tt and Ts that indicate similar portions of the images IMtg and IMsg (for example, the same portion of the same object) are matched. The matched feature points Tt and Ts are also called a matching pair. As will be described later, a more appropriate pair is selected from the plurality of pairs of feature points Tt and Ts formed in S385. Hereinafter, the pair of feature points Tt and Ts finally obtained is called a feature point pair. In S385, the processor 210 acquires candidate feature point pairs (also called candidate feature point pairs).

[0058] 10(B) shows an example of a candidate feature point pair. Multiple lines RL in the figure each represent a candidate feature point pair M. Each line RL connects feature points Tt and Ts that form the candidate feature point pair M. As shown in the figure, a pair of feature points Tt and Ts that indicate the same part may be selected as the candidate feature point pair M. For example, feature point Tt1 that indicates the upper right corner of the first object OB1 in the gray reference image IMtg may be associated with feature point Ts1 that indicates the upper right corner of the first object OB1 in the gray read image IMsg.

[0059] Furthermore, a pair of feature points Tt and Ts that indicate different portions may be selected as a candidate feature point pair M. For example, a feature point Tt1 that indicates the upper right corner of a first object OB1 in the gray reference image IMtg may be associated with a feature point Ts2 that indicates the upper right corner of a third object OB3 in the gray read image IMsg.

[0060] Various matching methods may be used. For example, the processor 210 may match feature points Tt and Ts whose distance obtained using feature quantities Ft and Fs is close. The distance is calculated so that a small distance indicates a high similarity between the two feature quantities Ft and Fs. Various methods may be used to calculate such distances, depending on the data structure of the feature quantities Ft and Fs. When the feature quantities Ft and Fs are expressed by binary vectors (vectors consisting of one or more binary elements), such as the A-KAZE feature descriptor, the processor 210 may use the Hamming distance. Instead of the Hamming distance, the processor 210 may use various other distances (e.g., norms such as the L1 norm and the L2 norm (also known as the Euclidean distance)). Norms are applicable to various feature descriptors.

[0061] The processor 210 associates the reference feature point Tt with the closest read feature point Ts among the multiple read feature points Ts. Alternatively, the processor 210 may associate the read feature point Ts with the closest reference feature point Tt among the multiple reference feature points Tt. Various methods may be used to search for the closest feature point. For example, the processor 210 may form multiple candidate feature point pairs M by brute-force matching. Alternatively, other methods, such as FLANN (Fast Library for Approximate Nearest Neighbor)-based matching, may be employed.

[0062] Here, the processor 210 may perform a process of removing pairs with low reliability. For example, the processor 210 may sort multiple pairs of feature points Tt and Ts in ascending order of distance and select some of the top pairs (e.g., 50% of the pairs).

[0063] In S385, the processor 210 stores in the storage device 215 (e.g., the non-volatile storage device 230) data representing the plurality of candidate feature points pairs M. Then, the processor 210 ends the processing in Fig. 6, i.e., the processing of S220 in Fig. 5.

[0064] In S230, the processor 210 executes a feature point pair selection process using color information. This selection process uses color information to select appropriate candidates M from multiple candidate feature point pairs M (i.e., the candidate feature point pairs M are verified). In this embodiment, the multiple candidate feature point pairs M selected by the process of S230 are used as final feature point pairs.

[0065] 11 is a flowchart showing an example of the selection process. The processor 210 executes a loop process S410 (including S420-S465) between a start L4s and an end L4e for each of the plurality of candidate feature points pairs M.

[0066] In S420, the processor 210 selects the unprocessed candidate M as a candidate of interest Mi, which is the candidate to be processed.

[0067] In S425, the processor 210 calculates a representative color value Ct of the first partial region Pt including the reference feature point Tt of the focused candidate Mi (the representative color value Ct is also referred to as the reference representative color value Ct). FIGS. 12A and 12B are diagrams showing an example of calculating a representative color value. FIG. 12A shows a portion of the reference image IMt including the feature point Tt. The figure also shows the first partial region Pt including the feature point Tt. The processor 210 calculates the reference representative color value Ct using multiple color values ​​of multiple pixels in the first partial region Pt. The method for calculating the representative color value Ct may be any of various methods for calculating a color representative of the first partial region Pt. In this embodiment, the processor 210 calculates the average value of each of red (R), green (G), and blue (B) in the first partial region Pt as the representative color value Ct. Note that various summary statistics representing the magnitude of the color value (e.g., median, mode, etc.) may be used instead of the average value. Furthermore, the configuration of the first partial region Pt (specifically, the shape of the first partial region Pt and the relative position of the first partial region Pt with respect to the position of the feature point Tt) may be various configurations that enable calculation of the representative color value Ct representing the color of the portion indicated by the feature point Tt. In this embodiment, the first partial region Pt is a region centered on the feature point Tt, for example, a region of P rows and Q columns and P*Q pixels. In order to reduce the dependency of the representative color value Ct on the angle of rotation of the object, P=Q is preferable (for example, P=Q=5). Instead of a region of P rows and Q columns, the first partial region Pt may be a region whose distance from the feature point Tt is equal to or less than a distance threshold.

[0068] In S430 (FIG. 11), the processor 210 calculates a representative color value Cs of a second partial region Ps including the read feature point Ts of the target candidate Mi (the representative color value Cs is also referred to as the read representative color value Cs). FIG. 12(B) shows a portion of the read image IMs including the feature point Ts. The figure also shows the second partial region Ps including the feature point Ts. The configuration of the second partial region Ps is the same as the configuration of the first partial region Pt. The second partial region Ps is a region of P rows and Q columns, P*Q pixels, centered on the feature point Ts. The calculation method of the read representative color value Cs is the same as the calculation method of the reference representative color value Ct. The processor 210 calculates the average values ​​of red (R), green (G), and blue (B) within the second partial region Ps as the read representative color value Cs.

[0069] In S435 (FIG. 11), the processor 210 calculates a first hue Ht from the reference representative color value Ct and calculates a second hue Hs from the read representative color value Cs. A known relationship can be used as the correspondence between the representative color value and the hue (for example, the correspondence between the RGB values ​​in the RGB color space and the H value in the HSV color space). Note that the first hue Ht, like the reference representative color value Ct, is an example of the representative color value CJt of the first partial region Pt. The second hue Hs, like the read representative color value Cs, is an example of the representative color value CJs of the second partial region Ps.

[0070] In S445, the processor 210 determines whether a hue condition CH is satisfied, indicating that the absolute value of the difference between the first hue Ht and the second hue Hs is less than a hue difference threshold dHth. The absolute value of the difference between the hues Ht and Hs (also referred to as the hue difference dH) is the value corresponding to the smaller angle difference between the first hue Ht and the second hue Hs on the hue circle. If the focused candidate Mi is a suitable pair of feature points Tt and Ts that represent the same part, the hue difference dH may be small. If the hue difference dH is less than the hue difference threshold dHth (S445: Yes), the processor 210 selects the focused candidate Mi as a candidate to be retained in S460. If the hue difference dH is equal to or greater than the hue difference threshold dHth (S445: No), the hues of the first partial region Pt and the second partial region Ps may differ significantly. In other words, there is a high possibility that the feature points Tt and Ts of the focused candidate Mi indicate different parts. In S465, the processor 210 excludes the focused candidate Mi from the candidates for the feature points pair.

[0071] In this way, the processor 210 excludes from the candidates any candidate Mi with a hue difference dH equal to or greater than the hue difference threshold dHth. FIG. 10C shows an example of a candidate feature point pair M remaining after the processing of FIG. 11. A pair of feature points Tt, Ts indicating portions with different hues may be excluded. For example, the hues are different between the first object OB1 and the third object OB3. The line RLa shown in FIG. 10B indicates the pair of the reference feature point Tt of the first object OB1 and the read feature point Ts of the third object OB3. As shown in FIG. 10C, this pair is excluded (FIG. 11: S445 (No), S465).

[0072] Note that the more lenient the conditions for retaining a focused candidate Mi, the greater the total number of appropriate candidates M that remain without being eliminated. However, the total number of inappropriate candidates may also be greater. The greater the total number of appropriate candidates, the more reliable the process using multiple feature point pairs (e.g., the smaller the error in the coordinate correspondence described below). If the total number of inappropriate candidates is large, the reliability of the process using multiple feature point pairs may decrease (e.g., the larger the error in the coordinate correspondence). The hue difference threshold dHth (S445) may be experimentally determined in advance so that the reliability of the process using multiple feature point pairs is acceptable. For example, the greater the hue difference threshold dHth, the more lenient the conditions for retaining a focused candidate Mi. Even if the focused candidate Mi is an appropriate pair of feature points Tt and Ts, the hue difference dH may be greater than zero. A hue difference threshold dHth greater than zero allows such a hue difference dH. The hue difference threshold dHth may be set to a value greater than zero and less than the maximum possible value of the hue difference dH.

[0073] After S460 or S465, the processor 210 proceeds to S420 and executes the loop process S410 for the next focused candidate Mi. When the loop process S410 for all candidates M has been completed, the processor 210 stores data representing the candidate feature points pairs M to be retained in the storage device 215 (e.g., the non-volatile storage device 230) in S470. The processor 210 then ends the process of FIG. 11, i.e., the process of S230 in FIG. 5. The multiple candidate feature points pairs M selected by the process of S230 are used as the final multiple feature points pairs (in this embodiment, the multiple feature points pairs are used to determine the correspondence between coordinates). Thus, in this embodiment, the feature point matching process includes S385 (FIG. 6) and S230 (FIG. 5).

[0074] In S250, the processor 210 determines the correspondence between coordinates on the reference image IMt and coordinates on the scanned image IMs. FIG. 13 is a diagram showing an example of a representation format for the coordinate correspondence. The diagram shows coordinates COt on the reference image IMt, coordinates COs on the scanned image IMs, and a matrix Mtx that associates these coordinates COt and COs. In this embodiment, the matrix Mtx represents a so-called affine transformation. In the diagram, the coordinates COt and COs and the matrix Mtx are expressed in a homogeneous coordinate system. Each of the coordinates COt and COs is represented by a three-dimensional vector having positions Xt and Xs in the first direction Dx on the images IMt and IMs, positions Yt and Ys in the second direction Dy, and a third component of 1. The matrix Mtx is a 3-row, 3-column matrix. As shown in the figure, the matrix Mtx is represented by six parameters a-f in two rows and three columns, and three elements (0, 0, 1) in the third row. Such a matrix Mtx can represent rotation, enlargement, reduction, translation, and skew. The matrix Mtx can be calculated by using three or more pairs of coordinates COt, COs (i.e., three or more pairs of feature points).

[0075] The processor 210 determines the matrix Mtx (here, six parameters a-f) using a plurality of feature point pairs MP (i.e., a plurality of candidate feature point pairs M obtained in S220-S230 of FIG. 5). The method for determining the matrix Mtx may be various. For example, the processor 210 may determine the matrix Mtx according to a method called Random Sample Consensus (RANSAC). To determine the matrix Mtx, for example, a function of OpenCV (Open Source Computer Vision Library) may be used.

[0076] Upon completion of S250 (FIG. 5), the process of FIG. 5, i.e., S120 of FIG. 4, ends. In S130, the processor 210 inspects the printed image. The inspection method may be any of various methods that use the coordinate correspondence (FIG. 13). In this embodiment, the processor 210 generates difference image data using the coordinate correspondence. FIGS. 14(A) and 14(B) are diagrams showing examples of difference images. FIG. 14(A) shows a case where the printed image has no defects, and FIG. 14(B) shows a case where the printed image has defects (here, missing Err).

[0077] The left side of FIGS. 14A and 14B shows scanned images IMs and IMs2, and a target image IMp arranged on the scanned images IMs and IMs2. As described with reference to FIGS. 6 and 9, the reference image IMt includes at least a portion of the target image IMp (here, the portion including the colored portion P1). The correspondence between coordinates COt on the reference image IMt and coordinates on the target image IMp is expressed by a translation that associates portions common to the images IMt and IMp. The processor 210 uses the coordinate correspondence between the reference image IMt and the target image IMp and the correspondence shown in FIG. 13 to determine the correspondence between coordinates on the target image IMp and coordinates COs on the scanned image IMs (referred to as the target coordinate relationship). In FIGS. 14A and 14B, the target image IMp is arranged on the scanned images IMs and IMs2 in accordance with the target correspondence relationship.

[0078] The images IMd and IMd2 on the right side of Figures 14(A) and 14(B) represent examples of difference images between the scanned images IMs and IMs2 and the reference image IMt. The processor 210 generates data for the difference images IMd and IMd2 that represent the color value differences (e.g., absolute values ​​of the brightness value differences) between the scanned images IMs and IMs2 and the target image IMp at positions associated by the target coordinate relationship. The scanned image IMs in Figure 14(A) represents a printed image without defects. Therefore, the difference image IMd does not have any areas that show large differences. The scanned image IMs2 in Figure 14(B) represents a printed image with missing bits Err. Therefore, in the difference image IMd2, the areas corresponding to the missing bits Err show larger differences than the other areas.

[0079] The processor 210 may use the reference image IMt instead of the target image IMp to generate the data of the difference images IMd and IMd2. In this case, the difference in the non-defective portions of the background BG will be smaller.

[0080] In S140 (FIG. 4), the processor 210 outputs the inspection results. Various methods for outputting the inspection results are possible. In this embodiment, the processor 210 displays a difference image on the display unit 240 (FIG. 1). By observing the display unit 240, the operator can easily recognize defects in the printed image. Alternatively, the processor 210 may output data representing the inspection results to a storage device (e.g., the non-volatile storage device 230 or an external storage device connected to the data processing device 200). The data representing the inspection results is thereby stored in the storage device. The data representing the inspection results can be used for various processes (e.g., inspection processing of the entire T-shirt 700). After S140, the processor 210 ends the inspection processing.

[0081] 15(A) and 15(B) are diagrams showing another example of an image to be processed. In FIGS. 15(A) and 15(B), color images IMpc2 and IMpc3 of target images IMp2 and IMp3 are shown. The only difference from the color image IMpc in FIG. 9 is that the backgrounds BG2 and BG3 extend outside the circumscribing rectangle RA. In the areas of the backgrounds BG2 and BG3, the opacity k indicates transparency. The objects OB1-OB4 in the color images IMpc2 and IMpc3 are the same as the objects OB1-OB4 in the color image IMpc in FIG. 9, respectively. When the target images IMp2 and IMp3 are used, a scanned image similar to the scanned image IMs (FIG. 3(B)) is also obtained.

[0082] The background BG2 of the color image IMpc2 in FIG. 15A is configured to surround each of the objects OB1-OB4. In S340 (FIG. 6), the processor 210 generates a temporary reference image IM21 by setting the color value of the transparent portion P02 of the color image IMpc2 to the medium color Cm. In S345, the processor 210 determines a peripheral portion PS2 that surrounds the circumscribing rectangle RA. The peripheral portion PS2 is formed inside the outline of the temporary reference image IM21. Therefore, in S350, no new region is added to the temporary reference image IM21. The temporary reference image processed in S350 is the same as the temporary reference image IM21. In S355, the processor 210 generates a temporary reference image IM22 by deleting the portion outside the peripheral portion PS2 from the temporary reference image IM21. The temporary reference image IM22 is used as a reference image (called reference image IMt2).

[0083] The background BG3 of the color image IMpc3 in FIG. 15B includes a portion on the first direction Dx side and a portion on the second direction Dy side of the circumscribing rectangle RA. In S340 ( FIG. 6 ), the processor 210 generates a temporary reference image IM31 by setting the color value of the transparent portion P03 of the color image IMpc2 to the medium color Cm. In S345, the processor 210 determines a peripheral portion PS3 that surrounds the circumscribing rectangle RA. The peripheral portion PS3 includes an outer portion Pu3 that is a portion outside the temporary reference image IM31 (specifically, the portion on the −Dx direction side and the portion on the −Dy direction side of the temporary reference image IM31). In S350, the processor 210 generates a temporary reference image IM32 by adding the outer portion Pu3 of the peripheral portion PS3 that is located outside the temporary reference image IM31 to the temporary reference image IM31. The processor 210 sets the color value of the outer portion Pu3 to the medium color Cm. In S355, the processor 210 generates a temporary reference image IM33 by deleting the portion outside the surrounding portion PS3 from the temporary reference image IM32. The temporary reference image IM33 is used as a reference image (referred to as reference image IMt3). In this way, the outer portion Pu3 has been added to the reference image IMt3.

[0084] In this way, the reference images IMt2 and IMt3 include peripheral portions PS2 and PS3 surrounding the circumscribing rectangle RA, similar to the reference image IMt ( FIG. 10A ). Therefore, from the reference images IMt2 and IMt3, similar to the reference image IMt, an appropriate candidate feature point pair M can be formed from the vicinity of the circumscribing contour LRA.

[0085] Furthermore, if the color images IMpc2 and IMpc3, and thus the target images IMp2 and IMp3, include portions outside the surrounding portions PS2 and PS3, the processor 210 deletes the portions outside the surrounding portions PS2 and PS3 (FIG. 6: S355). This reduces the size of the reference images IMt2 and IMt3, thereby reducing the burden of processes that use the reference images IMt2 and IMt3 (e.g., processes S360-S385 in FIG. 6). For example, the size of the memory area used to store the reference images IMt2 and IMt3 is reduced. Hereinafter, the portions of the transparent portions P0, P02, and P03 of the target images IMp, IMp2, and IMp3 (FIGS. 10, 15A, and 15B) that are included in the reference images IMt, IMt2, and IMt3 are referred to as reference transparent portions P0t, P02t, and P03t.

[0086] As described above, in this embodiment, the processor 210 executes the following process in accordance with the program 231. In S330 of FIG. 6 , the processor 210 executes a generation process to generate a reference image IMt using print image data representing the target image IMp ( FIG. 9 ) (the process of S330 is also referred to as generation process S330). The reference image IMt includes a reference transparent portion P0t, which is at least a part of the transparent portion P0 of the target image IMp, which indicates that no colorant is used, and a colored portion P1 of the target image IMp, which indicates that a colorant is used. The generation process S330 includes S340. In S340, the processor 210 sets the color of the transparent portion P0 (including the reference transparent portion P0t) to a medium color Cm, which represents a T-shirt 700, which is an example of a medium. In S385, the processor 210 performs a feature point matching process using a plurality of feature points Ts in the gray read image IMsg ( FIG. 10A ) (i.e., a plurality of feature points Ts in the read image IMs) and a plurality of feature points Tt in the gray reference image IMtg ( FIG. 10A ) (i.e., a plurality of feature points Tt in the reference image IMt). The read image IMs is an image generated by optically reading the target image IMp to be printed on the T-shirt 700 using the printing image data.

[0087] In this way, because the color of the reference transparent portion P0t included in the reference image IMt ( FIG. 9 ) is set to the medium color Cm, the color of the reference transparent portion P0t becomes close to the color of the corresponding portion in the read image IMs. Therefore, the processor 210 can appropriately perform feature point matching by using the reference feature point Tt and the read feature point Ts. The reference feature point Tt is a feature point in the gray reference image IMtg ( FIG. 10 ), i.e., a feature point in the reference image IMt. The read feature point Ts is a feature point in the gray read image IMsg, i.e., a feature point in the read image IMs. If the color of the reference transparent portion P0t in the reference image IMt is set to a color unrelated to the medium (here, the T-shirt 700), the distance between the feature quantities Ft and Fs may increase even if the reference feature point Tt and the read feature point Ts near the reference transparent portion P0t represent the same portion of the same object. As a result, the total number of suitable pairs of feature points Tt and Ts formed by feature point matching may be reduced. In this embodiment, the possibility of such a defect is reduced.

[0088] In this embodiment, in S335 (FIG. 6), the processor 210 executes a color determination process to determine a medium color Cm by analyzing the read image IMs (the process of S335 is also referred to as the color determination process S335). The processor 210 can reduce the difference between the color of the reference transparent portion P0t of the reference image IMt and the color of the corresponding portion of the read image IMs. Therefore, the processor 210 can appropriately perform feature point matching by using the reference feature points Tt and the read feature points Ts.

[0089] In this embodiment, the color determination process S335 ( FIG. 6 ) includes steps S910, S915, and S920 in FIG. 8A . In step S910, the processor 210 analyzes the scanned image IMs to obtain a color appearance frequency distribution (here, a three-dimensional histogram HST). In step S915, the processor 210 determines a representative color Cb using the color appearance frequency distribution. In step S920, the processor 210 determines the medium color Cm as the representative color Cb. This configuration allows the processor 210 to determine a medium color Cm that is suitable for the color of the scanned image IMs.

[0090] In this embodiment, in S230 (FIG. 5), the processor 210 selects a feature point pair MP using color information. That is, the process of matching the reference feature point Tt with the read feature point Ts includes S230. The process of S230 includes S445-S465 of FIG. 11. In S445-S465, the processor 210 selects candidate feature points M for the feature points Tt and Ts to be retained using the hue Ht associated with the feature point Tt and the hue Hs associated with the feature point Ts. Thus, the process of matching the reference feature point Tt with the read feature point Ts includes a process of determining the correspondence between the feature point Ts in the read image IMs and the feature point Tt in the reference image IMt using the hues Ht and Hs. The colors of the same portions of the same object may differ between the reference image IMt and the read image IMs. Here, the possibility of a difference in hue is smaller than that of brightness and saturation. The processor 210 can use the hues Ht and Hs to select a suitable candidate feature point pair M as the feature point pair MP.

[0091] In this embodiment, the generation process S330 ( FIG. 6 ) further includes S350. As described with reference to FIGS. 9 and 15(B), in S350, the processor 210 adds outer portions Pu and Pu3, which are outer portions of the color images IMpc and IMpc3 (i.e., the target images IMp and IMp3) and indicate the medium color Cm, to the temporary reference images IM11 and IM31. In this way, the processor 210 adds the outer portions Pu and Pu3 to the reference images IMt and IMt3 (the process of S350 is also referred to as the addition process S350). Therefore, the processor 210 can form appropriate candidate feature point pairs M from the vicinity of the contours of the target images IMp and IMp3.

[0092] In this embodiment, the reference images IMt, IMt2, and IMt3 (FIGS. 9, 15A, and 15B) include a colored portion P1 and surrounding portions PS, PS2, and PS3 that surround a circumscribing rectangle RA that circumscribes the colored portion P1. The reference images IMt, IMt2, and IMt3 are images that do not include the portions of the target images IMp, IMp2, and IMp3 that are outside the surrounding portions PS, PS2, and PS3 (note that the target image IMp does not include the portion outside the surrounding portion PS). The generation process S330 (FIG. 6) further includes S345. In S345, the processor 210 determines the surrounding portions PS, PS2, and PS3 using the colored portion P1. 9 and 15B, in the addition process S350, the processor 210 adds the outer portions Pu and Pu3 of the surrounding portions PS and PS3 that are located outside the target images IMp and IMp3 to the reference images IMt and IMt3. This configuration allows the processor 210 to generate an appropriate reference image regardless of the shape and position of the colored portion in the target image.

[0093] Furthermore, in this embodiment, the processor 210 executes the following processes in accordance with the program 231. In the generation process S330 in FIG. 6 , the processor 210 executes a generation process to generate reference images IMt and IMt3 using printing image data representing the target images IMp and IMp3 ( FIGS. 9 and 15B ). The generation process S330 ( FIG. 6 ) further includes an addition process S350. As described with reference to FIGS. 9 and 15B , in the addition process S350, the processor 210 adds outer portions Pu and Pu3, which are portions outside the color images IMpc and IMpc3 (i.e., the target images IMp and IMp3) and indicate the medium color Cm, to the reference images IMt and IMt3. The reference images IMt and IMt3 are used to form candidate feature point pairs M ( FIG. 10A ). From the gray reference image of the reference image IMt3, multiple reference feature points Tt are detected, similar to the multiple reference feature points Tt on the gray reference image IMtg of the reference image IMt. In S385 ( FIG. 6 ), the processor 210 performs a feature point matching process using multiple feature points Ts in the gray read image IMsg ( FIG. 10(A) ) (i.e., multiple feature points Ts in the read image IMs) and multiple feature points Tt in the gray reference images of the reference images IMt and IMt3 (i.e., multiple feature points Tt in the reference images IMt and IMt3). The read image IMs is an image generated by optically reading the target image IMp to be printed on the T-shirt 700 using the printing image data. As described above, when the target image IMp3 is used, a read image similar to the read image IMs is also obtained. As a result, the processor 210 can form suitable candidate feature points pairs M from the vicinity of the contours of the target images IMp and IMp3. The greater the total number of suitable candidate feature points pairs M, the greater the total number of suitable feature points pairs MP. The greater the total number of suitable feature points pairs MP, the more reliable the process (e.g., alignment) that uses multiple feature points pairs MP.

[0094] As described above, the processor 210 determines the medium color Cm by analyzing the scanned image IMs in the color determination process S335 (FIG. 6), and therefore, can appropriately perform feature point matching.

[0095] As described above, the color determination process S335 (FIG. 6) includes S910, S915, and S920 in FIG. 8A, so the processor 210 can determine the medium color Cm that is suitable for the color of the scanned image IMs.

[0096] As described above, the processor 210 selects a feature point pair MP using color information in S230 (FIG. 5). The process of S230 includes S445-S465 in FIG. 11. The process of matching the reference feature point Tt with the read feature point Ts includes a process of determining the correspondence between the feature point Ts in the read image IMs and the feature point Tt in the reference image IMt using the hues Ht and Hs. By using the hues Ht and Hs, the processor 210 can select an appropriate candidate feature point pair M as the feature point pair MP.

[0097] B. Second Embodiment: FIG. 16A is a portion of a flowchart of another embodiment of the process for determining the medium color Cm (S335 (FIG. 6)). This embodiment differs from the embodiment of FIG. 8A in that steps S905b and S910b are executed instead of S910. In this embodiment, the target area to be analyzed for determining the medium color Cm is specified by the user.

[0098] In S905b, the processor 210 acquires a user instruction specifying a target area for analysis. In this embodiment, the processor 210 displays the scanned image on the display unit 240 ( FIG. 1 ). FIG. 16B is a diagram illustrating an example of a scanned image IMs displayed by the display unit 240. By observing the displayed scanned image IMs, a user (e.g., an operator) can recognize the portion of the scanned image IMs representing the T-shirt 700 (particularly the blank portion without a printed image). The user operates the operation unit 250 to input information indicating a target area ARb representing the blank portion of the T-shirt 700 to the data processing device 200. In this embodiment, the target area ARb is a rectangular area. The user can specify the target area ARb by specifying the positions of two vertices forming diagonal corners of the rectangle on the scanned image IMs. The processor 210 acquires a user instruction specifying the target area ARb using information from the operation unit 250.

[0099] In S910b (FIG. 16A), the processor 210 obtains a color frequency distribution by analyzing a target area ARb of the scanned image IMs. In this embodiment, the processor 210 calculates the frequency of each bin of the three-dimensional histogram HST (FIG. 8B) using multiple color values ​​of multiple pixels in the target area ARb. The portion of the scanned image IMs other than the target area ARb is not used.

[0100] After S910b, the processor 210 executes S915 and S920 (FIG. 8A). As a result, the medium color Cm is determined to be the representative color Cb obtained from the target area ARb. The processor 210 then ends the process of determining the medium color Cm, i.e., S335 in FIG. 6.

[0101] As described above, in this embodiment, the color determination process S335 ( FIG. 6 ) includes S905b ( FIG. 16A ). In S905b, the processor 210 acquires a user instruction specifying the target area ARb of the scanned image IMs to be analyzed. As described in S910b, the target area ARb of the scanned image IMs is the target area ARb of the scanned image IMs that is the target of analysis in the color determination process S335. This configuration allows the processor 210 to determine an appropriate medium color Cm that represents the color of the T-shirt 700. For example, if the target image IMp ( FIG. 3A ) includes a large solid area, analysis of the entire scanned image IMs may determine a representative color Cb (and thus a medium color Cm) that represents the color of the solid area, rather than the color of the margins of the T-shirt 700. In this embodiment, the possibility of such a problem is reduced.

[0102] C. Third Embodiment: FIG. 17(A) is a portion of a flowchart of another embodiment of the process for determining the medium color Cm (S335 (FIG. 6)). The difference from the embodiment in FIG. 8(A) is that S910c is executed instead of S910. In this embodiment, a target area of ​​the scanned image is analyzed to determine the medium color Cm. FIG. 17(B) is a diagram showing an example of the target area. The scanned image IMs is shown in the figure. The target area ARc is a specific partial area on the side corresponding to the downward direction of the T-shirt 700 (here, the second direction Dy). In this embodiment, a predetermined area of ​​the scanned image IMs is used as the target area ARc.

[0103] In S910c, the processor 210 obtains a color frequency distribution by analyzing the target area ARc of the scanned image. In this embodiment, the processor 210 calculates the frequency of each bin of the three-dimensional histogram HST ( FIG. 8B ) using multiple color values ​​of multiple pixels in the target area ARc. The portion of the scanned image IMs other than the target area ARc is not used.

[0104] After S910c, the processor 210 executes S915 and S920 (FIG. 8A). As a result, the medium color Cm is determined to be the representative color Cb obtained from the target area ARc. The processor 210 then ends the process of determining the medium color Cm, i.e., S335 in FIG. 6.

[0105] As described above, in this embodiment, the printing medium is a T-shirt 700. The portion of the T-shirt 700 on which an image is printed may be a portion on the upper side of the T-shirt 700 (e.g., a portion corresponding to the chest). A portion on the lower side of the T-shirt 700 may be a blank portion on which no image is printed. The target of analysis of the scanned image IMs ( FIG. 17B ) by the color determination process S335 ( FIG. 6 ) is the target area ARc. The target area ARc is a specific partial area of ​​the scanned image IMs in the direction corresponding to the lower side of the T-shirt 700 (here, the second direction Dy). Such a target area ARc is likely to represent a blank portion of the T-shirt 700. Therefore, the processor 210 can determine an appropriate medium color Cm that represents the color of the T-shirt 700. For example, even if the target image IMp (Figure 3 (A)) includes a solid area with a large area, the processor 210 can determine the representative color Cb (and therefore the medium color Cm) that represents the color of the marginal area of ​​the T-shirt 700.

[0106] D. Modifications: (1) In the embodiment of FIG. 16A , the method of specifying the target area in the scanned image may be various methods other than the method of specifying the vertex positions of the target area ARb. For example, the processor 210 may adopt, as the target area, an area surrounded by an outline freely drawn by the user (such area selection is also called freehand selection). Furthermore, the processor 210 may use a designated point on the scanned image designated by the user to select, as the target area, a partial area that includes the designated point and exhibits a color similar to the color of the designated point (area selection using a designated point is also called a magic wand tool).

[0107] (2) In the examples of Figures 17(A) and 17(B), the target area ARc in the scanned image may be any of various partial areas of the scanned image IMs in a direction corresponding to the downward direction of the T-shirt 700 (here, the second direction Dy). Hereinafter, the direction of the scanned image IMs corresponding to the downward direction of the T-shirt 700 is also referred to as the medium downward direction. The target area ARc may be, for example, a partial area included in an area IMsL corresponding to half the length of the scanned image IMs in the medium downward direction Dy.

[0108] (3) The method of determining the medium color Cm may be any of various methods for determining the medium color by analyzing the scanned image, instead of the methods shown in Figures 8(A), 8(B), 16(A), 16(B), 17(A), and 17(B). For example, a reference color may be assigned in advance to each of the multiple bins in the three-dimensional histogram HST (e.g., the average color of the range of color values ​​in the bin). The processor 210 may then use the reference color assigned to the bin with the highest frequency as the representative color Cb.

[0109] The processor 210 may also use a trained machine learning model (e.g., a model that performs region division called instance segmentation (e.g., Mask R-CNN)) to divide the region representing the medium (here, the T-shirt 700) in the scanned image into a region representing the printed image and another region (referred to as a blank region). The processor 210 may then determine the medium color Cm by analyzing the blank region. Here, the processor 210 may calculate, as the medium color Cm, a summary statistic (e.g., median, mode, mean, etc.) that represents multiple color values ​​of multiple pixels included in the blank region.

[0110] The medium color Cm may also be determined without using the scanned image. For example, data representing the correspondence between the medium identification information (e.g., the medium model number) and the medium color Cm may be stored in advance in the storage device 215 (e.g., the non-volatile storage device 230). The processor 210 may adopt the medium color Cm associated with the medium identification information input by the user. The processor 210 may also adopt a color (e.g., RGB gradation values) specified by the user as the medium color Cm.

[0111] (4) The method of specifying the transparent portion of the target image that indicates that no colorant is to be used may be any method other than the method of using the opacity k of the alpha channel. For example, the portion of the target image to be used as the transparent portion may be a portion having a specific color (e.g., white).

[0112] (5) The generation process for generating a reference image using a target image may be various processes instead of the generation process S330 of FIG. 6 . For example, S350 may be executed after S355. The process for deleting the portion outside the peripheral portion from the reference image (e.g., S355) may be omitted. The process for setting the color of the reference transparent portion to the medium color, which is the color representing the medium (e.g., S340) may be omitted. The processor 210 may set the color of the transparent portion to a predetermined color (e.g., white). The process for adding the portion outside the target image that indicates the medium color (e.g., S350) to the reference image may be omitted. The processes for adding the peripheral portion surrounding the circumscribing rectangle that circumscribes the colored portion to the reference image (e.g., S345, S350) may be omitted. The processor 210 may generate a reference image that includes the target image and the peripheral portion surrounding the target image. In either case, the target image may be an image that does not include a transparent portion. Again, if the reference image includes a perimeter surrounding the target image, the processor 210 can form suitable pairs from near the edges of the reference image.

[0113] (6) In S425 and S435 ( FIG. 11 ), the processor 210 may calculate the first hue Ht by using the target image instead of the reference image. If the target image includes a transparent portion, the processor 210 may calculate the first hue Ht by using the remaining portion of the target image excluding the transparent portion. Note that the process of selecting candidate feature point pairs M using color information (e.g., S230 ( FIG. 5 )) may be omitted.

[0114] (7) The detection method for the feature points Tt and Ts (i.e., key points) may be any of various methods for detecting points representing portions of an object in an image, instead of the methods described in S365 and S370 (FIG. 6). The detection method may be selected in advance from, for example, a search for extrema (maximum and minimum) using Difference-of-Gaussian (DoG), Harris corner detection, Features from Accelerated Segment Test (FAST) corner detection, Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented Fast and Rotated Brief (ORB).

[0115] The calculation method of the feature quantities Ft and Fs (i.e., the feature descriptor) may be various methods for calculating information describing the features of keypoints, instead of the methods described in S375 and S380 (FIG. 6). The algorithm for calculating the feature descriptor may be selected in advance from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, ORB, KAZE, and A-KAZE. Furthermore, the calculation method of the distance between the feature quantities Ft and Fs may be various methods suitable for the data structure of the feature quantities Ft and Fs. When the feature quantities Ft and Fs are expressed by binary vectors, the distance may be the Hamming distance. Instead of the Hamming distance, various other distances (e.g., norms such as the L1 norm and the L2 norm (also known as Euclidean distance)) may be used. The norm is applicable to a variety of feature descriptors.

[0116] (8) The feature point matching process executed according to the program 231 may include a process for forming pairs of read feature points Ts in the read image IMs and reference feature points Tt in the reference image IMt (e.g., S385 ( FIG. 6 )), as well as a process for detecting multiple reference feature points Tt from the reference image IMt (e.g., S365), a process for detecting multiple read feature points Ts from the read image IMs (e.g., S370), a process for calculating feature values ​​Ft for each of the multiple reference feature points Tt (e.g., S375), and a process for calculating feature values ​​Fs for each of the multiple read feature points Ts (e.g., S380). Processes other than the generation process for generating a reference image using image data for printing representing a target image and the feature point matching process using multiple feature points in the read image and multiple feature points in the reference image may be executed according to a program other than the program 231. For example, S250 ( FIG. 5 ) may be executed according to another program. S130 and S140 (FIG. 4) may be executed according to another program. Processes other than the generation process and the feature point matching process may be performed by a device other than the data processing device 200. The generation process may include one or both of a process of setting the color of the reference transparent portion to a medium color that represents the medium, and an addition process of adding an external portion that is outside the target image and indicates the medium color that represents the medium, to the reference image. In either case, the image to be processed may be a grayscale image instead of a color image.

[0117] (9) The print medium is not limited to the T-shirt 700 and may be various types of clothing (for example, various shirts such as polo shirts, coats, slacks, etc.). The print medium may be various types of fabric such as clothing or bags. The print medium is not limited to fabric and may be various media such as paper, film, leather, etc. Furthermore, the printing device 900 may be a printing device of another type (for example, a laser type) instead of an inkjet type printing device.

[0118] (10) The results of feature point matching may be used for various processes, not just for determining coordinate correspondence. For example, if the printed image has defects, the total number of feature point pairs MP formed by feature point matching may be reduced. The total number of feature point pairs MP may be used to evaluate the quality of the printed image (the greater the total number of feature point pairs MP, the higher the quality).

[0119] (11) The coordinate correspondence ( FIG. 13 ) may represent various transformations, such as a homography transformation, instead of an affine transformation. Furthermore, the coordinate correspondence may be represented in various formats, such as a lookup table, instead of the matrix Mtx. Furthermore, the coordinate correspondence may be used in various processes, not limited to inspection. For example, when a single image is generated by combining a target image and a scanned image, the coordinate correspondence may be used to align the target image and the scanned image.

[0120] (12) The data processing device that performs the generation process and the feature point matching process is not limited to a personal computer (e.g., data processing device 200 (FIG. 1)), but may be various other devices (e.g., a smartphone, a tablet computer, a control device incorporated in a reading device, etc.). Furthermore, multiple devices (e.g., computers) that can communicate with each other via a network may share part of the data processing functions of the data processing device, and collectively provide the data processing functions (a system including these devices corresponds to a data processing device).

[0121] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the process of S385 in Fig. 6 may be executed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0122] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used while stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0123] The above-described examples and modifications can be combined as appropriate. The above-described examples and modifications are provided to facilitate understanding of the present disclosure and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof.

[0124] 100...reading device, 110...controller, 120...conveyor device, 122...position sensor, 130...table, 140...support section, 180...reading sensor, 190...casing, 200...data processing device, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 231...program, 240...display section, 250...operation section, 270...communication interface, 700...T-shirt, 900...printing device, ARb, AR c...target area, Cb...representative color, Cm...medium color, Hs, Ht...hue, IMp, IMp2, IMp3...target image, IMt, IMt2, IMt3...reference image, IMs, IMs2...scanned image, P0, P02, P03...transparent portion, P0t...reference transparent portion, P1...colored portion, PS, PS2, PS3...surrounding portion, Pu, Pu3...outside portion, RA...circumscribed rectangle, S330...generation process, S335...color determination process, S350...addition process, Ts, Tt...feature points

Claims

1. A program that causes a computer to realize the following functions: a function for executing a generation process to generate a reference image using image data for printing that represents a target image, the reference image including a reference transparent portion that is at least a part of a transparent portion of the target image that indicates that a coloring material is not used, and a colored portion of the target image that indicates that a coloring material is used, the generation process including a process for setting the color of the reference transparent portion to a medium color that represents the medium; and a function for executing a feature point matching process using a plurality of feature points in a read image that is generated by optically reading the target image that is printed on the medium using the image data for printing, and a plurality of feature points in the reference image.

2. A program according to claim 1, further causing a computer to realize a function of executing a color determination process for determining the medium color by analyzing the scanned image.

3. A program as described in claim 2, wherein the color determination process includes a process of obtaining a color occurrence frequency distribution by analyzing the scanned image, a process of determining a representative color using the color occurrence frequency distribution, and a process of determining the medium color as the representative color.

4. A program as described in claim 2 or 3, wherein the color determination process includes a process of acquiring a user instruction specifying a target area of ​​the read image to be analyzed, and the target area of ​​the read image to be analyzed by the color determination process is the target area of ​​the read image.

5. A program as claimed in claim 2 or 3, wherein the medium is a T-shirt, and the target of analysis of the read image by the colour determination process is a specific partial area of ​​the read image on a side corresponding to a downward direction of the T-shirt.

6. A program according to any one of claims 1 to 3, wherein the feature point matching process includes a process for determining a correspondence between feature points in the read image and feature points in the reference image using hue.

7. A program according to any one of claims 1 to 3, wherein the generation process further includes an addition process of adding an outer portion of the target image, which portion indicates the medium color, to the reference image.

8. A program as described in claim 7, wherein the reference image is an image including the colored portion and a surrounding portion surrounding a circumscribing rectangle that circumscribes the colored portion, and does not include a portion of the target image outside the surrounding portion, the generation process further including a process of determining the surrounding portion using the colored portion, and the addition process including a process of adding the outer portion of the surrounding portion that is located outside the target image to the reference image.

9. A program that enables a computer to realize the following functions: a function for executing a generation process to generate a reference image using image data for printing that represents a target image, the generation process including an additional process for adding an outer portion of the target image that represents a medium color, which is an outer portion of the target image and is a color that represents the medium, to the reference image; and a function for executing a feature point matching process using a plurality of feature points in a read image that is generated by optically reading the target image that is printed on the medium using the image data for printing, and a plurality of feature points in the reference image.

10. A program according to claim 9, further causing a computer to realize a function of executing a color determination process for determining the medium color by analyzing the scanned image.

11. A program as described in claim 10, wherein the color determination process includes a process of obtaining a color occurrence frequency distribution by analyzing the read image, a process of determining a representative color using the color occurrence frequency distribution, and a process of determining the medium color as the representative color.

12. A program as described in claim 10 or 11, wherein the color determination process includes a process of acquiring a user instruction that specifies a target area of ​​the read image to be analyzed, and the target area of ​​the read image to be analyzed by the color determination process is the target area of ​​the read image.

13. A program as claimed in claim 10 or 11, wherein the medium is a T-shirt, and the target of analysis of the read image by the color determination process is a specific partial area of ​​the read image on a side corresponding to a downward direction of the T-shirt.

14. A program according to any one of claims 9 to 11, wherein the feature point matching process includes a process of determining a correspondence between feature points in the read image and feature points in the reference image using hue.

15. A data processing device comprising: a generation unit that executes a generation process to generate a reference image using image data for printing representing a target image, the reference image including a reference transparent portion that is at least a part of a transparent portion of the target image that indicates that a coloring material is not used, and a colored portion of the target image that indicates that a coloring material is used, the generation process including a process of setting the color of the reference transparent portion to a medium color that represents a medium; and a matching unit that executes a feature point matching process using a plurality of feature points in a read image generated by optically reading the target image printed on the medium using the image data for printing, and a plurality of feature points in the reference image.

16. A data processing device comprising: a generation unit that executes a generation process to generate a reference image using image data for printing representing a target image, the generation process including an addition process that adds an outer portion of the target image that indicates a medium color, which is a color representing the medium, to the reference image; and a matching unit that executes a feature point matching process using a plurality of feature points in a read image generated by optically reading the target image printed on the medium using the image data for printing and a plurality of feature points in the reference image.

Images