Computer program and information processing device

Abstract

Provided is a new technology related to the relative position of a first object to a second object.　Feature point matching between a target image and a reference image indicating a second object which includes the first object is executed to determine a plurality of feature point pairs. A coordinate transformation matrix indicating the positional relationship between the reference image and the target image is calculated using, from among the plurality of feature point pairs, a feature point pair corresponding to a portion different from the first object. Difference information about the difference between the first object in the reference image and the first object in the target image is calculated using the coordinate transformation matrix and specific information about the position of the first object in the reference image and the target image. Alternatively, the coordinate transformation matrix is calculated using a feature point pair corresponding to the first object, from among the plurality of feature point pairs, and the difference information is calculated using the coordinate transformation matrix and specific information relating to the position of a portion that is different from the first object in each of the reference image and the target image.

Description

Computer program and information processing device

[0001] The present specification relates to a computer program and an information processing device.

[0002] There are known techniques for inspecting objects (e.g., label sheets affixed to products such as multifunction peripherals) using captured images of the object. For example, Patent Literature 1 discloses the following technique: Image data of a label sheet without defects is generated by inputting the image data of the label sheet into a machine learning model. Difference image data representing the difference between the input image and the generated image is used to inspect whether the object has any abnormalities.

[0003] Japanese Patent Application Laid-Open No. 2022-174516

[0004] However, there is room for improvement in processing the relative position of a first object (e.g., a label sheet) with respect to a second object (e.g., a product).

[0005] This specification discloses new techniques for processing the relative position of a first object with respect to a second object.

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] A computer program that causes a computer to realize the following: a reference image acquisition function that acquires a reference image showing a second object including a first object; a target image acquisition function that acquires a target image showing the second object including the first object; a pair determination function that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation function that calculates a first coordinate transformation matrix that indicates a positional relationship between the reference image and the target image, using feature point pairs among the plurality of feature point pairs that correspond to parts that are different from the first object; and a difference information calculation function that calculates difference information between the first object in the reference image and the first object in the target image, using specific information regarding the position of the first object in each of the reference image and the target image and the first coordinate transformation matrix, wherein the difference information includes information regarding a difference in the relative position of the first object with respect to the second object.

[0008] According to the above configuration, difference information is calculated using a first coordinate transformation matrix calculated using feature point pairs corresponding to portions different from the first object and specific information regarding the positions of the first object in the reference image and the target image. As a result, difference information that appropriately indicates the difference in the relative position of the first object with respect to the second object can be calculated. Therefore, for example, even if the position of the second object in the target image differs from the position of the second object in the reference image, the position of the first object included in the second object can be inspected.

[0009] [Application Example 2] A computer program that causes a computer to realize the following: a reference image acquisition function that acquires a reference image showing a second object including a first object; a target image acquisition function that acquires a target image showing the second object including the first object; a pair determination function that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, wherein each of the plurality of feature point pairs is a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation function that calculates a first coordinate transformation matrix that indicates a positional relationship between the reference image and the target image, using a feature point pair that corresponds to the first object out of the plurality of feature point pairs; and a difference information calculation function that calculates difference information between the first object in the reference image and the first object in the target image, using specific information on positions of portions of the reference image and the target image that are different from the first object, and the first coordinate transformation matrix, wherein the difference information includes information on a difference in relative position of the first object with respect to the second object.

[0010] According to the above configuration, difference information is calculated using a first coordinate transformation matrix calculated using a feature point pair corresponding to the first object and specific information regarding the positions of the portions of the reference image and the target image that are different from the first object. As a result, difference information that appropriately indicates the difference in the relative position of the first object with respect to the second object can be calculated. Therefore, for example, even if the position of the second object in the target image differs from the position of the second object in the reference image, the position of the first object included in the second object can be inspected.

[0011] The technology disclosed in this specification can be realized in various forms, such as an inspection device that inspects the position of a first object contained in a second object, a method for inspecting the position of a first object contained in a second object, a computer program for realizing these devices and methods, a recording medium on which the computer program is recorded, etc.

[0012] 1 is a block diagram showing the configuration of an inspection system 1000 according to the present embodiment; FIG. 2 is an explanatory diagram of a product 300; FIG. 3 is a flowchart of advance preparation processing; FIG. 4 is a diagram showing an example of an image used in advance preparation processing; FIG. 5 is a flowchart of position inspection processing according to the first embodiment; FIG. 6 is a diagram showing an example of an image used in position inspection processing; FIG. 7 is an explanatory diagram of a coordinate transformation matrix; FIG. 8 is an explanatory diagram of calculation of a positional deviation amount according to the first embodiment; FIG. 9 is a flowchart of position inspection processing according to the second embodiment; FIG. 10 is a flowchart of position inspection processing according to the third embodiment; FIG. 11 is a flowchart of position inspection processing according to the fourth embodiment; FIG. 12 is an explanatory diagram of calculation of a positional deviation amount according to the fourth embodiment;

[0013] A. First Example A-1. Configuration of the Inspection Apparatus Next, an embodiment will be described based on an example. FIG. 1 is a block diagram showing the configuration of an inspection system 1000 of this example. The inspection system 1000 includes a processing device 100 and an imaging device 400. The processing device 100 and the imaging device 400 are connected so as to be able to communicate with each other.

[0014] The processing device 100 is a computer such as a personal computer, and includes a CPU 110 as a controller of the processing device 100, a GPU 115, a volatile storage device 120 such as RAM, a non-volatile storage device 130 such as a hard disk drive, an operation unit 150 such as a mouse or keyboard, a display unit 140 such as a liquid crystal display, and a communication unit 170. The communication unit 170 includes a wired or wireless interface for communicatively connecting to an external device, for example, an image capture device 400.

[0015] The GPU (Graphics Processing Unit) 115 is a processor that performs image processing such as 3D graphics and calculation processing for machine learning models under the control of the CPU 110. In this embodiment, the GPU is used to execute calculation processing for an object detection model AN and a monocular depth estimation model MN, which will be described later.

[0016] The volatile storage device 120 provides a buffer area for temporarily storing various intermediate data generated when the CPU 110 performs processing. The non-volatile storage device 130 stores a computer program PG for the inspection device.

[0017] The computer program PG includes, as a module, a computer program that causes the CPU 110 and the GPU 115 to cooperate to realize the functions of an object detection model AN and a monocular depth estimation model MN, which will be described later. The computer program PG is provided, for example, by the manufacturer of the processing device 100. The computer program PG may be provided, for example, in a form that is downloaded from a server, or in a form that is stored on a DVD-ROM or the like. The CPU 110 executes the computer program PG to perform advance preparation processing and position inspection processing, which will be described later.

[0018] The image capturing device 400 is a digital camera that captures an image of a subject using a two-dimensional image sensor to generate a captured image representing the subject. The captured image data is bitmap data representing an image including a plurality of pixels, specifically, RGB image data that represents the color of each pixel using RGB values. The RGB values ​​are gradation values ​​(hereinafter also referred to as component values) of three color components, i.e., color values ​​of the RGB color system including R, G, and B values. The R, G, and B values ​​are, for example, gradation values ​​with a predetermined number of gradations (e.g., 256). The captured image data may also be luminance image data that represents the luminance of each pixel.

[0019] The photographing device 400 generates a photographed image under the control of the processing device 100, and transmits data of the photographed image to the processing device 100. In this embodiment, the photographing device 400 is used to photograph a product 300 to which a label L, which is an inspection target of the position inspection process, is affixed, and generate the photographed image.

[0020] 2A and 2B are explanatory diagrams of a product 300. Fig. 2A shows a perspective view of the product 300. In this embodiment, the product 300 is a printer having a substantially rectangular parallelepiped housing 30. In the manufacturing process, a rectangular label L is attached to a predetermined position on the front surface 31 (the surface on the +Y side) of the housing 30.

[0021] 2B shows a label L. The label L includes, for example, a background B and characters TX and a mark MK indicating various information such as the manufacturer's or product's brand logo, model number, lot number, etc.

[0022] A-2. Advance Preparation Processing FIG. 3 is a flowchart of the advance preparation processing. The advance preparation processing is a process that is executed only once prior to the position inspection processing described below. In the advance preparation processing, a reference image, label area information within the reference image, and a background mask image that indicates the background area of ​​the reference image are prepared. The advance preparation processing is started when a user (e.g., an inspection operator) inputs an instruction to start the processing to the processing device 100 via the operation unit 150. For example, the user inputs an instruction to start the advance preparation processing while placing the product 300, on which the label L to be inspected is affixed in the correct position, in a predetermined position for photographing using the photographing device 400.

[0023] In S110, the CPU 110 executes a process for generating a reference image RI. For example, the CPU 110 transmits a photographing instruction to the photographing device 400. In response to the photographing instruction, the photographing device 400 generates a reference image RI by photographing the product 300, and transmits data of the reference image RI to the processing device 100. As a result, the CPU 110 acquires data of the generated reference image RI.

[0024] 4A and 4B are diagrams showing examples of images used in the advance preparation process. An example of a reference image RI is shown in FIG. 4A. The reference image RI is an image showing the front of a product 300R, a label LR affixed to the front of the product 300R, and a background BR. The product and label shown in the reference image RI are referred to as product 300R and label LR, with an "R" added to the end of the reference image to distinguish them from the actual product 300 and label L (FIG. 2).

[0025] In S120, the CPU 110 saves the acquired reference image RI. For example, data of the acquired reference image RI is stored in the non-volatile storage device 130. The saved reference image RI is used in the position inspection process described later.

[0026] In S130, the CPU 110 identifies a label area LA using an object detection model AN. The label area LA is an area where the label LR is located. In this embodiment, the object detection model AN is a machine learning model including a trained convolutional neural network. For example, the object detection model AN is a machine learning model called YOLO (You only look once). YOLO is disclosed in, for example, the following paper: Alexey Bochkovskiy, Chien-Yao Wang, Hong-Yuan Mark Liao, "YOLOv4: Optimal Speed ​​and Accuracy of Object", arXiv:2004.10934, https: / / arxiv.org / abs / 2004.10934

[0027] The YOLO model predicts the area (also called a bounding box) where the object to be detected (in this example, label L) is located. The object detection model AN in this example is pre-trained to detect the label area LA where the label L is located.

[0028] In S130, the CPU 110 inputs the reference image RI to the object detection model AN. The object detection model AN uses the GPU 115 to perform calculations on the reference image RI using a trained parameter set, and outputs label area information indicating the label area LA. This identifies the label area LA ( FIG. 4A ) within the reference image RI. The label area LA is a rectangular area surrounding the label LR.

[0029] In S140, the CPU 110 saves label area information indicating the label area LA. For example, the acquired label area information is stored in the non-volatile memory device 130. The label area information includes, for example, the coordinates of the upper left vertex of the rectangular label area LA and the coordinates of the lower right vertex of the label area LA. The saved label area information is used in the position inspection process, which will be described later.

[0030] In S150, the CPU 110 generates a depth image (not shown) using the monocular depth estimation model MN. The depth image is an image that indicates the depth of each pixel of the input image. The depth is a value that corresponds to the distance from the shooting position (the position of the image capture device 400). In this embodiment, the closer the distance from the shooting position, the larger the depth value, and the farther the distance from the shooting position, the smaller the depth value. In this embodiment, the depth is a value of 256 gradations from 0 to 255.

[0031] In this embodiment, the monocular depth estimation model MN is a machine learning model including a trained convolutional neural network. For example, the monocular depth estimation model MN is a machine learning model called MiDaS. MiDaS is disclosed in, for example, the following paper: Ren? Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun, "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer", arXiv:1907.01341, https: / / doi.org / 10.48550 / arXiv.1907.01341

[0032] The MiDaS model is trained to generate a depth image that indicates the depth of each pixel in an input image.

[0033] In S150, the CPU 110 inputs the reference image RI to the monocular depth estimation model MN. The monocular depth estimation model MN uses the GPU 115 to perform calculations on the reference image RI using a trained parameter set and outputs a depth image.

[0034] In S160, the CPU 110 uses the depth image to calculate a binarization threshold value THb for identifying, in the reference image RI, a product area PA where the product 300 is located and a background area BA that is an area different from the product area PA. For example, the CPU 110 calculates the binarization threshold value THb by analyzing a histogram of the depth image.

[0035] FIG. 4B illustrates a histogram DH of the depth image. The horizontal axis indicates depth, and the vertical axis indicates the number (frequency) of pixels having each depth. For example, the histogram DH is obtained by classifying each pixel of the depth image into multiple classes according to the depth of each pixel. In FIG. 4B, for example, the histogram DH is generated by treating each of the 256 gradation values ​​(values ​​0 to 255) that the depth can take as one class. As shown in FIG. 4B, the histogram DH of the depth image is considered to have a bimodal distribution, including a first peak MT1 corresponding to the product region and a second peak MT2 corresponding to the background region.

[0036] For this reason, in this embodiment, the histogram DH is assumed to include a first peak MT1 corresponding to the product area PA and a second peak MT2 corresponding to the background area BA, and the average Dave and variance σd of the first peak MT1 are calculated. The first peak MT1 corresponding to the product area PA is the peak on the side closer to the image capture device 400 (the side with a larger depth value). The second peak MT2 corresponding to the background area BA is the peak on the side farther from the image capture device 400 (the side with a smaller depth value).

[0037] For example, it is assumed that the histogram DH of the depth image can be expressed by a Gaussian mixture model, which is a superposition of a first Gaussian distribution (also called a normal distribution) corresponding to the first peak MT1 and a second Gaussian distribution corresponding to the second peak MT2. Under this assumption, the CPU 110 calculates the mean Dave and variance σd of the first Gaussian distribution using a known likelihood estimation method, for example, the EM (Expectation-Maximization) algorithm.

[0038] The CPU 110 calculates the binarization threshold value THb as a value that is three times the variance σd away from the average Dave of the first peak MT1 (average Dave of the first Gaussian distribution) toward the second peak MT2 (FIG. 4B). That is, the binarization threshold value THb is expressed by the following equation (1): THb = (Dave - 3 × σd) (1)

[0039] In S170, the CPU 110 performs a binarization process on the depth image using a binarization threshold THb to generate a background mask image MI. For example, in the binarization process, among the multiple pixels of the depth image, pixels having a depth equal to or greater than the binarization threshold THb are classified as pixels constituting the product area PA, and pixels having a depth less than the binarization threshold THb are classified as pixels constituting the background area BA.

[0040] Fig. 4(C) shows a background mask image MI corresponding to the reference image RI in Fig. 4(A). In Fig. 4(C), a product area PA indicates the area in the reference image RI in Fig. 4(A) where the product 300R is located, and a background area BA indicates the area in the reference image RI in Fig. 4(A) where the background BR is located. In this way, by generating the background mask image MI, the product area PA and the background area BA are identified in the reference image RI.

[0041] In S180, the CPU 110 saves the generated background mask image MI and ends the advance preparation process. Specifically, data of the generated background mask image MI is stored in the non-volatile storage device 130. The saved background mask image MI is used in the position inspection process described below.

[0042] A-3. Position Inspection Processing Fig. 5 is a flowchart of the position inspection processing of the first embodiment. The position inspection processing is processing for inspecting whether or not the label L is affixed in the correct position on the product 300 to be inspected. The position inspection processing is started when a user (e.g., an inspection operator) inputs a start instruction to the processing device 100 via the operation unit 150. For example, the user inputs a start instruction for the position inspection processing while the product 300 to be inspected is placed in a predetermined position for photographing using the imaging device 400.

[0043] In S210, the CPU 110 executes a process for generating a target image SI. For example, the CPU 110 transmits a photographing instruction to the photographing device 400. In response to the photographing instruction, the photographing device 400 generates the target image SI by photographing the product 300 to be inspected, and transmits the data of the target image SI to the processing device 100. As a result, the CPU 110 acquires the data of the generated target image SI.

[0044] 6A and 6B are diagrams showing examples of images used in the position inspection process. FIG. 6A illustrates a target image SI. The target image SI is an image showing the front of a product 300S, a label LR affixed to the front of the product 300S, and a background BR. To distinguish the product and label shown in the target image SI from the actual product 300 and label L ( FIG. 2 ) and the product 300R and label LR ( FIG. 4A ) in the reference image RI, the product and label are represented as product 300S and label LS, using reference numerals with an "S" added to the end of the reference numerals.

[0045] Figure 6(B) shows the same reference image RI as Figure 4(A). The position of the product 300S in the target image SI may differ from the position of the product 300R in the reference image RI. This is because there is variation in the position at which the product 300 is placed when the product 300 to be inspected is photographed in S210. Furthermore, the position at which the label LS is affixed to the product 300S in the target image SI may differ from the position at which the label LS is affixed to the product 300R in the reference image RI. This is because there is variation in the position at which the label L is affixed to the product 300 during the manufacturing process of the product 300 to be inspected.

[0046] In S220, the CPU 110 reads out the data of the reference image RI saved in 120 in Fig. 3 from the non-volatile storage device 130. The read-out data of the reference image RI is stored in the volatile storage device 120.

[0047] In S230, a feature point extraction process is executed for each of the target image SI and the reference image RI. The feature point extraction process is a process of identifying multiple feature points in the image and calculating feature amounts of the feature points. The feature amounts are descriptors that describe the characteristics of the feature points and are expressed, for example, as vectors with a predetermined number of dimensions. A known method, specifically a method called SuperPoint, is used to identify the feature points and calculate the feature amounts. SuperPoint is disclosed, for example, in the following paper: Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich, "SuperPoint: Self-Supervised Interest Point Detection and Description", arXiv:1712.07629v4, https: / / doi.org / 10.48550 / arXiv.1712.07629

[0048] It should be noted that other methods such as SIFT (Scale-Invariant Feature Transform), DISK (DIScrete Keypoints), KAZE, and AKAZE (Accelerated-KAZE) may also be used as a method for identifying feature points and calculating feature amounts.

[0049] 5A illustrates a plurality of feature points Ps (Psb, Psl, Psp) identified in the target image SI. As shown in FIG. 5A, the identified plurality of feature points Ps includes a feature point Psb located in the background BS, a feature point Psl located on the label LS, and a feature point Psp on the product 300S (excluding the feature point on the label LS).

[0050] 5B illustrates a plurality of feature points Pr (Prb, Prl, Prp) identified in the reference image RI. As shown in FIG. 5B, the identified feature points Pr include a feature point Prb located in the background BR, a feature point Prl located on the label LR, and a feature point Prp on the product 300R (excluding the feature point on the label LR).

[0051] In addition, in Figures 5(A) and (B), only a small number of feature points are shown to avoid cluttering the figures, but in reality, many more feature points (e.g., several tens to several hundreds) are identified in both the reference image RI and the target image SI.

[0052] In S240, the CPU 110 performs feature point matching between multiple feature points in the target image SI and multiple feature points in the reference image RI. Multiple feature point pairs are determined through feature point matching. A feature point pair is a pair consisting of one feature point Ps in the target image SI and one feature point Pr in the reference image RI. The two feature points Ps and Pr constituting a feature point pair indicate similar portions of the target image SI and the reference image RI (e.g., the same portion of the product 300, the same portion of the label L, or the same portion of the background). Hereinafter, a feature point Pr in the reference image RI that constitutes a feature point pair with a specific feature point Ps in the target image SI will also be referred to as the feature point Pr corresponding to the specific feature point Ps.

[0053] The feature point matching method is a known method, specifically, a method called LightGlue. LightGlue is disclosed in, for example, the following paper: Philipp Lindenberger, Paul-Edouard Sarlin, Marc Pollefeys, "LightGlue: Local Feature Matching at Light Speed", arXiv:2306.13643v1, https: / / doi.org / 10.48550 / arXiv.2306.13643

[0054] Note that other methods such as brute force matching or SuperGlue may also be used as the feature point matching method.

[0055] 5(A) and 5(B), the two feature points Ps and Pr connected by a dashed dotted line constitute a feature point pair determined by feature point matching. Note that feature points that do not constitute a feature point pair, i.e., feature points for which no matching partner has been found, are deleted at this point.

[0056] 3 from the non-volatile storage device 130. The read data of the background mask image MI is stored in the volatile storage device 120.

[0057] In S260, the CPU 110 excludes feature points pairs corresponding to the background region from the determined plurality of feature points pairs. Specifically, the CPU 110 refers to the background mask image MI to identify the background region BA of the reference image RI ( FIG. 4C ). For each of the plurality of feature points pairs determined by the feature point matching in S240, the CPU 110 determines whether the feature points in the reference image RI are located within the background region BA. The CPU 110 identifies feature points pairs including feature points within the background region BA of the reference image RI as feature points pairs corresponding to the background region and excludes them from the processing target. Information on the excluded feature points pairs is deleted from, for example, the volatile storage device 120. In the example of FIG. 5, the feature points pair consisting of feature points Prb and Psb indicated by a square in FIGS. 5A and 5B is excluded from the processing target.

[0058] 3 from the non-volatile storage device 130. The read label area information is stored in the volatile storage device 120.

[0059] In S280, the CPU 110 classifies the plurality of feature points pairs to be processed, i.e., the plurality of feature points pairs that do not include feature points in the background region BA of the reference image RI, into label-corresponding pairs and product-corresponding pairs. A label-corresponding pair is a feature points pair that corresponds to a labeled region, and a product-corresponding pair is a feature points pair that corresponds to a portion of the product region that is different from the labeled region.

[0060] Specifically, the CPU 110 identifies a label area LA in the reference image RI by referring to the label area information ( FIG. 5B ). For each of the feature point pairs to be processed, the CPU 110 determines whether the feature point in the reference image RI is located within the label area LA. The CPU 110 classifies feature point pairs that include a feature point within the label area LA of the reference image RI as a label-corresponding pair. Feature point pairs that do not include a feature point within the label area LA of the reference image RI—in other words, feature point pairs that include a feature point located in a portion of the product area PA of the reference image RI that is different from the label area LA—as a product-corresponding pair. In the example of FIGS. 5A and 5B, the feature point pair consisting of feature points Prl and Psl, indicated by black circles in FIGS. 5A and 5B, is classified as a label-corresponding pair. The feature point pair consisting of feature points Prp and Psp, indicated by white circles in FIGS. 5A and 5B, is classified as a product-corresponding pair.

[0061] In S290, the CPU 110 selects, from the multiple feature points pairs to be processed, a feature points pair with the highest matching accuracy as the feature points pair to be used. The matching accuracy is indicated, for example, by the distance (e.g., Euclidean distance) between the feature amounts (vectors) of the two feature points that make up the feature points pair. The larger the distance, the lower the matching accuracy, and the smaller the distance, the higher the matching accuracy. The matching accuracy may also be expressed as the cosine similarity between the feature amounts (vectors) of the two feature points. The CPU 110 selects, from the multiple label-associated pairs, a predetermined number (e.g., 50) of feature points pairs in descending order of matching accuracy as the label-associated pairs to be used. The CPU 110 selects, from the multiple product-associated pairs, a predetermined number of feature points pairs in descending order of matching accuracy as the product-associated pairs to be used. As a result, a predetermined number of label-associated pairs and a predetermined number of product-associated pairs are selected as the feature points pairs to be used.

[0062] In S300, the CPU 110 calculates a coordinate transformation matrix Ha that indicates the positional relationship between the target image SI and the reference image RI using the selected predetermined number of product correspondence pairs. In this embodiment, the coordinate transformation matrix Ha is a matrix that transforms the coordinates of the target image SI so that the product 300S in the target image SI matches the product 300R in the reference image RI. In this way, the coordinate transformation matrix Ha is a matrix that transforms the coordinates of each pixel in the target image SI into coordinates in the coordinate system of the reference image RI.

[0063] FIG. 7 is an explanatory diagram of a coordinate transformation matrix. As shown in FIG. 7, the coordinate transformation matrix Ha represents the correspondence between the position Cis of a point on the target image SI and the position Cir of a point on the reference image RI. In the figure, the positions Cis and Cir are represented in so-called homogeneous coordinates (also called projected coordinates). The two values ​​x and y of the position Cis represent coordinates in the coordinate system of the target image SI. The third coordinate of the position Cis is fixed to 1. The same is true for the position Cir. The two values ​​x' and y' of the position Cir represent coordinates in the coordinate system of the reference image RI. By multiplying the coordinates (x, y, 1) of the position Cis by the coordinate transformation matrix Ha, the coordinates (x', y', 1) of the position CIr are obtained. As described above, it can be seen that the coordinate transformation matrix Ha is a matrix that transforms the coordinates of each pixel in the target image SI into coordinates in the coordinate system of the reference image RI.

[0064] As shown in FIG. 7 , the coordinate transformation matrix Ha is expressed as a matrix with 3 rows and 3 columns. Element hij indicates the element in row i and column j. The bottom right element h33 is fixed to 1. In this embodiment, the eight elements of the coordinate transformation matrix Ha are divided into three submatrices SM1, SM2, and SM3. The first submatrix SM1 is a submatrix consisting of four elements h11, h12, h21, and h22. The second submatrix SM2 is a submatrix consisting of two elements h13 and h23. The third submatrix SM3 is a submatrix consisting of two elements h31 and h32.

[0065] The four elements h11, h12, h21, and h22 of the first submatrix SM1 represent coordinate transformations that include rotation, scaling (enlargement or reduction), and skew (the latter may transform a rectangle into a parallelogram).

[0066] The two elements h13 and h23 of the second submatrix SM2 (Figure 7(A)) represent a parallel translation between a two-dimensional coordinate system representing coordinates x and y and a two-dimensional coordinate system representing coordinates x' and y'.

[0067] In this embodiment, an affine transformation matrix is ​​used as the coordinate transformation matrix Ha, and therefore the two elements h31 and h32 of the third submatrix SM3 are fixed to 0. Note that instead of an affine transformation matrix, a homography transformation matrix may be used as the coordinate transformation matrix Ha. In this case, the two elements h31 and h32 of the third submatrix SM3 may take values ​​other than 0.

[0068] The six elements h11-h13 and h21-h23 of the coordinate transformation matrix Ha can be determined by using three or more correspondences between the positions Cis and Cir, that is, three pairs of feature points.

[0069] The coordinates of each feature point constituting a feature point pair may not be completely accurate values ​​and may contain errors. In order to mitigate the influence of such errors and calculate the coordinate transformation matrix Ha with high accuracy, in this embodiment, six elements of the coordinate transformation matrix Ha (affine transformation matrix) are calculated using a predetermined number (e.g., 50) of feature point pairs that are greater than three, i.e., the predetermined number of corresponding product pairs to be used selected in S290.

[0070] The coordinate transformation matrix Ha (affine transformation matrix) is calculated by a known method, and in this embodiment, it is calculated using the least squares method. Instead of the least squares method, methods such as RANSAC (Random sample consensus) and LMedS (least-median of squares) may also be used.

[0071] In S310, the CPU 110 transforms the coordinates of the feature points of the target image SI that make up the predetermined number of label-correspondence pairs selected in S290 into the coordinate system of the reference image RI. Specifically, using the coordinate transformation matrix Ha calculated in S300, the coordinate transformation is performed according to the equation in Fig. 7. As a result, the coordinates of the two feature points that make up each label-correspondence pair are both expressed in the coordinate system of the reference image RI.

[0072] The coordinate transformation matrix Ha is calculated using a product correspondence pair so that the product 300R in the reference image RI and the product 300S in the target image SI match. Therefore, if the relative position of the label LR relative to the product 300R in the reference image RI matches the relative position of the label LS relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the label correspondence pair will match in the coordinate system of the reference image RI. If the relative position of the label LR relative to the product 300R in the reference image RI is misaligned with the relative position of the label LS relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the label correspondence pair will be misaligned in the coordinate system of the reference image RI. In other words, if the label LS is in the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the label correspondence pair will match in the coordinate system of the reference image RI. If the label LS is misaligned from the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points that make up the label-corresponding pair will be misaligned in the coordinate system of the reference image RI. This can be used to determine whether the label LS is in the correct position relative to the product 300S in the target image SI, i.e., whether the label L is affixed in the correct position relative to the product 300 being inspected.

[0073] In S320, the CPU 110 calculates the amount of misalignment between the two feature points that make up each label-correspondence pair in the coordinate system of the reference image RI. FIG. 8 is an explanatory diagram of the calculation of the amount of misalignment in the first embodiment. In FIG. 8, feature points Pr1 to Pr3 on the label LR of the reference image RI and feature points Ps1 to Ps3 on the label LS of the target image SI are plotted in the coordinate system of the reference image RI (a two-dimensional coordinate system with the X and Y axes in FIG. 8 as axes). The feature points Pr1 and Ps1 make up a label-correspondence pair. Similarly, the feature points Pr2 and Ps2, and the feature points Pr3 and Ps3, respectively, make up label-correspondence pairs.

[0074] In this embodiment, the misalignment amount is information indicating the misalignment of two feature points, and in this embodiment, includes parallel misalignment amounts Δx and Δy and an angular misalignment amount θ. The parallel misalignment amount Δx in the X direction is, for example, the difference in the X coordinates of two feature points that make up a label-corresponding pair. FIG. 8 illustrates the parallel misalignment amounts Δx1, Δx2, and Δx3 in the X direction for the label-corresponding pair Pr1 and Ps1, the label-corresponding pair Pr2 and Ps2, and the label-corresponding pair Pr3 and Ps3, respectively. The parallel misalignment amount Δy in the Y direction is the difference in the Y coordinates of two feature points that make up a label-corresponding pair. FIG. 8 illustrates the parallel misalignment amounts Δy1, Δy2, and Δy3 in the Y direction for the label-corresponding pair Pr1 and Ps1, the label-corresponding pair Pr2 and Ps2, and the label-corresponding pair Pr3 and Ps3, respectively. The parallel misalignment amounts Δx and Δy are calculated for each of a predetermined number of label-corresponding pairs.

[0075] The angular deviation θ is, for example, the angle formed by two vectors indicating the directions of two feature points constituting a label-correspondence pair. For example, a vector Vs2 ( FIG. 8 ) indicating the direction of feature point Ps2 on the label LS of the target image SI is a vector whose starting point is a reference feature point (e.g., Ps1 in FIG. 8 ) on the label LS of the target image SI and whose ending point is feature point Ps2. A vector Vr2 ( FIG. 8 ) indicating the direction of feature point Pr2 on the label LR of the reference image RI is a vector whose starting point is a reference feature point (e.g., Pr1 in FIG. 8 ) on the label LR of the reference image RI and whose ending point is feature point Pr2. The angle θ2 formed by the two vectors Vs2 and Vr2 is calculated, for example, based on a rotation matrix indicating the relationship between the two vectors. Alternatively, the angle θ2 may be calculated based on the cosine similarity of the two vectors. The angular deviation θ is calculated for each of a predetermined number of label-correspondence pairs.

[0076] As can be seen from the above explanation, the information indicating the amount of positional deviation (parallel deviation Δx, Δy and angular deviation θ) is information regarding the difference between the label LR in the reference image RI and the label LS in the target image SI, and can be said to be information regarding the difference in the relative position of the label (e.g., LR, LS) with respect to the product (e.g., 300R, 300S).

[0077] In S330, the CPU 110 determines whether the label attachment position is acceptable or unacceptable based on the calculated misalignment amounts. For example, the CPU 110 calculates average values ​​Δxa, Δya, and θa of the parallel misalignment amounts Δx, Δy, and angular misalignment amount θ calculated for each of a predetermined number of label-corresponding pairs. Note that instead of the average values, other statistical representative values, such as medians, may be calculated. The CPU 110 compares the average values ​​Δxa, Δya, and θa of the parallel misalignment amounts Δx, Δy, and angular misalignment amount θ with predetermined threshold values ​​THx, THy, and THθ, respectively. For example, if all of the average values ​​Δxa, Δya, and θa are less than the threshold value, the CPU 110 determines that the attachment position of the label L on the product 300 being inspected is acceptable. If at least one of the average values ​​Δxa, Δya, and θa is equal to or greater than the threshold value, the CPU 110 determines that the attachment position of the label L on the product 300 being inspected is unacceptable.

[0078] In S340, the CPU 110 outputs the result of the determination of the label application position and ends the position inspection process. For example, the CPU 110 displays the result of the determination of the label application position on the display unit 140.

[0079] According to the first embodiment described above, the CPU 110 acquires a reference image RI showing a product 300R including a label LR (S220 in FIG. 5 ). The CPU 110 acquires a target image SI showing a product 300S including a label LS (S210 in FIG. 5 ). The CPU 110 performs feature point matching between the reference image RI and the target image SI to determine multiple feature point pairs (S240 in FIG. 5 ). The CPU 110 calculates a coordinate transformation matrix Ha using product correspondence pairs that correspond to portions of the feature point pairs that are different from the labels LR and LS (S300 in FIG. 5 ). The CPU 110 calculates the positional deviation (parallel deviation Δx, Δy and angular deviation θ) indicating the positional difference between the label LR in the reference image RI and the label LS in the target image SI using the label correspondence pairs, which are feature point pairs corresponding to the labels LR and LS, and the coordinate transformation matrix Ha (S310 and S320 in FIG. 5 ). In this way, the positional misalignment amount of the labels LR and LS is calculated using the coordinate transformation matrix Ha calculated using the product correspondence pair and the label correspondence pair, which is information about the positions of the labels LR and LS. As a result, a positional misalignment amount that appropriately indicates the difference in the relative positions of the labels LR and LS with respect to the products 300R and 300S can be calculated. Therefore, even if the position of the product 300S in the target image SI differs from the position of the product 300R in the reference image RI, the relative position of the label LS with respect to the product 300S in the target image SI can be appropriately compared with the relative position of the label LR with respect to the product 300R in the reference image RI. Therefore, the position of the label LS included in the product 300S can be accurately inspected using the target image SI.

[0080] Furthermore, according to this embodiment, the CPU 110 identifies the product area PA and the background area BA in the reference image RI (S150-S170 in FIG. 3). The CPU 110 excludes feature points pairs corresponding to the background area from the multiple feature points pairs (S260 in FIG. 5), and calculates the coordinate transformation matrix Ha using product-corresponding pairs from the remaining feature points pairs (S320 in FIG. 5). That is, the CPU 110 calculates the coordinate transformation matrix Ha using product-corresponding pairs corresponding to portions of the product area PA that are different from the label LR, without using feature points pairs corresponding to the background area BA (S260 and S320 in FIG. 5). In this way, the background area BA is identified, and the coordinate transformation matrix Ha is calculated without using feature points pairs corresponding to the background area BA, thereby eliminating the influence of the background and allowing the coordinate transformation matrix Ha to be calculated with high accuracy.

[0081] Furthermore, according to this embodiment, the CPU 110 inputs the reference image RI into the monocular depth estimation model MN to generate a depth image (S150 in FIG. 3), and identifies the product area PA and the background area BA in the reference image RI based on the depth image (S160, S170 in FIG. 3). When the product 300 is positioned as the subject to be photographed and photographed, the photograph is usually taken with the product 300 closer to the photographing device 400 than its surroundings. Therefore, the product area PA and the background area BA can be accurately separated and identified based on the depth image obtained by inputting the reference image RI into the monocular depth estimation model MN.

[0082] Furthermore, according to this embodiment, the CPU 110 assumes that the histogram DH of the depth image has multiple peaks, including a first peak MT1 corresponding to the product area PA and a second peak MT2 corresponding to the background area BA. The CPU 110 calculates the average Dave and variance σd of the first peak MT1 and determines the binarization threshold THb using the average Dave and variance σd (S160 in FIG. 3 and FIG. 4B). The CPU 110 then binarizes the depth image using the binarization threshold THb to generate a background mask image MI (S170 in FIG. 3). This identifies the product area PA and the background area BA. When a product 300 is positioned as the subject to be photographed and a photograph is taken, a relatively large difference in depth typically occurs between the product 300 and its surroundings. For this reason, the histogram DH of the depth image is likely to be bimodal, with a first peak MT1 and a second peak MT2. Therefore, by assuming that the histogram DH has two peaks, the binarization threshold THb can be determined so as to separate the product area PA and the background area BA with high accuracy.

[0083] Furthermore, according to this embodiment, the CPU 110 inputs a reference image RI into the object detection model AN and identifies a label area LA within the reference image RI (S130 in FIG. 3). Among multiple feature point pairs, the CPU 110 identifies a product-correspondence pair, which is a feature point pair that does not include a feature point Prl within the identified label area LA (S280 in FIG. 5), and calculates a coordinate transformation matrix Ha using the product-correspondence pair (S280 and S300 in FIG. 5). As a result, by identifying the label area LA using the object detection model AN, it is possible to easily determine the product-correspondence pair to be used in calculating the coordinate transformation matrix Ha.

[0084] Furthermore, according to this embodiment, the CPU 110 selects a plurality of product correspondence pairs to be used from a plurality of product correspondence pairs using the matching accuracy of the feature point pairs (S290 in FIG. 5). The CPU 110 calculates the coordinate transformation matrix Ha using the plurality of product correspondence pairs to be used (S300 in FIG. 5). As a result, the coordinate transformation matrix Ha can be calculated using product correspondence pairs that are likely to show a correct correspondence relationship between feature points, and therefore, a highly accurate coordinate transformation matrix Ha can be calculated.

[0085] Furthermore, according to this embodiment, the CPU 110 calculates the amount of misalignment (parallel misalignment Δx, Δy and angular misalignment θ) using a label-corresponding pair corresponding to the labels LS, LS among the multiple feature point pairs (S320 in FIG. 5). As a result, the amount of misalignment can be calculated based on the misalignment of the feature points of the labels LS, LS that make up the feature point pair.

[0086] Furthermore, according to this embodiment, the CPU 110 selects multiple label-correspondence pairs to be used from the multiple label-correspondence pairs using the matching accuracy of the feature point pairs (S290 in FIG. 5). The CPU 110 calculates the amount of misregistration using the multiple label-correspondence pairs to be used (S320 in FIG. 5). As a result, the amount of misregistration can be calculated using label-correspondence pairs that are likely to show a correct correspondence between feature points, allowing the amount of misregistration to be calculated with high accuracy.

[0087] Furthermore, according to this embodiment, the CPU 110 converts the coordinates of the feature points in the target image SI that make up the label-correspondence pair into coordinates in the coordinate system of the reference image RI using the coordinate transformation matrix Ha (S310 in FIG. 5). The CPU 110 then calculates the amount of misalignment between the two feature points that make up the label-correspondence pair in the coordinate system of the reference image RI (S320 in FIG. 5). As a result, the relative position of the label LS with respect to the product 300S in the target image SI and the relative position of the label LR with respect to the product 300R in the reference image RI can be compared in the same coordinate system, allowing an appropriate amount of misalignment to be calculated.

[0088] As can be seen from the above explanation, the labels L, LR, and LS in this embodiment are examples of first objects, and the products 300, 300R, and 300S are examples of second objects. The positional deviation information (parallel deviation amounts Δx and Δy and angular deviation amount θ) in this embodiment is an example of difference information, and the coordinate transformation matrix Ha is an example of a first coordinate transformation matrix. The label-correspondence pairs in this embodiment are an example of identification information.

[0089] B. Second Example In the second example, the content of the position inspection process differs from that of the first example shown in Fig. 5. The hardware configuration (Fig. 1) of the inspection system 1000 in the second example and the content of the advance preparation process (Fig. 3) are the same as those in the first example.

[0090] FIG. 9 is a flowchart of the position inspection process of the second embodiment. In S200B, the CPU 110 executes S210 to S290 of FIG. 5. In S300B, the CPU 110 calculates a coordinate transformation matrix Ht using a predetermined number of selected product correspondence pairs. In this embodiment, the coordinate transformation matrix Ht is a matrix that transforms the coordinates of the reference image RI so that the product 300R in the reference image RI matches the product 300S in the target image SI. In this manner, the coordinate transformation matrix Ht is a matrix that transforms the coordinates of each pixel in the reference image RI into coordinates in the coordinate system of the target image SI. In other words, the coordinate transformation matrix Ht calculated in S300B of the second embodiment is the inverse matrix of the coordinate transformation matrix Ha calculated in S300 of the first embodiment.

[0091] The coordinate transformation matrix Ht calculated in the second embodiment is an affine transformation matrix, similar to the coordinate transformation matrix Ha calculated in the first embodiment. The coordinate transformation matrix Ht (affine transformation matrix) is calculated using a known method, such as the least squares method, similar to the first embodiment.

[0092] In S310B, the CPU 110 transforms the coordinates of the feature points of the reference image RI that make up the predetermined number of label-correspondence pairs selected in S290 into the coordinate system of the target image SI. Specifically, the coordinate transformation is performed using the coordinate transformation matrix Ht calculated in S300B. As a result, the coordinates of the two feature points that make up each label-correspondence pair are both expressed in the coordinate system of the target image SI.

[0093] Like the coordinate transformation matrix Ha in the first embodiment, the coordinate transformation matrix Ht is calculated using a product correspondence pair, so that the product 300R in the reference image RI and the product 300S in the target image SI match. Therefore, if the label LS is in the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points that make up the label correspondence pair will match in the coordinate system of the target image SI. If the label LS is shifted from the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points that make up the label correspondence pair will be shifted in the coordinate system of the target image SI. This can be used to determine whether the label LS is in the correct position relative to the product 300S in the target image SI, i.e., whether the label L is affixed to the product 300 being inspected in the correct position.

[0094] In S320B, the CPU 110 calculates the positional shift amounts of the two feature points constituting each label-correspondence pair in the coordinate system of the target image SI. As in the first embodiment, the positional shift amounts include the parallel shift amounts Δx and Δy and the angular shift amount θ, and are calculated for each of a predetermined number of label-correspondence pairs according to the method described in the first embodiment with reference to FIG.

[0095] In S330B, the CPU 110 determines whether the label application position is acceptable or not based on the calculated amount of misalignment. The method of determining whether the label application position is acceptable or not is the same as in S330 of FIG.

[0096] In S340B, similarly to S340 in FIG. 5, the CPU 110 outputs the determination result of the label attachment position, and ends the position inspection process.

[0097] According to the second embodiment described above, the CPU 110 converts the coordinates of the feature points in the reference image RI that make up the label-correspondence pair into coordinates in the coordinate system of the target image SI using the coordinate transformation matrix Ht (S310B in FIG. 9). The CPU 110 then calculates the amount of misalignment between the two feature points that make up the label-correspondence pair in the coordinate system of the target image SI (S320B in FIG. 9). As a result, as in the first embodiment, the relative position of the label LS with respect to the product 300S in the target image SI and the relative position of the label LR with respect to the product 300R in the reference image RI can be compared in the same coordinate system, allowing an appropriate amount of misalignment to be calculated.

[0098] C. Third Embodiment In the third embodiment, the content of the position inspection process differs from the position inspection process of the first embodiment shown in Fig. 5. The hardware configuration of the inspection system 1000 of the third embodiment (Fig. 1) and the content of the advance preparation process (Fig. 3) are the same as those of the first embodiment.

[0099] 10 is a flowchart of the position inspection process of Example 3. In S200C, the CPU 110 executes S210 to S290 of FIG.

[0100] 5, the CPU 110 calculates a coordinate transformation matrix Ha using the predetermined number of product correspondence pairs selected in S290. As in the first embodiment, the coordinate transformation matrix Ha is a matrix that transforms the coordinates of the target image SI so that the product 300S in the target image SI matches the product 300R in the reference image RI. In this way, the coordinate transformation matrix Ha is a matrix that transforms the coordinates of each pixel in the target image SI into coordinates in the coordinate system of the reference image RI.

[0101] In S310C, the CPU 110 calculates a coordinate transformation matrix Hb using the predetermined number of label correspondence pairs selected in S290 of Fig. 5. Unlike the coordinate transformation matrix Ha, the coordinate transformation matrix Hb is calculated using label correspondence pairs, and is therefore a matrix that transforms the coordinates of the target image SI so that the label LS of the target image SI matches the label LR of the reference image RI.

[0102] Like the coordinate transformation matrix Ha, the coordinate transformation matrix Hb is an affine transformation matrix with 3 rows and 3 columns. Like the coordinate transformation matrix Ha, the coordinate transformation matrix Hb (affine transformation matrix) is calculated using a known method, specifically, the least squares method. Instead of the least squares method, methods such as RANSAC (Random sample consensus) and LMedS (least-median of squares) may be used.

[0103] The coordinate transformation matrix Ha and the coordinate transformation matrix Hb are both matrices that transform the coordinates of each pixel in the target image SI into coordinates in the coordinate system of the reference image RI.

[0104] The coordinate transformation matrix Ha is calculated using a product correspondence pair so that the product 300R in the reference image RI matches the product 300S in the target image SI. The coordinate transformation matrix Hb is calculated using a label correspondence pair so that the label LR in the reference image RI matches the label LS in the target image SI. Therefore, if the relative position of the label LR relative to the product 300R in the reference image RI matches the relative position of the label LS relative to the product 300S in the target image SI, the coordinate transformation matrix Ha and the coordinate transformation matrix Hb are the same matrix. If the relative position of the label LR relative to the product 300R in the reference image RI matches the relative position of the label LS relative to the product 300S in the target image SI, the coordinate transformation matrices Ha and Hb have element values ​​that differ by an amount corresponding to the misalignment. By utilizing this and comparing the coordinate transformation matrix Ha with the coordinate transformation matrix Hb, it is possible to determine whether the label LS is in the correct position relative to the product 300S in the target image SI, i.e., whether the label L is attached in the correct position relative to the product 300 being inspected.

[0105] In S320C, the CPU 110 calculates difference information indicating the difference between the coordinate transformation matrices Ha and Hb. As shown in FIG. 7 , a total of six elements h11, h12, h13, h21, h22, and h23 of the first submatrix SM1 and the second submatrix SM2 of the coordinate transformation matrices may differ between the coordinate transformation matrices Ha and Hb. For this reason, for example, the mean square error of the six elements of the two coordinate transformation matrices Ha and Hb may be calculated as the difference information. Alternatively, for example, the mean absolute error of the six elements may be calculated as the difference information, or the two coordinate transformation matrices Ha and Hb may be analyzed to calculate the error in the translation component and the error in the angle component.

[0106] The difference information indicating the difference between the two coordinate transformation matrices Ha and Hb in this embodiment is information regarding the difference between the label LR in the reference image RI and the label LS in the target image SI, similar to the positional deviation amounts (parallel deviation amounts Δx, Δy and angular deviation amount θ) in the first and second embodiments, and can be said to be information regarding the difference in the relative position of the label (e.g., LR, LS) with respect to the product (e.g., 300R, 300S).

[0107] In S330C, the CPU 110 determines whether the label attachment position is acceptable or unacceptable based on the calculated difference information. For example, the CPU 110 compares the mean square error calculated as the difference information with a predetermined threshold. For example, if the mean square error is less than the threshold, the CPU 110 determines that the attachment position of the label L on the product 300 being inspected is acceptable. If the mean square error is equal to or greater than the threshold, the CPU 110 determines that the attachment position of the label L on the product 300 being inspected is unacceptable.

[0108] In S340C, the CPU 110 outputs the determination result of the label attachment position, similar to S340 (FIG. 5) in the first embodiment, and ends the position inspection process.

[0109] According to the third embodiment described above, the CPU 110 calculates a coordinate transformation matrix Ha using the product-corresponding pair corresponding to the products 300R and 300S among the multiple feature point pairs (S300C in FIG. 10 ), and calculates a coordinate transformation matrix Hb using the label-corresponding pair corresponding to the labels LR and LS (S310C in FIG. 10 ). The CPU 110 calculates difference information indicating the difference between the coordinate transformation matrix Ha and the coordinate transformation matrix Hb (S320C in FIG. 10 ). As a result, the difference between the relative position of the label LS with respect to the product 300S in the target image SI and the relative position of the label LR with respect to the product 300R in the reference image RI can be evaluated by comparing the two coordinate transformation matrices. Therefore, for example, even if the position of product 300S in target image SI is different from the position of product 300R in reference image RI, the relative position of label LS to product 300S in target image SI can be appropriately compared with the relative position of label LR to product 300R in reference image RI.

[0110] As can be seen from the above explanation, the coordinate transformation matrix Ha in the third embodiment is an example of a first coordinate transformation matrix, and the coordinate transformation matrix Hb is an example of a second coordinate transformation matrix.

[0111] D. Fourth Embodiment In the fourth embodiment, the content of the position inspection process differs from that of the first embodiment shown in Fig. 5. The hardware configuration (Fig. 1) of the inspection system 1000 of the fourth embodiment and the content of the advance preparation process (Fig. 3) are the same as those of the first embodiment.

[0112] FIG. 11 is a flowchart of the position inspection process of the fourth embodiment. In S200D, the CPU 110 executes S210 to S290 of FIG. 5. In S300D, the CPU 110 calculates a coordinate transformation matrix Hb using a predetermined number of label-correspondence pairs selected in S290 of FIG. 5. Unlike the first embodiment, the coordinate transformation matrix Hb is calculated using label-correspondence pairs, and is therefore a matrix that transforms the coordinates of the target image SI so that the label LS of the target image SI matches the label LR of the reference image RI. In this way, the coordinate transformation matrix Hb is a matrix that transforms the coordinates of each pixel of the target image SI into coordinates in the coordinate system of the reference image RI. Like the coordinate transformation matrix H of FIG. 7, the coordinate transformation matrix Hb is a 3-row, 3-column affine transformation matrix.

[0113] The coordinate transformation matrix Hb (affine transformation matrix) is calculated using a known method, specifically, the least squares method, as in the first embodiment. Instead of the least squares method, methods such as RANSAC (Random sample consensus) and LMedS (least-median of squares) may also be used.

[0114] In S310D, the CPU 110 transforms the coordinates of the feature points of the target image SI that make up the predetermined number of product-corresponding pairs selected in S290 of Fig. 5 into the coordinate system of the reference image RI. Specifically, the coordinate transformation is performed using the coordinate transformation matrix Hb calculated in S300D. As a result, the coordinates of both of the two feature points that make up each product-corresponding pair are expressed in the coordinate system of the reference image RI.

[0115] The coordinate transformation matrix Hb is calculated using a label correspondence pair, so that the label LR in the reference image RI and the label LS in the target image SI match. Therefore, if the relative position of the label LR relative to the product 300R in the reference image RI matches the relative position of the label LS relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the product correspondence pair will match in the coordinate system of the reference image RI. If the relative position of the label LR relative to the product 300R in the reference image RI is misaligned with the relative position of the label LS relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the product correspondence pair will be misaligned in the coordinate system of the reference image RI. In other words, if the label LS is in the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the product correspondence pair will match in the coordinate system of the reference image RI. If the label LS is misaligned from the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points that make up the product-corresponding pair will be misaligned in the coordinate system of the reference image RI. This can be used to determine whether the label LS is in the correct position relative to the product 300S in the target image SI, i.e., whether the label L is affixed in the correct position relative to the product 300 being inspected.

[0116] In S320D, the CPU 110 calculates the amount of misalignment between the two feature points constituting each product-corresponding pair in the coordinate system of the reference image RI. FIG. 12 is an explanatory diagram of the calculation of the amount of misalignment in the fourth embodiment. In FIG. 12, feature points Pr4 to Pr6 on the product 300R (excluding the label LR) in the reference image RI and feature points Ps4 to Ps6 on the product 300S (excluding the label LS) in the target image SI are plotted in the coordinate system of the reference image RI (a two-dimensional coordinate system with the X and Y axes in FIG. 12 as axes). In the second embodiment, the coordinate transformation matrix Hb is calculated using the label-corresponding pair, so that the label LS of the target image SI matches the label LR of the reference image RI, as shown in FIG. 12. The feature points Pr4 and Ps4 form a product-corresponding pair. Similarly, the feature points Pr5 and Ps5, and the feature points Pr6 and Ps6, respectively, form a product-corresponding pair.

[0117] In the second embodiment, the misalignment amount includes parallel misalignment amounts Δx and Δy in the X and Y directions of the two feature points that make up the product-corresponding pair, and an angular misalignment amount θ formed by two vectors that indicate the directions of the two feature points that make up the product-corresponding pair. The parallel misalignment amounts Δx and Δy and the angular misalignment amount θ are calculated using the same method as that described for the two feature points that make up the label-corresponding pair in the first embodiment ( FIG. 8 ). The parallel misalignment amounts Δx and Δy and the angular misalignment amount θ are calculated for each of a predetermined number of product-corresponding pairs.

[0118] As can be seen from the above explanation, similar to the first embodiment, the information indicating the positional deviation amount in the second embodiment (parallel deviation amount Δx, Δy and angular deviation amount θ) is information regarding the difference between the label LR in the reference image RI and the label LS in the target image SI, and can be said to be information regarding the difference in the relative position of the label (e.g., LR, LS) with respect to the product (e.g., 300R, 300S).

[0119] In S330D, the CPU 110 determines whether the label attachment position is acceptable based on the calculated misalignment amounts. For example, the CPU 110 calculates average values ​​Δxa, Δya, and θa of the parallel misalignment amounts Δx, Δy, and angular misalignment amount θ calculated for each of a predetermined number of product-corresponding pairs. Note that instead of the average values, other statistical representative values, such as medians, may be calculated. As in S330 ( FIG. 5 ) of the first embodiment, the CPU 110 compares the average values ​​Δxa, Δya, and θa of the parallel misalignment amounts Δx, Δy, and angular misalignment amount θ with predetermined threshold values ​​THx, THy, and THθ, respectively. For example, if all of the average values ​​Δxa, Δya, and θa are less than the threshold values, the CPU 110 determines that the attachment position of the label L on the product 300 being inspected is acceptable. If at least one of the average values ​​Δxa, Δya, and θa is equal to or greater than the threshold value, the CPU 110 determines that the position of the label L attached to the product 300 being inspected is unacceptable.

[0120] In S340D, the CPU 110 outputs the determination result of the label application position, as in S340 (FIG. 5) in the first embodiment, and ends the position inspection process. For example, the CPU 110 displays the determination result of the label application position on the display unit 140.

[0121] According to the fourth embodiment described above, the CPU 110 acquires a reference image RI showing a product 300R including a label LR (S200D in FIG. 11 , S220 in FIG. 5 ). The CPU 110 acquires a target image SI showing a product 300S including a label LS (S200D in FIG. 11 , S210 in FIG. 5 ). The CPU 110 performs feature point matching between the reference image RI and the target image SI to determine multiple feature point pairs (S200D in FIG. 11 , S240 in FIG. 5 ). The CPU 110 calculates a coordinate transformation matrix Hb using a label-corresponding pair corresponding to the labels LR and LS among the multiple feature point pairs (S300D in FIG. 11 ). The CPU 110 calculates the misalignment amount (parallel misalignment amount Δx, Δy and angular misalignment amount θ) indicating the difference in position between the label LR in the reference image RI and the label LS in the target image SI using the product-correspondence pair, which is a feature point pair corresponding to a portion different from the labels LR and LS, and the coordinate transformation matrix Hb (S310D, S320D in FIG. 11 ). In this way, the misalignment amount of the labels LR and LS is calculated using the coordinate transformation matrix Hb calculated using the label-correspondence pair and the product-label-correspondence pair, which is information regarding the position of the portion different from the labels LR and LS (products 300R and 300S). As a result, as in the first embodiment, the misalignment amount can be calculated that appropriately indicates the difference in the relative position of the labels LR and LS with respect to the products 300R and 300S. Therefore, even if the position of the product 300S in the target image SI differs from the position of the product 300R in the reference image RI, the relative position of the label LS to the product 300S in the target image SI can be appropriately compared with the relative position of the label LR to the product 300R in the reference image RI. Therefore, the position of the label LS included in the product 300S can be inspected with high accuracy using the target image SI.

[0122] Furthermore, according to the fourth embodiment, the CPU 110 identifies the product area PA and the background area BA in the reference image RI (S200D in FIG. 11 , S150-S170 in FIG. 3 ). The CPU 110 excludes feature points pairs corresponding to the background area from the multiple feature points pairs (S200D in FIG. 11 , S260 in FIG. 5 ), and calculates the misregistration amount using product-corresponding pairs from the remaining feature points pairs (S320D in FIG. 11 ). That is, the CPU 110 calculates the difference information (misregistration amount) using product-corresponding pairs corresponding to portions of the product area PA that are different from the label LR, without using feature points pairs corresponding to the background area BA (S260 in FIG. 5 , S320D in FIG. 11 ). In this way, the background area BA is identified and the difference information is calculated without using feature points pairs corresponding to the background area BA, thereby eliminating the influence of the background and allowing the difference information to be calculated with high accuracy.

[0123] Furthermore, according to the fourth embodiment, the CPU 110 inputs a reference image RI into the object detection model AN and identifies a label area LA within the reference image RI (S200D in FIG. 11 , S130 in FIG. 3 ). Among multiple feature point pairs, the CPU 110 identifies a label-correspondence pair, which is a feature point pair that includes a feature point Prl within the identified label area LA (S200D in FIG. 11 , S280 in FIG. 5 ), and calculates a coordinate transformation matrix Hb using the label-correspondence pair (S300 in FIG. 11 ). As a result, by identifying the label area LA using the object detection model AN, it is possible to easily determine the label-correspondence pair to be used in calculating the coordinate transformation matrix Hb.

[0124] Furthermore, according to this embodiment, the CPU 110 calculates the amount of misalignment (parallel misalignment Δx, Δy and angular misalignment θ) using a product-corresponding pair, which is a feature point pair that corresponds to a portion different from the labels LS and LR, among the multiple feature point pairs (S320D in FIG. 11). As a result, the amount of misalignment can be calculated based on the misalignment of the feature points of the products 300S and 300R that make up the feature point pair.

[0125] As can be seen from the above explanation, the labels L, LR, and LS in this embodiment are examples of first objects, and the products 300, 300R, and 300S are examples of second objects. The positional deviation information (parallel deviation amounts Δx and Δy and angular deviation amount θ) in this embodiment is an example of difference information, and the coordinate transformation matrix Hb is an example of a first coordinate transformation matrix. The product correspondence pairs in this embodiment are an example of identification information.

[0126] E. Modifications (1) In the above embodiment, the advance preparation process of Fig. 3 is performed only once, and the position inspection process of Fig. 5 is performed for each of the multiple products 300 to be inspected. Alternatively, the advance preparation process of Fig. 3 and the position inspection process of Fig. 5 may be performed as a whole for each of the multiple products 300.

[0127] (2) In the above embodiment, the background area BA and the product area PA are identified in the reference image RI (S150 to S170 in FIG. 3). Alternatively, in the position inspection process, the same processes as S150 to S170 in FIG. 3 may be performed on the target image SI to identify the background area and the product area. In this case, for example, in S260 of the position inspection process in FIG. 5, feature point pairs including feature points located in the background area of ​​the target image SI may be excluded from the feature point pairs to be used. Furthermore, the background area and the product area may be identified in both the target image SI and the reference image RI. In this case, for example, in S260 of the position inspection process in FIG. 5, pairs of feature points located in the background area of ​​the target image SI and feature points located in the background area of ​​the reference image RI may be excluded from the feature point pairs to be used. However, as in the above embodiments, when the background area BA and the product area PA are identified only for the reference image RI, the background area BA and the product area PA only need to be identified once in the preparatory processing, so when inspecting the position of the label L of multiple products 300 using multiple target images SI, the amount of processing required for inspection can be reduced.

[0128] (3) In the above embodiment, the label area LA is identified in the reference image RI (S130 in FIG. 3 ). Alternatively, in the position inspection process, the label area may be identified by performing a process similar to S130 in FIG. 3 on the target image SI. In this case, for example, in S280 of the position inspection process in FIG. 5 , feature point pairs including feature points located in the label area of ​​the target image SI may be classified as label-corresponding pairs, and feature point pairs not including feature points located in the label area of ​​the target image SI may be classified as product-corresponding pairs. Alternatively, the label area may be identified in both the target image SI and the reference image RI. In this case, for example, in S280 of the position inspection process in FIG. 5 , pairs of feature points located in the label area of ​​the target image SI and feature points located in the label area of ​​the reference image RI may be classified as label-corresponding pairs, and other feature point pairs may be classified as product-corresponding pairs. However, when the label area LA is identified only for the reference image RI, as in the above embodiments, it is only necessary to identify the label area LA once in the preparatory process. Therefore, when inspecting the positions of the labels L of multiple products 300 using multiple target images SI, the amount of processing required for inspection can be reduced.

[0129] (4) In the first embodiment, the misalignment amount is calculated using a label-corresponding pair, which is a feature point corresponding to the labels LR and LS (S310 and S320 in FIG. 5 ). Alternatively, the misalignment amount may be calculated using rectangular labeled areas identified in the target image SI and the reference image RI. In this case, for example, the CPU 110 identifies labeled areas in each of the target image SI and the reference image RI using the object detection model described above. The CPU 110 transforms the positional information of the labeled area in the target image SI (e.g., the four vertices of the rectangular labeled area) into the coordinate system of the reference image RI using the coordinate transformation matrix Ha. The CPU 110 calculates the misalignment amount indicating the misalignment between the labeled area in the target image SI and the labeled area in the reference image RI in the coordinate system of the reference image RI. In this modification, the positional information of the labeled areas identified in each of the target image SI and the reference image RI is an example of the identification information.

[0130] (5) In the fourth embodiment, the misalignment amount is calculated using a product correspondence pair, which are feature points corresponding to the products 300R and 300S (S310D and S320D in FIG. 11 ). This is not limiting, and the misalignment amount may be calculated using, for example, product regions identified in the target image SI and the reference image RI. In this case, for example, the CPU 110 identifies the product regions in each of the target image SI and the reference image RI using the monocular depth estimation model MN described above. The CPU 110 converts position information of the product region in the target image SI (e.g., coordinates of the edge indicating the boundary between the product region and the background region) into the coordinate system of the reference image RI using the coordinate transformation matrix Ha. The CPU 110 calculates the misalignment amount indicating the misalignment between the product region in the target image SI and the product region in the reference image RI in the coordinate system of the reference image RI. In this modification, the position information of the product regions identified in each of the target image SI and the reference image RI is an example of identification information.

[0131] (6) In the fourth embodiment described above, the CPU 110 transforms the coordinates of the feature points of the target image SI that constitute the product-corresponding pair into coordinates in the coordinate system of the reference image RI using the coordinate transformation matrix Hb (S310D in FIG. 11), and calculates the positional shift of the two feature points that constitute the product-corresponding pair in the coordinate system of the reference image RI (S320D in FIG. 11). Alternatively, the CPU 110 may transform the coordinates of the feature points of the reference image RI that constitute the product-corresponding pair into coordinates in the coordinate system of the target image SI using the inverse matrix of the coordinate transformation matrix Hb, and calculate the positional shift of the feature points that constitute the product-corresponding pair in the coordinate system of the target image SI.

[0132] (7) In the above embodiment, the CPU 110 identifies the label area LA of the reference image RI using the object detection model AN (S130 in FIG. 3). Alternatively, the CPU 110 may identify the label area LA of the reference image RI using another method. The other method may be, for example, a method of identifying a rectangular label area using pattern matching using pattern data that indicates the characteristics of the label L.

[0133] (8) In the above embodiment, the CPU 110 identifies the background area BA and the product area PA of the reference image RI using the monocular depth estimation model MN (S150 in FIG. 3). Alternatively, the CPU 110 may identify the background area BA and the product area PA of the reference image RI using another method. The other method may be a method of identifying the product area using pattern matching using pattern data that indicates the characteristics of the product 300, and identifying the area surrounding the product area as the background area.

[0134] (9) Some of the processes in the advance preparation process and position inspection process in each of the above embodiments may be omitted as appropriate. For example, if the target image SI is captured so as to include only the front surface 31 of the product 300 and not the background, or if the area of ​​the background is extremely small, steps S150 to S180 in Fig. 3 related to identifying the background area BA may be omitted, and S260 in Fig. 5 may also be omitted. Furthermore, the process of selecting feature point pairs to be used using the matching accuracy (S290 in Fig. 5) may be omitted, and the processes of S300 to S320 in Fig. 5 may be performed using all feature point pairs.

[0135] (10) In S290 of FIG. 5 , a predetermined number of feature point pairs with the highest matching accuracy are selected as feature point pairs to be used from among multiple feature point pairs. Instead of matching accuracy, feature point pairs to be used may be selected based on another index. For example, the target image SI and the reference image RI are both images obtained by photographing the same type of product 300 and label L. For this reason, for example, the coordinate difference (ΔX1s, ΔX1r) between the two feature points constituting a product-corresponding pair is likely to be relatively close for each of the multiple product-corresponding pairs. For example, if the coordinate difference for one product-corresponding pair is significantly different from the coordinate difference for the other product-corresponding pairs, the matching of that product-corresponding pair is likely to be incorrect. For this reason, for example, the CPU 110 may calculate the coordinate difference for each of the multiple product-corresponding pairs and exclude product-corresponding pairs whose coordinate difference is greater than a predetermined amount (e.g., three times the variance σ) from the average of the coordinate differences. The same applies to label-corresponding pairs.

[0136] (11) Furthermore, the preparatory process and position inspection process in each of the above embodiments can be automatically executed by the CPU 110. Alternatively, some of the preparatory process and position inspection process may include processes that require operator operation. For example, in the preparatory process of FIG. 3, the identification of the label area LA in the reference image RI (S130 in FIG. 3) is automatically executed using the object detection model AN. Alternatively, for example, an operator may operate a pointing device such as a mouse to specify the label area LA on the reference image RI displayed on the display unit 140, and the CPU 110 may identify the area specified by the operator as the label area LA. Similarly, the processes of S150 to S170 in FIG. 3 for identifying the background area BA and the product area PA in the reference image RI may also be similarly specified based on the operator's designation.

[0137] (12) The method of calculating the binarization threshold value THb for generating the background mask image MI (S160 in FIG. 3) described with reference to FIG. 4B is an example and is not limiting. The binarization threshold value THb may be calculated using other methods, such as Otsu's binarization method.

[0138] (13) The object detection model AN and the monocular depth estimation model MN used in the above embodiment are merely examples, and other models may be used. For example, the object detection model AN may be another YOLO model such as "YOLO v3" or "YOLO v5" instead of the "YOLO v4" ​​model. Furthermore, other models such as SSD R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN may be used as the object detection model AN.

[0139] Similarly, the monocular depth estimation model MN may be another model instead of the MiDaS model. For example, the monocular depth estimation model MN may be a model called SfMLearner or a model called ZoeDepth (Zero-shot Transfer by Combining Relative and Metric Depth).

[0140] (14) The second object to be inspected is not limited to a printer, but may be other objects such as a multifunction peripheral, a sewing machine, a cutting machine, or a mobile terminal. Furthermore, the first object whose position relative to the second object is inspected is not limited to a label attached to a product, but may be any object. For example, the first object may be a label image printed on the first object, or any part of a product, such as a tag, accessory, part, stamp, or nameplate attached to the first object.

[0141] (15) In the above embodiment, the pre-preparation process and the position inspection process are executed by the processing device 100 of FIG. 1 . Alternatively, the pre-preparation process and the position inspection process may each be executed by a separate device. In this case, for example, data on the reference image RI, label area LA, and background mask image MI prepared by the pre-preparation process are stored in a storage device of the device that executes the position inspection process. Furthermore, all or part of the pre-preparation process and the position inspection process may be executed by multiple computers (e.g., so-called cloud servers) that can communicate with each other via a network. Furthermore, the computer program that performs the position inspection process and the computer program that performs the pre-preparation process may be different computer programs.

[0142] (16) In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, all or part of the advance preparation process and the position inspection process may be executed by a hardware circuit such as an ASIC (Application Specific Integrated Circuit).

[0143] The present invention has been described above based on examples and modifications, but the above-described embodiments of the invention are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit and scope of the claims, and equivalents thereof are also included in the present invention.

[0144] 1000... inspection system, 100... processing device, 110... CPU, 120... volatile storage device, 130... non-volatile storage device, 140... display unit, 150... operation unit, 170... communication unit, 300, 300S, 300R... product, 400... imaging device, AN... object detection model, Ha, Hb, Ht... coordinate transformation matrix, L, LR, LS... label, MI... background mask image, MN... monocular depth estimation model, PG... computer program, RI... reference image, SI... target image

Claims

1. A computer program that causes a computer to realize the following: a reference image acquisition function that acquires a reference image showing a second object including a first object; a target image acquisition function that acquires a target image showing the second object including the first object; a pair determination function that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation function that calculates a first coordinate transformation matrix that indicates the positional relationship between the reference image and the target image, using feature point pairs among the plurality of feature point pairs that correspond to parts different from the first object; and a difference information calculation function that calculates difference information between the first object in the reference image and the first object in the target image, using specific information regarding the position of the first object in each of the reference image and the target image and the first coordinate transformation matrix, wherein the difference information includes information regarding the difference in the relative position of the first object with respect to the second object.

2. A computer program as claimed in claim 1, further comprising: a computer-implemented area identification function for identifying an object area in which the second object is located and a background area different from the object area in at least one of the reference image and the target image; and a matrix calculation function for calculating the first coordinate transformation matrix using the feature point pair corresponding to a part of the object area different from the first object, without using the feature point pair corresponding to the background area.

3. A computer program as claimed in claim 2, further comprising: the region identification function inputting at least one of the reference image and the target image into a monocular depth estimation model to generate a depth image; and identifying the object region and the background region in at least one of the reference image and the target image based on the depth image.

4. A computer program as described in claim 3, wherein the region identification function calculates the mean and variance of the first peaks, assuming that the histogram of the depth image has a plurality of peaks including a first peak corresponding to the object region and a second peak corresponding to the background region; determines a binarization threshold using the mean and variance of the first peaks; and identifies the object region and the background region by binarizing the depth image using the binarization threshold.

5. A computer program as described in claim 2, wherein the region identification function identifies the object region and the background region in the reference image, and the matrix calculation function calculates the first coordinate transformation matrix using the feature point pair corresponding to a portion of the object region of the reference image that is different from the first object, without using the feature point pair corresponding to the background region of the reference image.

6. A computer program as claimed in claim 1, further comprising: causing a computer to implement an object identification function that inputs at least one of the reference image and the target image into an object detection model and identifies an area in the at least one image where the first object is located; and wherein the matrix calculation function calculates the first coordinate transformation matrix using, of the plurality of feature point pairs, feature point pairs that do not include feature points in the area where the identified first object is located.

7. A computer program as claimed in claim 1, wherein a pair selection function is implemented in a computer to select a plurality of pairs to be used from a plurality of feature point pairs corresponding to parts different from the first object using the matching accuracy of the feature point pairs, and wherein the matrix calculation function calculates the first coordinate transformation matrix using the selected plurality of use pairs.

8. A computer program according to claim 1, wherein the specific information is information on the feature point pair corresponding to the first object among the plurality of feature point pairs.

9. A computer program according to claim 8, wherein the difference information calculation function uses the first coordinate transformation matrix to transform the coordinates of the feature points of the target image that make up the feature points pair corresponding to the first object into coordinates in the coordinate system of the reference image, and calculates, as the difference information, information indicating the positional deviation between the two feature points that make up the feature points pair corresponding to the first object in the coordinate system of the reference image.

10. A computer program according to claim 8, wherein the difference information calculation function uses the first coordinate transformation matrix to transform the coordinates of the feature points of the reference image that make up the feature points pair corresponding to the first object into coordinates in the coordinate system of the target image, and calculates, as the difference information, information indicating the positional deviation between the two feature points that make up the feature points pair corresponding to the first object in the coordinate system of the target image.

11. A computer program as described in claim 8, wherein the difference information calculation function calculates a second coordinate transformation matrix indicating the positional relationship between the reference image and the target image using the feature point pair corresponding to the first object among the plurality of feature point pairs, and calculates information indicating the difference between the first coordinate transformation matrix and the second coordinate transformation matrix as the difference information.

12. A computer program that causes a computer to realize the following: a reference image acquisition function that acquires a reference image showing a second object including a first object; a target image acquisition function that acquires a target image showing the second object including the first object; a pair determination function that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation function that calculates a first coordinate transformation matrix that indicates the positional relationship between the reference image and the target image, using a feature point pair corresponding to the first object out of the plurality of feature point pairs; and a difference information calculation function that calculates difference information between the first object in the reference image and the first object in the target image, using specific information regarding the positions of parts of the reference image and the target image that differ from the first object and the first coordinate transformation matrix, the difference information including information regarding the difference in the relative position of the first object with respect to the second object.

13. A computer program as described in claim 12, further comprising: a computer-implemented area identification function that identifies an object area in which the second object is located and a background area different from the object area in at least one of the reference image and the target image; and a difference information calculation function that calculates the difference information using the identification information corresponding to a portion of the object area that is different from the first object, without using the identification information corresponding to the background area.

14. A computer program as described in claim 13, further comprising: the region identification function inputting at least one of the reference image and the target image into a monocular depth estimation model to generate a depth image; and identifying the object region and the background region in at least one of the reference image and the target image based on the depth image.

15. A computer program as described in claim 14, wherein the region identification function calculates the mean and variance of the first peaks, assuming that the histogram of the depth image has a plurality of peaks including a first peak corresponding to the object region and a second peak corresponding to the background region; determines a binarization threshold using the mean and variance of the first peaks; and identifies the object region and the background region by binarizing the depth image using the binarization threshold.

16. A computer program as described in claim 13, wherein the region identification function identifies the object region and the background region in the reference image, and the difference information calculation function calculates the difference information using the identification information corresponding to a portion of the object region of the reference image that is different from the first object, without using the identification information corresponding to the background region of the reference image.

17. A computer program as claimed in claim 12, further comprising: causing a computer to implement an object identification function that inputs at least one of the reference image and the target image into an object detection model and identifies the area in the at least one image where the first object is located; and wherein the matrix calculation function calculates the first coordinate transformation matrix using, of the plurality of feature point pairs, a feature point pair that includes the feature point in the area where the identified first object is located.

18. A computer program as claimed in claim 12, wherein a pair selection function is implemented in a computer to select a plurality of pairs of feature points to be used from among a plurality of pairs of feature points corresponding to the first object using the matching accuracy of the feature points pairs, and the matrix calculation function calculates the first coordinate transformation matrix using the selected plurality of pairs of feature points to be used.

19. A computer program according to claim 12, wherein the specific information is information on a feature point pair among the plurality of feature point pairs that corresponds to a portion different from the first object.

20. A computer program as set forth in claim 19, wherein the difference information calculation function uses the first coordinate transformation matrix to transform the coordinates of the feature points of the target image that make up the feature points pair corresponding to a part different from the first object into coordinates in the coordinate system of the reference image, and calculates, as the difference information, information indicating the positional deviation between the two feature points that make up the feature points pair that correspond to a part different from the first object in the coordinate system of the reference image.

21. A computer program as described in claim 19, wherein the difference information calculation function uses the first coordinate transformation matrix to transform the coordinates of the feature points of the reference image that make up the feature points pair corresponding to a part different from the first object into coordinates in the coordinate system of the target image, and calculates, as the difference information, information indicating the positional deviation between the two feature points that make up the feature points pair that correspond to a part different from the first object in the coordinate system of the target image.

22. A computer program as described in claim 19, wherein the difference information calculation function calculates a second coordinate transformation matrix indicating the positional relationship between the reference image and the target image using feature point pairs that correspond to parts of the plurality of feature point pairs that are different from the first object, and calculates information indicating the difference between the first coordinate transformation matrix and the second coordinate transformation matrix as the difference information.

23. An information processing device comprising: a reference image acquisition unit that acquires a reference image showing a second object including a first object; a target image acquisition unit that acquires a target image showing the second object including the first object; a pair determination unit that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation unit that calculates a first coordinate transformation matrix indicating a positional relationship between the reference image and the target image, using feature point pairs among the plurality of feature point pairs that correspond to parts different from the first object; and a difference information calculation unit that calculates difference information between the first object in the reference image and the first object in the target image, using specific information regarding the position of the first object in each of the reference image and the target image and the first coordinate transformation matrix, wherein the difference information includes information regarding a difference in the relative position of the first object with respect to the second object.

24. An information processing device comprising: a reference image acquisition unit that acquires a reference image showing a second object including a first object; a target image acquisition unit that acquires a target image showing the second object including the first object; a pair determination unit that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation unit that calculates a first coordinate transformation matrix indicating a positional relationship between the reference image and the target image using a feature point pair corresponding to the first object out of the plurality of feature point pairs; and a difference information calculation unit that calculates difference information between the first object in the reference image and the first object in the target image using specific information regarding positions of parts of the reference image and the target image that are different from the first object and the first coordinate transformation matrix, the difference information including information regarding a difference in relative position of the first object with respect to the second object.

Images