Computer program and information processing device

Abstract

To provide a new technique relating to a relative position of a first object with respect to a second object.SOLUTION: Feature point matching between a reference image and a target image indicating a second object including a first object is performed to determine a plurality of feature point pairs. Of the plurality of feature point pairs, a feature point pair corresponding to a portion different from the first object is used to calculate a coordinate transformation matrix indicative of the positional relationship between the reference image and the target image, and identification information concerning the positions of the first object in the reference image and the target image and the coordinate transformation matrix are used to calculate difference information between the first object in the reference image and the first object in the target image. Alternatively, of the plurality of feature point pairs, a feature point pair corresponding to the first object is used to calculate the coordinate transformation matrix, and identification information concerning the positions of a portion different from the first object in the reference image and the target image and the coordinate transformation matrix are used to calculate the difference information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present specification relates to a computer program and an information processing device. [Background technology]

[0002] There is known technology for inspecting an object (for example, a label sheet affixed to a product such as a multifunction printer) using a captured image of the object. For example, Patent Document 1 discloses the following technology: Image data of the label sheet is input into a machine learning model to generate image data of a label sheet without defects. Differential image data representing the difference between the input image and the generated image is used to inspect the object for abnormalities. [Prior art documents] [Non-patent literature]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-174516 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, there is room for improvement in processing the relative position of a first object (for example, a label sheet) with respect to a second object (for example, a product).

[0005] This specification discloses new techniques for processing the relative position of a first object with respect to a second object. [Means for solving the problem]

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] A computer program, a reference image acquisition function for acquiring a reference image showing a second object including the first object; a target image acquisition function for acquiring a target image showing the second object including the first object; a pair determination function that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation function that calculates a first coordinate transformation matrix indicating a positional relationship between the reference image and the target image by using, among the plurality of feature point pairs, feature point pairs that correspond to portions different from the first object; a difference information calculation function that calculates difference information between the first object in the reference image and the first object in the target image using specific information related to the position of the first object in each of the reference image and the target image and the first coordinate transformation matrix, the difference information including information related to a difference in the relative position of the first object with respect to the second object; A computer program that makes a computer realize the above.

[0008] According to the above configuration, difference information is calculated using a first coordinate transformation matrix calculated using feature point pairs corresponding to portions different from the first object and specific information relating to the positions of the first object in the reference image and the target image. As a result, difference information that appropriately indicates the difference in the relative position of the first object with respect to the second object can be calculated. Therefore, for example, even if the position of the second object in the target image differs from the position of the second object in the reference image, the position of the first object included in the second object can be inspected.

[0009] [Application Example 2] A computer program, a reference image acquisition function for acquiring a reference image showing a second object including the first object; a target image acquisition function for acquiring a target image showing the second object including the first object; a pair determination function that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation function that calculates a first coordinate transformation matrix indicating a positional relationship between the reference image and the target image by using the feature point pair corresponding to the first object among the plurality of feature point pairs; a difference information calculation function that calculates difference information between the first object in the reference image and the first object in the target image using specific information related to positions of portions of the reference image and the target image that are different from the first object and the first coordinate transformation matrix, the difference information including information related to a difference in the relative position of the first object with respect to the second object; A computer program that makes a computer realize the above.

[0010] According to the above configuration, difference information is calculated using a first coordinate transformation matrix calculated using a feature point pair corresponding to the first object and specific information relating to the positions of parts of the reference image and the target image that differ from the first object. As a result, difference information that appropriately indicates the difference in the relative position of the first object with respect to the second object can be calculated. Therefore, for example, even if the position of the second object in the target image differs from the position of the second object in the reference image, the position of the first object included in the second object can be inspected.

[0011] The technology disclosed in this specification can be realized in various forms, such as an inspection device that inspects the position of a first object contained in a second object, a method for inspecting the position of a first object contained in a second object, a computer program for realizing these devices and methods, a recording medium on which the computer program is recorded, etc. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of an inspection system 1000 according to the present embodiment. [Figure 2] 1 is an explanatory diagram of a product 300. [Figure 3] 10 is a flowchart of advance preparation processing. [Figure 4] FIG. 10 is a diagram showing an example of an image used in the advance preparation process. [Figure 5] 10 is a flowchart of a position inspection process according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of an image used in the position inspection process. [Figure 7] FIG. 10 is an explanatory diagram of a coordinate transformation matrix. [Figure 8] FIG. 4 is an explanatory diagram of calculation of a positional deviation amount according to the first embodiment. [Figure 9] 10 is a flowchart of a position inspection process according to the second embodiment. [Figure 10] 10 is a flowchart of a position inspection process according to the third embodiment. [Figure 11] 13 is a flowchart of a position inspection process according to the fourth embodiment. [Figure 12] FIG. 13 is an explanatory diagram of calculation of a positional deviation amount according to the fourth embodiment.

[0013] A. First Example A-1. Inspection equipment configuration Next, an embodiment will be described based on an example. Fig. 1 is a block diagram showing the configuration of an inspection system 1000 of this example. The inspection system 1000 includes a processing device 100 and an imaging device 400. The processing device 100 and the imaging device 400 are connected so as to be able to communicate with each other.

[0014] The processing device 100 is, for example, a computer such as a personal computer. The processing device 100 includes a CPU 110 as a controller of the processing device 100, a GPU 115, a volatile storage device 120 such as RAM, a non-volatile storage device 130 such as a hard disk drive, an operation unit 150 such as a mouse or keyboard, a display unit 140 such as a liquid crystal display, and a communication unit 170. The communication unit 170 includes a wired or wireless interface for communicatively connecting to an external device, for example, an imaging device 400.

[0015] The GPU (Graphics Processing Unit) 115 is a processor that performs image processing such as 3D graphics and calculation processing for machine learning models under the control of the CPU 110. In this embodiment, the GPU is used to execute calculation processing for an object detection model AN and a monocular depth estimation model MN, which will be described later.

[0016] The volatile storage device 120 provides a buffer area for temporarily storing various intermediate data generated when the CPU 110 performs processing. The non-volatile storage device 130 stores a computer program PG for the inspection device.

[0017] The computer program PG includes, as modules, computer programs that cause the CPU 110 and the GPU 115 to cooperate to realize the functions of an object detection model AN and a monocular depth estimation model MN, which will be described later. The computer program PG is provided, for example, by the manufacturer of the processing device 100. The computer program PG may be provided, for example, in a form that is downloaded from a server or stored on a DVD-ROM or the like. The CPU 110 executes the computer program PG to perform advance preparation processing and position inspection processing, which will be described later.

[0018] The photographing device 400 is a digital camera that generates a photographed image representing a subject by photographing the subject using a two-dimensional image sensor. The photographed image data is bitmap data representing an image including a plurality of pixels, and more specifically, RGB image data that represents the color of each pixel using RGB values. The RGB values ​​are the gradation values ​​(hereinafter also referred to as component values) of three color components, i.e., color values ​​of the RGB color system including R, G, and B values. The R, G, and B values ​​are, for example, gradation values ​​with a predetermined number of gradations (e.g., 256). The photographed image data may also be luminance image data that represents the luminance of each pixel.

[0019] The photographing device 400 generates a photographed image under the control of the processing device 100, and transmits data of the photographed image to the processing device 100. In this embodiment, the photographing device 400 is used to photograph a product 300 to which a label L, which is an inspection target of the position inspection process, is affixed, and generate the photographed image.

[0020] 2 is an explanatory diagram of a product 300. Fig. 2(A) shows a perspective view of the product 300. In this embodiment, the product 300 is a printer having a substantially rectangular parallelepiped housing 30. In the manufacturing process, a rectangular label L is attached to a predetermined position on the front surface 31 (the surface on the +Y side) of the housing 30.

[0021] 2(B) shows a label L. The label L includes, for example, a background B and letters TX and a mark MK indicating various information such as the manufacturer's or product's brand logo, model number, lot number, etc.

[0022] A-2. Advance preparation process FIG. 3 is a flowchart of the advance preparation process. The advance preparation process is a process that is executed only once prior to the position inspection process described below. In the advance preparation process, a reference image, label area information in the reference image, and a background mask image that indicates the background area of ​​the reference image are prepared. The advance preparation process is started when a user (e.g., an inspection operator) inputs an instruction to start the process to the processing device 100 via the operation unit 150. For example, the user inputs an instruction to start the advance preparation process while placing a product 300, on which a label L to be inspected is attached in a normal position, in a predetermined position for photographing using the photographing device 400.

[0023] In S110, the CPU 110 executes a process for generating a reference image RI. For example, the CPU 110 transmits a photographing instruction to the photographing device 400. In response to the photographing instruction, the photographing device 400 generates a reference image RI by photographing the product 300, and transmits data of the reference image RI to the processing device 100. As a result, the CPU 110 acquires data of the generated reference image RI.

[0024] 4A and 4B are diagrams showing examples of images used in the advance preparation process. An example of a reference image RI is shown in FIG. 4A. The reference image RI is an image showing the front of a product 300R, a label LR affixed to the front of 300R, and a background BR. The product and label shown in the reference image RI are represented as product 300R and label LR, with "R" added to the end of the reference image to distinguish them from the actual product 300 and label L (FIG. 2).

[0025] In S120, the CPU 110 saves the acquired reference image RI. For example, data of the acquired reference image RI is stored in the nonvolatile storage device 130. The saved reference image RI is used in the position inspection process described later.

[0026] In S130, the CPU 110 identifies a label area LA using the object detection model AN. The label area LA is an area where the label LR is located. In this embodiment, the object detection model AN is a machine learning model including a trained convolutional neural network. For example, the object detection model AN is a machine learning model called YOLO (You only look once). YOLO is disclosed in, for example, the following paper: Alexey Bochkovskiy, Chien-Yao Wang, Hong-Yuan Mark Liao, "YOLOv4: Optimal Speed ​​and Accuracy of Object", arXiv:2004.10934, https: / / arxiv.org / abs / 2004.10934

[0027] The YOLO model predicts the area (also called a bounding box) where the object to be detected (in this example, label L) is located. The object detection model AN in this example is pre-trained to detect the label area LA where the label L is located.

[0028] In S130, the CPU 110 inputs the reference image RI to the object detection model AN. The object detection model AN uses the GPU 115 to perform a calculation on the reference image RI using a trained parameter set, and outputs label area information indicating the label area LA. This identifies the label area LA (FIG. 4(A)) within the reference image RI. The label area LA is a rectangular area surrounding the label LR.

[0029] In S140, the CPU 110 saves label area information indicating the label area LA. For example, the acquired label area information is stored in the nonvolatile storage device 130. The label area information includes, for example, the coordinates of the upper left vertex of the rectangular label area LA and the coordinates of the lower right vertex of the label area LA. The saved label area information is used in the position inspection process, which will be described later.

[0030] In S150, the CPU 110 generates a depth image (not shown) using the monocular depth estimation model MN. The depth image is an image that indicates the depth of each pixel of the input image. The depth is a value that corresponds to the distance from the shooting position (the position of the imaging device 400). In this embodiment, the closer the distance from the shooting position, the larger the depth value, and the farther the distance from the shooting position, the smaller the depth value. In this embodiment, the depth is a value of 256 gradations from 0 to 255.

[0031] In this embodiment, the monocular depth estimation model MN is a machine learning model including a trained convolutional neural network. For example, the monocular depth estimation model MN is a machine learning model called MiDaS. MiDaS is disclosed in, for example, the following paper: Ren? Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun, "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer", arXiv:1907.01341, https: / / doi.org / 10.48550 / arXiv.1907.01341

[0032] The MiDaS model is trained to generate a depth image that indicates the depth of each pixel in an input image.

[0033] In S150, the CPU 110 inputs the reference image RI to the monocular depth estimation model MN. The monocular depth estimation model MN uses the GPU 115 to perform calculations on the reference image RI using the trained parameter set, and outputs a depth image.

[0034] In S160, the CPU 110 uses the depth image to calculate a binarization threshold THb for identifying, in the reference image RI, a product area PA where the product 300 is located and a background area BA that is an area different from the product area PA. For example, the CPU 110 calculates the binarization threshold THb by analyzing a histogram of the depth image.

[0035] FIG. 4B shows a histogram DH of the depth image. The horizontal axis indicates depth, and the vertical axis indicates the number (frequency) of pixels having each depth. For example, the histogram DH is obtained by classifying each pixel of the depth image into multiple classes according to the depth of each pixel. In FIG. 4B, for example, the histogram DH is generated by treating each of the 256 gradation values ​​(values ​​from 0 to 255) that the depth can take as one class. As shown in FIG. 4B, the histogram DH of the depth image is considered to have a bimodal distribution including a first peak MT1 corresponding to the product region and a second peak MT2 corresponding to the background region.

[0036] For this reason, in this embodiment, the histogram DH is assumed to include a first peak MT1 corresponding to the product area PA and a second peak MT2 corresponding to the background area BA, and the average Dave and variance σd of the first peak MT1 are calculated. The first peak MT1 corresponding to the product area PA is the peak on the side closer to the image capture device 400 (the side with a larger depth value). The second peak MT2 corresponding to the background area BA is the peak on the side farther from the image capture device 400 (the side with a smaller depth value).

[0037] For example, it is assumed that the histogram DH of the depth image can be expressed by a Gaussian Mixture Model, which is a superposition of a first Gaussian distribution (also called a normal distribution) corresponding to the first peak MT1 and a second Gaussian distribution corresponding to the second peak MT2. Under this assumption, the CPU 110 calculates the mean Dave and variance σd of the first Gaussian distribution using a known likelihood estimation method, for example, the EM (Expectation-Maximization) algorithm.

[0038] The CPU 110 calculates the binarization threshold value THb as a value that is three times the variance σd away from the average Dave of the first peak MT1 (average Dave of the first Gaussian distribution) toward the second peak MT2 (FIG. 4(B)). That is, the binarization threshold value THb is expressed by the following equation (1). THb = (Dave - 3 × σd) … (1)

[0039] In S170, the CPU 110 performs binarization processing on the depth image using a binarization threshold THb to generate a background mask image MI. For example, in the binarization processing, among the multiple pixels of the depth image, pixels having a depth equal to or greater than the binarization threshold THb are classified as pixels constituting the product area PA, and pixels having a depth less than the binarization threshold THb are classified as pixels constituting the background area BA.

[0040] Figure 4(C) shows a background mask image MI corresponding to the reference image RI in Figure 4(A). In Figure 4(C), a product area PA indicates the area where the product 300R is located in the reference image RI in Figure 4(A), and a background area BA indicates the area where the background BR is located in the reference image RI in Figure 4(A). In this way, by generating the background mask image MI, the product area PA and the background area BA are identified in the reference image RI.

[0041] In S180, the CPU 110 saves the generated background mask image MI and ends the advance preparation process. Specifically, data of the generated background mask image MI is stored in the nonvolatile storage device 130. The saved background mask image MI is used in the position inspection process described later.

[0042] A-3. Position inspection process 5 is a flowchart of the position inspection process of the first embodiment. The position inspection process is a process for inspecting whether or not the label L is affixed in the correct position on the product 300 to be inspected. The position inspection process is started when a user (e.g., an inspection operator) inputs a start instruction to the processing device 100 via the operation unit 150. For example, the user inputs a start instruction for the position inspection process while the product 300 to be inspected is placed in a predetermined position for photographing using the photographing device 400.

[0043] In S210, the CPU 110 executes a process for generating a target image SI. For example, the CPU 110 transmits a photographing instruction to the photographing device 400. In response to the photographing instruction, the photographing device 400 generates the target image SI by photographing the product 300 to be inspected, and transmits the data of the target image SI to the processing device 100. As a result, the CPU 110 acquires the data of the generated target image SI.

[0044] 6A and 6B are diagrams showing examples of images used in the position inspection process. Fig. 6A illustrates a target image SI. The target image SI is an image showing the front of a product 300S, a label LR affixed to the front of 300S, and a background BR. The product and label shown in the target image SI are represented as product 300S and label LS using reference numerals with an "S" added to the end to distinguish them from the actual product 300 and label L (Fig. 2) and the product 300R and label LR (Fig. 4A) in the reference image RI.

[0045] FIG. 6(B) shows the same reference image RI as FIG. 4(A). The position of the product 300S in the target image SI may differ from the position of the product 300R in the reference image RI. This is because there is variation in the position at which the product 300 is placed when the product 300 to be inspected is photographed in S210. Furthermore, the position at which the label LS is affixed to the product 300S in the target image SI may differ from the position at which the label LS is affixed to the product 300R in the reference image RI. This is because there is variation in the position at which the label L is affixed to the product 300 during the manufacturing process of the product 300 to be inspected.

[0046] In S220, the CPU 110 reads out the data of the reference image RI saved in 120 in Fig. 3 from the nonvolatile storage device 130. The read data of the reference image RI is stored in the volatile storage device 120.

[0047] In S230, a feature point extraction process is executed for each of the target image SI and the reference image RI. The feature point extraction process is a process of identifying multiple feature points in an image and calculating the feature amounts of the feature points. The feature amounts are descriptors that describe the characteristics of the feature points and are expressed, for example, as vectors with a predetermined number of dimensions. A known method, specifically a method called SuperPoint, is used to identify the feature points and calculate the feature amounts. SuperPoint is disclosed, for example, in the following paper: Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich, “SuperPoint: Self-Supervised Interest Point Detection and Description”, arXiv:1712.07629v4, https: / / doi.org / 10.48550 / arXiv.1712.07629

[0048] It should be noted that other methods such as SIFT (Scale-Invariant Feature Transform), DISK (DIScrete Keypoints), KAZE, and AKAZE (Accelerated-KAZE) may also be used as a method for identifying feature points and calculating feature amounts.

[0049] 5A illustrates a plurality of feature points Ps (Psb, Psl, Psp) identified in the target image SI. As shown in FIG. 5A, the identified plurality of feature points Ps includes a feature point Psb located in the background BS, a feature point Psl located in the label LS, and a feature point Psp on the product 300S (excluding the feature point on the label LS).

[0050] 5(B) illustrates a plurality of feature points Pr (Prb, Prl, Prp) identified in the reference image RI. As shown in FIG. 5(B), the identified plurality of feature points Pr include a feature point Prb located in the background BR, a feature point Prl located on the label LR, and a feature point Prp on the product 300R (excluding the feature point on the label LR).

[0051] In addition, in Figures 5(A) and (B), only a small number of feature points are shown to avoid cluttering the figures, but in reality, many more feature points (for example, several tens to several hundreds) are identified in both the reference image RI and the target image SI.

[0052] In S240, CPU 110 performs feature point matching between multiple feature points in target image SI and multiple feature points in reference image RI. Multiple feature point pairs are determined by feature point matching. A feature point pair is a pair consisting of one feature point Ps in target image SI and one feature point Pr in reference image RI. The two feature points Ps and Pr constituting a feature point pair indicate similar parts of target image SI and reference image RI (e.g., the same part of product 300, the same part of label L, the same part of the background). Hereinafter, a feature point Pr in reference image RI that constitutes a feature point pair with a specific feature point Ps in target image SI will also be referred to as a feature point Pr corresponding to the specific feature point Ps.

[0053] The feature point matching method is a known method, specifically a method called LightGlue. LightGlue is disclosed in, for example, the following paper: Philipp Lindenberger, Paul-Edouard Sarlin, Marc Pollefeys, "LightGlue: Local Feature Matching at Light Speed", arXiv:2306.13643v1, https: / / doi.org / 10.48550 / arXiv.2306.13643

[0054] It should be noted that other methods such as brute force matching or SuperGlue may also be used as the feature point matching method.

[0055] In Figures 5(A) and 5(B), the two feature points Ps and Pr connected by a dashed dotted line form a feature point pair determined by feature point matching. Note that feature points that do not form a feature point pair, i.e., feature points for which no matching partner was found, are deleted at this point.

[0056] 3 from the nonvolatile storage device 130. The read data of the background mask image MI is stored in the volatile storage device 120.

[0057] In S260, the CPU 110 excludes feature points pairs corresponding to the background region from the determined plurality of feature points pairs. Specifically, the CPU 110 identifies the background region BA of the reference image RI with reference to the background mask image MI (FIG. 4C). For each of the plurality of feature points pairs determined by the feature point matching in S240, the CPU 110 determines whether the feature point in the reference image RI is located within the background region BA. The CPU 110 identifies feature points pairs including feature points within the background region BA of the reference image RI as feature points pairs corresponding to the background region, and excludes them from the processing target. Information on the excluded feature points pairs is deleted from, for example, the volatile storage device 120. In the example of FIG. 5, the feature points pair consisting of feature points Prb and Psb indicated by a square in FIGS. 5A and 5B is excluded from the processing target.

[0058] 3 from the nonvolatile storage device 130. The read label area information is stored in the volatile storage device 120.

[0059] In S280, the CPU 110 classifies the plurality of feature points pairs to be processed, i.e., the plurality of feature points pairs that do not include feature points in the background region BA of the reference image RI, into label-corresponding pairs and product-corresponding pairs. A label-corresponding pair is a feature points pair that corresponds to a labeled region, and a product-corresponding pair is a feature points pair that corresponds to a part of the product region that is different from the labeled region.

[0060] Specifically, the CPU 110 identifies the label area LA in the reference image RI by referring to the label area information (FIG. 5B). For each of the multiple feature point pairs to be processed, the CPU 110 determines whether the feature point in the reference image RI is located within the label area LA. The CPU 110 classifies feature point pairs that include a feature point within the label area LA of the reference image RI as a label-corresponding pair. The CPU 110 classifies feature point pairs that do not include a feature point within the label area LA of the reference image RI, in other words, feature point pairs that include a feature point located in a part of the product area PA of the reference image RI that is not within the label area LA, as a product-corresponding pair. In the example of FIG. 5, the feature point pair consisting of feature points Prl and Psl indicated by black circles in FIGS. 5A and 5B is classified as a label-corresponding pair. The feature point pair consisting of feature points Prp and Psp indicated by white circles in FIGS. 5A and 5B is classified as a product-corresponding pair.

[0061] In S290, the CPU 110 selects, from the multiple feature points pairs to be processed, a feature points pair with the highest matching accuracy as the feature points pair to be used. The matching accuracy is indicated, for example, by the distance (e.g., Euclidean distance) between the feature amounts (vectors) of the two feature points constituting the feature points pair. The larger the distance, the lower the matching accuracy, and the smaller the distance, the higher the matching accuracy. The matching accuracy may also be expressed by the cosine similarity between the feature amounts (vectors) of the two feature points. The CPU 110 selects, from the multiple label-associated pairs, a predetermined number (e.g., 50) of feature points pairs in descending order of matching accuracy as the label-associated pairs to be used. The CPU 110 selects, from the multiple product-associated pairs, a predetermined number of feature points pairs in descending order of matching accuracy as the product-associated pairs to be used. As a result, a predetermined number of label-associated pairs and a predetermined number of product-associated pairs are selected as the feature points pairs to be used.

[0062] In S300, the CPU 110 calculates a coordinate transformation matrix Ha indicating the positional relationship between the target image SI and the reference image RI using the selected predetermined number of product correspondence pairs. In this embodiment, the coordinate transformation matrix Ha is a matrix that transforms the coordinates of the target image SI so that the product 300S in the target image SI matches the product 300R in the reference image RI. In this way, the coordinate transformation matrix Ha is a matrix that transforms the coordinates of each pixel in the target image SI into coordinates in the coordinate system of the reference image RI.

[0063] FIG. 7 is an explanatory diagram of a coordinate transformation matrix. As shown in FIG. 7, the coordinate transformation matrix Ha represents the correspondence between the position Cis of a point on the target image SI and the position Cir of a point on the reference image RI. In the figure, the positions Cis and Cir are represented by so-called homogeneous coordinates (also called projective coordinates). The two values ​​x and y of the position Cis represent coordinates in the coordinate system of the target image SI. The third coordinate of the position Cis is fixed to 1. The same is true for the position Cir. The two values ​​x' and y' of the position Cir represent coordinates in the coordinate system of the reference image RI. The coordinates (x', y', 1) of the position Cis are multiplied by the coordinate transformation matrix Ha to obtain the coordinates (x', y', 1) of the position CIr. As described above, the coordinate transformation matrix Ha is understood to be a matrix that transforms the coordinates of each pixel in the target image SI into coordinates in the coordinate system of the reference image RI.

[0064] As shown in FIG. 7, the coordinate transformation matrix Ha is expressed as a matrix with 3 rows and 3 columns. Element hij indicates the element in row i and column j. The bottom right element h33 is fixed to 1. In this embodiment, the eight elements of the coordinate transformation matrix Ha are divided into three submatrices SM1, SM2, and SM3. The first submatrix SM1 is a submatrix consisting of four elements h11, h12, h21, and h22. The second submatrix SM2 is a submatrix consisting of two elements h13 and h23. The third submatrix SM3 is a submatrix consisting of two elements h31 and h32.

[0065] The four elements h11, h12, h21, and h22 of the first submatrix SM1 represent coordinate transformations that include rotation, scaling (enlargement or reduction), and skew (the latter may transform a rectangle into a parallelogram).

[0066] The two elements h13 and h23 of the second submatrix SM2 (Figure 7(A)) represent a parallel translation between a two-dimensional coordinate system representing the coordinates x and y and a two-dimensional coordinate system representing the coordinates x' and y'.

[0067] In this embodiment, an affine transformation matrix is ​​used as the coordinate transformation matrix Ha, and therefore the two elements h31 and h32 of the third submatrix SM3 are fixed to 0. Note that instead of an affine transformation matrix, a homography transformation matrix may be used as the coordinate transformation matrix Ha. In this case, the two elements h31 and h32 of the third submatrix SM3 may take values ​​other than 0.

[0068] The six elements h11-h13 and h21-h23 of the coordinate transformation matrix Ha can be determined by using three or more correspondences between the positions Cis and Cir, that is, three pairs of feature points.

[0069] The coordinates of each feature point constituting a feature point pair may not be completely accurate values ​​and may contain errors. In order to mitigate the effects of such errors and calculate the coordinate transformation matrix Ha with high accuracy, in this embodiment, six elements of the coordinate transformation matrix Ha (affine transformation matrix) are calculated using a predetermined number (e.g., 50) of feature point pairs that are greater than three, i.e., the predetermined number of product-corresponding pairs to be used selected in S290.

[0070] The coordinate transformation matrix Ha (affine transformation matrix) is calculated by a known method, and in this embodiment, it is calculated using the least squares method. Instead of the least squares method, methods such as RANSAC (Random sample consensus) and LMedS (Least-median of squares) may be used.

[0071] In S310, the CPU 110 transforms the coordinates of the feature points of the target image SI that make up the predetermined number of label-correspondence pairs selected in S290 into the coordinate system of the reference image RI. Specifically, using the coordinate transformation matrix Ha calculated in S300, the coordinate transformation is performed according to the equation in Fig. 7. As a result, the coordinates of the two feature points that make up each label-correspondence pair are both expressed in the coordinate system of the reference image RI.

[0072] The coordinate transformation matrix Ha is calculated using a product correspondence pair so that the product 300R in the reference image RI matches the product 300S in the target image SI. Therefore, when the relative position of the label LR relative to the product 300R in the reference image RI matches the relative position of the label LS relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the label correspondence pair match in the coordinate system of the reference image RI. When the relative position of the label LR relative to the product 300R in the reference image RI is misaligned with the relative position of the label LS relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the label correspondence pair will be misaligned in the coordinate system of the reference image RI. In other words, when the label LS is in the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the label correspondence pair match in the coordinate system of the reference image RI. If the label LS is misaligned from the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points that make up the label-corresponding pair will be misaligned in the coordinate system of the reference image RI. This can be used to determine whether the label LS is in the correct position relative to the product 300S in the target image SI, i.e., whether the label L is affixed in the correct position relative to the product 300 being inspected.

[0073] In S320, the CPU 110 calculates the amount of misalignment between two feature points constituting each label-correspondence pair in the coordinate system of the reference image RI. FIG. 8 is an explanatory diagram of the calculation of the amount of misalignment in the first embodiment. In FIG. 8, feature points Pr1 to Pr3 on the label LR of the reference image RI and feature points Ps1 to Ps3 on the label LS of the target image SI are plotted in the coordinate system of the reference image RI (a two-dimensional coordinate system with the X-axis and Y-axis in FIG. 8 as axes). The feature points Pr1 and Ps1 form a label-correspondence pair. Similarly, the feature points Pr2 and Ps2, and the feature points Pr3 and Ps3 form label-correspondence pairs, respectively.

[0074] In this embodiment, the misalignment amount is information indicating the misalignment of two feature points, and in this embodiment, includes parallel misalignment amounts Δx and Δy and an angular misalignment amount θ. The parallel misalignment amount Δx in the X direction is, for example, the difference in the X coordinates of two feature points that make up a label-correspondence pair. FIG. 8 illustrates the parallel misalignment amounts Δx1, Δx2, and Δx3 in the X direction for the label-correspondence pair Pr1 and Ps1, the label-correspondence pair Pr2 and Ps2, and the label-correspondence pair Pr3 and Ps3, respectively. The parallel misalignment amount Δy in the Y direction is the difference in the Y coordinates of two feature points that make up a label-correspondence pair. FIG. 8 illustrates the parallel misalignment amounts Δy1, Δy2, and Δy3 in the Y direction for the label-correspondence pair Pr1 and Ps1, the label-correspondence pair Pr2 and Ps2, and the label-correspondence pair Pr3 and Ps3, respectively. The parallel misalignment amounts Δx and Δy are calculated for each of a predetermined number of label-correspondence pairs.

[0075] The angular deviation θ is, for example, the angle formed by two vectors indicating the directions of two feature points constituting a label-correspondence pair. For example, a vector Vs2 (FIG. 8) indicating the direction of feature point Ps2 on label LS of target image SI is a vector whose start point is a reference feature point (e.g., Ps1 in FIG. 8) on label LS of target image SI and whose end point is feature point Ps2. A vector Vr2 (FIG. 8) indicating the direction of feature point Pr2 on label LR of reference image RI is a vector whose start point is a reference feature point (e.g., Pr1 in FIG. 8) on label LR of reference image RI and whose end point is feature point Pr2. The angle θ2 formed by the two vectors Vs2 and Vr2 is calculated, for example, based on a rotation matrix indicating the relationship between the two vectors. Alternatively, the angle θ2 may be calculated based on the cosine similarity of the two vectors. The angular deviation θ is calculated for each of a predetermined number of label-correspondence pairs.

[0076] As can be seen from the above explanation, the information indicating the amount of positional deviation (parallel deviation Δx, Δy and angular deviation θ) is information regarding the difference between the label LR in the reference image RI and the label LS in the target image SI, and can be said to be information regarding the difference in the relative position of the label (e.g., LR, LS) with respect to the product (e.g., 300R, 300S).

[0077] In S330, the CPU 110 determines whether the label attachment position is acceptable or unacceptable based on the calculated misalignment amounts. For example, the CPU 110 calculates average values ​​Δxa, Δya, and θa of the parallel misalignment amounts Δx, Δy, and angular misalignment amount θ calculated for each of a predetermined number of label-corresponding pairs. Note that instead of the average values, other statistical representative values, such as the median, may be calculated. The CPU 110 compares the average values ​​Δxa, Δya, and θa of the parallel misalignment amounts Δx, Δy, and angular misalignment amount θ with predetermined threshold values ​​THx, THy, and THθ, respectively. For example, if all of the average values ​​Δxa, Δya, and θa are less than the threshold value, the CPU 110 determines that the attachment position of the label L on the product 300 to be inspected is acceptable. If at least one of the average values ​​Δxa, Δya, and θa is equal to or greater than the threshold value, the CPU 110 determines that the attachment position of the label L on the product 300 to be inspected is unacceptable.

[0078] In S340, the CPU 110 outputs the determination result of the label application position and ends the position inspection process. For example, the CPU 110 displays the determination result of the label application position on the display unit 140.

[0079] According to the first embodiment described above, the CPU 110 acquires a reference image RI showing a product 300R including a label LR (S220 in FIG. 5). The CPU 110 acquires a target image SI showing a product 300S including a label LS (S210 in FIG. 5). The CPU 110 performs feature point matching between the reference image RI and the target image SI to determine multiple feature point pairs (S240 in FIG. 5). The CPU 110 calculates a coordinate transformation matrix Ha using product correspondence pairs that correspond to portions different from the labels LR and LS among the multiple feature point pairs (S300 in FIG. 5). The CPU 110 calculates a positional deviation amount (parallel deviation amounts Δx and Δy and angle deviation amount θ) that indicates the positional difference between the label LR of the reference image RI and the label LS in the target image SI using the label correspondence pairs that are feature point pairs corresponding to the labels LR and LS and the coordinate transformation matrix Ha (S310 and S320 in FIG. 5). In this way, the amount of misalignment of the labels LR and LS is calculated using the coordinate transformation matrix Ha calculated using the product correspondence pair and the label correspondence pair, which is information about the positions of the labels LR and LS. As a result, a misalignment amount that appropriately indicates the difference in the relative positions of the labels LR and LS with respect to the products 300R and 300S can be calculated. Therefore, even if the position of the product 300S in the target image SI differs from the position of the product 300R in the reference image RI, the relative position of the label LS with respect to the product 300S in the target image SI can be appropriately compared with the relative position of the label LR with respect to the product 300R in the reference image RI. Therefore, the position of the label LS included in the product 300S can be accurately inspected using the target image SI.

[0080] Furthermore, according to this embodiment, the CPU 110 further identifies a product area PA and a background area BA in the reference image RI (S150 to S170 in FIG. 3). The CPU 110 excludes feature points pairs corresponding to the background area from the plurality of feature points pairs (S260 in FIG. 5), and calculates the coordinate transformation matrix Ha using product-corresponding pairs of the remaining feature points pairs (S320 in FIG. 5). That is, the CPU 110 calculates the coordinate transformation matrix Ha using product-corresponding pairs corresponding to a portion of the product area PA that is different from the label LR, without using the feature points pair corresponding to the background area BA (S260 and S320 in FIG. 5). In this way, the background area BA is identified, and the coordinate transformation matrix Ha is calculated without using the feature points pair corresponding to the background area BA, so that the influence of the background is eliminated, and the coordinate transformation matrix Ha can be calculated with high accuracy.

[0081] Furthermore, according to this embodiment, the CPU 110 inputs the reference image RI into the monocular depth estimation model MN to generate a depth image (S150 in FIG. 3), and identifies the product area PA and the background area BA in the reference image RI based on the depth image (S160, S170 in FIG. 3). When the product 300 is positioned as a subject to be photographed and photographed, the photograph is usually taken with the product 300 closer to the photographing device 400 than its surroundings. Therefore, the product area PA and the background area BA can be accurately separated and identified based on the depth image obtained by inputting the reference image RI into the monocular depth estimation model MN.

[0082] Furthermore, according to this embodiment, the CPU 110 assumes that the histogram DH of the depth image has multiple peaks, including a first peak MT1 corresponding to the product area PA and a second peak MT2 corresponding to the background area BA. The CPU 110 calculates the mean Dave and variance σd of the first peak MT1, and determines the binarization threshold THb using the mean Dave and variance σd (S160 in FIG. 3, FIG. 4(B)). The CPU 110 generates a background mask image MI by binarizing the depth image using the binarization threshold THb (S170 in FIG. 3). This identifies the product area PA and the background area BA. When a product 300 is positioned as a subject to be photographed and photographed, a relatively large difference in depth typically occurs between the product 300 and its surroundings. For this reason, the histogram DH of the depth image is likely to be bimodal, having a first peak MT1 and a second peak MT2. Therefore, by assuming that the histogram DH has two peaks, the binarization threshold THb can be determined so as to separate the product area PA and the background area BA with high accuracy.

[0083] Furthermore, according to this embodiment, CPU 110 inputs reference image RI to object detection model AN and identifies label area LA within reference image RI (S130 in FIG. 3). CPU 110 identifies, from among multiple feature point pairs, a product-correspondence pair that is a feature point pair that does not include feature point Prl within the identified label area LA (S280 in FIG. 5), and calculates a coordinate transformation matrix Ha using the product-correspondence pair (S280, S300 in FIG. 5). As a result, by identifying the label area LA using object detection model AN, it is possible to easily determine the product-correspondence pair to be used in calculating the coordinate transformation matrix Ha.

[0084] Furthermore, according to this embodiment, the CPU 110 selects a plurality of product correspondence pairs to be used from the plurality of product correspondence pairs using the accuracy of matching of feature point pairs (S290 in FIG. 5). The CPU 110 calculates the coordinate transformation matrix Ha using the plurality of product correspondence pairs to be used (S300 in FIG. 5). As a result, the coordinate transformation matrix Ha can be calculated using product correspondence pairs that are likely to show a correct correspondence relationship between feature points, and therefore, it is possible to calculate a highly accurate coordinate transformation matrix Ha.

[0085] Furthermore, according to this embodiment, the CPU 110 calculates the amount of misalignment (parallel misalignment amounts Δx, Δy and angular misalignment amount θ) using a label-corresponding pair corresponding to the labels LS, LS among the multiple feature point pairs (S320 in FIG. 5). As a result, the amount of misalignment can be calculated based on the misalignment of the feature points labeled LS, LS that make up the feature point pair.

[0086] Furthermore, according to this embodiment, the CPU 110 selects multiple label correspondence pairs to be used from the multiple label correspondence pairs using the matching accuracy of the feature point pairs (S290 in FIG. 5). The CPU 110 calculates the amount of misregistration using the multiple label correspondence pairs to be used (S320 in FIG. 5). As a result, the amount of misregistration can be calculated using label correspondence pairs that are likely to show a correct correspondence relationship between feature points, so the amount of misregistration can be calculated with high accuracy.

[0087] Furthermore, according to this embodiment, the CPU 110 converts the coordinates of the feature points of the target image SI that make up the label-correspondence pair into coordinates in the coordinate system of the reference image RI using the coordinate transformation matrix Ha (S310 in FIG. 5). The CPU 110 calculates the amount of misalignment between the two feature points that make up the label-correspondence pair in the coordinate system of the reference image RI (S320 in FIG. 5). As a result, the relative position of the label LS with respect to the product 300S in the target image SI and the relative position of the label LR with respect to the product 300R in the reference image RI can be compared in the same coordinate system, making it possible to calculate an appropriate amount of misalignment.

[0088] As can be seen from the above explanation, the labels L, LR, and LS in this embodiment are examples of the first object, and the products 300, 300R, and 300S are examples of the second object. The positional deviation information (parallel deviation amounts Δx and Δy and angular deviation amount θ) in this embodiment is an example of difference information, and the coordinate transformation matrix Ha is an example of the first coordinate transformation matrix. The label correspondence pair in this embodiment is an example of identification information.

[0089] B. Second Example In the second embodiment, the content of the position inspection process differs from that of the first embodiment shown in Fig. 5. The hardware configuration (Fig. 1) of the inspection system 1000 of the second embodiment and the content of the advance preparation process (Fig. 3) are the same as those of the first embodiment.

[0090] FIG. 9 is a flowchart of the position inspection process of the second embodiment. In S200B, the CPU 110 executes S210 to S290 of FIG. 5. In S300B, the CPU 110 calculates a coordinate transformation matrix Ht using a predetermined number of selected product correspondence pairs. In this embodiment, the coordinate transformation matrix Ht is a matrix that transforms the coordinates of the reference image RI so that the product 300R in the reference image RI matches the product 300S in the target image SI. In this way, the coordinate transformation matrix Ht is a matrix that transforms the coordinates of each pixel in the reference image RI into coordinates in the coordinate system of the target image SI. In other words, the coordinate transformation matrix Ht calculated in S300B of the second embodiment is the inverse matrix of the coordinate transformation matrix Ha calculated in S300 of the first embodiment.

[0091] The coordinate transformation matrix Ht calculated in the second embodiment is an affine transformation matrix, similar to the coordinate transformation matrix Ha calculated in the first embodiment. The coordinate transformation matrix Ht (affine transformation matrix) is calculated using a known method, such as the least squares method, similar to the first embodiment.

[0092] In S310B, the CPU 110 transforms the coordinates of the feature points of the reference image RI that make up the predetermined number of label-correspondence pairs selected in S290 into the coordinate system of the target image SI. Specifically, the coordinate transformation is performed using the coordinate transformation matrix Ht calculated in S300B. As a result, the coordinates of the two feature points that make up each label-correspondence pair are both expressed in the coordinate system of the target image SI.

[0093] Like the coordinate transformation matrix Ha in the first embodiment, the coordinate transformation matrix Ht is calculated using a product correspondence pair, so that the product 300R in the reference image RI matches the product 300S in the target image SI. Therefore, if the label LS is in the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points that make up the label correspondence pair will match in the coordinate system of the target image SI. If the label LS is shifted from the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points that make up the label correspondence pair will be shifted in the coordinate system of the target image SI. This can be used to determine whether the label LS is in the correct position relative to the product 300S in the target image SI, i.e., whether the label L is affixed to the product 300 being inspected in the correct position.

[0094] In S320B, the CPU 110 calculates the positional shift amounts of the two feature points constituting each label-correspondence pair in the coordinate system of the target image SI. As in the first embodiment, the positional shift amounts include parallel shift amounts Δx and Δy and angular shift amount θ, and are calculated for each of a predetermined number of label-correspondence pairs according to the method described in the first embodiment with reference to FIG.

[0095] In S330B, the CPU 110 determines whether the label application position is acceptable or not based on the calculated amount of misalignment. The method of determining whether the label application position is acceptable or not is the same as in S330 of FIG.

[0096] In S340B, similarly to S340 in FIG. 5, the CPU 110 outputs the determination result of the label attachment position, and ends the position inspection process.

[0097] According to the second embodiment described above, the CPU 110 uses the coordinate transformation matrix Ht to transform the coordinates of the feature points of the reference image RI that make up the label-correspondence pair into coordinates in the coordinate system of the target image SI (S310B in FIG. 9). The CPU 110 calculates the amount of misalignment between the two feature points that make up the label-correspondence pair in the coordinate system of the target image SI (S320B in FIG. 9). As a result, as in the first embodiment, the relative position of the label LS with respect to the product 300S in the target image SI and the relative position of the label LR with respect to the product 300R in the reference image RI can be compared in the same coordinate system, making it possible to calculate an appropriate amount of misalignment.

[0098] C. Third Example In the third embodiment, the content of the position inspection process differs from the position inspection process of the first embodiment shown in Fig. 5. The hardware configuration (Fig. 1) of the inspection system 1000 of the third embodiment and the content of the advance preparation process (Fig. 3) are the same as those of the first embodiment.

[0099] 10 is a flowchart of the position inspection process of the third embodiment. In S200C, the CPU 110 executes S210 to S290 in FIG.

[0100] 5, the CPU 110 calculates a coordinate transformation matrix Ha using the predetermined number of product correspondence pairs selected in S290. As in the first embodiment, the coordinate transformation matrix Ha is a matrix that transforms the coordinates of the target image SI so that the product 300S in the target image SI matches the product 300R in the reference image RI. In this way, the coordinate transformation matrix Ha is a matrix that transforms the coordinates of each pixel in the target image SI into coordinates in the coordinate system of the reference image RI.

[0101] In S310C, the CPU 110 calculates a coordinate transformation matrix Hb using the predetermined number of label correspondence pairs selected in S290 of Fig. 5. Unlike the coordinate transformation matrix Ha, the coordinate transformation matrix Hb is calculated using label correspondence pairs, and is therefore a matrix that transforms the coordinates of the target image SI so that the label LS of the target image SI matches the label LR of the reference image RI.

[0102] Like the coordinate transformation matrix Ha, the coordinate transformation matrix Hb is a 3-row, 3-column affine transformation matrix. Like the coordinate transformation matrix Ha, the coordinate transformation matrix Hb (affine transformation matrix) is calculated using a known method, specifically, the least squares method. Instead of the least squares method, methods such as RANSAC (Random sample consensus) and LMedS (Least-Median of Squares) may be used.

[0103] The coordinate transformation matrix Ha and the coordinate transformation matrix Hb are both matrices that transform the coordinates of each pixel of the target image SI into coordinates in the coordinate system of the reference image RI.

[0104] The coordinate transformation matrix Ha is calculated using a product correspondence pair so that the product 300R in the reference image RI matches the product 300S in the target image SI. The coordinate transformation matrix Hb is calculated using a label correspondence pair so that the label LR in the reference image RI matches the label LS in the target image SI. Therefore, when the relative position of the label LR to the product 300R in the reference image RI matches the relative position of the label LS to the product 300S in the target image SI, the coordinate transformation matrix Ha and the coordinate transformation matrix Hb are the same matrix. When the relative position of the label LR to the product 300R in the reference image RI matches the relative position of the label LS to the product 300S in the target image SI, the coordinate transformation matrix Ha and the coordinate transformation matrix Hb are matrices whose element values ​​differ by an amount corresponding to the misalignment. By utilizing this and comparing the coordinate transformation matrix Ha with the coordinate transformation matrix Hb, it is possible to determine whether the label LS is in the correct position relative to the product 300S in the target image SI, i.e., whether the label L is attached in the correct position relative to the product 300 being inspected.

[0105] In S320C, the CPU 110 calculates difference information indicating the difference between the coordinate transformation matrices Ha and Hb. As shown in FIG. 7, a total of six elements h11, h12, h13, h21, h22, and h23 of the first submatrix SM1 and the second submatrix SM2 of the coordinate transformation matrices may differ between the coordinate transformation matrices Ha and Hb. For this reason, for example, the mean square error of the six elements of the two coordinate transformation matrices Ha and Hb may be calculated as the difference information. Alternatively, for example, the mean absolute error of the six elements may be calculated as the difference information, or the two coordinate transformation matrices Ha and Hb may be analyzed to calculate the error in the translation component and the error in the angle component.

[0106] The difference information indicating the difference between the two coordinate transformation matrices Ha and Hb in this embodiment is information about the difference between the label LR in the reference image RI and the label LS in the target image SI, similar to the positional deviation amounts (parallel deviation amounts Δx, Δy and angular deviation amount θ) in the first and second embodiments, and can be said to be information about the difference in the relative position of the label (e.g., LR, LS) with respect to the product (e.g., 300R, 300S).

[0107] In S330C, the CPU 110 determines whether the label attachment position is acceptable or unacceptable based on the calculated difference information. For example, the CPU 110 compares the mean square error calculated as the difference information with a predetermined threshold. For example, if the mean square error is less than the threshold, the CPU 110 determines that the attachment position of the label L on the product 300 being inspected is acceptable. If the mean square error is equal to or greater than the threshold, the CPU 110 determines that the attachment position of the label L on the product 300 being inspected is unacceptable.

[0108] In S340C, the CPU 110 outputs the determination result of the label attachment position, similarly to S340 (FIG. 5) in the first embodiment, and ends the position inspection process.

[0109] According to the third embodiment described above, the CPU 110 calculates the coordinate transformation matrix Ha using the product correspondence pair corresponding to the products 300R and 300S among the multiple feature point pairs (S300C in FIG. 10), and calculates the coordinate transformation matrix Hb using the label correspondence pair corresponding to the labels LR and LS (S310C in FIG. 10). The CPU 110 calculates difference information indicating the difference between the coordinate transformation matrix Ha and the coordinate transformation matrix Hb (S320C in FIG. 10). As a result, the difference between the relative position of the label LS with respect to the product 300S in the target image SI and the relative position of the label LR with respect to the product 300R in the reference image RI can be evaluated by comparing the two coordinate transformation matrices. Therefore, for example, even if the position of the product 300S in the target image SI is different from the position of the product 300R in the reference image RI, the relative position of the label LS to the product 300S in the target image SI can be appropriately compared with the relative position of the label LR to the product 300R in the reference image RI.

[0110] As can be seen from the above explanation, the coordinate transformation matrix Ha in the third embodiment is an example of a first coordinate transformation matrix, and the coordinate transformation matrix Hb is an example of a second coordinate transformation matrix.

[0111] D. Fourth Example In the fourth embodiment, the content of the position inspection process differs from that of the first embodiment shown in Fig. 5. The hardware configuration (Fig. 1) of the inspection system 1000 of the fourth embodiment and the content of the advance preparation process (Fig. 3) are the same as those of the first embodiment.

[0112] 11 is a flowchart of the position inspection process of the fourth embodiment. In S200D, the CPU 110 executes S210 to S290 of FIG. 5. In S300D, the CPU 110 calculates a coordinate transformation matrix Hb using a predetermined number of label correspondence pairs selected in S290 of FIG. 5. Unlike the first embodiment, the coordinate transformation matrix Hb is calculated using label correspondence pairs, and is therefore a matrix that transforms the coordinates of the target image SI so that the label LS of the target image SI matches the label LR of the reference image RI. In this way, the coordinate transformation matrix Hb is a matrix that transforms the coordinates of each pixel of the target image SI into coordinates in the coordinate system of the reference image RI. The coordinate transformation matrix Hb is a 3-row, 3-column affine transformation matrix, similar to the coordinate transformation matrix H of FIG. 7.

[0113] The coordinate transformation matrix Hb (affine transformation matrix) is calculated using a known method, specifically, the least squares method, as in the first embodiment. Instead of the least squares method, methods such as RANSAC (Random sample consensus) and LMedS (Least-Median of Squares) may also be used.

[0114] In S310D, CPU 110 transforms the coordinates of the feature points of target image SI constituting the predetermined number of product-corresponding pairs selected in S290 of Fig. 5 into the coordinate system of reference image RI. Specifically, the coordinate transformation is performed using the coordinate transformation matrix Hb calculated in S300D. As a result, the coordinates of the two feature points constituting each product-corresponding pair are both expressed in the coordinate system of reference image RI.

[0115] The coordinate transformation matrix Hb is calculated using a label correspondence pair so that the label LR of the reference image RI matches the label LS of the target image SI. Therefore, when the relative position of the label LR relative to the product 300R in the reference image RI matches the relative position of the label LS relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the product correspondence pair match in the coordinate system of the reference image RI. When the relative position of the label LR relative to the product 300R in the reference image RI is misaligned with the relative position of the label LS relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the product correspondence pair will be misaligned in the coordinate system of the reference image RI. In other words, when the label LS is in the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points constituting the product correspondence pair match in the coordinate system of the reference image RI. If the label LS is misaligned from the correct position relative to the product 300S in the target image SI, the coordinates of the two feature points that make up the product-corresponding pair will be misaligned in the coordinate system of the reference image RI. This can be used to determine whether the label LS is in the correct position relative to the product 300S in the target image SI, i.e., whether the label L is affixed in the correct position relative to the product 300 being inspected.

[0116] In S320D, the CPU 110 calculates the amount of misalignment between two feature points constituting each product-corresponding pair in the coordinate system of the reference image RI. FIG. 12 is an explanatory diagram of calculation of the amount of misalignment in the fourth embodiment. FIG. 12 plots feature points Pr4 to Pr6 on the product 300R (excluding the label LR) of the reference image RI and feature points Ps4 to Ps6 on the product 300S (excluding the label LS) of the target image SI in the coordinate system of the reference image RI (a two-dimensional coordinate system with the X-axis and Y-axis in FIG. 12). In the second embodiment, the coordinate transformation matrix Hb is calculated using the label-corresponding pair, so that the label LS of the target image SI matches the label LR of the reference image RI, as shown in FIG. 12. The feature points Pr4 and Ps4 form a product-corresponding pair. Similarly, the feature points Pr5 and Ps5, and the feature points Pr6 and Ps6 form product-corresponding pairs, respectively.

[0117] In the second embodiment, the misalignment amount includes parallel misalignment amounts Δx and Δy in the X and Y directions of the two feature points that make up the product-associated pair, and an angular misalignment amount θ formed by two vectors that indicate the directions of the two feature points that make up the product-associated pair. The parallel misalignment amounts Δx and Δy and the angular misalignment amount θ are calculated using the same method as that described for the two feature points that make up the label-associated pair in the first embodiment ( FIG. 8 ). The parallel misalignment amounts Δx and Δy and the angular misalignment amount θ are calculated for each of a predetermined number of product-associated pairs.

[0118] As can be seen from the above explanation, similar to the first embodiment, the information indicating the positional deviation amount in the second embodiment (parallel deviation amount Δx, Δy and angular deviation amount θ) is information regarding the difference between the label LR in the reference image RI and the label LS in the target image SI, and can be said to be information regarding the difference in the relative position of the label (e.g., LR, LS) with respect to the product (e.g., 300R, 300S).

[0119] In S330D, the CPU 110 determines whether the label attachment position is acceptable based on the calculated misalignment amounts. For example, the CPU 110 calculates average values ​​Δxa, Δya, and θa of the parallel misalignment amounts Δx, Δy, and angular misalignment amount θ calculated for each of a predetermined number of product-corresponding pairs. Note that instead of the average values, other statistical representative values, such as the median, may be calculated. As in S330 (FIG. 5) of the first embodiment, the CPU 110 compares the average values ​​Δxa, Δya, and θa of the parallel misalignment amounts Δx, Δy, and angular misalignment amount θ with predetermined threshold values ​​THx, THy, and THθ, respectively. For example, if all of the average values ​​Δxa, Δya, and θa are less than the threshold values, the CPU 110 determines that the attachment position of the label L on the product 300 being inspected is acceptable. When at least one of the average values ​​Δxa, Δya, and θa is equal to or greater than the threshold value, the CPU 110 determines that the position of the label L attached to the product 300 being inspected is unacceptable.

[0120] In S340D, the CPU 110 outputs the determination result of the label attachment position, as in S340 (FIG. 5) in the first embodiment, and ends the position inspection process. For example, the CPU 110 displays the determination result of the label attachment position on the display unit 140.

[0121] According to the fourth embodiment described above, the CPU 110 acquires a reference image RI showing a product 300R including a label LR (S200D in FIG. 11, S220 in FIG. 5). The CPU 110 acquires a target image SI showing a product 300S including a label LS (S200D in FIG. 11, S210 in FIG. 5). The CPU 110 performs feature point matching between the reference image RI and the target image SI to determine a plurality of feature point pairs (S200D in FIG. 11, S240 in FIG. 5). The CPU 110 calculates a coordinate transformation matrix Hb using a label-corresponding pair corresponding to the labels LR and LS among the plurality of feature point pairs (S300D in FIG. 11). CPU 110 calculates the misalignment amount (parallel misalignment amount Δx, Δy and angular misalignment amount θ) indicating the difference in position between label LR of reference image RI and label LS in target image SI using a product correspondence pair, which is a feature point pair corresponding to a portion different from labels LR, LS, and a coordinate transformation matrix Hb (S310D, S320D in FIG. 11). In this way, the misalignment amount of labels LR, LS is calculated using the coordinate transformation matrix Hb calculated using the label correspondence pair and the product-label correspondence pair, which is information regarding the position of a portion different from labels LR, LS (products 300R, 300S). As a result, as in the first embodiment, it is possible to calculate the misalignment amount that appropriately indicates the difference in the relative position of labels LR, LS with respect to products 300R, 300S. Therefore, even if the position of the product 300S in the target image SI differs from the position of the product 300R in the reference image RI, the relative position of the label LS to the product 300S in the target image SI can be appropriately compared with the relative position of the label LR to the product 300R in the reference image RI. Therefore, the position of the label LS included in the product 300S can be inspected with high accuracy using the target image SI.

[0122] Furthermore, according to the fourth embodiment, the CPU 110 identifies the product area PA and the background area BA in the reference image RI (S200D in FIG. 11, S150 to S170 in FIG. 3). The CPU 110 excludes feature points pairs corresponding to the background area from the plurality of feature points pairs (S200D in FIG. 11, S260 in FIG. 5), and calculates the amount of misregistration using product-corresponding pairs among the remaining feature points pairs (S320D in FIG. 11). That is, the CPU 110 calculates the difference information (amount of misregistration) using product-corresponding pairs corresponding to portions of the product area PA that are different from the label LR, without using feature points pairs corresponding to the background area BA (S260 in FIG. 5, S320D in FIG. 11). In this way, the background area BA is identified and the difference information is calculated without using feature points pairs corresponding to the background area BA, thereby eliminating the influence of the background and allowing the difference information to be calculated with high accuracy.

[0123] Furthermore, according to the fourth embodiment, the CPU 110 inputs the reference image RI to the object detection model AN and identifies the label region LA in the reference image RI (S200D in FIG. 11, S130 in FIG. 3). The CPU 110 identifies, from among multiple feature point pairs, a label correspondence pair that is a feature point pair that includes the feature point Prl in the identified label region LA (S200D in FIG. 11, S280 in FIG. 5), and calculates the coordinate transformation matrix Hb using the label correspondence pair (S300 in FIG. 11). As a result, by identifying the label region LA using the object detection model AN, it is possible to easily determine the label correspondence pair to be used in calculating the coordinate transformation matrix Hb.

[0124] Furthermore, according to this embodiment, the CPU 110 calculates the amount of misalignment (parallel misalignment amounts Δx, Δy and angular misalignment amount θ) by using a product-corresponding pair, which is a feature point pair corresponding to a portion different from the labels LS and LR, among the multiple feature point pairs (S320D in FIG. 11). As a result, the amount of misalignment can be calculated based on the misalignment of the feature points of the products 300S and 300R that make up the feature point pair.

[0125] As can be seen from the above explanation, the labels L, LR, and LS in this embodiment are examples of first objects, and the products 300, 300R, and 300S are examples of second objects. The positional deviation information (parallel deviation amounts Δx and Δy and angular deviation amount θ) in this embodiment is an example of difference information, and the coordinate transformation matrix Hb is an example of a first coordinate transformation matrix. The product correspondence pairs in this embodiment are an example of identification information.

[0126] E. Variations (1) In the above embodiment, the advance preparation process of Fig. 3 is performed only once, and the position inspection process of Fig. 5 is performed for each of the multiple products 300 to be inspected. Alternatively, the advance preparation process of Fig. 3 and the position inspection process of Fig. 5 may be performed as a whole for each of the multiple products 300.

[0127] (2) In the above embodiment, the background area BA and the product area PA are identified in the reference image RI (S150 to S170 in FIG. 3). Alternatively, in the position inspection process, the same processes as S150 to S170 in FIG. 3 may be performed on the target image SI to identify the background area and the product area. In this case, for example, in S260 of the position inspection process in FIG. 5, feature points pairs including feature points located in the background area of ​​the target image SI may be excluded from feature points pairs to be used. Furthermore, the background area and the product area may be identified in both the target image SI and the reference image RI. In this case, for example, in S260 of the position inspection process in FIG. 5, pairs of feature points located in the background area of ​​the target image SI and feature points located in the background area of ​​the reference image RI may be excluded from feature points pairs to be used. However, as in the above embodiments, when the background area BA and the product area PA are identified only for the reference image RI, the background area BA and the product area PA only need to be identified once in the preliminary preparation process, so when inspecting the positions of the labels L of multiple products 300 using multiple target images SI, the amount of processing required for inspection can be reduced.

[0128] (3) In the above embodiment, the label area LA is identified for the reference image RI (S130 in FIG. 3). Alternatively, in the position inspection process, the label area may be identified by performing a process similar to S130 in FIG. 3 for the target image SI. In this case, for example, in S280 of the position inspection process in FIG. 5, feature point pairs including feature points located in the label area of ​​the target image SI may be classified as label-corresponding pairs, and feature point pairs not including feature points located in the label area of ​​the target image SI may be classified as product-corresponding pairs. Alternatively, the label area may be identified for both the target image SI and the reference image RI. In this case, for example, in S280 of the position inspection process in FIG. 5, pairs of feature points located in the label area of ​​the target image SI and feature points located in the label area of ​​the reference image RI may be classified as label-corresponding pairs, and other feature point pairs may be classified as product-corresponding pairs. However, when the label area LA is identified only for the reference image RI, as in the above embodiments, it is only necessary to identify the label area LA once in the preparatory process. Therefore, when inspecting the positions of the labels L of multiple products 300 using multiple target images SI, the amount of processing required for inspection can be reduced.

[0129] (4) In the first embodiment, the misalignment amount is calculated using a label-corresponding pair, which is a feature point corresponding to the labels LR and LS (S310 and S320 in FIG. 5). This is not limiting, and the misalignment amount may be calculated using, for example, rectangular labeled areas identified in the target image SI and the reference image RI. In this case, for example, the CPU 110 identifies labeled areas in each of the target image SI and the reference image RI using the object detection model described above. The CPU 110 transforms position information of the labeled area in the target image SI (e.g., four vertices of the rectangular labeled area) into the coordinate system of the reference image RI using the coordinate transformation matrix Ha. The CPU 110 calculates the misalignment amount indicating the misalignment between the labeled area in the target image SI and the labeled area in the reference image RI in the coordinate system of the reference image RI. In this modification, the position information of the labeled areas identified in each of the target image SI and the reference image RI is an example of the identification information.

[0130] (5) In the fourth embodiment described above, the misalignment amount is calculated using a product correspondence pair, which are feature points corresponding to the products 300R and 300S (S310D and S320D in FIG. 11). This is not limiting, and the misalignment amount may be calculated using, for example, a product area identified in the target image SI and the reference image RI. In this case, for example, the CPU 110 identifies the product area in each of the target image SI and the reference image RI using the monocular depth estimation model MN described above. The CPU 110 transforms position information of the product area in the target image SI (for example, coordinates of the edge indicating the boundary between the product area and the background area) into the coordinate system of the reference image RI using the coordinate transformation matrix Ha. The CPU 110 calculates the misalignment amount indicating the deviation between the product area in the target image SI and the product area in the reference image RI in the coordinate system of the reference image RI. In this modification, the position information of the product area identified in each of the target image SI and the reference image RI is an example of identification information.

[0131] (6) In the fourth embodiment described above, the CPU 110 transforms the coordinates of the feature points of the target image SI that constitute the product-corresponding pair into coordinates in the coordinate system of the reference image RI using the coordinate transformation matrix Hb (S310D in FIG. 11), and calculates the positional shift of the two feature points that constitute the product-corresponding pair in the coordinate system of the reference image RI (S320D in FIG. 11). Alternatively, the CPU 110 may transform the coordinates of the feature points of the reference image RI that constitute the product-corresponding pair into coordinates in the coordinate system of the target image SI using the inverse matrix of the coordinate transformation matrix Hb, and calculate the positional shift of the feature points that constitute the product-corresponding pair in the coordinate system of the target image SI.

[0132] (7) In the above embodiment, the CPU 110 identifies the label area LA of the reference image RI using the object detection model AN (S130 in FIG. 3). Alternatively, the CPU 110 may identify the label area LA of the reference image RI using another method. The other method may be, for example, a method of identifying a rectangular label area using pattern matching using pattern data that indicates the characteristics of the label L.

[0133] (8) In the above embodiment, the CPU 110 identifies the background area BA and the product area PA of the reference image RI using the monocular depth estimation model MN (S150 in FIG. 3). Alternatively, the CPU 110 may use another method to identify the background area BA and the product area PA of the reference image RI. The other method may be a method of identifying the product area using pattern matching using pattern data that indicates the characteristics of the product 300, and identifying the area surrounding the product area as the background area.

[0134] (9) Some of the processes in the advance preparation process and position inspection process in each of the above embodiments may be omitted as appropriate. For example, if the target image SI is captured so as to include only the front surface 31 of the product 300 and not the background, or if the area of ​​the background is extremely small, steps S150 to S180 in Fig. 3 related to identifying the background area BA may be omitted, and S260 in Fig. 5 may also be omitted. Furthermore, the process of selecting feature points pairs to be used using the matching accuracy (S290 in Fig. 5) may be omitted, and the processes of S300 to S320 in Fig. 5 may be performed using all feature points pairs.

[0135] (10) In S290 of FIG. 5, a predetermined number of feature point pairs with the highest matching accuracy are selected as feature point pairs to be used from among multiple feature point pairs. Instead of matching accuracy, feature point pairs to be used may be selected based on another index. For example, the target image SI and the reference image RI are both images obtained by photographing the same type of product 300 and label L. For this reason, for example, the coordinate difference (ΔX1s, ΔX1r) between the two feature points constituting a product-corresponding pair is likely to be relatively close for each of the multiple product-corresponding pairs. For example, if the coordinate difference for one product-corresponding pair is significantly different from the coordinate difference for the other product-corresponding pairs, the matching of that product-corresponding pair is likely to be incorrect. For this reason, for example, the CPU 110 may calculate the coordinate difference for each of the multiple product-corresponding pairs and exclude product-corresponding pairs whose coordinate difference is more than a predetermined amount (e.g., three times the variance σ) away from the average of the coordinate differences. The same applies to label-corresponding pairs.

[0136] (11) Furthermore, the advance preparation process and position inspection process in each of the above embodiments are processes that can be automatically executed by the CPU 110. Alternatively, a portion of the advance preparation process and position inspection process may include processes that require operator operation. For example, in the advance preparation process of FIG. 3, the identification of the label area LA in the reference image RI (S130 in FIG. 3) is automatically executed using the object detection model AN. Alternatively, for example, an operator may operate a pointing device such as a mouse to specify the label area LA on the reference image RI displayed on the display unit 140, and the CPU 110 may identify the area specified by the operator as the label area LA. Similarly, the processes of S150 to S170 in FIG. 3 for identifying the background area BA and the product area PA in the reference image RI may also be similarly specified based on the operator's designation.

[0137] (12) The method of calculating the binarization threshold value THb for generating the background mask image MI (S160 in FIG. 3) described with reference to FIG. 4(B) is an example and is not limiting. The binarization threshold value THb may be calculated using other methods, such as Otsu's binarization method.

[0138] (13) The object detection model AN and the monocular depth estimation model MN used in the above embodiments are merely examples, and other models may be used. For example, the object detection model AN may be another YOLO model such as "YOLO v3" or "YOLO v5" instead of the "YOLO v4" ​​model. Furthermore, other models such as SSD R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN may be used as the object detection model AN.

[0139] Similarly, the monocular depth estimation model MN may be another model instead of the MiDaS model. For example, the monocular depth estimation model MN may be a model called SfMLearner or a model called ZoeDepth (Zero-shot Transfer by Combining Relative and Metric Depth).

[0140] (14) The second object to be inspected is not limited to a printer, but may be other objects such as a multifunction peripheral, a sewing machine, a cutting machine, or a mobile terminal. Furthermore, the first object, the position of which is inspected relative to the second object, is not limited to a label attached to a product, but may be any object. For example, the first object may be a label image printed on the first object, or any part of the product, such as a tag, accessory, part, stamp, or nameplate attached to the first object.

[0141] (15) In the above embodiment, the advance preparation process and the position inspection process are executed by the processing device 100 of FIG. 1. Alternatively, the advance preparation process and the position inspection process may each be executed by a different device. In this case, for example, data on the reference image RI, the label area LA, and the background mask image MI prepared by the advance preparation process are stored in a storage device of the device that executes the position inspection process. Also, all or part of the advance preparation process and the position inspection process may be executed by multiple computers (for example, so-called cloud servers) that can communicate with each other via a network. Also, the computer program that performs the position inspection process and the computer program that performs the advance preparation process may be different computer programs.

[0142] (16) In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, all or part of the preparation process and the position inspection process may be executed by a hardware circuit such as an ASIC (Application Specific Integrated Circuit).

[0143] The present invention has been described above based on examples and modifications, but the above-described embodiments of the invention are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit and scope of the claims, and equivalents thereof are also included in the present invention. [Explanation of symbols]

[0144] 1000... inspection system, 100... processing device, 110... CPU, 120... volatile storage device, 130... non-volatile storage device, 140... display unit, 150... operation unit, 170... communication unit, 300, 300S, 300R... product, 400... imaging device, AN... object detection model, Ha, Hb, Ht... coordinate transformation matrix, L, LR, LS... labels, MI... background mask image, MN... monocular depth estimation model, PG... computer program, RI... reference image, SI... target image

Claims

1. A computer program comprising: a reference image acquisition function for acquiring a reference image showing a second object including the first object; a target image acquisition function for acquiring a target image showing the second object including the first object; a pair determination function that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation function that calculates a first coordinate transformation matrix indicating a positional relationship between the reference image and the target image by using, among the plurality of feature point pairs, feature point pairs that correspond to portions different from the first object; a difference information calculation function that calculates difference information between the first object in the reference image and the first object in the target image using specific information related to the position of the first object in each of the reference image and the target image and the first coordinate transformation matrix, the difference information including information related to a difference in the relative position of the first object with respect to the second object; A computer program that makes the computer realize the above.

2. 2. The computer program of claim 1, further comprising: a computer realizing a region specifying function for specifying an object region in which the second object is located and a background region different from the object region in at least one of the reference image and the target image; the matrix calculation function calculates the first coordinate transformation matrix using the feature points pairs corresponding to a part of the object region that is different from the first object, without using the feature points pairs corresponding to the background region.

3. 3. The computer program of claim 2, further comprising: The region identification function is inputting at least one of the reference image and the target image into a monocular depth estimation model to generate a depth image; A computer program that identifies the object region and the background region in at least one of the reference image and the target image based on the depth image.

4. 4. A computer program according to claim 3, comprising: The region identification function is Assuming that the histogram of the depth image has a plurality of peaks including a first peak corresponding to the object region and a second peak corresponding to the background region, calculating the mean and variance of the first peaks; determining a binarization threshold using the average and variance of the first peak; A computer program that identifies the object region and the background region by binarizing the depth image using the binarization threshold.

5. 3. A computer program according to claim 2, comprising: the region identification function identifies the object region and the background region in the reference image; the matrix calculation function calculates the first coordinate transformation matrix using the feature points pair corresponding to a part of the object region of the reference image that is different from the first object, without using the feature points pair corresponding to the background region of the reference image.

6. 2. The computer program of claim 1, further comprising: inputting at least one of the reference image and the target image into an object detection model, and causing the computer to realize an object identification function that identifies an area in the at least one image in which the first object is located; The matrix calculation function calculates the first coordinate transformation matrix using, among the plurality of feature point pairs, feature point pairs that do not include feature points within an area in which the identified first object is located.

7. 2. The computer program of claim 1, a pair selection function is implemented in the computer to select a plurality of pairs to be used from the plurality of feature point pairs corresponding to portions different from the first object, using a matching accuracy of the feature point pairs; The matrix calculation function calculates the first coordinate transformation matrix using the selected plurality of use pairs.

8. 2. The computer program of claim 1, the specific information is information on the feature points pair corresponding to the first object among the plurality of feature points pairs.

9. 9. A computer program according to claim 8, comprising: The difference information calculation function is transforming, using the first coordinate transformation matrix, coordinates of the feature points of the target image that make up the feature point pair corresponding to the first object into coordinates of a coordinate system of the reference image; a computer program that calculates, as the difference information, information indicating a positional deviation between two feature points that constitute the feature point pair corresponding to the first object in a coordinate system of the reference image;

10. 9. A computer program according to claim 8, comprising: The difference information calculation function is transforming, using the first coordinate transformation matrix, coordinates of the feature points of the reference image that make up the feature point pair corresponding to the first object into coordinates of a coordinate system of the target image; a computer program that calculates, as the difference information, information indicating a positional shift between two feature points that constitute the feature point pair corresponding to the first object in a coordinate system of the target image;

11. 9. A computer program according to claim 8, comprising: The difference information calculation function is calculating a second coordinate transformation matrix indicating a positional relationship between the reference image and the target image using the feature point pair corresponding to the first object among the plurality of feature point pairs; A computer program product that calculates, as the difference information, information indicating a difference between the first coordinate transformation matrix and the second coordinate transformation matrix.

12. A computer program comprising: a reference image acquisition function for acquiring a reference image showing a second object including the first object; a target image acquisition function for acquiring a target image showing the second object including the first object; a pair determination function that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation function that calculates a first coordinate transformation matrix indicating a positional relationship between the reference image and the target image by using the feature point pair corresponding to the first object among the plurality of feature point pairs; a difference information calculation function that calculates difference information between the first object in the reference image and the first object in the target image using specific information related to positions of portions of the reference image and the target image that are different from the first object and the first coordinate transformation matrix, the difference information including information related to a difference in the relative position of the first object with respect to the second object; A computer program that makes the computer realize the above.

13. 13. The computer program of claim 12, further comprising: a computer realizing a region specifying function for specifying an object region in which the second object is located and a background region different from the object region in at least one of the reference image and the target image; The difference information calculation function calculates the difference information using the specific information corresponding to a part of the object region that is different from the first object, without using the specific information corresponding to the background region.

14. 14. The computer program of claim 13, further comprising: The region identification function is inputting at least one of the reference image and the target image into a monocular depth estimation model to generate a depth image; A computer program that identifies the object region and the background region in at least one of the reference image and the target image based on the depth image.

15. 15. A computer program according to claim 14, comprising: The region identification function is Assuming that the histogram of the depth image has a plurality of peaks including a first peak corresponding to the object region and a second peak corresponding to the background region, calculating the mean and variance of the first peaks; determining a binarization threshold using the average and variance of the first peak; A computer program that identifies the object region and the background region by binarizing the depth image using the binarization threshold.

16. 14. A computer program according to claim 13, comprising: the region identification function identifies the object region and the background region in the reference image; The difference information calculation function calculates the difference information using the specific information corresponding to a part of the object region of the reference image that is different from the first object, without using the specific information corresponding to the background region of the reference image.

17. 13. The computer program of claim 12, further comprising: inputting at least one of the reference image and the target image into an object detection model, and causing the computer to realize an object identification function that identifies an area in the at least one image in which the first object is located; The matrix calculation function calculates the first coordinate transformation matrix using, among the plurality of feature point pairs, feature point pairs that include the feature points within an area in which the identified first object is located.

18. 13. A computer program according to claim 12, comprising: a pair selection function that selects a plurality of pairs to be used from the plurality of feature point pairs corresponding to the first object using matching accuracy of the feature point pairs, The matrix calculation function calculates the first coordinate transformation matrix using the selected plurality of use pairs.

19. 13. A computer program according to claim 12, comprising: The specific information is information on feature points pairs that correspond to parts of the plurality of feature points pairs that are different from the first object.

20. 20. A computer program according to claim 19, comprising: The difference information calculation function is transforming, using the first coordinate transformation matrix, coordinates of feature points of the target image that make up the feature point pair corresponding to a portion different from the first object into coordinates of a coordinate system of the reference image; a computer program that calculates, as the difference information, information indicating a positional shift between two feature points that constitute the feature point pair and that correspond to a part different from the first object in a coordinate system of the reference image.

21. 20. A computer program according to claim 19, comprising: The difference information calculation function is transforming, using the first coordinate transformation matrix, coordinates of feature points of the reference image that constitute the feature point pair corresponding to a portion different from the first object into coordinates of a coordinate system of the target image; a computer program that calculates, as the difference information, information indicating a positional shift between two feature points that constitute the feature point pair and that correspond to a part different from the first object in a coordinate system of the target image.

22. 20. A computer program according to claim 19, comprising: The difference information calculation function is calculating a second coordinate transformation matrix indicating a positional relationship between the reference image and the target image using feature point pairs corresponding to portions different from the first object among the plurality of feature point pairs; A computer program product that calculates, as the difference information, information indicating a difference between the first coordinate transformation matrix and the second coordinate transformation matrix.

23. An information processing device, a reference image acquisition unit that acquires a reference image showing a second object including the first object; a target image acquisition unit that acquires a target image showing the second object including the first object; a pair determination unit that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation unit that calculates a first coordinate transformation matrix indicating a positional relationship between the reference image and the target image by using, among the plurality of feature point pairs, feature point pairs that correspond to portions different from the first object; a difference information calculation unit that calculates difference information between the first object in the reference image and the first object in the target image using specific information related to the position of the first object in each of the reference image and the target image and the first coordinate transformation matrix, the difference information including information related to a difference in the relative position of the first object with respect to the second object; An information processing device comprising:

24. An information processing device, a reference image acquisition unit that acquires a reference image showing a second object including the first object; a target image acquisition unit that acquires a target image showing the second object including the first object; a pair determination unit that performs feature point matching between the reference image and the target image to determine a plurality of feature point pairs, each of the plurality of feature point pairs being a pair of a feature point in the reference image and a feature point in the target image; a matrix calculation unit that calculates a first coordinate transformation matrix indicating a positional relationship between the reference image and the target image by using the feature point pair corresponding to the first object among the plurality of feature point pairs; a difference information calculation unit that calculates difference information between the first object in the reference image and the first object in the target image using specific information related to positions of portions of the reference image and the target image that are different from the first object and the first coordinate transformation matrix, the difference information including information related to a difference in the relative position of the first object with respect to the second object; An information processing device comprising:

Images