Computer program and abnormality detection device

Abstract

To suppress abnormality detection using an auto-encoder from decreasing in detection precision.SOLUTION: A computer program causes a computer to function to: acquire an input image including a target object and a background; perform key point matching between a reference image of the target object and the input image; calculate a homography conversion matrix showing the position relation between the input image and the reference image based upon a result of the key point matching; use the homography conversion matrix to specify an object region in the input image which corresponds to the target object; input the input image to an auto-encoder to let the auto-encoder generate an output image corresponding to the input image; and compare the object region in the input image with an object corresponding region in the output image corresponding to the object region to detect abnormality of the target object in the input image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present specification relates to a computer program for detecting an anomaly and an anomaly detection device. [Background technology]

[0002] There is a known technology for detecting anomalies in industrial products using an autoencoder such as a variational autoencoder (VAE) or a normal autoencoder (AE) (for example, Non-Patent Document 1). In this technology, input image data obtained by photographing an industrial product to be inspected is input to an autoencoder, and output image data is generated. Anomalies are detected using the input image data and the output image data. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Ryosuke Tachibana, Takashi Matsubara, Kuniaki Uehara, “Anomaly Detection in Industrial Products Using Non-Regularized Anomaly Measures by Deep Generative Models”, Proceedings of the 32nd Annual Conference of the Japanese Society for Artificial Intelligence (2018), p.2A103, published on 2018 / 07 / 30, https: / / doi.org / 10.11517 / pjsai.JSAI2018.0_2A103 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above technique does not take into account the case where the image represented by the input image data includes a background different from the object to be inspected, and the detection accuracy may be reduced due to the influence of the background.

[0005] This specification discloses a technique for suppressing a decrease in detection accuracy of anomaly detection using an autoencoder. [Means for solving the problem]

[0006] The technology disclosed in this specification can be realized in the following application examples.

[0007] [Application Example 1] A computer program that causes a computer to realize the following: an input image acquisition function that acquires an input image including a target object and a background; a matching function that performs keypoint matching between a reference image of the target object and the input image; a calculation function that calculates a homography transformation matrix that indicates the positional relationship between the input image and the reference image based on a result of the keypoint matching; a region identification function that uses the homography transformation matrix to identify an object region in the input image that corresponds to the target object; a first generation function that inputs the input image to an autoencoder and causes the autoencoder to generate an output image that corresponds to the input image; and a first detection function that detects an abnormality in the target object in the input image by comparing the object region in the input image with an object corresponding region in the output image that corresponds to the object region.

[0008] According to the above configuration, an object region in an input image is identified using a homography transformation matrix calculated based on the result of keypoint matching. Anomalies in a target object in the input image are detected by comparing the object region in the input image with an object corresponding region in an output image generated by an autoencoder. As a result, the influence of the background in the input image can be suppressed, and a decrease in the detection accuracy of anomaly detection using an autoencoder can be suppressed.

[0009] [Application Example 2] A computer program that causes a computer to realize the following: an input image acquisition function that acquires an input image including a target object and a background; a matching function that performs keypoint matching between a reference image of the target object and the input image; a calculation function that calculates a homography transformation matrix based on the result of the keypoint matching so as to match the input image with the reference image; a transformation function that transforms the input image using the homography transformation matrix to generate a transformed image; a generation function that inputs the transformed image to an autoencoder to generate a transformed output image corresponding to the transformed image; and a detection function that compares the transformed image with the transformed output image to detect an abnormality of the target object in the input image.

[0010] According to the above configuration, the input image is transformed using a homography transformation matrix calculated based on the result of keypoint matching to generate a transformed image. The transformed image is compared with a transformed output image generated by inputting the transformed image into an autoencoder to detect an abnormality in a target object in the input image. Since the shape of the target object in the transformed image is almost the same as the shape of the target object in the reference image, it is possible to prevent the transformed image from including more background than expected. As a result, it is possible to suppress the influence of the background in the input image, and therefore it is possible to suppress a decrease in the detection accuracy of anomaly detection using an autoencoder.

[0011] The technology disclosed in this specification can be realized in various forms, such as an anomaly detection device, an anomaly detection method, a computer program for realizing these devices and methods, a recording medium on which the computer program is recorded, etc. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram showing a data processing device according to an embodiment; [Diagram 2] FIG. 3 is an explanatory diagram of a product 300 and labels L1 and L2. [Diagram 3]FIG. 2 is an explanatory diagram of a reference image RI and a reference area image RA. [Figure 4] An explanatory diagram of autoencoder GN. [Diagram 5] 4 is a flowchart of an inspection process according to the first embodiment. [Figure 6] FIG. [Figure 7] FIG. 1 is an explanatory diagram of a homography transformation matrix. [Figure 8] FIG. 2 is an explanatory diagram of transformation using a homography transformation matrix H. [Figure 9] 4 is a diagram showing an example of a calculation formula for an approximate angle AG and an approximate magnification SC. [Figure 10] 4A and 4B are explanatory diagrams of an output image GI and a difference image DI. [Figure 11] 13 is a flowchart of an inspection process according to a second embodiment. [Figure 12] FIG. 13 is an explanatory diagram of a transformed output image GTI and a difference image DTI. [Figure 13] 13 is a flowchart of an inspection process according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A. First Example: A1. Equipment configuration: FIG. 1 is an explanatory diagram showing a data processing device as an embodiment. The data processing device 100 is, for example, a personal computer. In this embodiment, a label L is affixed to a product 300 (for example, a multifunction peripheral). The data processing device 100 processes image data representing the appearance of the label L in order to inspect the appearance of the label L. The data processing device 100 is used as an anomaly detection device that detects anomalies (for example, stains, scratches) in a target object (in this embodiment, the label L).

[0014] The data processing device 100 includes a processor 110, a storage device 115, a display unit 140, an operation unit 150, and a communication interface 170. These elements are connected to each other via a bus. The storage device 115 includes a volatile storage device 120 and a non-volatile storage device 130.

[0015] The processor 110 is a device configured to process data. The processor 110 may be, for example, a central processing unit (CPU) or a system on a chip (SoC). The volatile storage device 120 is, for example, a DRAM, and the non-volatile storage device 130 is, for example, a flash memory or a hard disk. The non-volatile storage device 130 stores data of a computer program PG, reference keypoints RK, and a reference area image RA. The processor 110 executes the computer program PG to perform an inspection process, which will be described later. The reference keypoints RK and the reference area image RA will be described later.

[0016] The display unit 140 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 150 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel arranged overlaid on the display unit 140. A user can input various instructions to the data processing device 100 by operating the operation unit 150. The communication interface 170 is an interface for communicating with other devices. The communication interface 170 includes, for example, one or more of a USB interface, a wired LAN interface, an IEEE802.11 wireless interface, and an industrial camera interface (for example, CameraLink, CoaXPress, etc.). The imaging device 400 is connected to the communication interface 170.

[0017] The photographing device 400 photographs the subject using a two-dimensional image sensor, thereby generating two-dimensional photographed image data representing the subject. In this embodiment, the photographing device 400 is a so-called digital camera. The photographed image data is bitmap data representing an image including a plurality of pixels. In this embodiment, the photographed image data is RGB image data representing the color of each pixel by the gradation values ​​of three color components of red R, green G, and blue B. The R value, G value, and B value are expressed, for example, in 256 gradations from zero to 255. Note that the photographed image data may represent the color of each pixel by the gradation values ​​of other color components (for example, luminance). The photographing device 400 photographs the product 300 to which the label L is attached, and generates photographed image data representing a photographed image of a portion of the product 300 including the label L.

[0018] A2. Label and photo 2 is an explanatory diagram of a product 300 and labels L1 and L2. The product 300 (FIG. 2(A)) of this embodiment is a multifunction peripheral. In the manufacturing process of the product 300, a label L is attached to a predetermined position on the outer surface of the product 300. Then, in order to inspect the appearance of the label L, the photographing device 400 photographs the label L on the product 300. When photographing the label L, the position and orientation of the photographing device 400 relative to the product 300 are adjusted so that the label L is located within the photographing range CV of the photographing device 400.

[0019] FIG. 2(B) shows a rectangular label L1, which is a rectangular sheet-like label, as an example of the label L. FIG. 2(C) shows a circular label L2, which is a circular sheet-like label, as an example of the label L. The label L is not limited to a circle or a rectangle, and may have other shapes such as a trapezoid, a parallelogram, a triangle, or a polygon with five or more sides. The surface of these labels L (L1, L2) includes at least one of characters, symbols, and figures indicating various information such as the manufacturer's name, the manufacturer's logo, a brand logo, and a certification mark. Appropriate information to be represented by the label L is associated with the product 300 in advance. An adhesive is applied to the back side of the label L for attaching the label L to the surface of the product 300.

[0020] A3. Reference image, reference area image and reference keypoints FIG. 3 is an explanatory diagram of a reference image RI and a reference area image RA. The reference image RI is a rectangular image of a predetermined size that indicates a normal label without any abnormalities. For example, an image used to produce the label L, such as a copy image or a design drawing, is used as the reference image RI. FIG. 2(A) shows a reference image RI1 of a rectangular label L1. The reference image RI1 indicates a normal rectangular label L1 (also called a normal rectangular label L1s) that is free of any abnormalities such as scratches. FIG. 2(B) shows a reference image RI2 of a circular label L2. The reference image RI2 indicates a normal circular label L2 (also called a normal circular label L2s).

[0021] The reference image RI is a digital image composed of a plurality of pixels arranged in a matrix, and is therefore rectangular. The rectangle label L1 is rectangular, so the reference image RI1 for the rectangle label L1 includes only the normal rectangle label L1s and does not include the background. The circle label L2 is circular, so the reference image RI2 for the circle label L2 includes the normal rectangle label L1s and the background BI around the normal rectangle label L1s.

[0022] The reference keypoints RK are characteristic points of a normal label shown in the reference image RI. The reference keypoints RK are, for example, points located at corners or edges of characters or marks of a normal label. FIG. 3(A) illustrates the reference keypoints RK detected in the reference image RI1 of the rectangular label L1, and FIG. 3(B) illustrates the reference keypoints RK detected in the reference image RI2 of the circular label L2. In FIG. 3(A), the three reference keypoints RKa-RKc are given the suffixes a, b, and c for the explanation to be described later. Note that a larger number of reference keypoints RK are detected (for example, tens or hundreds) than the number shown in FIGS. 3(A) and (B).

[0023] The plurality of reference keypoints RK are detected by analyzing the data of the reference image RI according to a predetermined detection method. The detection method of the reference keypoints RK may be various methods. For example, the detection method may be selected in advance from a search for extreme values ​​(maximum and minimum values) using DoG (Difference-of-Gaussian), Harris corner detection, and FAST (Features from Accelerated Segment Test) corner detection. In addition, an algorithm for detecting keypoints and calculating feature descriptors may be adopted for detecting keypoints. The feature descriptor is information describing the features of the keypoints. The algorithm for detecting keypoints and calculating feature descriptors may be selected in advance from, for example, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF). In this embodiment, the processor 110 detects keypoints according to the ORB algorithm. To detect keypoints using the ORB algorithm, for example, the function “detect” or the function “detectAndCompute” of OpenCV (Open Source Computer Vision Library) can be used.

[0024] The reference area image RA is an image showing an area (also called a label area) showing the label L in the reference image RI. For example, the reference area image RA is a rectangular image of the same size as the reference image RI. FIG. 3(C) illustrates a reference area image RA1 showing the label area OI1 of the reference image RI1 of the rectangular label L1, and FIG. 3(D) illustrates a reference area image RA2 showing the label area OI2 of the reference image RI2 of the circular label L2. The hatched areas in FIG. 3(C) and (D) indicate the label areas. The reference image RI1 of the rectangular label L1 includes only the normal rectangular label L1s and does not include the background, so the entire reference area image RA1 of the reference image RI1 is the label area OI1. The reference image RI2 of the circular label L2 includes the normal circular label L2s and the background BI, so the reference area image RA2 of the reference image RI2 includes the circular label area OI2 and the background area BA around the label area OI2.

[0025] The reference area image RA is a binary image obtained by, for example, setting the value of pixels constituting the label area to "1" and the value of pixels constituting the background area to "0." The reference area image RA is generated, for example, by an operator designating an area on the reference image RI where the label L is located, and the designated area is set as the label area and the area excluding the designated area is set as the background area.

[0026] The data of the reference keypoints RK and the data of the reference area image RA are generated in advance by a computer (the data processing device 100 or another computer) and stored in the non-volatile storage device 130 (FIG. 1). Note that the data of the reference image RI is not used in the inspection process described later, so it does not need to be stored in the non-volatile storage device 130.

[0027] A4. Autoencoder As shown in Fig. 1, the computer program PG includes a computer program that realizes the function of the autoencoder GN as a module. Fig. 4 is an explanatory diagram of the autoencoder GN. Fig. 4(A) is a schematic diagram showing an example of the configuration of the autoencoder GN. In this embodiment, the autoencoder GN includes an encoder Ve and a decoder Vd.

[0028] The encoder Ve performs a dimensionality reduction process on the input image II to extract features of the input image II and generate a feature map. In this embodiment, the encoder Ve has p (m is an integer equal to or greater than 1) convolution layers Ve21-Ve2p. A pooling layer (e.g., a max pooling layer) is provided immediately after each convolution layer. The activation function of each of the p convolution layers is, for example, ReLU.

[0029] The decoder Vd performs a dimension restoration process on the feature map to generate an output image GI, which represents an image reconstructed based on the feature map. The image size and color components of the color values ​​of each pixel of the output image GI are the same as those of the input image II.

[0030] In this embodiment, the decoder Vd has q (q is an integer equal to or greater than 1) convolution layers Vd21-Vd2q. An upsampling layer is provided immediately after each of the remaining convolution layers except for the last convolution layer Vd2q. The activation function of the last convolution layer Vd2q is a function suitable for generating an output image GI (e.g., a Sigmoid function or a Tanh function). The activation function of each of the other convolution layers is, for example, ReLU.

[0031] The convolution layers Ve21-Ve2p and Vd21-Vd2q perform processing including convolution processing and bias addition processing on the input image or feature map. Each of these convolution layers has a parameter set including multiple weights and multiple biases of multiple filters used in the convolution processing.

[0032] In this way, the autoencoder GN is a machine learning model that performs calculation processing using a plurality of parameters (weights and biases) on the data of the input image II to generate an output image GI.

[0033] Next, the training process of the autoencoder GN will be described. A trained autoencoder GN is created by this training process. The training process is executed by the data processing device 100 or another computer prior to the inspection process. In the following, the training process is executed by the data processing device 100. In this embodiment, the training process is executed for each label L to be inspected, and one autoencoder GN is trained for one type of label. For example, the autoencoder GN for inspecting the rectangular label L1 and the autoencoder GN for inspecting the label L2 are different from each other.

[0034] FIG. 4(B) is a flowchart of the training process of the autoencoder GN. Through training, a plurality of calculation parameters (including a plurality of calculation parameters used in the calculations of the convolution layers Ve21-Ve2p and Vd21-Vd2q) used in the calculations of the autoencoder GN are adjusted. Before the training process, the plurality of calculation parameters are set to initial values ​​such as random values.

[0035] In S10, the processor 110 acquires a plurality of normal images corresponding to the batch size from the non-volatile storage device 130. The normal image is an image indicating a normal label L that does not include an abnormality. For example, in the training process of the autoencoder GN for the rectangle label L1, a normal image indicating a normal rectangle label L1 is acquired. The plurality of normal images may include, for example, a photographed image generated by photographing the normal rectangle label L1 with a digital camera. The plurality of normal images may include an image generated by performing a known augmentation process on a photographed image indicating the normal rectangle label L1 or a reference image RI1. The augmentation process is a process of generating a plurality of different training images from one original image by performing image processing such as translation, enlargement / reduction, rotation, noise addition, contrast adjustment, and brightness adjustment on the original image. In S20, the processor 110 inputs a plurality of normal images to the autoencoder GN as input images II, and generates a plurality of output images GI corresponding to the plurality of normal images.

[0036] In S30, the processor 110 calculates a loss value using a plurality of normal images and a plurality of output image GIs corresponding to the plurality of normal images. Specifically, the processor 110 calculates an evaluation value indicating the difference between the normal image and the corresponding output image GI for each normal image. The loss value is, for example, the total value of the cross entropy error of the component values ​​for each pixel and each color component. The calculation of the loss value may use other known loss functions for calculating a loss value according to the difference between the component values, for example, square error and absolute error.

[0037] In S40, the processor 110 adjusts a plurality of calculation parameters of the autoencoder GN using the calculated loss value. Specifically, the processor 110 adjusts the calculation parameters according to a predetermined algorithm so that the total of the loss values ​​calculated for each normal image is reduced. As the predetermined algorithm, for example, an algorithm using the backpropagation method and the gradient descent method is used.

[0038] In S50, the processor 110 determines whether a training end condition is satisfied. The training end condition may be any of a variety of conditions, such as the loss value being equal to or less than a reference value, the amount of change in the loss value being equal to or less than a reference value, or the number of times the adjustment of the calculation parameters in S40 has been repeated being equal to or greater than a predetermined number.

[0039] If the training end condition is not satisfied (S50: NO), the processor 110 returns to S10 and continues training. If the training end condition is satisfied (S50: YES), the processor 110 stores the trained autoencoder GN data including the adjusted calculation parameters in the non-volatile storage device 130 in S60, and ends the training process.

[0040] The output image GI generated by the trained autoencoder GN indicates a reproduced image reproduced by reconstructing the features of the normal rectangle label L1 as the input image II. For this reason, the reproduced image generated by the trained autoencoder GN is an image almost similar to the image of the normal label L1 input. The trained autoencoder GN is trained to reconstruct only the features of the normal rectangle label L1. For this reason, it is expected that the reproduced image generated when an image of a rectangle label L1 including an abnormality such as a scratch is input to the trained autoencoder GN will show the normal rectangle label L1. In other words, whether an image of a normal rectangle label L1 is input to the autoencoder GN or an image of a rectangle label L1 including an abnormality is input, the reproduced image will be an image reproducing the normal rectangle label L1.

[0041] The above describes the training process of the autoencoder GN for the rectangular label L1. When training the autoencoder GN for the circular label L2, the same process is performed using a normal image indicating a normal circular label L2 instead of a normal image indicating a normal rectangular label L1.

[0042] A5. Inspection processing FIG. 5 is a flowchart of the inspection process of the first embodiment. The processor 110 of the data processing device 100 (FIG. 1) executes the computer program PG to execute the inspection process. As described in FIG. 2(A), the relative arrangement between the product 300 and the photographing device 400 is adjusted for inspection. After the product 300 is arranged, an instruction to start the inspection process is input to the data processing device 100. In this embodiment, the worker inputs the start instruction by operating the operation unit 150. The processor 110 starts the inspection process in response to the start instruction. The process of FIG. 5 shows the inspection process of one product 300. The processor 110 repeats the process of FIG. 5 to inspect each of the multiple products 300.

[0043] In S100, the processor 110 acquires data of a captured image PI including the label L to be processed. In this embodiment, the processor 110 supplies a capture instruction to the photographing device 400. In response to the capture instruction, the photographing device 400 captures an image of a part of the product 300 including the label L, and generates data of the captured image PI. The processor 110 acquires the data of the captured image PI from the photographing device 400.

[0044] Fig. 6 is an explanatory diagram of the inspection process. Fig. 6(A) shows, as an example of the captured image PI, a captured image PI1 in which the label to be processed is a rectangular label L1. The captured image PI is a rectangular image having two sides parallel to a first direction Dx and two sides parallel to a second direction Dy. The captured image PI is represented by the color values ​​(in this embodiment, the gradation values ​​of R, G, and B) of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy.

[0045] The captured image PI includes a label L to be processed and a background BK. For example, the captured image PI1 in FIG. 6(A) includes a rectangular label L1 and a background BK1 showing the surface of the product 300. The background BK1 is not necessarily a uniform image, and may include background objects Bo such as structures, patterns, and part boundaries present on the surface of the product 300. In addition, unlike the normal rectangular label L1s in the reference image RI, the rectangular label L1 may include various abnormalities such as scratches, stains, and erroneous information (e.g., typos). The rectangular label L1 in the captured image PI1 in FIG. 6(A) includes an abnormality AP (here, stains).

[0046] In S110 (FIG. 5), the processor 110 detects key points in the captured image PI by analyzing data of the captured image PI. The key points in the captured image PI are also called captured key points PK to distinguish them from reference key points RK in the reference image RI. The captured key points PK are characteristic points in the captured image PI, such as points located at corners or edges of characters or marks in a rectangular label L1 in the captured image PI, or points located at characteristic parts of a background object.

[0047] FIG. 6(B) shows the photographic key points PK detected from the photographic image PI1. In the figure, the three photographic key points PKa-PKc are given the suffixes a, b, and c for the explanation that will be described later. Although not shown in the figure, in reality, many more photographic key points PK are detected (for example, several tens or several hundreds). The method of detecting the reference key points RK may be various methods, and the same method as the above-mentioned reference key points RK (FIGS. 3(A) and (B)) is used.

[0048] In S120, the processor 110 executes keypoint matching between the photographed image PI and the reference image RI. That is, matching is performed between the multiple photographed keypoints PK in FIG. 6B and the multiple reference keypoints RK in FIG. 3A. The reference keypoints RK are detected in advance as described above, and the data of the reference keypoints RK are stored in the non-volatile storage device 130. By the keypoint matching, the photographed keypoints PK and the reference keypoints RK indicating the corresponding parts (for example, the same corners of the same character) of the rectangular label L1 in the photographed image PI and the normal rectangular label L1s in the reference image RI are associated with each other. The processor 110 extracts multiple pairs of the photographed keypoints PK and the reference keypoints RK that are associated with each other. Hereinafter, the pairs of the photographed keypoints PK and the reference keypoints RK that are associated with each other are also called matching pairs.

[0049] FIG. 6(C) is a diagram showing an example of a result of keypoint matching. Each of the lines RL in the diagram indicates a proper matching pair. Each line RL connects keypoints PK and RK that form a matching pair. The pair of keypoints PKa and RKa indicates the same corner of the same square mark. The pair of keypoints PK2 and RK2 indicates the same end of the same letter "L". The line RLz indicates an inappropriate matching pair. The pair of keypoints PK3 and RK3 indicated by the line RLz indicates different marks from each other. In this way, the result of keypoint matching may include errors.

[0050] The method of keypoint matching may be various methods. For example, the processor 110 may perform keypoint matching using a feature descriptor of the keypoint. The feature descriptor may be various information describing the features of the keypoint. The feature descriptor is calculated so as to change according to the distribution of color values ​​of multiple pixels around the keypoint, for example. The method of calculating the feature descriptor may be various methods. The algorithm for calculating the feature descriptor may be selected in advance from, for example, BRIEF (Binary Robust Independent Elementary Features), BRISK (Binary Robust Invariant Scalable Keypoints), SIFT, SURF, and ORB. In this embodiment, the processor 110 uses a feature descriptor based on the ORB algorithm.

[0051] For example, the processor 110 calculates the feature descriptor of each captured key point PK by analyzing the data of the captured image PI1. For example, the OpenCV function "compute" can be used to calculate the feature descriptor based on the ORB algorithm.

[0052] The data of the reference keypoints RK (FIG. 1) includes the coordinates of each reference keypoint RK and a feature descriptor of each reference keypoint RK. The feature descriptor of the reference keypoints RK is calculated in advance by the same method as the method for calculating the feature descriptor of the photographed keypoints PK. The data of the reference keypoints RK (FIG. 1) represents the number of reference keypoints RK and the feature descriptor of each reference keypoint RK. The processor 110 refers to the data of the reference keypoints RK to acquire the feature descriptor of each reference keypoint RK.

[0053] The method of keypoint matching using the feature descriptor may be various methods. For example, the matching method may be selected in advance from FLANN (Fast Library for Approximate Nearest Neighbor) based matching and brute force matching. For FLANN based matching, for example, matching by "cv2.FlannBasedMatcher" of OpenCV can be used. For brute force matching, for example, matching by "cv2.BFMatcher" of OpenCV can be used. In this embodiment, the processor 110 performs brute force matching.

[0054] In the brute force matching, the processor 110 associates the reference keypoint RK, which has the shortest distance to the photographic keypoint PK of interest, with the photographic keypoint PK of interest among the multiple reference keypoints RK. The processor 110 calculates the distance between the two associated keypoints PK and RK using the two feature descriptors of the two keypoints PK and RK. This distance is calculated so that a small distance indicates a high similarity between the two feature descriptors. The method of calculating such a distance may be various methods suitable for the data configuration of the feature descriptor. When the feature descriptor is expressed by a binary vector (a vector composed of one or more binary elements), such as the feature descriptors of ORB, BRIEF, and BRISK, the processor 110 may use the Hamming distance. The processor 110 may use various other distances (for example, norms such as L1 norm and L2 norm (also called Euclidean distance)) instead of the Hamming distance. The norm is applicable to various feature descriptors. In this embodiment, the processor 110 uses the Hamming distance as the distance of the feature descriptor of the ORB. The processor 110 associates each photographic keypoint PK with the reference keypoint RK with the shortest distance by brute-force matching.

[0055] Here, the processor 110 may execute a process of removing pairs with low reliability. For example, the processor 110 may sort a plurality of pairs of keypoints PK and RK in ascending order of distance, and select some of the top pairs (for example, 15% of pairs) as final pairs. The processor 110 may also perform a cross-check of the matching. When the matching result for the u-th photographed keypoint PK is the v-th reference keypoint RK, the matching result for the v-th reference keypoint RK may differ from the u-th photographed keypoint PK. The cross-check removes a pair of keypoints PK and RK when the matching results are not consistent between the pair.

[0056] In S130 (FIG. 5), the processor 110 calculates a homography transformation matrix H that matches the captured image PI with the reference image RI according to the result of keypoint matching. The homography transformation matrix is ​​a matrix that represents a homography (also called a projective transformation). The homography indicates a mapping between two projection planes of the same scene. By the homography, a straight line is transformed into a straight line, and a square is transformed into a square. A rectangle can be transformed into a trapezoid. The homography transformation matrix H can also be said to be a matrix that indicates the positional relationship between the label L in the captured image PI and the label L in the reference image RI.

[0057] FIG. 7 is an explanatory diagram of a homography transformation matrix. The homography transformation matrix H in FIG. 7(A) represents the correspondence relationship between the position Cip of a point on the captured image PI and the position Cir of a point on the reference image RI. In the figure, the positions Cip and Cir are represented by so-called homogeneous coordinates (also called projective coordinates). The two coordinates x and y of the position Cip indicate the positions (for example, pixel positions) in two mutually perpendicular directions Dx and Dy on the captured image PI (FIG. 6(C)). These coordinates x and y represent the coordinates of a two-dimensional Cartesian coordinate system representing the position of a point on the captured image PI. The third coordinate of the position Cip is fixed to 1. The same is true for the position Cir. The two coordinates x' and y' of the position Cir represent the positions in two mutually perpendicular directions Dx and Dy on the reference image RI (FIG. 6(C)). These coordinates x' and y' represent the coordinates of a two-dimensional Cartesian coordinate system representing the position of a point on the reference image RI. The coordinates (Z*x', Z*y', Z) are obtained by multiplying the coordinates (x, y, 1) of position Cip by the homography transformation matrix H. (The "*" is the multiplication symbol.) The coordinates x', y' are obtained by dividing the first and second components (Z*x', Z*y') by the third component Z.

[0058] The homography transformation matrix H represents the correspondence between two positions Cip and Cir expressed in homogeneous coordinates. The homography transformation matrix H is expressed as a matrix with 3 rows and 3 columns. The element hij indicates the element in the i-th row and j-th column. The bottom right element h33 is fixed to 1. Thus, the homography transformation matrix H has 8 degrees of freedom. The remaining 8 elements are divided into three submatrices SM1, SM2, and SM3. The first submatrix SM1 is a submatrix consisting of four elements h11, h12, h21, and h22. The second submatrix SM2 is a submatrix consisting of two elements h13 and h23. The third submatrix SM3 is a submatrix consisting of two elements h31 and h32.

[0059] The four elements h11, h12, h21, and h22 of the first submatrix SM1 represent a coordinate transformation that includes rotation, scaling (enlargement or reduction), and skew (the skew may transform a rectangle into a parallelogram). This coordinate transformation is between a two-dimensional coordinate system that indicates positions on the captured image PI (i.e., a two-dimensional coordinate system that indicates coordinates x, y) and a two-dimensional coordinate system that indicates positions on the reference image RI (i.e., a two-dimensional coordinate system that indicates coordinates x', y').

[0060] 7B shows homography transformation matrices H1-H4 that represent coordinate transformation. In these homography transformation matrices H1-H4, the elements of submatrices SM2 and SM3 are zero.

[0061] The first homography transformation matrix H1 represents a rotation of angle T. h11=h22=cos(T), h12=-sin(T), and h21=sin(T).

[0062] The second homography transformation matrix H2 represents the scaling of the magnification U. h11=h22=U, and h12=h21=zero. Note that the magnification may be different between the x-axis and the y-axis. h11 represents the magnification of the x-axis, and h22 represents the magnification of the y-axis.

[0063] The homography transformation matrices H3 and H4 represent a skew of angle V. In the third homography transformation matrix H3, h12=tan(V), h11=h22=1, and h21=zero. The third homography transformation matrix H3 tilts the y-axis by angle V. The fourth homography transformation matrix H4 is a matrix obtained by exchanging h12 and h21 of the third homography transformation matrix H3. The fourth homography transformation matrix H4 tilts the x-axis by angle V.

[0064] The first submatrix SM1 of the homography transformation matrix H (Figure 7(A)) can be decomposed into a product of multiple matrices (e.g., the first submatrix SM1 of the homography transformation matrices H1, H2, H3, and H4) each representing multiple types of coordinate transformation including rotation, scaling, and skew.

[0065] The two elements h13 and h23 of the second submatrix SM2 (FIG. 7(A)) indicate a translation between a two-dimensional coordinate system showing coordinates x and y and a two-dimensional coordinate system showing coordinates x' and y'. The two elements h31 and h32 of the third submatrix SM3 change the component Z according to the coordinates x and y. The elements h31 and h32 can transform a rectangle into a trapezoid.

[0066] The eight elements h11-h13, h21-h23, h31, and h32 of the homography transformation matrix H can be determined by using four or more correspondences of the positions Cip and Cir. For example, the homography transformation matrix H can be determined by using four matching pairs of the key points PK and RK (FIG. 6(C)).

[0067] The coordinates of the keypoints PK and RK of the matching pair (FIG. 6C) may include errors. For example, the position of the photographic keypoint PK on the rectangular label L1 may deviate from the position of the reference keypoint RK associated with the photographic keypoint PK on the normal rectangular label L1s. To mitigate the effect of such errors, in this embodiment, the processor 110 uses more than four matching pairs to calculate eight elements of the homography transformation matrix H.

[0068] There may be various methods for calculating the homography transformation matrix H. For example, the method for calculating the homography transformation matrix H may be a simple least squares method or a robust method.

[0069] In the least squares method, the homography transformation matrix H is calculated so that the sum of squares of the error is minimized. Here, as the sum of squares of the error, the sum of the square of the difference in the position in the first direction Dx and the square of the difference in the position in the second direction Dy may be used. These position differences indicate the difference in position between the transformed point obtained by transforming the photographic key point PK by the homography transformation matrix H and the reference key point RK associated with the photographic key point PK. From one matching pair, the square of the Euclidean distance between the transformed point and the reference key point RK is calculated. The sum of squares of the error may be the sum of squares of multiple distances calculated from multiple matching pairs. In this embodiment, all of the multiple matching pairs are used to calculate the sum of squares of the error. Instead of this, a part of the multiple matching pairs having a small distance of the feature descriptor may be used.

[0070] The robust method may be various methods capable of mitigating the influence of inappropriate matching pairs. For example, a method based on RANSAC (Random sample consensus) may be adopted as the robust method. In the RANSAC-based method, four matching pairs are randomly selected. Candidates for the homography transformation matrix H are calculated using the four matching pairs. A method for calculating the candidates for the homography transformation matrix H may be, for example, the least squares method. A plurality of captured key points PK are transformed according to the candidates for the homography transformation matrix H. An error (e.g., Euclidean distance) between the transformed point and the reference key point RK is calculated for each of the plurality of matching pairs. A score indicating the quality of the candidate for the homography transformation matrix H is calculated using a plurality of errors of the plurality of matching pairs. The score may be, for example, the total number of matching pairs (also called "inliers") having an error equal to or less than a predetermined threshold. The process including the selection of the four matching pairs, the calculation of the candidates for the homography transformation matrix H, and the calculation of the score is performed multiple times. Among the multiple candidates for the homography transformation matrix H, the candidate having the highest score is adopted as the homography transformation matrix H.

[0071] In this embodiment, the processor 110 calculates the homography transformation matrix H according to a method based on RANSAC. For example, the OpenCV function "cv2.findHomography" can be used to calculate the homography transformation matrix H based on RANSAC. In this function, various algorithms different from RANSAC can be used depending on the setting of a flag. For example, a simple least squares method can be selected. Also, as a robust method, LMedS (least-median of squares) or RHO can be selected instead of RANSAC. The algorithm for calculating the homography transformation matrix H may be selected from these algorithms instead of RANSAC.

[0072] FIG. 8 is an explanatory diagram of transformation using a homography transformation matrix H. FIG. 8(A) shows an explanatory diagram of a homography transformation matrix Ha calculated when the label L to be processed is a rectangular label L1. The homography transformation matrix Ha is calculated so as to match a captured image PI1 (FIG. 8(A)) including a rectangular label L1 and a background BK1 with a reference image RI1 (FIG. 3(A)). FIG. 8(A) shows a transformed image TI1 generated by transforming the captured image PI1 according to the homography transformation matrix Ha. The position and size of a rectangular label L1t in the transformed image TI1 are approximately the same as the position and size of a normal rectangular label L1s in the reference image RI1.

[0073] FIG. 8(B) shows an explanatory diagram of the homography transformation matrix Hb calculated when the label L to be processed is a circular label L2. The homography transformation matrix Hb is calculated so as to match the captured image PI2 (FIG. 8(B)) including the circular label L2 and the background BK2 with the reference image RI2 (FIG. 3(C)). FIG. 8(B) shows a transformed image TI2 generated by performing homography transformation of the captured image PI2 according to the homography transformation matrix Hb. The position and size of the circular label L2t in the transformed image TI2 are almost the same as the position and size of the normal circular label L2s in the reference image RI2. Note that the transformed images TI1 and TI2 in FIGS. 8(A) and (B) are shown for the purpose of explaining the homography transformation matrix H, but do not need to be generated in the first embodiment.

[0074] In S135, processor 110 determines whether or not the calculated homography transformation matrix H satisfies a specific tolerance condition. Specifically, processor 110 calculates an index value indicating the degree of abnormality of homography transformation matrix H using values ​​constituting homography transformation matrix H. If the index value of homography transformation matrix H is within a predetermined tolerance range, processor 110 determines that homography transformation matrix H satisfies the tolerance condition, and if the index value of homography transformation matrix H is outside the predetermined tolerance range, processor 110 determines that homography transformation matrix H does not satisfy the tolerance condition.

[0075] In this embodiment, an approximate angle AG, which is an approximation of the angle of rotation by the homography transformation matrix H, and an approximate magnification SC, which is an approximation of the scaling magnification by the homography transformation matrix H, are calculated as index values ​​indicating the degree of abnormality of the homography transformation matrix H.

[0076] FIG. 9 is a diagram showing examples of calculation formulas for the approximate angle AG and the approximate magnification SC. Five formulas EQ1-EQ5 are shown in the figure. The first formula EQ1 shows the correspondence relationship between the homography transformation matrix H, the approximate angle AG, and the approximate magnification SC. Here, it is assumed that the first submatrix SM1 of the homography transformation matrix H is expressed as the product of a matrix (matrix H1 (FIG. 7(B))) indicating the rotation of the approximate angle AG and a matrix (matrix H2 (FIG. 7(B)) indicating the scaling of the approximate magnification SC. In other words, it is assumed that the influence of other types of coordinate transformation (e.g., skew) is sufficiently small. In this case, as shown in the figure, the elements h11 and h22 are expressed as "SC*cos(AG)", the element h12 is expressed as "-SC*sin(AG)", and the element h21 is expressed as "SC*sin(AG)".

[0077] In reality, the homography transformation matrix H is affected by various parameters such as the orientation of the image capture device 400 with respect to the label L to be processed and the distortion of a lens (not shown) of the image capture device 400. The elements h11, h12, h21, and h22 of the first submatrix SM1 may deviate from values ​​calculated only from rotation and scaling. Therefore, in this embodiment, the processor 110 calculates the approximate angle AG and the approximate magnification SC using all four elements h11, h12, h21, and h22 of the first submatrix SM1.

[0078] In this embodiment, as shown in the second formula EQ2, the average of "h11" and "h22" is adopted as SC*cos(AG) (this value is called the first value Pc). As shown in the third formula EQ3, the average of "h21" and "-h12" is adopted as SC*sin(AG) (this value is called the second value Ps). From these formulas EQ2 and EQ3, formulas EQ4 and EQ5 are derived. As shown in the fourth formula EQ4, the approximation magnification SC is calculated by the square root of the sum of the square of the first value Pc and the square of the second value Ps. As shown in the fifth formula EQ5, the approximation angle AG is calculated by the inverse function of the tangent function (argument is "Ps / Pc"). In this way, SC*cos(AG) and SC*sin(AG) are each calculated using the average of the corresponding two elements of the first submatrix SM1. Therefore, the influence of parameters other than the angle of rotation and the scaling magnification can be mitigated.

[0079] Processor 110 calculates the approximate magnification SC and the approximate angle AG from the homography transformation matrix H according to equations EQ4 and EQ5 in Fig. 9. Processor 110 determines that the index value is within the allowable range when the approximate magnification SC is within the allowable magnification range (a range not less than a lower limit SC1 and not more than an upper limit SC2) and the approximate angle AG is within the allowable angle range (a range not less than a lower limit AG1 and not more than an upper limit AG2). The allowable magnification range and the allowable angle range are determined in advance.

[0080] In this embodiment, the relative position between the product 300 and the photographing device 400 when photographing the label L to be processed is adjusted to a predetermined position suitable for photographing. Therefore, usually, the approximate magnification SC is approximately constant, and the approximate angle AG is approximately constant. The approximate magnification SC is determined, for example, according to the ratio of the resolution (i.e., pixel density) between the photographed image PI and the reference image RI, and the distance between the photographing device 400 and the label L to be processed. The approximate angle AG can be, for example, approximately zero degrees.

[0081] In this embodiment, the allowable angle range and the allowable magnification range are set to ranges larger than the ranges of the approximate angle AG and the approximate magnification SC that may occur due to variations in the affixing position of the label L to be processed on the product 300 and variations in the relative position with respect to the imaging device 400. For this reason, if the approximate angle AG and the approximate magnification SC are outside the allowable ranges and the homography transformation matrix H does not satisfy the allowable conditions, it is considered that keypoint matching has not been performed appropriately in the first place.

[0082] If the homography transformation matrix H does not satisfy the tolerance condition (S135: NO), the processor 110 determines in S190 that the label L in the captured image PI is different from the normal label Ls in the reference image RI. In other words, it is determined that a label has been mislabeled, and a label different from the label L that should have been mislabeled (the label shown in the reference image RI) is affixed to the product 300. As described above, if the homography transformation matrix H does not satisfy the tolerance condition, it is considered that keypoint matching has not been performed properly, and therefore the characteristics of the label L in the captured image PI and the characteristics of the normal label Ls in the reference image RI are different, and the labels are considered to be different from each other. The processor 110 stores the inspection result indicating that a mislabel has occurred in the storage device 115 (for example, the non-volatile storage device 130) and ends the inspection process.

[0083] If the homography transformation matrix H satisfies the tolerance condition (S135: YES), it can be determined that the label L in the captured image PI is the same as the normal label Ls in the reference image RI. In this case, the process proceeds to abnormality detection. Specifically, in S140, the processor 110 calculates the inverse matrix H of the homography transformation matrix H. -1 Calculate the inverse matrix H -1 The calculation of is performed using a known calculation formula.

[0084] In S150, the processor 110 acquires data of the reference area image RA from the non-volatile storage device 130. When the label L to be processed is a rectangular label L1, data of the reference area image RA1 (FIG. 3(C)) of the rectangular label L1 is acquired, and when the label L to be processed is a circular label L2, data of the reference area image RA2 (FIG. 3(D)) of the circular label L2 is acquired.

[0085] In S160, the processor 110 calculates an inverse matrix H of the homography transformation matrix H for the reference region image RA. -1 A mask image MI is generated by performing homography transformation according to the following. FIG. 8C shows the inverse matrix Ha -1 As shown in FIG. 8C, the mask image MI is obtained by applying the inverse matrix Ha -1 The mask image MI1 is an image that is obtained by performing a homography transformation (inverse transformation) according to the following. The mask image MI1 is an image that shows a label area OA1. The label area OA1 shows an area in which a rectangular label L1 (FIG. 8(A)) is placed in the captured image PI1.

[0086] FIG. 8D shows the inverse matrix Hb -1 As shown in FIG. 8D, the mask image MI2 is obtained by subtracting the inverse matrix Hb -1 The mask image MI2 is an image obtained by performing homography transformation according to the following. The mask image MI2 is an image showing the label area OA2. The label area OA2 shows the area in which the circular label L2 (FIG. 8(B)) in the captured image PI2 is placed.

[0087] As can be seen from the above explanation, generating a mask image MI is equivalent to identifying a label area OA in which a label L is to be placed within a captured image PI.

[0088] In S170, the processor 110 inputs the captured image PI to the autoencoder GN as an input image II to generate an output image GI. FIG. 10 is an explanatory diagram of an output image and a difference image. FIG. 10 shows an output image GI1 corresponding to the captured image PI1 of the rectangular label L1 as an example of the output image GI. The output image GI1 corresponding to the captured image PI1 means the output image GI1 obtained by inputting the captured image PI1 to the autoencoder GN.

[0089] As described above, the output image GI1 includes a reproduced label L1g that reproduces the rectangle label L1 of the input photographed image PI1. For example, even if the rectangle label L1 of the input photographed image PI1 includes an abnormal AP, the reproduced label L1g of the output image GI1 is an image that does not include the abnormal AP. Here, the background BKo of the output image GI1 of FIG. 10 includes a background object Bx that corresponds to the background object Bo of the background BK1 of the photographed image PI1. The autoencoder GN is trained to reproduce the rectangle label L1, but is not trained to reproduce the background object Bo, so it cannot predict what kind of image the background object Bx will be. In the example of FIG. 10, the background object Bx is an image having a different shape from the background object Bo.

[0090] In S180 and S185, the processor 110 compares the label area OA in the captured image PI with the label area OAg in the output image GI to detect an abnormality in the label L in the captured image PI. Specifically, in S180, the processor 110 generates a difference image DI indicating the difference between the captured image PI and the output image GI. For example, the processor 110 calculates a difference value (v1-v2) between a component value v1 of a pixel in the captured image PI and a component value v2 of a corresponding pixel in the output image GI, and normalizes the difference value to a value in the range of 0 to 1. The processor 110 calculates the difference value for each pixel and for each color component, and generates a difference image DI in which the difference value is the color value of the pixel.

[0091] 10 shows, as an example of the difference image DI, a difference image DI1 between a photographed image PI1 of a rectangular label L1 and an output image GI1 corresponding to the photographed image PI1. The photographed image PI1 includes an abnormal AP, and the output image GI1 does not include the abnormal AP, so the difference image DI1 in FIG. 10 includes the abnormal AP. If the photographed image PI1 does not include the abnormal AP, the difference image DI1 does not include the abnormal AP. The photographed image PI1 includes a background object Bo in the background BK1, and the output image GI1 includes a background object Bx different from the background object Bo in the background BKo. For this reason, the difference image DI1 includes a difference object Bz indicating the difference between the background object Bo of the photographed image PI1 and the background object Bx of the output image GI1.

[0092] In S185, the processor 110 detects an abnormality in the label L based on an image in the label area OA of the difference image DI. For example, the processor 110 identifies an abnormal pixel included in the label area OA of the difference image DI. The label area OA is identified by referring to the label area OA in the mask image MI described above. An abnormal pixel is, for example, a pixel of a plurality of pixels included in the label area OA, in which at least one component value of the RGB value is equal to or greater than a threshold value TH1. An image in an area (background area) different from the label area OA of the difference image DI is not used to detect an abnormality.

[0093] For example, when the difference image DI1 in FIG. 10 is the processing target, a plurality of pixels constituting the abnormality AP are identified as abnormal pixels in the label area OA1. If the rectangular label L1 in the photographed image PI1 in FIG. 10 does not include an abnormality, the label area OA1 of the difference image DI1 does not include an abnormality, so no abnormal pixels are identified in the label area OA1. When the number of abnormal pixels identified in the difference image DI is equal to or greater than the threshold value TH2, the processor 110 determines that the label L in the photographed image PI is a defective product including an abnormality. When the number of abnormal pixels identified in the difference image DI is less than the threshold value TH2, the processor 110 determines that the label L in the photographed image PI is a normal product including no abnormality. Alternatively, the processor 110 may identify the size and position of the abnormality in the label L based on the number and position of the abnormal pixels in the difference image DI, in addition to the presence or absence of an abnormality. The processor 110 stores the inspection result (e.g., the presence or absence of an abnormality) in the storage device 115 (e.g., the non-volatile storage device 130) and ends the inspection process.

[0094] According to the present embodiment described above, the processor 110 acquires a captured image PI including a label L and a background BK (S100 in FIG. 5), and performs keypoint matching between a reference image RI of the label L and the captured image PI (S110, S120 in FIG. 5). The processor 110 calculates a homography transformation matrix H indicating a positional relationship between the captured image PI and the reference image RI based on the result of the keypoint matching (S130 in FIG. 5). The processor 110 uses the homography transformation matrix H to identify a label area OA in the captured image PI (S140-S160 in FIG. 5). The processor 110 inputs the captured image PI to an autoencoder GN, and causes the autoencoder GN to generate an output image GI corresponding to the captured image PI (S170 in FIG. 5). The processor 110 compares the label area OA in the captured image PI with the label area OAg in the output image GI (the area in the output image GI corresponding to the label area OA) to detect an abnormality in the label L in the captured image PI (S180, S185 in FIG. 5).

[0095] Since the label area OA in the captured image PI is compared with the label area OAg in the output image GI, the influence of the background BK of the captured image PI can be suppressed compared to, for example, comparing the entire captured image PI with the entire output image GI, and therefore a decrease in the detection accuracy of anomaly detection using the autoencoder GN can be suppressed.

[0096] For example, in the example of FIG. 10, the background BK1 of the captured image PI1 includes the surface pattern of the product 300 as a background object Bo. For this reason, the background object Bx also appears in the background BKo of the output image GI. Since the autoencoder GN is not trained to reproduce the surface pattern of the product 300, the background object Bx of the output image GI is likely not the same as the background object Bo of the captured image PI. In this case, the difference object Bz caused by the background objects Bo and Bx also appears outside the label area OA in the difference image DI1. In such a case, it is assumed that the entire captured image PI1 and the entire output image GI1 are compared to detect an abnormality (in other words, the abnormality is detected based on the entire difference image DI1). In this case, since the difference object Bz exists in the difference image DI1, it may be erroneously determined that there is an abnormality even if there is no abnormality AP in the rectangular label L1 of the captured image PI1. According to this embodiment, the label area OA in the captured image PI is compared with the label area OAg in the output image GI, in other words, the image in the label area OA in the difference image DI is considered, and the abnormality detection is performed without considering the image outside the label area OA. As a result, the abnormality detection is performed in the difference image DI1 without considering the difference object Bz caused by the background objects Bo and Bx, so that the influence of the background BK in the captured image PI can be suppressed, and the detection accuracy of the abnormality detection can be suppressed from decreasing.

[0097] Furthermore, according to this embodiment, when the homography transformation matrix H satisfies a specific allowable condition (YES in S135 of FIG. 5), the processor 110 determines that the label L in the captured image PI and the normal label Ls in the reference image RI are the same label, and when the homography transformation matrix H does not satisfy the specific allowable condition (NO in S135 of FIG. 5), the processor 110 determines that the label L in the captured image PI and the normal label Ls in the reference image RI are the same label (S190 of FIG. 5). As a result, it is possible to determine whether the label L in the captured image PI and the label in the reference image RI are the same label, using the homography transformation matrix H. As a result, it is possible to determine that the label L has been misapplied without executing S140 to S180, and therefore it is possible to efficiently inspect the label L.

[0098] Furthermore, according to this embodiment, when it is determined that the label L in the photographed image PI and the label in the reference image RI are the same label (YES in S135 in FIG. 5), the processor 110 detects an abnormality in the label L in the photographed image PI (S140-S185 in FIG. 5). When it is determined that the label L in the photographed image PI and the label L in the reference image RI are different labels (NO in S135 in FIG. 5), the processor 110 does not detect an abnormality in the label L in the photographed image PI (does not execute S140-S185 in FIG. 5). As a result, when the label L in the photographed image PI and the label in the reference image RI are the same label, it is possible to perform an abnormality detection of the label L, and when the label L in the photographed image PI and the label in the reference image RI are different labels, it is possible to suppress unnecessary abnormality detection.

[0099] Furthermore, according to this embodiment, the label L to be processed includes a circular label L2 having a shape other than a rectangle (FIG. 3(B)). In this case, the reference image RI2 includes a normal circular label L2s and a background BI (FIG. 3(B)). In this way, when the label L to be processed has a shape other than a rectangle, both the reference image RI1 and the captured image PI2 include the background, so that the accuracy of anomaly detection is likely to decrease due to the influence of the background. Even in such a case, by identifying the label area OA2 (FIGS. 8(B) and (D)) in the captured image PI2 and comparing the captured image PI2 with the output image (not shown) in the label area OA2, it is possible to suppress the accuracy of anomaly detection from decreasing due to the influence of the background.

[0100] Furthermore, the processor 110 acquires a reference area image RA indicating the label area OI in the reference image RI (S150 in FIG. 5), and identifies the label area OA in the captured image PI using the homography transformation matrix H and the reference area image RA (S140, S160 in FIG. 5). In this way, by using the reference area image RA, the label area OA in the captured image PI can be appropriately identified. In particular, even if the label L to be processed has a shape other than a rectangle, the label area OA in the captured image PI can be identified by transforming the reference area image RA using the inverse matrix of the homography transformation matrix H.

[0101] Furthermore, in this embodiment, the processor 110 calculates a homography transformation matrix H so as to match the label L in the captured image PI with the normal label Ls in the reference image RI (S130 in FIG. 5). The processor 110 converts the region of the label L in the reference image RI, i.e., the label region OI shown in the reference region image RA (FIGS. 3C and 3D), into the inverse matrix Ha of the homography transformation matrix H. -1 (FIGS. 8C and 8D, S140-S160 in FIG. 5). As a result, the label area OA of the captured image PI can be appropriately identified based on the area of ​​the label L in the reference image RI.

[0102] As can be seen from the above explanation, the label L in this embodiment is an example of a target object, the label area OA in the captured image PI is an example of an object area, and the label area OAg in the output image GI is an example of an object corresponding area.

[0103] B. Second Example Fig. 11 is a flowchart of the inspection process of the second embodiment. In the second embodiment, S160B-S185B in Fig. 11 are executed instead of S135-S185 in Fig. 5. Other processes in the inspection process of the second embodiment are the same as the processes with the same reference numerals in the inspection process of Fig. 5.

[0104] In S160B, the processor 110 performs homography transformation on the captured image PI according to the homography transformation matrix H calculated in S130 to generate a transformed image TI. If the label L to be processed is a rectangular label L1, homography transformation is performed on the captured image PI1 (FIG. 8(A)) according to the homography transformation matrix Ha to generate a transformed image TI1 (FIG. 8(A)). The position and size of the rectangular label L1t in the transformed image TI1 are almost the same as the position and size of the normal rectangular label L1s in the reference image RI1. The transformed image TI1 does not include a background, like the reference image RI1. If the rectangular label L1 of the captured image PI includes an abnormal AP, the rectangular label L1t in the transformed image TI also includes an abnormal APt.

[0105] When the label L to be processed is a circular label L2, homography transformation is performed on the captured image PI2 (FIG. 8(B)) according to the homography transformation matrix Hb to generate a transformed image TI2 (FIG. 8(B)). The position and size of the circular label L2 in the transformed image TI2 are almost the same as the position and size of the normal circular label L2s in the reference image RI2. The transformed image TI2 includes a background BK2t with an area similar to that of the background BI of the reference image RI2. The area of ​​the background BK2t included in the transformed image TI2 is smaller than the background BK2 included in the captured image PI2.

[0106] In S170B, the processor 110 inputs the transformed image TI to the autoencoder GN as an input image II to generate a transformed output image GTI. FIG. 12 is an explanatory diagram of a transformed output image and a difference image. FIG. 11 shows a transformed output image GTI1 corresponding to the transformed image TI1 (FIG. 8(A)) of the rectangular label L1t as an example of the transformed output image GTI. The transformed output image GTI1 corresponding to the transformed image TI1 means an output image obtained by inputting the transformed image TI1 to the autoencoder GN.

[0107] The transformed output image GTI1 includes a reproduced label L1tg that reproduces the input transformed output image GTI1 (i.e., the rectangle label L1t). For example, even if the input transformed image TI1 (rectangle label L1t) includes an abnormal APt, the reproduced label L1tg of the transformed output image GTI1 is an image that does not include the abnormal APt. Since the transformed output image GTI1 does not include a background, the transformed output image GTI1 does not include a background either.

[0108] In S180B and S190B, the processor 110 compares the converted image TI with the converted output image GTI1 to detect an abnormality in the label Lt in the converted image TI. Since the abnormality in the label Lt in the converted image TI is also included in the label L in the captured image PI, detecting an abnormality in the label Lt in the converted image TI is equivalent to detecting an abnormality in the label L in the captured image PI.

[0109] Specifically, in S180B, the processor 110 generates a difference image DTI indicating the difference between the transformed output image GTI1 and the transformed output image GTI1. The method of generating the difference image is the same as the method of generating the difference image DI between the captured image PI and the output image GI in the first embodiment.

[0110] 12 shows, as an example of the difference image DTI, a difference image DTI1 between a transformed image TI1 of a rectangular label L1t and a transformed output image GTI1 corresponding to the transformed output image GTI1. Since the rectangular label L1t of the transformed image TI1 includes an abnormal APt, the difference image DTI1 in FIG. 12 includes the abnormal APt. Note that since neither the transformed image TI1 nor the transformed output image GTI1 includes a background, the influence of the background of the captured image PI1 does not appear in the difference image DTI1.

[0111] In S185B, the processor 110 detects anomalies in the label L based on the entire difference image DI. For example, the processor 110 identifies anomalous pixels included in the difference image DI. As in the first embodiment, an anomalous pixel is a pixel in which at least one component value of the RGB value is equal to or greater than the threshold value TH1.

[0112] For example, when the difference image DTI1 in Fig. 12 is the processing target, a plurality of pixels constituting the abnormal APt are identified as abnormal pixels. As in the first embodiment, when the number of abnormal pixels identified in the difference image DTI is equal to or greater than the threshold value TH2, the processor 110 determines that the label L in the photographed image PI is a defective product containing an abnormality. When the number of abnormal pixels identified in the difference image DI is less than the threshold value TH2, the processor 110 determines that the label L in the photographed image PI is a normal product containing no abnormality. The processor 110 stores the inspection result (e.g., the presence or absence of an abnormality) in the storage device 115 (e.g., the non-volatile storage device 130) and ends the inspection process.

[0113] According to the present embodiment described above, the processor 110 acquires a captured image PI including a label L and a background BK (S100 in FIG. 11), and performs keypoint matching between a reference image RI of the label L and the captured image PI (S110, S120 in FIG. 11). The processor 110 calculates a homography transformation matrix H indicating a positional relationship between the captured image PI and the reference image RI based on the result of the keypoint matching (S130 in FIG. 11). The processor 110 transforms the captured image PI using the homography transformation matrix H to generate a transformed image TI (S160B in FIG. 11), inputs the transformed image TI to an autoencoder GN, and generates a transformed output image GTI corresponding to the transformed image TI (S170B in FIG. 11). The processor 110 compares the transformed image TI with the transformed output image GTI1 to detect an abnormality of the label L in the captured image PI (S180B, S185B in FIG. 11).

[0114] According to the above configuration, the shape and size of the label L in the converted image TI are almost the same as the shape and size of the normal label Ls in the reference image RI, so that the converted image TI can be prevented from including more background than expected. As a result, the influence of the background BK in the captured image PI can be suppressed, so that the detection accuracy of the anomaly detection using the autoencoder GN can be prevented from decreasing.

[0115] In particular, when the label L to be processed is a rectangular label L1, the reference image RI1 (FIG. 3(A)) and the transformed image TI1 (FIG. 8(A)) are images that include the rectangular label L1 and do not include a background. In this case, the transformed image TI1 that does not include a background is input to the autoencoder GN, so that the influence of the background can be eliminated and the accuracy of anomaly detection can be improved. Also, in this case, it is possible to ensure that an image that includes a background is not input to the autoencoder GN, so that the autoencoder GN can be trained using a group of normal images for learning that include only the rectangular label L1 and do not include a background. As a result, the autoencoder GN can be trained so that it can reproduce the rectangular label L1 with higher accuracy. Therefore, the accuracy of anomaly detection using the autoencoder GN can be further improved.

[0116] Even when the label L to be processed is a circular label L2, the area of ​​the background BK2t included in the converted image TI2 (FIG. 8(B)) can be minimized by using a rectangular reference image RI2 (FIG. 3(B)) circumscribing the label L2. As a result, compared to inputting the captured image PI2 directly to the autoencoder GN, the influence of the background BK2 of the captured image PI2 can be suppressed, and therefore a decrease in the detection accuracy of anomaly detection using the autoencoder GN can be suppressed.

[0117] C. Third Example Fig. 13 is a flowchart of the inspection process of the third embodiment. In S300, the processor 110 executes the processes of S100-S130 in Fig. 5. In S320, the processor 110 determines whether the label L to be inspected is a rectangular label L1 or a circular label L2.

[0118] If the label L to be inspected is a rectangular label L1 (S320: YES), the processor 110 executes the processes of S160B-S185B in Fig. 11 in S340. If the label L to be inspected is a circular label L2 (S320: NO), the processor 110 executes the processes of S140-S185 in Fig. 5 in S330.

[0119] According to the third embodiment described above, when the label L of the inspection target is a rectangular label L1, the anomaly detection of the second embodiment is executed. That is, the captured image PI1 is transformed according to the homography transformation matrix H to generate a transformed image TI1, and the transformed image TI1 is input to the autoencoder GN to generate a transformed output image GTI1. Then, the transformed image TI1 is compared with the transformed output image GTI1 to detect an anomaly.

[0120] Also, when the label L of the inspection target is not rectangular (when it is a circular label L2 in this embodiment), the anomaly detection of the first embodiment is executed. That is, the reference area image RA2 is transformed according to the inverse matrix of the homography transformation matrix Hb to generate a mask image MI2, in other words, the label area OA2 in the captured image PI2 is specified. The captured image PI2 is input to the autoencoder GN to generate an output image GI2. Then, the label area OA2 in the captured image PI2 is compared with the label area OAg of the output image GI2 to detect an anomaly.

[0121] As a result, regardless of whether the shape of the label L to be processed is rectangular or not rectangular, the influence of the background in the photographed image PI can be suppressed using the homography transformation matrix H, so that the detection accuracy of the anomaly detection using the autoencoder GN can be suppressed from decreasing. For example, when the shape of the label L to be processed is rectangular, a converted image TI1 that does not include the background can be input to the autoencoder GN, so that the influence of the background can be eliminated and the accuracy of the anomaly detection can be improved. In addition, since it is possible to ensure that an image including the background is not input to the autoencoder GN, the autoencoder GN can be trained using a group of normal images for learning that includes only the rectangular label L1 and does not include the background. As a result, the autoencoder GN can be trained so that the rectangular label L1 can be reproduced with higher accuracy. When the shape of the label L to be processed is not rectangular, the label area OA2 in the photographed image PI2 is identified, and then the label area OA2 in the photographed image PI2 is compared with the label area OAg of the output image GI2, so that the influence of the background can be eliminated and the accuracy of the anomaly detection can be improved. Furthermore, since the captured image PI2 is directly input to the autoencoder GN, the risk of a decrease in the accuracy of anomaly detection due to changes in the label L2 in the image caused by the homography transformation (e.g., changes such as enlargement / reduction or rotation) can be reduced, compared to, for example, when the image obtained by performing homography transformation on the captured image PI2 is input to the autoencoder GN.

[0122] D. Variations (1) The method of detecting key points may be various other methods instead of the above method. For example, the processor 110 may detect multiple key points by template matching using multiple template images prepared in advance. As the template image, an image of a characteristic part of the normal label Ls can be used. The processor 110 may also adopt, as a matching pair, a pair of key points detected using the same template image from two images (for example, the captured image PI and the reference image RI).

[0123] (2) The inspection process of each of the above embodiments can be modified as appropriate. For example, in the inspection process of Fig. 5, S135 and S190 may be omitted, and the processes of S140 to S185 may always be executed after S130.

[0124] 5, the processor 110 acquires data of the reference area image RA from the non-volatile storage device 130. Without being limited to this, the processor 110 may acquire the reference area image RA by generating it in S150. For example, when the label L to be processed is a rectangular label L1, the processor 110 may generate a rectangular monochromatic image of the same size as the reference image RI1 as the reference area image RA.

[0125] (4) The autoencoder GN in each of the above embodiments is not limited to a normal autoencoder, but may be, for example, a VQ-VAE (Vector Quantized Variational Auto Encoder), a VAE (Variational Autoencoder), or an autoencoder included in so-called GANs (Generative Adversarial Networks).

[0126] (5) In the inspection process of each of the above embodiments, the processor 110 generates difference images DI and DTI between an input image (a photographed image PI or a converted image TI) and an image (an output image GI or a converted output image GTI) obtained by inputting the input image to an autoencoder GN, and identifies abnormal pixels in the difference images, thereby performing abnormality detection. This is not the only method, and other methods may be used for abnormality detection. For example, the processor 110 may perform abnormality detection using a method called PaDiM. In the PaDiM method, for example, the processor 110 inputs the difference images DI and DTI to the encoder Ve of the autoencoder GN, and generates feature maps of the difference images DI and DTI. Then, the processor 110 detects abnormalities by calculating the Mahalanobis distance between the feature map of the difference image and the feature map of the difference images of multiple normal labels. The difference images of multiple normal labels are generated, for example, by inputting multiple images indicating normal labels to an autoencoder GN to generate an output image, and taking the difference between the image of the normal label and the output image. A feature map of the difference images of multiple normal labels is generated in advance by inputting multiple normal label difference images to the encoder Ve of the autoencoder GN. The PaDiM method is disclosed in, for example, the paper "T. Defard, A. Setkov, A. Loesch, and R. Audigier, "Padim: a patch distribution modeling framework for anomaly detection and localization", arXiv:2011.08785(2020),https: / / arxiv.org / abs / 2011.08785,Posted on 17 Nov 2020".

[0127] (6) The calculation method of the approximation magnification SC and the approximation angle AG may be various methods other than the method of FIG. 9. The approximation magnification SC and the approximation angle AG may be calculated based on elements forming the first submatrix SM1. That is, the approximation magnification SC and the approximation angle AG may be calculated based on one or more elements of the four elements of the first submatrix SM1. For example, the average of h11 and h22 may be used as the approximation magnification SC. The argument of the inverse function of the tangent function of the fifth equation EQ5 may be "h21 / h11" or "-h12 / h22".

[0128] 5, only one of the approximate magnification SC and the approximate angle AG may be used. That is, the processor 110 may determine that the homography transformation matrix H does not satisfy the tolerance condition in a preselected case between a first case in which the approximate angle AG is outside the allowable angle range and a second case in which the approximate magnification SC is outside the allowable magnification range.

[0129] (7) In S130 of Fig. 5, the processor 110 calculates the homography transformation matrix H so as to make the label L in the captured image PI match the normal label Ls in the reference image RI. Alternatively, in S130, the processor 110 may calculate the homography transformation matrix H so as to make the normal label Ls in the reference image RI match the label L in the captured image PI. In this case, the inverse matrix Ha in the first embodiment is -1 is calculated as the homography transformation matrix, S140 in FIG. 5 may be omitted.

[0130] (8) The process of acquiring the captured image PI used in the inspection process may be various other processes instead of the above processes. For example, the processor 110 may acquire the captured image PI by performing a correction process for correcting the characteristics of the image capturing device 400 (FIG. 2(A)), such as lens distortion correction (not shown), on the output image output from the image capturing device 400. The parameters used in the correction process are also called camera internal parameters. In addition, the captured image PI may be supplied to the data processing device 100 by another device (e.g., a computer such as a smartphone) different from the data processing device 100. The processor 110 may execute the inspection process (e.g., the inspection process of each embodiment of FIG. 5, FIG. 11, and FIG. 13) using the supplied captured image PI.

[0131] (9) The target object is not limited to the label L (FIG. 2(B)), and may be any object. The target object may be an object provided on a product (e.g., a multifunction printer). The target object may include one or more elements selected from a plurality of elements including a label sheet, a nameplate (e.g., a three-dimensional inscription), and a painted pattern.

[0132] (10) In the above third embodiment (FIG. 13), the autoencoder GN used when S330 in FIG. 13 is executed (i.e., the autoencoder to which the captured image is input (S170 in FIG. 5)) and the autoencoder GN used when S340 in FIG. 13 is executed (i.e., the autoencoder to which the converted image is input (S170B in FIG. 11)) are the same autoencoder. Alternatively, the autoencoder used when S330 in FIG. 13 is executed and the autoencoder used when S340 in FIG. 13 is executed may be different autoencoders.

[0133] Specifically, a first autoencoder trained for a label different from a rectangle and a second autoencoder trained for a label of a rectangle are prepared. Then, the computer program PG in Fig. 1 includes computer programs that realize the functions of the first autoencoder and the second autoencoder as modules. In S330 in Fig. 13, the processor 110 inputs a captured image to the first autoencoder to generate an output image from the first autoencoder (S170 in Fig. 5), and in S340 in Fig. 13, inputs a converted image to the second autoencoder to generate a converted output image from the second autoencoder (S170B in Fig. 11).

[0134] (11) The inspection process of each of the above embodiments may be executed by various other devices (e.g., digital camera, smartphone) instead of the above-mentioned data processing device 100 (FIG. 1). In addition, the inspection process of each of the above embodiments may be shared and executed by a plurality of devices (e.g., computers) that can communicate with each other via a network.

[0135] In the above embodiment and modified examples, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the function of calculating the homography transformation matrix may be realized by a dedicated hardware circuit.

[0136] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored in a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used in a state stored in the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0137] The above-mentioned examples and modifications can be combined as appropriate. The above-mentioned examples and modifications are provided to facilitate understanding of the present disclosure, and do not limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and the present invention includes equivalents thereof. [Explanation of symbols]

[0138] 100...data processing device, 110...processor, 115...storage device, 120...volatile storage device, 130...non-volatile storage device, 140...display unit, 150...operation unit, 170...communication interface, 300...product, 400...imaging device, GI...output image, GN...autoencoder, GTI...converted output image, L...label, MI...mask image, PG...computer program, PI...captured image, RA...reference area image, RI...reference image PK, RK...keypoint, TI...converted image

Claims

1. A computer program comprising: an input image acquisition function for acquiring an input image including a target object and a background; A matching function for performing keypoint matching between a reference image of the target object and the input image; A calculation function for calculating a homography transformation matrix indicating a positional relationship between the input image and the reference image based on a result of the keypoint matching; a region specifying function that specifies an object region in which the target object is located in the input image using the homography transformation matrix; a first generating function for inputting the input image to an autoencoder and causing the autoencoder to generate an output image corresponding to the input image; a first detection function for detecting an anomaly of the target object in the input image by comparing the object region in the input image with an object corresponding region in the output image corresponding to the object region; A computer program that enables a computer to realize the above.

2. 2. A computer program according to claim 1, comprising: A computer program that causes a computer to realize an item determination function that determines that the target object in the input image and the target object in the reference image are the same item if the homography transformation matrix satisfies certain tolerance conditions, and determines that the target object in the input image and the target object in the reference image are different items if the homography transformation matrix does not satisfy certain tolerance conditions.

3. 3. A computer program according to claim 2, comprising: The first detection function is detecting an anomaly in the target object in the input image when the target object in the input image and the target object in the reference image are determined to be the same item; A computer program that does not detect an abnormality in the target object in the input image when the target object in the input image and the target object in the reference image are determined to be different items.

4. 2. A computer program according to claim 1, comprising: the target object has a shape other than a rectangle; The reference image includes the target object and a background.

5. 5. A computer program as claimed in claim 4, further comprising: A reference area information acquisition function is provided to acquire reference area information indicating an area of ​​the target object in the reference image. A computer program, wherein the region identification function identifies the object region in the input image using the homography transformation matrix and the reference region information.

6. 2. The computer program of claim 1, further comprising: a shape determination function for determining whether the shape of the target object is rectangular; a transformation function for transforming the input image using the homography transformation matrix to generate a transformed input image when the shape of the target object is rectangular; a second generating function for inputting the transformed input image to the autoencoder and causing the autoencoder to generate a transformed output image corresponding to the transformed input image; a second detection function for comparing the transformed input image and the transformed output image to detect anomalies of the target object in the input image; This is realized on a computer. If the target object is not rectangular, The region identification function identifies the object region using the homography transformation matrix; The first generating function inputs the input image to the autoencoder to cause the autoencoder to generate the output image; A computer program, wherein the first detection function detects an abnormality in the target object in the input image by comparing the object region in the input image with an object corresponding region in the output image that corresponds to the object region.

7. 7. A computer program according to claim 6, comprising: The autoencoder includes a first autoencoder and a second autoencoder different from the first autoencoder; The first generating function inputs the input image to the first autoencoder to cause the first autoencoder to generate the output image; The second generating function is a computer program that inputs the transformed input image to the second autoencoder and causes the second autoencoder to generate the transformed output image.

8. 2. A computer program according to claim 1, comprising: the calculation function calculates the homography transformation matrix to match the target object in the input image with the target object in the reference image; A computer program, wherein the region identification function identifies the object region in the input image by transforming the region of the target object in the reference image using an inverse matrix of the homography transformation matrix.

9. A computer program comprising: an input image acquisition function for acquiring an input image including a target object and a background; A matching function for performing keypoint matching between a reference image of the target object and the input image; A calculation function for calculating a homography transformation matrix so as to match the input image with the reference image based on a result of the keypoint matching; a transformation function that transforms the input image using the homography transformation matrix to generate a transformed image; A generation function of inputting the transformed image to an autoencoder to generate a transformed output image corresponding to the transformed image; a detection function for detecting anomalies of the target object in the input image by comparing the transformed image with the transformed output image; A computer program that enables a computer to realize the above.

10. 10. A computer program as claimed in claim 9, comprising: A computer program, wherein the reference image and the transformed image are each an image that includes the rectangular target object and does not include a background.

11. An abnormality detection device, comprising: an input image acquisition unit that acquires an input image including a target object and a background; a matching unit that performs keypoint matching between a reference image of the target object and the input image; a calculation unit that calculates a homography transformation matrix indicating a positional relationship between the input image and the reference image based on a result of the keypoint matching; a region specifying unit that specifies an object region in which the target object is located within the input image using the homography transformation matrix; a generation unit that inputs the input image to an autoencoder and causes the autoencoder to generate an output image corresponding to the input image; a detection unit that detects an abnormality of the target object in the input image by comparing the object region in the input image with an object corresponding region in the output image that corresponds to the object region; An abnormality detection device comprising:

12. An abnormality detection device, comprising: an input image acquisition unit that acquires an input image including a target object and a background; a matching unit that performs keypoint matching between a reference image of the target object and the input image; a calculation unit that calculates a homography transformation matrix based on a result of the keypoint matching so as to match the input image with the reference image; a transformation unit that transforms the input image using the homography transformation matrix to generate a transformed image; a generation unit that inputs the transformed image to an autoencoder to generate a transformed output image corresponding to the transformed image; a detection unit that detects an anomaly of the target object in the input image by comparing the transformed image with the transformed output image; An abnormality detection device comprising:

Images