Program and inspection apparatus
A machine learning-based defect inspection system addresses the challenge of detecting streak-like defects in textiles by resizing images to emphasize their length, enhancing detection accuracy and improving defect identification.
Patent Information
- Application Number
- PCT/JP2024/016443
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-04-26
- Publication Date
- 2024-12-05
AI Technical Summary
Current defect inspection techniques, particularly for streak-like defects in textiles, face challenges in accurately detecting defects due to their elongated nature, making it difficult to distinguish them from non-defective areas, especially when the defect length is short.
A machine learning-based approach using a data processing device that acquires and processes images of textiles, employing a horizontal and vertical detection model to identify streak-like defects by resizing images to emphasize their length, thereby enhancing defect detection accuracy.
The system effectively identifies streak-like defects by outputting different results based on the presence or absence of defects, improving detection accuracy and reducing the likelihood of missing defects due to their elongated nature.
Smart Images

Figure JP2024016443_05122024_PF_FP_ABST
Abstract
Description
Program and inspection device
[0001] The present specification relates to a technique for inspecting an object for defects using a machine learning model.
[0002] Techniques for detecting defects using image data have been proposed. For example, in the technique described in Patent Document 1, defects in an object to be inspected are comprehensively determined using a minute defect inspection unit, a binarization inspection unit, an unevenness inspection unit, and a streak defect inspection unit.
[0003] Japanese Patent Application Publication No. 8-145907
[0004] However, there is room for improvement when inspecting for streak-like defects on an object to be inspected.
[0005] This specification discloses a technique for inspecting an object for streak-like defects.
[0006] The techniques disclosed in this specification can be implemented in the following application examples.
[0007] [Application Example 1] A program that causes a computer to realize the second acquisition function, which acquires a target image representing an optically read object to be inspected, and a second acquisition function that uses a machine learning model to acquire a result that is associated with a patch image that is a part of the target image, wherein the patch image has an aspect ratio that is different from the aspect ratio of the reference size of an image input to the machine learning model and an aspect ratio of 1:1, and the machine learning model is trained to output different results depending on the presence or absence of a streak-like defect extending in the longer of the vertical and horizontal directions of the patch image in the object to be inspected represented by the patch image, and an execution function that executes specific processing based on the result.
[0008] According to this configuration, the patch image, which is a part of the target image, has an aspect ratio that is different from the aspect ratio of the reference size of the image input to the machine learning model and an aspect ratio of 1:1, and the machine learning model is trained to output different results depending on the presence or absence of a defect in the object to be inspected represented by the patch image, which is a streak-like defect extending in the longer of the vertical and horizontal directions of the patch image, so that inspection of the object to be inspected can be performed appropriately.
[0009] The technology disclosed in this specification can be realized in various forms, such as an inspection method and an inspection device, a computer program for realizing the functions of the method or device, a recording medium (e.g., a non-transitory recording medium) on which the computer program is recorded, and the like.
[0010] 1 is an explanatory diagram showing a data processing device; 2 is a perspective view of a digital camera 110, a woven fabric 700, and a conveying device 900; 3A-3E are explanatory diagrams showing an example of a defect in the woven fabric 700; 4 is a flowchart showing an example of a training process; 5A-5D are diagrams showing information processed in training and inspection; 6 is a block diagram showing an example of a horizontal model 310h; 7 is a diagram showing an overview of a feature map; 8 is a flowchart showing an example of an inspection process; 9A-9C are diagrams showing examples of patch images and an anomaly map; 10A-11F are diagrams showing information processed in training and inspection; 11 is a flowchart showing an example of a training process; 12 is a diagram showing an overview of a feature map; 13 is a flowchart showing an example of an inspection process; 14A is a diagram showing a reference example of a positional relationship; 15B is a diagram showing an example of an arrangement of horizontal patch images 91s; 16 is a flowchart showing an example of an inspection process; 17 is a diagram showing an overview of a feature map; 18 is a flowchart showing an example of an inspection process; 19 is a diagram showing an example of an overall feature map; 20 is a diagram showing an example of an arrangement of horizontal patch images 91s; 21 is a diagram showing another embodiment of a horizontal patch image. 1 is a block diagram showing another example of an anomaly detection model. 2 is a diagram showing the architecture of the "Swin Transformer." 3 (A)-(C) are diagrams showing examples of information to be processed.
[0011] A. First Example: A1. Device Configuration: Fig. 1 is an explanatory diagram showing a data processing device as one example. Data processing device 200 is, for example, a personal computer. Data processing device 200 performs various data processing for inspecting the appearance of an object (for example, a woven or knitted fabric). Hereinafter, it is assumed that the appearance of a woven fabric 700 is to be inspected.
[0012] The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, a graphics processing unit 260 (referred to as GPU 260), and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.
[0013] The processor 210 is a device configured to process data, and is, for example, a central processing unit (CPU) or a system on a chip (SoC). The volatile storage device 220 is, for example, a DRAM, and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores data for a training program 232, an inspection program 233, and an anomaly detection model 310. The anomaly detection model 310 includes a horizontal detection model 310h and a vertical detection model 310v. Hereinafter, the horizontal detection model 310h will also be simply referred to as the horizontal model 310h. The vertical detection model 310v will also be simply referred to as the vertical model 310v. The detection models 310h and 310v are program modules that form a machine learning model. Details of the programs 232-233 and the detection models 310h and 310v will be described later.
[0014] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel overlaid on the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250.
[0015] The GPU 260 is a computing device configured to perform various numerical calculations such as image processing and machine learning. The GPU 260 performs various calculations in accordance with instructions from the processor 210. A driver program (not shown) for controlling the GPU 260 may be provided by the manufacturer of the GPU 260.
[0016] The communication interface 270 is an interface for communicating with other devices (for example, it includes one or more of a USB interface, a wired LAN interface, an IEEE802.11 wireless interface, and an industrial camera interface (for example, CameraLink, CoaXPress, etc.)). The digital camera 110 and the conveying device 900 are connected to the communication interface 270. The conveying device 900 is a device that conveys the textile 700. The digital camera 110 is used to photograph the textile 700.
[0017] 2 is a perspective view of the digital camera 110, the fabric 700, and the conveying device 900. In this embodiment, the fabric 700 is conveyed by the conveying device 900. The conveying device 900 has two rollers 910 and 920. The fabric 700 is guided by the two rollers 910 and 920. Between the two rollers 910 and 920, the fabric 700 forms a flat portion. The light source 120 irradiates light onto the flat portion of the fabric 700. The shooting direction D110 of the digital camera 110 faces the flat portion of the fabric 700.
[0018] The conveying device 900 has a rotary encoder 912 connected to the first roller 910 and a drive motor 922 connected to the second roller 920. The drive motor 922 rotates the second roller 920 to convey the fabric 700 from the first roller 910 to the second roller 920. The data processing device 200 can obtain the conveying position of the fabric 700 using data from the rotary encoder 912. The data processing device 200 can control the conveyance of the fabric 700 by controlling the drive motor 922.
[0019] An example of a pattern of the woven fabric 700 is shown at the bottom of FIG. 2. The woven fabric 700 is woven using warp threads Wp and weft threads Wf. The woven fabric 700 can be woven in various ways. Patterns W1, W2, and W3 in the figure represent "plain weave," "twill weave," and "satin weave," respectively. In either case, the woven fabric 700 is configured so that patterns associated with the weave are regularly arranged. When multiple specific patterns are arranged at approximately equal intervals in a specific direction, it can be said that the specific patterns are regularly arranged in a specific direction. The captured image of the woven fabric 700 generated by the digital camera 110 shows the detailed pattern of the woven fabric 700.
[0020] A2. Defects: Figures 3(A) to 3(E) are explanatory diagrams showing examples of defects in the woven fabric 700. Figure 3(A) shows an example of a captured image of the woven fabric 700. The captured image Ma is a rectangular image having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy perpendicular to the first direction Dx.
[0021] Assume that the woven fabric 700 represented by the captured image Ma has a streak-like defect 701. In the captured image Ma, the defect 701 is a linear defect extending in the first direction Dx. Such a defect 701 may be formed due to a defect in a yarn (e.g., a weft yarn Wf or a warp yarn Wp ( FIG. 2 )). For example, if the tension of a yarn during the production of the woven fabric 700 is inappropriate, a streak-like defect may be formed along the yarn. Furthermore, if the thickness of a single yarn is inappropriate, a streak-like defect may be formed along the yarn. Furthermore, the streak-like defect 701 may be formed due to contact between the woven fabric 700 and another member (e.g., a device for transporting the woven fabric 700 (not shown)).
[0022] Rectangular partial images 901, 901s, 902, and 902s are shown on the captured image Ma. The first partial image 901 represents a defect-free portion of the fabric 700. The shape of the first partial image 901 is a rectangle that is long in the first direction Dx. The first small image 901s is a square image that forms the end of the first partial image 901 on the -Dx direction side. The second partial image 902 represents a portion of the fabric 700 that has a defect 701. The shape of the second partial image 902 is the same as the shape of the first partial image 901. The defect 701 extends from the end of the second partial image 902 on the -Dx direction side to the end of the second partial image 902 on the +Dx direction side. The shape of the second small image 902s is a square image that forms the end of the second partial image 902 on the -Dx direction side. The shape of the second small image 902s is the same as the shape of the first small image 901s. Note that a square is a rectangle with four sides of the same length. A rectangle has two side pairs (a side pair consists of two opposite sides), and in a rectangle, the length of one side pair is longer than the length of the other side pair.
[0023] FIG. 3B shows an example of an enlarged view of the first small image 901s. As shown, the first small image 901s (and thus the captured image Ma) includes a first portion WP1 and a second portion WP2, which have different colors. Here, the first portion WP1 is an island-shaped portion, and the second portion WP2 is the remaining portion. The captured image Ma shows a plurality of regularly-spaced first portions WP1. The pattern represented by the first portion WP1 and the second portion WP2 represents the pattern formed by the weft thread Wf and the warp thread Wp ( FIG. 2 ). For example, in the captured image Ma, the first portion WP1 may represent the weft thread Wf, and the second portion WP2 may represent the warp thread Wp.
[0024] 3C shows an example of an enlarged view of the first partial image 901. The first partial image 901 shows a large number of first portions WP1.
[0025] FIG. 3D shows an example of an enlarged view of the second small image 902s. Similar to the first small image 901s (FIG. 3B), the second small image 902s shows multiple first portions WP1. The streak-like defect is represented by a linear portion of the fabric. In the example of FIG. 3D, the defect 701 is represented by a first portion WP1d on the line representing the defect 701 (hereinafter, these first portions WP1d are also referred to as defect portions WP1d). The appearance of the defect portion WP1d may differ from the appearance of a normal first portion WP1 that does not represent the defect 701. For example, the size of the defect portion WP1d may be larger than the size of the normal first portion WP1. Furthermore, the color of the defect portion WP1d may differ from the color of the normal first portion WP1. In either case, the appearance of the second portion WP2 on the line representing the defect 701 may be the same as the appearance of the normal second portion WP2.
[0026] Streak-like defects can be formed due to various causes. For example, defects can be formed due to defective yarns. In this case, the appearance of a portion of a pattern formed by a plurality of yarns (e.g., patterns W1-W3 in FIG. 2) formed by defective yarns can differ from the appearance of normal portions. The appearance of other portions can be the same as normal portions. Furthermore, the plurality of yarns forming the woven fabric can form small concave and convex portions on the surface of the woven fabric. When defects are formed due to contact between the woven fabric and other components, the appearance of the convex portions that are likely to come into contact with other components can differ from the appearance of normal portions, while the appearance of the concave portions can be the same as normal portions. In this way, a portion of a linear portion can exhibit a defect. That is, the appearance of a portion of the linear portion can differ from the appearance of normal portions, while the appearance of the remaining portion of the linear portion can be the same as normal portions. Furthermore, the entire linear portion can exhibit a defect. That is, the appearance of the entire linear portion can differ from the appearance of normal portions without defects.
[0027] In this way, the linear portion of the fabric corresponding to the defect may include a portion that appears normal. If the portion of the image representing the defect that appears inappropriate is small, it is not easy to detect the defect from that image. For example, if the length of the defect 701 in the second small image 902s is short, the total number of defect portions WP1d included in the second small image 902s is small. In this case, it is not easy to detect the defect 701 from the second small image 902s. To make it easier to detect the defect from the image, it is preferable that the defect included in the image is long.
[0028] FIG. 3E shows an example of an enlarged view of the second partial image 902. The defect 701 in the second partial image 902 is longer (i.e., the portion representing the defect is larger) than the defect 701 in the second small image 902s ( FIG. 3D ). Therefore, when the second partial image 902 is referenced, the defect 701 is more likely to be detected than when the second small image 902s is referenced. For example, the total number of defect portions WP1d in the second partial image 902 is greater than the total number of defect portions WP1d in the second small image 902s. As a result, the defect 701 is more likely to be detected. In this embodiment, inspection for streak-like defects is performed using images having an aspect ratio different from 1:1 (e.g., rectangular images), such as the partial images 901 and 902.
[0029] A4. Anomaly Detection Models and Their Training Process: FIG. 4 is a flowchart showing an example of the training process for the detection models 310h and 310v (FIG. 1). The detection models 310h and 310v are used to detect anomalies in the textile 700 in the inspection process described below. The horizontal model 310h is trained to detect streak-like defects extending in the first direction Dx (FIG. 3(A)). The vertical model 310v is trained to detect streak-like defects extending in the second direction Dy. The detection models 310h and 310v may be any models that detect anomalies. Furthermore, the training process for each of the detection models 310h and 310v may be any process suitable for the model.
[0030] In this embodiment, the detection models 310h and 310v are trained using images of training textiles without abnormalities (details of the detection models 310h and 310v will be described later). Here, it is assumed that photographed images of the training textiles are used. As described with reference to FIGS. 3A to 3E, to detect streak-like defects, it is preferable to use an image of an area that is long in the direction in which the defect extends. In this embodiment, a partial image that is long in the first direction Dx is used for the horizontal model 310h, and a partial image that is long in the second direction Dy is used for the vertical model 310v.
[0031] The operator attaches a training fabric (also referred to as a training fabric) to the conveying device 900. The training fabric is the fabric 700 without any abnormalities. The operator then operates the operation unit 250 of the data processing device 200 (FIG. 1) to input an instruction to start the training process. In response to the instruction, the processor 210 starts the training process in accordance with the training program 232.
[0032] In S110, the processor 210 provides a photographing instruction to the digital camera 110. The digital camera 110 photographs the training fabric 700t. The processor 210 acquires photographed image data from the digital camera 110.
[0033] 5(A)-5(D) are diagrams showing information processed in training and testing. FIGS. 5(A) and 5(B) show examples of captured images. In each figure, a training image Mt and a test image Ms, described later, are shown superimposed on each other (the reference symbols for elements related to the test image Ms are enclosed in parentheses). The captured image Mt is a rectangular image having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy perpendicular to the first direction Dx. The captured image Mt represents the training fabric 700t. Although not shown, the captured image Mt represents the first portion WP1 and the second portion WP2 described in FIG. 3(B) and elsewhere. The data of the captured image Mt is grayscale bitmap data representing the brightness value of each pixel. The color components represented by the captured image data may be other color components (for example, the three color components R (red), G (green), and B (blue)) instead of luminance.
[0034] In S130 (FIG. 4), the processor 210 acquires a plurality of horizontal patch images and a plurality of vertical patch images from the captured image Mt. FIG. 5A shows an example of a plurality of horizontal patch images 91t. The shape of the horizontal patch image 91t is a predetermined rectangle, with the long side parallel to the first direction Dx and the short side parallel to the second direction Dy. In this embodiment, the processor 210 acquires data for each horizontal patch image 91t by dividing the captured image Mt into a plurality of horizontal patch images 91t. The arrangement of the plurality of horizontal patch images 91t on the captured image Mt may be predetermined. The plurality of horizontal patch images 91t are used to train the horizontal model 310h.
[0035] FIG. 5C shows an example of a horizontal patch image 91t. Similar to the partial image 901s in FIG. 3B, the horizontal patch image 91t shows a first portion WP1 and a second portion WP2. The height Hs indicates the number of pixels in the second direction Dy, and the width W1u indicates the number of pixels in the first direction Dx. The height Hs is the same as the height accepted by the horizontal model 310h (the height Hs is also referred to as the reference height Hs). The width W1u is larger than the width accepted by the horizontal model 310h (referred to as the reference width). In this embodiment, the reference width is the same as the reference height Hs.
[0036] FIG. 5B shows an example of multiple vertical patch images 92t. The shape of the vertical patch image 92t is the same as the shape obtained by rotating the horizontal patch image 91t by 90 degrees. The long side is parallel to the second direction Dy, and the short side is parallel to the first direction Dx. In this embodiment, the processor 210 acquires data for each vertical patch image 92t by dividing the captured image Mt into multiple vertical patch images 92t. The multiple vertical patch images 92t are used to train the vertical model 310v. Although not shown, the vertical patch image 92t represents a first portion WP1 and a second portion WP2, similar to the horizontal patch image 91t ( FIG. 5B ).
[0037] In S140 (FIG. 4), the processor 210 calculates the feature amount for each patch image. In this embodiment, S140 includes S142, S144, S146, and S148.
[0038] In S142, the processor 210 generates compressed horizontal patch image data by compressing (i.e., resizing) the horizontal patch image in the horizontal direction. FIG. 5C shows an example of a compressed horizontal patch image. The compressed horizontal patch image 91tc is generated by horizontally compressing (here, resizing in the first direction Dx) the horizontal patch image 91t. The width Ws of the compressed horizontal patch image 91tc indicates the number of pixels in the first direction Dx. The width Ws is the same as the reference width accepted by the horizontal model 310h (the width Ws is also referred to as the reference width Ws; in this embodiment, Ws = Hs). The height Hs of the compressed horizontal patch image 91tc is the same as the height Hs of the horizontal patch image 91t. The image resizing (also called resolution conversion) may be any known method (e.g., bicubic method, bilinear method, etc.). The processor 210 generates a compressed horizontal patch image 91tc for each of the multiple horizontal patch images 91t.
[0039] In S144 (FIG. 4), the processor 210 calculates the feature quantities of each of the compressed horizontal patch images 91tc using the horizontal model 310h (FIG. 1).
[0040] FIG. 6 is a block diagram illustrating an example of the horizontal model 310h. In this embodiment, the horizontal model 310h is configured using an anomaly detection technology called "PaDiM" (a Patch Distribution Modeling Framework for Anomaly Detection and Localization). "PaDiM" is disclosed, for example, in the paper "T. Defard, A. Setkov, A. Loesch, and R. Audigier, "Padim: a patch distribution modeling framework for anomaly detection and localization", arXiv:2011.08785(2020), https: / / arxiv.org / abs / 2011.08785, posted on 17 Nov 2020."
[0041] In the "PaDiM" technology, a feature vector is generated from image data using a feature extraction model. Multiple feature vectors are generated in advance from multiple image data representing normal objects, and a mean vector and covariance matrix indicating the distribution of the multiple feature vectors are calculated (the set of mean vector and covariance matrix is also called Gaussian parameters). The Gaussian parameters are calculated through a training process. In an inspection process, a feature vector is generated from image data representing the object to be inspected. The distance (here, the Mahalanobis distance) obtained from the feature vector and the Gaussian parameters is then calculated as an anomaly score.
[0042] In this embodiment, a portion of an image classification model 310t for classifying images into multiple classes is used as the feature extraction model 310a. Input image data Id is input to the image classification model 310t. The input image data Id is bitmap data representing a rectangular image having two sides parallel to a first direction Dx and two sides parallel to a second direction Dy perpendicular to the first direction Dx. The image classification model 310t uses the input image data Id to generate output data Od indicating the type of the input image data Id (e.g., the confidence level for each class).
[0043] Various models can be used as the image classification model 310t. In this embodiment, a known model called ResNet18 is used as the image classification model 310t. ResNet18 is disclosed in, for example, the paper "K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in ICML, 2016."
[0044] In this example, the image classification model 310t includes multiple processing blocks BC1-BC4 and BF. Each of the processing blocks BC1-BC4 is a convolutional neural network (CNN). Each of the processing blocks BC1-BC4 includes one or more convolutional layers. The processing block BF includes one or more fully connected layers. The processing block BF generates output data Od by reducing the dimension of the feature map from the previous processing block.
[0045] The image classification model 310t has multiple calculation parameters. For example, each convolution layer has filter weights and biases used in the convolution calculation. Each fully connected layer has weights and biases used in the calculation. The multiple calculation parameters are adjusted by training the image classification model 310t. Note that data of various trained image classification models is publicly available on the Internet. In this embodiment, data of a publicly available trained image classification model is used as the image classification model 310t. The trained image classification model has been trained using a large amount of image data (e.g., a dataset called ImageNet, which is classified into 1,000 classes). Such a trained model can appropriately extract image features of the textile 700.
[0046] To generate the feature vector, multiple feature maps obtained from multiple convolutional layers of the image classification model 310t are used. In this embodiment, the processor 210 uses a first feature map fm1 from the convolutional layer included in the first processing block BC1, a second feature map fm2 from the convolutional layer included in the second processing block BC2, and a third feature map fm3 from the convolutional layer included in the third processing block BC3. The feature extraction model 310a included in the horizontal model 310h is the first part of the image classification model 310t, and the part including the processing blocks BC1-BC3 is used. Note that the feature extraction model 310a may include the entire image classification model 310t.
[0047] The feature maps fm1, fm2, and fm3 each represent the feature values of a plurality of positions (corresponding to image pixels) aligned along the first direction Dx and the second direction Dy. The width W indicates the number of positions in the first direction Dx of the first feature map fm1, and the height H indicates the number of positions in the second direction Dy of the first feature map fm1. The feature maps fm1, fm2, and fm3 have different resolutions. The order of increasing resolution is fm1, fm2, and fm3.
[0048] The numbers c1, c2, and c3 in the figure indicate the number of channels of the feature maps fm1, fm2, and fm3, respectively, where c1<c2<c3.
[0049] The processor 210 converts the feature maps fm2 and fm3 into feature maps fm2r and fm3r having the same resolution as the largest feature map fm1 by a known enlargement process (i.e., a resolution conversion process, e.g., nearest neighbor). The resolution conversion is performed for each channel. These feature maps fm1, fm2r, and fm3r collectively represent W*H*(c1+c2+c3) feature quantities.
[0050] The processor 210 forms a feature matrix FM using R channels among the c1+c2+c3 channels of the feature maps fm1, fm2r, and fm3r (R<c1+c2+c3). The R channels may be selected in advance. Alternatively, the processor 210 may select the R channels randomly (for example, using random numbers). The feature matrix FM is composed of H*W*R elements. The feature matrix FM represents the feature vectors of the H*W positions. The feature vector xij in the figure represents the feature vector at position (i, j) (i represents the position in the first direction Dx, and j represents the position in the second direction Dy). The feature vector xij is an R-dimensional vector having R elements.
[0051] In this manner, in S144 ( FIG. 4 ), the processor 210 calculates the feature matrix F M for each of the multiple compressed horizontal patch images 91tc (the feature matrix F M is an example of a feature quantity). The processor 210 stores the calculated feature matrix F M data in the storage device 215 (e.g., the non-volatile storage device 230).
[0052] FIG. 7 is a diagram illustrating an overview of feature maps processed by the horizontal model 310h. The diagram shows examples of feature maps fm1 and fm2 obtained from the compressed horizontal patch image 91tc. In the diagram, the respective sizes of the image, filter, and map are expressed as "height * width" or "height * width * number of channels." FIG. 7 illustrates an example in which the following configurations CA1-CA5 are applied: (CA1) Hs = Ws = 256. (CA2) The first processing block BC1 is a convolution layer using 3*3 convolution filters. The total number of convolution filters is c1. The first processing block BC1 generates a first feature map fm1 from the compressed horizontal patch image 91tc. The size of the first feature map fm1 is 256 * 256 * c1. (CA3) The second processing block BC2 is composed of a downsampling layer BC2a and a convolution layer BC2b. The downsampling layer is also called a pooling layer. (CA4) The downsampling layer BC2a generates an intermediate feature map tc2a by reducing the height and width of the first feature map fm1 by half. The size of the intermediate feature map tc2a is 128*128*c1. The downsampling layer BC2a may be a variety of pooling layers, such as max pooling or average pooling. (CA5) The convolution layer BC2b is a convolution layer that uses 3*3 convolution filters. The total number of convolution filters is c2. The convolution layer BC2b generates a second feature map fm2 from the intermediate feature map tc2a. Each convolution filter may reference one or more channels of the intermediate feature map tc2a. The size of the second feature map fm2 is 128*128*c2. Note that the configurations of the processing blocks BC1 and BC2 are not limited to the above configurations CA1-CA5, and various configurations may be used.
[0053] In S146 ( FIG. 4 ), the processor 210 generates data of a compressed vertical patch image by compressing (i.e., resizing) the vertical patch image in the vertical direction. The processing of S146 is the same as the processing of S142, except that the image to be processed is a vertical patch image and the resizing direction is the second direction Dy. Although not shown, the compressed vertical patch image has a reference height Hs and a reference width Ws, similar to the compressed horizontal patch image 91tc ( FIG. 5(B) ).
[0054] In S148 (FIG. 4), the processor 210 calculates feature quantities for each of the multiple compressed vertical patch images. The feature quantities for the compressed vertical patch images are calculated using the vertical model 310v (FIG. 1). Although not shown, the configuration of the vertical model 310v is the same as the configuration of the horizontal model 310h (FIG. 6). The processor 210 calculates feature maps fm1, fm2, and fm3 for each of the multiple compressed vertical patch images. The processor 210 calculates a feature matrix FM using the calculated feature maps fm1, fm2, and fm3. The processor 210 stores the data of the feature matrix FM in the storage device 215 (e.g., the non-volatile storage device 230).
[0055] Here, the total number of horizontal patch images 91t and the total number of vertical patch images 92t are each assumed to be N. The total number N may be any number suitable for training the detection models 310h and 310v (e.g., several hundred or several thousand). In S110, the processor 210 may acquire data of multiple captured images to obtain N horizontal patch images 91t and N vertical patch images 92t. Then, in S130, the processor 210 may acquire multiple horizontal patch images 91t and multiple vertical patch images 92t from each of the multiple captured images. It is preferable that the portions of the training textile 700t represented by the captured images differ between the multiple captured images. For example, the processor 210 may cause the conveying device 900 to convey the textile 700t each time the textile 700t is captured.
[0056] In S290, the processor 210 trains the detection models 310h and 310v using a plurality of feature quantities (here, the feature matrix F) of a plurality of patch images. In this example, S290 includes training the horizontal model 310h (S292) and training the vertical model 310v (S294).
[0057] At S292, the processor 210 trains the landscape model 310h using the N feature matrices F for the N compressed landscape patch images 91tc. The N compressed landscape patch images 91tc are shown at the bottom of Figure 6. At S144 (Figure 4), the processor 210 calculates N feature matrices F from the N compressed landscape patch images 91tc. This results in N feature vectors xij being calculated for each of the H*W positions.
[0058] The lower part of FIG. 6 shows N feature vectors xij,1-xij,N calculated from the N compressed horizontal patch images 91tc. These N feature vectors correspond to positions (i, j). The processor 210 calculates a mean vector μij and a covariance matrix Σij using the N feature vectors xij,1-xij,N. The mean vector μij indicates the average feature vector xij obtained from the image of the normal textile 700t. The covariance matrix Σij is a matrix of covariances between the elements of multiple feature vectors xij obtained from the image of the normal textile 700t. The covariance matrix Σij is calculated according to the calculation formula described in the above-mentioned paper on "PaDiM." The set of the mean vector μij and the covariance matrix Σij is also called Gaussian parameters.
[0059] The processor 210 calculates Gaussian parameters for each of the H*W positions. As a result, the processor 210 determines the value of each element of the Gaussian matrix GM, which is composed of the H*W Gaussian parameters. The processor 210 stores data for the horizontal model 310h, which includes the feature extraction model 310a and the Gaussian matrix GM, in the storage device 215 (here, the non-volatile storage device 230). The processor 210 then completes S292 ( FIG. 4 ), i.e., ends training of the horizontal model 310h. As described above, in this embodiment, the value of each element of the Gaussian matrix GM is determined by the training process.
[0060] In S294, the processor 210 trains the vertical model 310v. The training method for the vertical model 310v is the same as the training method for the horizontal model 310h (S292). As described above, in S148, the processor 210 calculates N feature matrices FM from the N compressed vertical patch images. As a result, N feature vectors xij are calculated for each of the H*W positions. In S294, the processor 210 uses the N feature vectors xij to calculate Gaussian parameters (i.e., a set of mean vectors μij and covariance matrices Σij). The processor 210 calculates the Gaussian parameters for each of the H*W positions. As a result, the processor 210 determines the value of each element of the Gaussian matrix GM. The processor 210 stores data for the vertical model 310v, including the feature extraction model 310a and the Gaussian matrix GM, in the storage device 215 (here, the non-volatile storage device 230). Then, the processor 210 ends S294 (FIG. 4), i.e., the training of the vertical model 310v, and S290 ends, ending the training process of FIG.
[0061] The feature extraction model 310 a may be common to the horizontal model 310 h and the vertical model 310 v. Alternatively, the feature extraction model 310 a may be different between the horizontal model 310 h and the vertical model 310 v. The Gaussian matrix G M is different between the horizontal model 310 h and the vertical model 310 v.
[0062] A5. Inspection Process: Figure 8 is a flowchart showing an example of the inspection process. The operator attaches the fabric 700 to be inspected (also referred to as the target fabric 700) to the conveying device 900 (Figure 2). The operator then operates the operation unit 250 of the data processing device 200 (Figure 1) to input an instruction to start the inspection process. In response to the instruction, the processor 210 starts the inspection process in accordance with the inspection program 233.
[0063] In S710, the processor 210 controls the drive motor 922 (FIG. 2) to transport the target fabric 700 so as to capture images of uninspected portions of the target fabric 700. Note that in the first S710 after the start of the inspection process, transportation of the target fabric 700 may be omitted.
[0064] In S720, the processor 210 provides a photographing instruction to the digital camera 110. The digital camera 110 photographs the target fabric 700. The processor 210 acquires photographed image data from the digital camera 110.
[0065] As shown in Figures 5(A) and 5(B), in this embodiment, the shape of the test image Ms is the same as the shape of the training image Mt. Like the training image Mt, the test image Ms includes a plurality of horizontal patch images 91s and a plurality of vertical patch images 92s. In this embodiment, the shape and arrangement of the test horizontal patch images 91s are the same as the shape and arrangement of the training horizontal patch images 91t. The shape and arrangement of the test vertical patch images 92s are the same as the shape and arrangement of the training vertical patch images 92t.
[0066] The arrangement of the horizontal patch images 91s in the photographed image Ms may be different from the arrangement of the horizontal patch images 91t in the photographed image Mt. The arrangement of the vertical patch images 92s in the photographed image Ms may be different from the arrangement of the vertical patch images 92t in the photographed image Mt.
[0067] Fig. 5(D) shows an example of a horizontal patch image 91s. Similar to the horizontal patch image 91t in Fig. 5(C), the horizontal patch image 91s represents a first portion WP1 and a second portion WP2. The horizontal patch image 91s in Fig. 5(D) represents a portion of the fabric 700 that has a defect 701.
[0068] In S730 (FIG. 8), the processor 210 performs horizontal compression on the entire captured image. The horizontal compression method is the same as the horizontal compression method in S142 (FIG. 4). The processor 210 generates horizontally compressed captured image data by resizing in the first direction Dx (the horizontally compressed captured image is also referred to as a horizontally compressed image). Here, the ratio of the length in the first direction Dx after compression to the length in the first direction Dx before compression is the same as the ratio in S142 (FIG. 4).
[0069] FIG. 7 shows an example of a compressed captured image. FIG. 7 shows an example in which the height and width of the horizontally compressed image Msc are 1024. The horizontally compressed image Msc represents a plurality of compressed horizontal patch images 91sc. The compressed horizontal patch image 91sc is the same as the image obtained by horizontally compressing (here, resizing in the first direction Dx) the horizontal patch image 91s (FIG. 5(A)). In S730 (FIG. 8), the processor 210 generates a compressed horizontal patch image 91sc for each of the plurality of horizontal patch images 91s by resizing the entire captured image.
[0070] FIG. 5D shows an example of a compressed horizontal patch image. The compressed horizontal patch image 91sc is generated by horizontally compressing the horizontal patch image 91s (here, resizing in the first direction Dx). When the horizontal patch image 91s represents a defect 701, the compressed horizontal patch image 91sc also represents the defect 701. The uncompressed horizontal patch image 91s in the figure shows a partial image 91ss, which is a part of the horizontal patch image 91s. The size of the partial image 91ss is the same as the size of the compressed horizontal patch image 91sc. The partial image 91ss represents the defect 701 having the same length as the reference width Ws. On the other hand, the compressed horizontal patch image 91sc represents the defect 701 having the same length as the width W1u (however, in the compressed horizontal patch image 91sc, the length of the defect 701 has been compressed from the width W1u to the reference width Ws). As such, since the length of the defect 701 contained in the compressed horizontal patch image 91sc (i.e., the length before compression) is long, detecting the defect 701 in the compressed horizontal patch image 91sc is easier than detecting the defect 701 in the uncompressed partial image 91ss of the same size.
[0071] In S740 ( FIG. 8 ), the processor 210 calculates the feature amounts of each of the multiple compressed horizontal patch images 91sc by performing feature amount calculation processing on the entire horizontally compressed image Msc. The feature amount calculation method is similar to the calculation method in S144 ( FIG. 4 ). The processor 210 executes the operations (here, operations using convolution filters) of the processing blocks BC1, BC2, and BC3 ( FIGS. 6 and 7 ) of the horizontal model 310h using the horizontally compressed image Msc. As a result, the processor 210 generates data of feature maps fm1, fm2, and fm3 corresponding to each of the multiple compressed horizontal patch images 91sc (i.e., the multiple horizontal patch images 91s ( FIG. 5(A))).
[0072] Here, the generation of feature maps fm1, fm2, and fm3 will be described assuming that the configuration of FIG. 7 (configurations CA1-CA5) is applied. The processor 210 generates data for feature map Msc1 by performing the calculations of the first processing block BC1 (here, calculations using a convolution filter) on the entire horizontally compressed image Msc (the feature map Msc1 is also referred to as the first overall feature map Msc1). In the example of FIG. 7, the size of the first overall feature map Msc1 is 1024*1024*c1. The first overall feature map Msc1 has multiple first partial regions sc1 corresponding to multiple compressed horizontal patch images 91sc of the horizontally compressed image Msc. One first partial region sc1 represents the first feature map fm1 of the corresponding compressed horizontal patch image 91sc.
[0073] The processor 210 generates data for an intermediate feature map Msc2a by performing the calculations using the downsampling layer BC2a on the entire first global feature map Msc1. The size of the intermediate feature map Msc2a is 512*512*c1. The intermediate feature map Msc2a has a plurality of intermediate partial regions sc2a. The plurality of intermediate partial regions sc2a correspond to the plurality of first partial regions sc1 of the first global feature map Msc1, i.e., the plurality of compressed horizontal patch images 91sc of the horizontally compressed image Msc.
[0074] The processor 210 generates data for the feature map Msc2 by performing the calculations using the convolutional layer BC2b on the entire intermediate feature map Msc2a (the feature map Msc2 is also referred to as the second global feature map Msc2). The size of the second global feature map Msc2 is 512*512*c2. The second global feature map Msc2 has multiple second partial regions sc2. The multiple second partial regions sc2 correspond to the multiple intermediate partial regions sc2a of the intermediate feature map Msc2a, i.e., the multiple compressed horizontal patch images 91sc of the horizontally compressed image Msc. One second partial region sc2 represents the second feature map fm2 of the corresponding compressed horizontal patch image 91sc.
[0075] Although not shown, the processor 210 generates data for a third overall feature map by performing the calculations of the third processing block BC3 ( FIG. 6 ) on the entire second overall feature map Msc2. The third overall feature map has multiple subregions corresponding to the multiple compressed horizontal patch images 91sc of the horizontally compressed image Msc. Each subregion represents a third feature map fm3 of the corresponding compressed horizontal patch image 91sc.
[0076] In this manner, in S740 (FIG. 8), the processor 210 executes feature calculation processing (here, processing of processing blocks BC1, BC2, and BC3) for the entire horizontally compressed image Msc (FIG. 7). As a result, the processor 210 generates data for feature maps fm1, fm2, and fm3 for each of the multiple compressed horizontal patch images 91sc (i.e., the multiple horizontal patch images 91s (FIG. 5A)). The processor 210 calculates the feature matrix FM using the feature maps fm1, fm2, and fm3. The method for calculating the feature matrix FM is the same as the method described in S144 (FIG. 4). Here, the R channels used during training are used. The processor 210 stores the data for the feature matrix FM for each of the multiple compressed horizontal patch images 91sc (i.e., the multiple horizontal patch images 91s) in the storage device 215 (e.g., the non-volatile storage device 230).
[0077] In S750, the processor 210 performs vertical compression of the entire captured image. The only difference between the vertical compression and the horizontal compression in S730 is that the direction of vertical compression (i.e., resizing) is the second direction Dy. Although not shown, the vertically compressed captured image represents a plurality of compressed vertical patch images. The compressed vertical patch images are the same as the image obtained by vertically compressing the vertical patch image 92s (FIG. 5B).
[0078] In S760, the processor 210 calculates feature amounts (here, a feature matrix FM) corresponding to each of the multiple compressed vertical patch images by performing feature amount calculation processing on the entire vertically compressed captured image. The processing in S760 is similar to the processing in S740. The processor 210 executes calculations of multiple processing blocks (here, three processing blocks corresponding to processing blocks BC1, BC2, and BC3 in FIG. 6 ) of the vertical model 310v using the vertically compressed captured image. As a result, the processor 210 generates data of feature maps fm1, fm2, and fm3 corresponding to each of the multiple compressed vertical patch images (i.e., the multiple vertical patch images 92s ( FIG. 5(B))). The processor 210 calculates the feature matrix FM using the feature maps fm1, fm2, and fm3. The processor 210 stores the data of the feature matrix FM for each of the multiple compressed vertical patch images (i.e., the multiple vertical patch images 92s) in the storage device 215 (e.g., the non-volatile storage device 230).
[0079] In S890, the processor 210 determines whether the target textile 700 has an anomaly (i.e., a defect) using the feature matrix F of each of the multiple compressed horizontal patch images 91sc (FIG. 7) and the horizontal model 310h. The determination method may be any method suitable for the horizontal model 310h. In this embodiment, the processor 210 calculates an anomaly score for each of the H*W positions (FIG. 6) using the method described in the above-mentioned "PaDiM" paper. Specifically, the processor 210 obtains a feature vector xij represented by the feature matrix F associated with the compressed horizontal patch image 91sc (and thus the horizontal patch image 91s (FIG. 5(A))). The processor 210 calculates the anomaly score using the feature vector xij and the Gaussian parameters (μij, Σij). The anomaly score is calculated as the Mahalanobis distance between the feature vector xij and the Gaussian parameters (μij, Σij). (The anomaly score is calculated using the formula described in the above-mentioned paper in "PaDiM.") The anomaly score is calculated for each of the H*W positions. The anomaly score set SS in FIG. 6 indicates a set of H*W anomaly scores.
[0080] As described in S290 of Figure 4, the Gaussian parameters are trained using images of normal, defect-free training fabric 700t. Thus, if lateral patch image 91s represents a defect-free portion of target fabric 700, a small anomaly score may be calculated. If lateral patch image 91s represents a defect-free portion of target fabric 700, a large anomaly score may be calculated. A large anomaly score indicates a high probability of an anomaly.
[0081] The anomaly score set SS forms a so-called anomaly map (also called a heat map). FIGS. 9A to 9C are diagrams showing examples of patch images and anomaly maps. Note that the resolution of the anomaly map is lower than that of the patch images. In the figures, to facilitate comparison between the patch images and the anomaly map, the sizes of the patch images and the anomaly map are adjusted so that the sizes in the first direction Dx are the same between the patch images and the anomaly map, and the sizes in the second direction Dy are the same between the patch images and the anomaly map.
[0082] 9A shows a horizontal patch image 91sa representing a normal portion of the target fabric 700a, and an abnormality map 91Ha. As shown, the abnormality map 91Ha does not show any abnormal portions with large abnormality scores.
[0083] 9B shows a horizontal patch image 91sb representing a portion of the target fabric 700b having a linear defect 701b extending in the first direction Dx, and an abnormality map 91Hb. The abnormality map 91Hb shows a linear abnormal portion ANb representing the defect 701b.
[0084] The processor 210 calculates an anomaly score for each of a plurality of positions (H*W positions ( FIG. 6 )) for each horizontal patch image 91s. The processor 210 uses the plurality of anomaly scores to determine whether an anomaly condition for determining that the target textile 700 has an anomaly is met. The anomaly condition may be various conditions. For example, the anomaly condition may be that the largest anomaly score is equal to or greater than a first threshold. The first threshold may be determined experimentally in advance so as to appropriately detect an anomaly in the target textile 700. The anomaly condition may be that the total number of pixels having an anomaly score equal to or greater than a second threshold is equal to or greater than a third threshold. The second and third thresholds may be determined experimentally in advance so as to appropriately detect an anomaly in the target textile 700. The anomaly condition may be determined for each patch image. Alternatively, the anomaly condition may be determined using all of the anomaly scores of the captured image.
[0085] In S892 (FIG. 8), the processor 210 uses the feature matrix F of each of the multiple compressed vertical patch images and the vertical model 310v to determine whether the target textile 700 has an anomaly (i.e., a defect). The determination method is similar to the determination method in S890. The processor 210 obtains a feature vector x represented by the feature matrix F associated with the compressed vertical patch image (and thus the vertical patch image 92s (FIG. 5(B))). The processor 210 calculates an anomaly score using the feature vector x and the Gaussian parameters (μ, Σ). The anomaly score is calculated for each of the H*W positions.
[0086] 9C shows a vertical patch image 92sc representing a portion of the target fabric 700c having a linear defect 701c extending in the second direction Dy, and an abnormality map 92Hc. The abnormality map 92Hc shows a linear abnormal portion ANc representing the defect 701c.
[0087] Although not shown, when the vertical patch image represents a normal portion of the target fabric, an abnormality map is generated that does not show abnormal portions with large abnormality scores, similar to the abnormality map 91Ha in FIG. 9(A).
[0088] The processor 210 calculates an anomaly score for each of a plurality of positions (H*W positions (FIG. 6)) for each vertical patch image 92s. The processor 210 uses the plurality of anomaly scores to determine whether an anomaly condition for determining that the target textile 700 has an anomaly is met. The anomaly condition in S892 (FIG. 8) is the same as the anomaly condition in S890.
[0089] In S894, the processor 210 determines whether the abnormal condition is satisfied. If the abnormal condition is not satisfied (S894: No), in S896, the processor 210 determines whether the conveyance is completed. If an uninspected portion of the target fabric 700 remains, the determination result in S896 is No. In this case, the processor 210 proceeds to S710 and inspects the uninspected portion of the target fabric 700. If the conveyance is completed (S896: Yes), the processor 210 ends the inspection process.
[0090] If the abnormality condition is met in S894 (S894: Yes), the processor 210 executes a specific process in S898. The specific process may be any process related to the abnormality. In this embodiment, the processor 210 stops the conveyance of the target fabric 700 and notifies the operator of the abnormality. Various methods may be used to notify the operator of the abnormality. For example, the processor 210 may display an error message or an abnormality map on the display unit 240. Alternatively, the processor 210 may output a warning sound from a speaker (not shown). After S898, the processor 210 ends the inspection process.
[0091] As described above, in this embodiment, the processor 210 of the data processing device 200 executes the following process to inspect the target fabric 700. In S720 ( FIG. 8 ), the processor 210 acquires data of a captured image of the target fabric 700. The target fabric 700 is an example of an object to be inspected. The captured image is an example of an object image that represents the object to be inspected as optically read.
[0092] In steps S730, S740, and S890 (FIG. 8), the processor 210 uses the horizontal model 310h to obtain an abnormality score set SS associated with a horizontal patch image 91s, which is a part of the captured image. As described in FIG. 6, in this embodiment, H*W abnormality scores are associated with one horizontal patch image 91s. The entire H*W abnormality scores (i.e., the abnormality score set SS) is an example of a result output through the horizontal model 310h.
[0093] As shown in FIG. 5D, the aspect ratio of the horizontal patch image 91s is W1u / Hs (here, size in the first direction Dx / size in the second direction Dy). The width W1u is greater than the height Hs (=reference width Ws). Thus, the horizontal patch image 91s has an aspect ratio that is different from the aspect ratio of the reference size (Ws / Hs (FIG. 6)) and the aspect ratio of 1:1. The reference size is the size of the image input to the horizontal model 310h (in this embodiment, Hs*Ws).
[0094] As described in Figures 4, 6, 9A, and 9B, the horizontal model 310h is trained to output a different anomaly score set SS depending on whether or not a streak defect extending in the first direction Dx is present. Here, the first direction Dx is the longer of the vertical and horizontal directions of the horizontal patch image 91s. The defect is a defect in the target fabric 700 represented by the horizontal patch image 91s.
[0095] In S894 and S898 (FIG. 8), the processor 210 executes the identification process based on the abnormality score set SS. According to the above configuration, the processor 210 can appropriately inspect the target fabric 700 for streak-like defects.
[0096] 6, 7, and 8, the processor 210 uses the compressed horizontal patch image 91sc to acquire an anomaly score set SS associated with the horizontal patch image 91s. Here, the compressed horizontal patch image 91sc is an example of a standard-sized image representing the horizontal patch image 91s compressed in the long direction (here, the first direction Dx) of the horizontal patch image 91s. As described in FIG. 5(D), the compressed horizontal patch image 91sc may represent a longer defect than an uncompressed image 91ss of the same size (however, the defect is compressed within the compressed horizontal patch image 91sc). Therefore, the processor 210 can appropriately determine whether or not the target fabric 700 has a streak-like defect.
[0097] The height Hs and width W1u of the horizontal patch image 91s may each have various values. The ratio (W1u / Hs) of the length of the long side (here, the width W1u) to the length of the short side (here, the height Hs) of the horizontal patch image 91s may be experimentally determined in advance so as to appropriately detect streak-like defects extending in a direction parallel to the long side (here, the first direction Dx). The ratio (W1u / Hs) is preferably, for example, 2 or greater. This configuration makes it easier to detect defects from the horizontal patch image 91s compared to when the ratio (W1u / Hs) is less than 2. The ratio (W1u / Hs) may be 3 or greater, or may be 4 or greater. The larger the ratio (W1u / Hs), the easier it is to detect defects from the horizontal patch image 91s.
[0098] If the ratio (W1u / Hs) is excessively large, the proportion of the width of the portion representing the defect in the width W1u of the horizontal patch image 91s may decrease, and the proportion of the width of the portion without the defect may increase. It may not be easy to detect defects from such a horizontal patch image 91s. Therefore, the ratio (W1u / Hs) may be limited so as to enable appropriate detection of defects. The upper limit of the ratio (W1u / Hs) may be determined experimentally so as to enable appropriate detection of defects. For example, the ratio (W1u / Hs) may be 30 or less, preferably 25 or less, and more preferably 20 or less.
[0099] In this embodiment, the process using the vertical patch images 92t, 92s and the vertical model 310v is similar to the process using the horizontal patch images 91t, 91s and the horizontal model 310h, except for the direction of compression (resizing). Therefore, the process using the vertical patch images 92t, 92s and the vertical model 310v can provide the same various advantages as the process using the horizontal patch images 91t, 91s and the horizontal model 310h.
[0100] In this embodiment, the object to be inspected is the woven fabric 700. In S894 and S898 ( FIG. 8 ), the processor 210 executes a specification process based on the abnormality score set SS. The specification process includes stopping the transport of the woven fabric 700. Therefore, the processor 210 can reduce the possibility of the target woven fabric 700 being unintentionally transported when inspecting the target woven fabric 700 for streak defects.
[0101] The identification process (S898) may be executed when the abnormality score set SS satisfies the abnormality condition, as described in S894. The abnormality condition may be any of various conditions for determining that the target fabric 700 has an abnormality.
[0102] B. Second Example: B1. Training Process: FIGS. 10(A)-10(F) are diagrams showing information processed in training and testing in the second example. In each figure, elements of a training image Mt and elements of a test image Ms, which will be described later, are shown superimposed (reference numerals for elements related to the test image Ms are enclosed in parentheses). FIGS. 10(A) and 10(D) show examples of the test images Mt and Ms. In this example, the feature matrix of the patch image is calculated using the feature matrices FM of the multiple block images 93t and 93s included in the patch image. The size of the block images 93t and 93s is the same as the reference size of the detection models 310h and 310v (Hs*Ws (FIG. 6) in this example). As shown in FIG. 10(A), the horizontal patch images 91t and 91s are composed of L block images 93t and 93s (L is an integer greater than or equal to 2) arranged in the first direction Dx. 10D, vertical patch images 92t, 92s are composed of L block images 93t, 93s arranged in the second direction Dy. In this embodiment, compression of each of the patch images 91t, 91s, 92t, 92s is omitted.
[0103] Fig. 11 is a flowchart showing an example of the training process of Example 2. S110 is the same as S110 in Fig. 4. The processor 210 acquires data of the captured image Mt from the digital camera 110 (Fig. 2).
[0104] In S150b, the processor 210 acquires a plurality of block images 93t from the photographed image Mt (FIGS. 10A and 10D). In this embodiment, the processor 210 acquires data of each block image 93t by dividing the photographed image Mt into a plurality of block images 93t. The arrangement of the plurality of block images 93t on the photographed image Mt may be determined in advance.
[0105] In S160b, the processor 210 calculates feature quantities (here, feature matrix FM) for each block image 93t. The feature quantity calculation method is the same as the feature quantity calculation method in S144 and S148 ( FIG. 4 ). Here, the block image 93t is used instead of the compressed patch image. In this embodiment, the feature extraction model 310a ( FIG. 6 ) is assumed to be common to the horizontal model 310h and the vertical model 310v. The processor 210 calculates feature maps fm1, fm2, and fm3 by performing calculations of the feature extraction model 310a, which is common to the detection models 310h and 310v, using the block image 93t. The processor 210 generates data for the feature matrix FM using the feature maps fm1, fm2, and fm3. The feature maps fm1, fm2, and fm3 and the feature matrix FM are examples of feature quantities for the block image 93t. The feature maps fm1, fm2, and fm3 are examples of feature amounts of the block image 93t calculated by the feature extraction model 310a.
[0106] FIG. 12 is a diagram showing an overview of feature maps. The diagram shows examples of feature maps fm1 and fm2 obtained from a block image 93t. The difference from FIG. 7 is that a block image 93t is used instead of the compressed horizontal patch image 91tc. Similar to FIG. 7, FIG. 12 shows an example in which the above configurations CA1-CA5 are applied.
[0107] In S170b (FIG. 11), the processor 210 calculates the feature amount for each patch image. In this embodiment, S170b includes S172b and S174b.
[0108] In S172b, the processor 210 calculates the feature quantities of the horizontal patch image 91t using multiple feature matrices FM of multiple block images 93t included in the horizontal patch image 91t. FIG. 10B shows a feature matrix FM1t (also referred to as the horizontal feature matrix FM1t), which is an example of the feature quantities of the horizontal patch image 91t. The processor 210 generates data of the horizontal feature matrix FM1t of the horizontal patch image 91t by concatenating L feature matrices FM of L block images 93t included in the horizontal patch image 91t. As shown in FIG. 6, one feature matrix FM is composed of H*W*R elements. The horizontal feature matrix FM1t is composed of H*W*(L*R) elements, where L*R indicates the total number of channels. The processor 210 uses the L*R channels of the L feature matrices FM as the L*R channels of the horizontal feature matrix FM1t.
[0109] The processor 210 generates data of a horizontal feature matrix FM1t for each of the horizontal patch images 91t (FIG. 10A). The processor 210 stores the data of the horizontal feature matrices FM1t in the storage device 215 (e.g., the non-volatile storage device 230).
[0110] In S174b (FIG. 11), the processor 210 calculates feature quantities of the vertical patch image 92t using multiple feature matrices FM of multiple block images 93t included in the vertical patch image 92t. FIG. 10E shows a feature matrix FM2t (also referred to as the vertical feature matrix FM2t), which is an example of feature quantities of the vertical patch image 92t. The method for calculating the vertical feature matrix FM2t is similar to the method for calculating the horizontal feature matrix FM1t (FIG. 10B). The processor 210 generates data of the vertical feature matrix FM2t of the vertical patch image 92t by concatenating L feature matrices FM of L block images 93t included in the vertical patch image 92t. The vertical feature matrix FM2t is composed of H*W*(L*R) elements, where (L*R) indicates the number of channels. The processor 210 stores the data of the multiple vertical feature matrices FM2t in the storage device 215 (e.g., the non-volatile storage device 230).
[0111] Here, the total number of horizontal patch images 91t and the total number of vertical patch images 92t are assumed to be N. As in the embodiment of FIG. 4 , the processor 210 may acquire data of a plurality of captured images in S110 ( FIG. 11 ) to obtain N horizontal patch images 91t and N vertical patch images 92t. In S150b, the processor 210 may acquire a plurality of block images 93t from each of the plurality of captured images. In S170b, the processor 210 may process the N horizontal patch images 91t and the N vertical patch images 92t obtained from the plurality of captured images.
[0112] In S290b, the processor 210 trains the detection models 310h and 310v using a plurality of feature quantities (here, feature matrices FM1t and FM2t) of a plurality of patch images. In this example, S290b includes training the horizontal model 310h (S292b) and training the vertical model 310v (S294b).
[0113] In S292, the processor 210 trains the horizontal model 310h using N horizontal feature matrices FM1t (FIG. 10B) of the N horizontal patch images 91t (FIG. 10A). The N horizontal patch images 91t are shown in FIG. 10C. In S172b (FIG. 11), the processor 210 calculates N horizontal feature matrices FM1t using the N horizontal patch images 91t. As a result, N feature vectors xij are calculated for each of the H*W positions (FIG. 6). In this example, the feature vectors xij are L*R-dimensional vectors having L*R elements.
[0114] 10(C) shows N feature vectors xij,1-xij,N calculated from N horizontal patch images 91t. These N feature vectors correspond to positions (i, j). The processor 210 calculates a mean vector μij and a covariance matrix Σij using the N feature vectors xij,1-xij,N. In this embodiment, the mean vector μij and the covariance matrix Σij (i.e., Gaussian parameters) represent the mean and covariance of the L*R-dimensional feature vector xij.
[0115] The processor 210 calculates Gaussian parameters for each of the H*W positions. As a result, the processor 210 determines the value of each element of a Gaussian matrix GM composed of the H*W Gaussian parameters. The processor 210 stores data for the horizontal model 310h, including the feature extraction model 310a and the Gaussian matrix GM, in the storage device 215 (here, the non-volatile storage device 230). The processor 210 then completes S292b ( FIG. 11 ), i.e., ends training of the horizontal model 310h.
[0116] In S294b, the processor 210 trains a vertical model 310v. The training method for the vertical model 310v is the same as the training method for the horizontal model 310h (S292b). As described above, in S174b, the processor 210 calculates N vertical feature matrices FM2t using N vertical patch images 92t. As a result, N feature vectors xij are calculated for each of the H*W positions (FIG. 6). FIG. 10(F) shows the N feature vectors xij,1-xij,N calculated from the N vertical patch images 92t. The processor 210 calculates the mean vector μij and the covariance matrix Σij (i.e., Gaussian parameters) using the N feature vectors xij,1-xij,N. The processor 210 calculates the Gaussian parameters for each of the H*W positions. As a result, the processor 210 generates a Gaussian matrix GM consisting of H*W Gaussian parameters. The processor 210 stores data of the vertical model 310v, including the feature extraction model 310a and the Gaussian matrix GM, in the storage device 215 (here, the non-volatile storage device 230). Then, the processor 210 ends the process of S294b, i.e., the training process of the vertical model 310v. Then, S290b ends, and the training process of FIG. 11 ends.
[0117] B2. Inspection Process: Figure 13 is a flowchart showing an example of the inspection process of the second embodiment. Steps S710 and S720 are the same as steps S710 and S720 in Figure 8. The processor 210 acquires photographed image data of the target fabric 700 from the digital camera 110.
[0118] As shown in Figures 10(A) and 10(B), the test image Ms includes a plurality of horizontal patch images 91s and a plurality of vertical patch images 92s, similar to the training image Mt. Each of the horizontal patch images 91s and the vertical patch image 92s is composed of L block images 93s. In this embodiment, the shape and arrangement of the test horizontal patch image 91s are the same as those of the training horizontal patch image 91t. The shape and arrangement of the test vertical patch image 92s are the same as those of the training vertical patch image 92t. The shape and arrangement of the test block image 93s are the same as those of the training block image 93t.
[0119] The arrangement of the block images 93s in the photographed image Ms may be different from the arrangement of the block images 93t in the photographed image Mt. That is, the arrangement of the horizontal patch images 91s in the photographed image Ms may be different from the arrangement of the horizontal patch images 91t in the photographed image Mt. The arrangement of the vertical patch images 92s in the photographed image Ms may be different from the arrangement of the vertical patch images 92t in the photographed image Mt.
[0120] In S770b, the processor 210 calculates feature amounts (here, a feature matrix F M ) for each of the multiple block images 93s by performing feature amount calculation processing on the entire captured image. The processor 210 executes calculations of the feature extraction model 310a ( FIG. 6 ) that is common to the detection models 310h and 310v, using the captured image Ms ( FIGS. 10A and 10D ). As a result, the processor 210 generates data for feature maps fm1, fm2, and fm3 corresponding to each of the multiple block images 93s.
[0121] Here, generation of feature maps fm1, fm2, and fm3 will be described assuming that the configuration of Fig. 12 (configurations CA1-CA5 above) is applied. Fig. 12 shows examples of a captured image Ms and feature maps fm1 and fm2. Fig. 12 shows an example in which the height and width of the captured image Ms are k*256 (k is an integer equal to or greater than 2). The captured image Ms represents a plurality of block images 93s.
[0122] The processor 210 generates data for a feature map Ms1 by performing an operation by the first processing block BC1 (here, an operation using a convolution filter) on the entire captured image Ms (the feature map Ms1 is also referred to as a first overall feature map Ms1). In the example of FIG. 12, the size of the first overall feature map Ms1 is (k*256)*(k*256)*c1 (k is an integer equal to or greater than 2). The first overall feature map Ms1 has a plurality of first partial regions s1 corresponding to a plurality of block images 93s. One first partial region s1 represents a first feature map fm1 of the corresponding block image 93s.
[0123] The processor 210 generates data for the intermediate feature map Ms2a by performing the calculations using the downsampling layer BC2a on the entire first overall feature map Ms1. The size of the intermediate feature map Ms2a is (k*128)*(k*128)*c1. The intermediate feature map Ms2a has a plurality of intermediate partial regions s2a. The plurality of intermediate partial regions s2a correspond to the plurality of first partial regions s1 of the first overall feature map Ms1, i.e., the plurality of block images 93s of the captured image Ms, respectively.
[0124] The processor 210 generates data for the feature map Ms2 by performing calculations using the convolutional layer BC2b on the entire intermediate feature map Ms2a (the feature map Ms2 is also referred to as the second overall feature map Ms2). The size of the second overall feature map Ms2 is (k*128)*(k*128)*c2. The second overall feature map Ms2 has multiple second partial regions s2. The multiple second partial regions s2 correspond to the multiple intermediate partial regions s2a of the intermediate feature map Ms2a, i.e., the multiple block images 93s of the captured image Ms. One second partial region s2 represents the second feature map fm2 of the corresponding block image 93s.
[0125] Although not shown, the processor 210 generates data for the third overall feature map by performing the calculations of the third processing block BC3 ( FIG. 6 ) on the entire second overall feature map Ms2. The third overall feature map has multiple partial regions corresponding to multiple block images 93s of the captured image Ms. Each partial region represents the third feature map fm3 for the corresponding block image 93s.
[0126] As described above, in S770b (FIG. 13), the processor 210 executes feature calculation processing (here, processing of processing blocks BC1, BC2, and BC3) for the entire captured image Ms. As a result, the processor 210 generates data for feature maps fm1, fm2, and fm3 for each of the multiple block images 93s. The processor 210 calculates the feature matrix FM using the feature maps fm1, fm2, and fm3. The method for calculating the feature matrix FM is the same as the method described in S160b (FIG. 11). Here, the R channels used during training are used. The processor 210 stores the data for the feature matrix FM for each of the multiple block images 93s in the storage device 215 (e.g., the non-volatile storage device 230).
[0127] The feature maps fm1, fm2, and fm3 and the feature matrix FM are examples of feature amounts of the block image 93s. The feature maps fm1, fm2, and fm3 are examples of feature amounts of the block image 93s calculated by the feature extraction model 310a.
[0128] In S782b, the processor 210 calculates the feature quantities of each of the multiple horizontal patch images 91s (FIG. 10A). FIG. 10B shows a feature matrix FM1s (referred to as the horizontal feature matrix FM1s), which is an example of the feature quantities of the horizontal patch image 91s. The method for calculating the horizontal feature matrix FM1s is the same as the method for calculating the training horizontal feature matrix FM1t. The processor 210 generates data of the horizontal feature matrix FM1s of the horizontal patch image 91s by concatenating L feature matrices FM of L block images 93s included in the horizontal patch image 91s. The processor 210 generates data of the horizontal feature matrix FM1s of each of the multiple horizontal patch images 91s and stores the data of the multiple horizontal feature matrices FM1s in the storage device 215 (e.g., the non-volatile storage device 230).
[0129] In S784b, the processor 210 calculates the feature quantities of each of the multiple vertical patch images 92s ( FIG. 10B ). FIG. 10E shows a feature matrix FM2s (referred to as the vertical feature matrix FM2s), which is an example of the feature quantities of the vertical patch image 92s. The method for calculating the vertical feature matrix FM2s is the same as the method for calculating the training vertical feature matrix FM2t. The processor 210 generates data of the vertical feature matrix FM2s of the vertical patch image 92s by concatenating L feature matrices FM of L block images 93s included in the vertical patch image 92s. The processor 210 generates data of the vertical feature matrix FM2s of each of the multiple vertical patch images 92s and stores the data of the multiple vertical feature matrices FM2s in the storage device 215 (e.g., the non-volatile storage device 230).
[0130] In S890b, the processor 210 determines whether the target textile 700 has an anomaly using the respective lateral feature matrices FM1s (FIG. 10B) of the plurality of lateral patch images 91s (FIG. 10A) and the lateral model 310h. The determination method is similar to the determination method in S890 of FIG. 8. The processor 210 obtains the feature vector xij (FIG. 6) represented by the lateral feature matrix FM1s associated with the lateral patch image 91s. In this embodiment, the feature vector xij is an L*R dimensional vector. The processor 210 calculates an anomaly score using the feature vector xij and Gaussian parameters (μij, Σij). The anomaly score is calculated for each of the H*W positions. As described in S290b of FIG. 11, the Gaussian parameters are trained using images of a normal training textile 700t without defects. Thus, a small anomaly score may be calculated if the horizontal patch image 91s represents a defect-free portion of the target textile 700. A large anomaly score may be calculated if the horizontal patch image 91s represents a defect-containing portion of the target textile 700.
[0131] As described in S172b, FIG. 10B, and other references, the horizontal feature matrix FM1s is generated by concatenating L feature matrices FM of L block images 93s arranged in the first direction Dx. Therefore, the anomaly score when the L block images 93s (i.e., horizontal patch images 91s) represent a streak-like defect extending in the first direction Dx may differ from the anomaly score when the horizontal patch image 91s does not represent such a defect. The set of anomaly scores at H*W positions represents an anomaly map. In this embodiment, an anomaly map that does not show an anomaly, such as the anomaly map 91Ha, can be generated from the horizontal patch image 91sa in FIG. 9A. An anomaly map that shows a linear anomaly indicating the defect 701b, such as the anomaly map 91Hb, can be generated from the horizontal patch image 91sb in FIG. 9B.
[0132] In S890b, the processor 210 calculates an anomaly score for each of a plurality of positions (H*W positions (FIG. 6)) for each horizontal patch image 91s. The processor 210 uses the plurality of anomaly scores to determine whether an anomaly condition for determining that the target textile 700 has an anomaly is satisfied. The anomaly condition may be the same as the anomaly condition used in S890 of FIG. 8.
[0133] In S892b, the processor 210 determines whether the target textile 700 has an abnormality by using the vertical feature matrix FM2s ( FIG. 10(E) ) of each of the vertical patch images 92s ( FIG. 10(D) ) and the vertical model 310v. The determination method is similar to the determination method in S892 of FIG. 8. The processor 210 acquires a feature vector xij ( FIG. 6 ) represented by the vertical feature matrix FM2s associated with the vertical patch image 92s. The processor 210 calculates an abnormality score using the feature vector xij and Gaussian parameters (μij, Σij). The abnormality score is calculated for each of the H*W positions. The set of anomaly scores for the H*W positions represents an anomaly map. In this embodiment, an anomaly map showing a linear abnormality portion indicating a defect 701c, like the anomaly map 92Hc, can be generated from the vertical patch image 92sc in FIG. 9(C).
[0134] The processor 210 calculates an anomaly score for each of a plurality of positions (H*W positions (FIG. 6)) for each vertical patch image 92s. The processor 210 uses the plurality of anomaly scores to determine whether an anomaly condition for determining that the target textile 700 has an anomaly is satisfied. The anomaly condition in S892b is the same as the anomaly condition in S890b.
[0135] S894, S896, and S898 after S892b are the same as S894, S896, and S898, respectively, in Fig. 8. If the determination result in S896 is No, the processor 210 proceeds to S710.
[0136] As described above, in this embodiment, similar to the first embodiment, the processor 210 of the data processing device 200 performs the following process to inspect the target textile 700. In S720 ( FIG. 13 ), the processor 210 acquires data of a captured image of the target textile 700. In S770b, S782b, and S890b, the processor 210 uses the horizontal model 310h to acquire a set of anomaly scores associated with a horizontal patch image 91s, which is a part of the captured image. As in the embodiment of FIG. 5(D) , the horizontal patch image 91s has an aspect ratio different from the reference size aspect ratio (Ws / Hs ( FIG. 6 )) and the 1:1 aspect ratio. As described in FIGS. 9(A), 9(B), 11, S890b ( FIG. 13 ), etc., the horizontal model 310h is trained to output different sets of anomaly scores depending on the presence or absence of a streak-like defect extending in the first direction Dx. In S894 and S898 (FIG. 13), the processor 210 executes the identification process based on the set of abnormality scores. With the above configuration, the processor 210 can appropriately inspect the target fabric 700 for streak-like defects.
[0137] In this embodiment, the horizontal model 310h ( FIG. 6 ) includes a feature extraction model 310a and a Gaussian matrix GM. The feature extraction model 310a is an example of a feature calculation unit trained to calculate feature quantities (here, feature maps fm1, fm2, and fm3) of an image of a standard size (here, Hs*Ws). The Gaussian matrix GM is used to output a set of abnormality scores, as described in S782b and S890b. As described in FIG. 10(B), S782b, and S890b ( FIG. 13 ), the set of abnormality scores is associated with the horizontal patch image 91s, i.e., the entire L number of block images 93s. The set of abnormality scores is output using L feature quantities (here, L sets of feature maps fm1, fm2, and fm3) of the L number of block images 93s of the L number of standard sizes and the Gaussian matrix GM. The Gaussian matrix GM is an example of an output unit that is trained to output results associated with the L features, which are associated with the entire set of L reference-sized images.
[0138] 10B , the horizontal patch image 91s includes L block images 93s. Each of the L block images 93s has a reference size. The L block images 93s include two or more block images 93s that are consecutive in a first direction Dx. Here, the first direction Dx is the longer of the vertical and horizontal directions of the horizontal patch image 91s.
[0139] In S770b (FIG. 13), the processor 210 uses the feature extraction model 310a to acquire L feature amounts (here, L feature matrices F) of L block images 93s of the horizontal patch image 91s. In S782b and S890b, the processor 210 uses the Gaussian matrix G and the L feature amounts of the L block images 93s to acquire a set of anomaly scores associated with the horizontal patch image 91s. With the above configuration, the processor 210 can appropriately inspect the target fabric 700 for streak defects.
[0140] In this embodiment, the process using the vertical patch images 92t, 92s and the vertical model 310v is similar to the process using the horizontal patch images 91t, 91s and the horizontal model 310h, except that the long directions of the patch images are different. For example, the feature extraction model 310a of the vertical model 310v is an example of a feature calculation unit, similar to the feature extraction model 310a of the horizontal detection model 310h. The Gaussian matrix GM of the vertical model 310v is an example of an output unit, similar to the Gaussian matrix GM of the horizontal model 310h. The process using the vertical patch images 92t, 92s and the vertical model 310v can provide various advantages similar to those provided by the process using the horizontal patch images 91t, 91s and the horizontal model 310h.
[0141] In this embodiment, as shown in FIGS. 10A and 10D , the horizontal patch image 91s and the vertical patch image 92s partially overlap each other. For example, the horizontal patch image 91sm (referred to as the first patch image 91sm) in the upper left of FIG. 10A partially overlaps the vertical patch image 92sm (referred to as the second patch image 92sm) in the upper left of FIG. 10B. Specifically, the first block image 93sp in the upper left corner is included in both the patch images 91sm and 92sm. The second block image 93sq, which is located closer to the second direction Dy than the first block image 93sp ( FIG. 10D ), is not included in the first patch image 91sm but is included in the second patch image 92sm.
[0142] In S770b ( FIG. 13 ), the processor 210 stores data representing information (here, the feature matrix FM) associated with the feature amounts (here, the feature maps fm1, fm2, and fm3) of the first block image 93sp in the storage device 215 (e.g., the non-volatile storage device 230). The first block image 93sp is included in the L block images 93s of the first patch image 91sm ( FIG. 10(A)). In S784b and S892b, the processor 210 acquires a set of anomaly scores associated with the second patch image 92sm ( FIG. 10(D)). The second patch image 92sm includes the first block image 93sp and a second block image 93sq that is not included in the L block images 93s of the first patch image 91sm ( FIG. 10(A)). Here, the processor 210 does not newly acquire the feature quantities of the first block image 93sp using the feature extraction model 310a, but instead uses information (here, the feature matrix FM) represented by the data stored in the non-volatile storage device 230. This configuration reduces the number of repetitions of the same calculation using the feature extraction model 310a (for example, the number of repetitions of calculating the feature quantities of the first block image 93sp), thereby speeding up the inspection process.
[0143] The data (also referred to as feature data) stored in the storage device 215 in S770b may represent various information associated with the features calculated by the feature extraction model 310a. The feature data may represent information generated using the features (e.g., feature matrix FM, feature maps fm1, fm2r, fm3r, etc.). Alternatively, the feature data may represent the features themselves (e.g., feature maps fm1, fm2, fm3, etc.). In either case, the processor 210 may use the information represented by the feature data to obtain a set of anomaly scores associated with the patch image.
[0144] C. Third Example: FIG. 14A is a diagram illustrating a reference example of the positional relationship between a horizontal patch image 91s and a defect 701d extending in the first direction Dx on a captured image Ms. The defect 701d can be formed in various positions. In the figure, the defect 701d is located at the connection portion of two horizontal patch images 91s aligned in the first direction Dx. When a single streak-like defect is located at the connection portion of two patch images 91s, the portion representing the defect in each patch image may be small. For example, in the example of FIG. 14A, the lengths L1 and L2 of the portion representing the defect 701d in each horizontal patch image 91s are approximately half the length Ld of the defect 701d. When the portion representing the defect in a patch image is small, the defect may not be detected from that patch image.
[0145] In this embodiment, multiple horizontal patch images 91s arranged in the first direction Dx are arranged so that they partially overlap. FIG. 14B is a diagram illustrating an example of the arrangement of horizontal patch images 91s. The diagram illustrates the positions of three horizontal patch images 91sp, 91sq, and 91sr arranged in the first direction Dx. The difference dp in the diagram indicates the difference in position in the first direction Dx between two adjacent horizontal patch images 91s. The difference dp is smaller than the width W1u of one block image 93s. Therefore, two adjacent horizontal patch images 91s partially overlap. For example, the first horizontal patch image 91sp and the second horizontal patch image 91sq include a first portion P1. The first portion P1 is the overlapping portion of the two images 91sp and 91sq. The second horizontal patch image 91sq includes a second portion P2 that is not included in the first horizontal patch image 91sp. The first horizontal patch image 91sp includes a third portion P3 that is not included in the second horizontal patch image 91sq.
[0146] When multiple patch images 91s are arranged so that they partially overlap each other, the maximum length of a defect in one patch image 91s may be increased compared to when the multiple patch images 91s are arranged so that they do not overlap each other. For example, in the example of FIG. 14A , the maximum length of the defect 701d in one patch image 91s is length L1 or length L2. In the example of FIG. 14B , the second horizontal patch image 91sq represents the entire defect 701d. The maximum length of the defect 701d in one patch image 91s is length Ld, which is longer than lengths L1 and L2. Typically, the longer the defect 701d in the patch image 91s, the easier it is to detect the defect 701 from the patch image 91s. When multiple horizontal patch images 91s arranged as in FIG. 14B are used for inspection, the possibility of overlooking defects that may be formed in various positions is reduced.
[0147] Although not shown, the arrangement of the horizontal patch images 91s in the second direction Dy is the same as that shown in Fig. 5A. Two horizontal patch images 91s adjacent to each other in the second direction Dy are in contact with each other without overlapping.
[0148] The multiple vertical patch images are arranged in the same manner as the multiple horizontal patch images 91s. Unlike the embodiment of FIG. 5B , in this embodiment, the multiple vertical patch images 92s aligned in the second direction Dy are arranged so as to partially overlap. Although not shown, the difference in position in the second direction Dy between two vertical patch images 92s adjacent to each other in the second direction Dy is smaller than the length of one vertical patch image 92s in the second direction Dy. As a result, the two adjacent vertical patch images 92s partially overlap. The arrangement of the multiple vertical patch images 92s in the first direction Dx is the same as the arrangement in FIG. 5B . The two vertical patch images 92s adjacent to each other in the first direction Dx are adjacent to each other without overlapping.
[0149] In this embodiment, the plurality of horizontal patch images 91s and the plurality of vertical patch images 92s arranged in this manner are used in the inspection process. Note that defect detection is not performed in the training process. Therefore, the training process may be the same as the training process in the embodiment of FIG. 4. That is, the arrangements of the horizontal patch images 91t and the vertical patch images 92t in FIGS. 5(A) and 5(B) may be used. However, the respective arrangements of the horizontal patch images 91t and the vertical patch images 92t for training may be the same as the respective arrangements of the horizontal patch images 91s and the vertical patch images 92s for inspection.
[0150] Fig. 15 is a flowchart showing an example of the inspection process of the third embodiment. There are three differences from the inspection process of Fig. 8. The first difference is that the arrangement of multiple horizontal patch images (i.e., multiple compressed horizontal patch images) is different. The second difference is that the arrangement of multiple vertical patch images (i.e., multiple compressed vertical patch images) is different. The third difference is that feature calculation processing is performed for each patch image.
[0151] S710, S720, and S730 are the same as S710, S720, and S730 in FIG. 8, respectively.
[0152] In S740c, the processor 210 calculates the feature quantities of each of the multiple compressed horizontal patch images using the compressed captured image and the horizontal model 310h. FIG. 16 is a diagram showing an overview of the feature maps. Similar to FIG. 7, FIG. 16 shows an example in which the above configurations CA1-CA5 are applied. The process of acquiring the feature maps fm1, fm2, and fm3 (FIGS. 6 and 12) from the training compressed horizontal patch image 91tc is the same as the acquisition process in FIG. 7.
[0153] 16 also shows a horizontally compressed image Msc for inspection. The horizontally compressed image Msc is the same as the horizontally compressed image Msc in FIG. 7. In this embodiment, unlike the embodiment in FIG. 7, multiple compressed horizontal patch images 91sc aligned in the first direction Dx are arranged so as to partially overlap. The difference dpc in position in the first direction Dx between two adjacent compressed horizontal patch images 91sc in the first direction Dx is calculated by multiplying the difference dp in position before compression ( FIG. 14B ) by the rate of change in length due to compression.
[0154] In this embodiment, the processor 210 executes feature calculation processing for each compressed horizontal patch image 91sc. Specifically, the processor 210 acquires the compressed horizontal patch image 91sc from the horizontally compressed image Msc. The processor 210 uses the compressed horizontal patch image 91sc to perform the operations of processing blocks BC1, BC2, and BC3 (FIGS. 6 and 16) to acquire feature maps fm1, fm2, and fm3. The processor 210 calculates the feature matrix FM using the feature maps fm1, fm2, and fm3. The processor 210 calculates the feature matrix FM for each of the multiple compressed horizontal patch images 91sc arranged on the horizontally compressed image Msc. The processor 210 stores the data of the feature matrix FM for each of the multiple compressed horizontal patch images 91sc (i.e., the multiple horizontal patch images 91s) in the storage device 215 (e.g., the non-volatile storage device 230).
[0155] S750 (FIG. 15) is the same as S750 in FIG.
[0156] In S760c, the processor 210 calculates feature quantities for each of the multiple compressed vertical patch images using the compressed captured image and the vertical model 310v. The processing of S760c is similar to the processing of S740c. The processor 210 executes feature quantity calculation processing for each compressed vertical patch image. As a result, the processor 210 calculates a feature matrix FM for each of the multiple compressed vertical patch images. The processor 210 stores the data of the feature matrix FM for each of the multiple compressed vertical patch images (i.e., the multiple vertical patch images) in the storage device 215 (e.g., the non-volatile storage device 230).
[0157] S890 and S892 are the same as S890 and S892, respectively, in Fig. 8. S894, S896, and S898 are the same as S894, S896, and S898, respectively, in Fig. 8. If the determination result in S896 is No, the processor 210 proceeds to S710.
[0158] As described above, in this embodiment, the horizontal model 310h and the vertical model 310v are trained in the same manner as in the embodiment of Fig. 4. In the inspection process (Fig. 15), the processor 210 executes the identification process based on the anomaly score sets SS for each of the horizontal patch images 91s and the anomaly score sets SS for each of the vertical patch images 92s (S894, S898). The method for calculating the feature matrix FM is the same as the calculation method in the embodiment of Fig. 8. Therefore, this embodiment can provide the same various advantages as those provided by the embodiment of Fig. 8.
[0159] In this embodiment, the arrangement of the horizontal patch images 91s and the arrangement of the vertical patch images 92s described in FIGS. 5A, 5B, 14B, etc. are applied to the inspection process. In S890 ( FIG. 15 ), the processor 210 uses the feature matrix FM of the horizontal patch image 91s and the horizontal model 310h to acquire an abnormality score set SS for the horizontal patch image 91s ( FIGS. 6 and 14B ). The processor 210 acquires an abnormality score set SS for each of the multiple horizontal patch images 91s. For example, the processor 210 acquires an abnormality score set SS associated with a second horizontal patch image 91sq ( FIG. 14B ). The second horizontal patch image 91sq is a horizontal patch image 91s that is part of the captured image Ms and is different from the first horizontal patch image 91sp. The second horizontal patch image 91sq has the same shape as the first horizontal patch image 91sp. The second horizontal patch image 91sq includes a first portion P1 that is a part of the first horizontal patch image 91sp. The second horizontal patch image 91sq includes a second portion P2 of the captured image Ms that is not included in the first horizontal patch image 91sp. The horizontal patch images 91sp and 91sq configured in this manner can increase the maximum length of defects in one patch image 91s. This reduces the possibility of overlooking defects that may be formed in various positions.
[0160] In this embodiment, the process using the vertical patch image and the vertical model 310v is similar to the process using the horizontal patch images 91t, 91s and the horizontal model 310h, except for the direction of compression (resizing). Therefore, the process using the vertical patch image and the vertical model 310v can provide the same various advantages as the process using the horizontal patch images 91t, 91s and the horizontal model 310h.
[0161] D. Fourth Embodiment: Fig. 17 is a flowchart showing an example of an inspection process of a fourth embodiment. The difference from the embodiment of Fig. 15 is that a global feature map (e.g., global feature maps Msc1 and Msc2 in Fig. 7) is used to calculate the feature matrix FM for each of a plurality of compressed patch images. The inspection process of this embodiment may be executed instead of the inspection process of Fig. 15.
[0162] S710, S720, and S730 are the same as S710, S720, and S730 in FIG. 15, respectively.
[0163] In S740d, the processor 210 generates data of an overall feature map by performing feature calculation processing on the entire horizontally compressed image. The method for generating the overall feature map is the same as the method for generating it in S740 (FIG. 8).
[0164] FIG. 18 is a diagram showing an example of an overall feature map. Assume that the configuration of FIG. 7 (configurations CA1-CA5) is applied here. The diagram shows a captured image Ms, a horizontally compressed image Msc, and overall feature maps Msc1, Msc2, and Msc3 (each of which is a portion of the maps Msc1, Msc2, and Msc3). The horizontally compressed image Msc and the overall feature maps Msc1 and Msc2 are the same as the horizontally compressed image Msc and the overall feature maps Msc1 and Msc2 in FIG. 7, respectively. The third overall feature map Msc3 is obtained by performing the calculation by the third processing block BC3 (FIG. 6) on the entire second overall feature map Msc2.
[0165] The first overall feature map Msc1 represents the "c1"-dimensional feature values of each of the multiple positions FP1 defined by its resolution (e.g., 1024*1024). Similarly, the second overall feature map Msc2 represents the "c2"-dimensional feature values of each of the multiple positions FP2 defined by its resolution (e.g., 512*512), and the third overall feature map Msc3 represents the "c3"-dimensional feature values of each of the multiple positions FP3 defined by its resolution (e.g., 256*256). In the figure, multiple positions FP1, FP2, and FP3 are represented in partial areas of the feature maps Msc1, Msc2, and Msc3. The processor 210 stores the data of the overall feature maps Msc1, Msc2, and Msc3 in the storage device 215 (e.g., the non-volatile storage device 230).
[0166] Positions FP1, FP2, and FP3 on the overall feature maps Msc1, Msc2, and Msc3 correspond to position PC on the horizontally compressed image Msc and, in turn, position PS on the captured image Ms. Positions FP1, FP2, and FP3 on the overall feature maps Msc1, Msc2, and Msc3 represent position PS on the captured image Ms at the same or different resolution as the resolution of the captured image Ms.
[0167] S750 (FIG. 17) is the same as S750 in FIG.
[0168] In S760d, the processor 210 generates data of an overall feature map by performing feature calculation processing on the entire vertically compressed image. The processing of S760d is the same as the processing of S740d. The processor 210 executes the calculations of the processing blocks of the vertical model 310v (blocks corresponding to processing blocks BC1, BC2, and BC3 in FIG. 6 ) using the vertically compressed captured image. As a result, the processor 210 generates data of three overall feature maps similar to the three overall feature maps Msc1, Msc2, and Msc3 in FIG. 18 . The processor 210 stores the data of the three overall feature maps in the storage device 215 (e.g., the non-volatile storage device 230).
[0169] In S890d, the processor 210 performs an abnormality determination process using compressed horizontal patch images. S890d includes S890d2 and S890d4. In S890d2, the processor 210 acquires feature quantities (here, feature matrix FM) for each of the multiple compressed horizontal patch images. Two compressed horizontal patch images 91sc are shown on the horizontal compressed image Msc in FIG. 18 (one is referred to as the first compressed patch 91sc1, and the other is referred to as the second compressed patch 91sc2). The compressed patches 91sc1 and 91sc2 correspond to the horizontal patch images 91s1 and 91s2, respectively, on the captured image Ms. The horizontal patch images 91s1 and 91s2 partially overlap. A first portion P1d is the overlapping portion of the horizontal patch images 91s1 and 91s2. Similarly, the compressed patches 91sc1 and 91sc2 partially overlap each other. The first portion P1cd is the overlapping portion of the compressed patches 91sc1 and 91sc2.
[0170] The feature matrix (referred to as the first feature matrix FM1) of the first compressed patch 91sc1 is obtained as follows. The global feature maps Msc1, Msc2, and Msc3 respectively indicate first partial regions scl1, sc21, and sc31 associated with the first compressed patch 91sc1. The portion of the first partial region scl1 in the first global feature map Msc1 represents the first feature map fm1. The portion of the first partial region sc21 in the second global feature map Msc2 represents the second feature map fm2. The portion of the first partial region sc31 in the third global feature map Msc3 represents the third feature map fm3. The processor 210 obtains the feature maps fm1, fm2, and fm3 associated with the first compressed patch 91sc1 from the global feature maps Msc1, Msc2, and Msc3 obtained in S890d2 ( FIG. 17 ). The processor 210 calculates the first feature matrix FM1 using the feature maps fm1, fm2, and fm3. The method of calculating the first feature matrix FM1 from the feature maps fm1, fm2, and fm3 is the same as the method described with reference to FIG.
[0171] The global feature maps Msc1, Msc2, and Msc3 indicate second partial regions sc12, sc22, and sc32, respectively, that correspond to the second compressed patch 91sc2. The processor 210 obtains a feature matrix (referred to as a second feature matrix FM2) for the second compressed patch 91sc2 using the feature maps fm1, fm2, and fm3 obtained from the second partial regions sc12, sc22, and sc32. In this manner, the processor 210 obtains the feature matrix FM for each of the multiple compressed horizontal patch images 91sc.
[0172] S890d4 ( FIG. 17 ) is the same as S890 in FIG. 15 . The processor 210 calculates an anomaly score set SS using the feature quantities (here, the feature matrix FM) of each of the compressed horizontal patch images 91sc (i.e., the horizontal patch images 91s) and the horizontal model 310h. For example, a first anomaly score set SS1 is calculated from the first feature matrix FM1 ( FIG. 18 ), and a second anomaly score set SS2 is calculated from the second feature matrix FM2. The processor 210 determines whether the target textile 700 has an anomaly using the multiple anomaly score sets SS of the multiple horizontal patch images 91s.
[0173] In S892d ( FIG. 17 ), the processor 210 performs an anomaly determination process using the compressed vertical patch images. S892d includes S892d2 and S892d4. The process of S892d2 is similar to the process of S890d2. The processor 210 acquires feature quantities (here, a feature matrix F) for each of the multiple compressed vertical patch images. S892d4 is the same as S892 in FIG. 15 . The processor 210 determines whether the target fabric 700 has an anomaly using the feature quantities (here, the feature matrix F) for each of the multiple compressed vertical patch images (i.e., multiple vertical patch images) and the vertical model 310v.
[0174] S894, S896, and S898 are the same as S894, S896, and S898, respectively, in Fig. 15. If the determination result in S896 is No, the processor 210 proceeds to S710.
[0175] As described above, in this embodiment, the horizontal model 310h and the vertical model 310v are trained in the same manner as in the embodiment of FIG. 4. In the inspection process (FIG. 17), similar to the inspection process of FIG. 15, the processor 210 executes identification processing (S894, S898) based on the anomaly score sets SS for each of the horizontal patch images 91s (FIG. 18) and the anomaly score sets SS for each of the vertical patch images. The method for calculating the feature matrix FM is the same as the calculation method in the embodiments of FIGS. 8 and 15. Therefore, this embodiment can provide the same various advantages as those provided by the embodiments of FIGS. 8 and 15.
[0176] 15, the arrangement of the horizontal patch images 91s and the arrangement of the vertical patch images 92s described in Fig. 5(A), Fig. 5(B), Fig. 14(B), etc. are applied to the inspection process. Therefore, the present embodiment can provide the same various advantages as those provided by the embodiment of Fig. 15.
[0177] In this embodiment, the horizontal model 310h ( FIG. 6 ) includes a feature extraction model 310a and a Gaussian matrix GM. The feature extraction model 310a is an example of a first processing unit that processes an image of a reference size (here, Hs*Ws). In this embodiment, the feature extraction model 310a outputs feature maps fm1, fm2, and fm3. In S890d ( FIG. 17 ), the processor 210 calculates an anomaly score set SS using the Gaussian matrix GM and the feature maps fm1, fm2, and fm3 of the compressed horizontal patch image 91sc (and thus the horizontal patch image 91s). The Gaussian matrix GM is an example of a second processing unit that outputs a result (here, the anomaly score set SS) associated with information (here, the feature maps fm1, fm2, and fm3) output from the feature extraction model 310a.
[0178] 18 , feature amounts of the first partial regions sc11, sc21, and sc31 (i.e., feature maps fm1, fm2, and fm3) are used to obtain the first anomaly score set SS1 associated with the first compressed patch 91sc1. The feature maps fm1, fm2, and fm3 of the first partial regions sc11, sc21, and sc31 are included in the overall feature maps Msc1, Msc2, and Msc3 output from the feature extraction model 310a. In this way, the feature maps fm1, fm2, and fm3 of the first partial regions sc11, sc21, and sc31 are information obtained using the information output from the feature extraction model 310a and are an example of first information used to obtain the first anomaly score set SS1.
[0179] To obtain the second anomaly score set SS2 associated with the second compressed patch 91sc2, the feature amounts of the second partial regions sc12, sc22, and sc32 (i.e., the feature maps fm1, fm2, and fm3) are used. In Fig. 18, first portions P11, P21, and P31, which are overlapping portions of the first partial regions sc11, sc21, and sc31 and the second partial regions sc12, sc22, and sc32, are hatched. The feature amounts of the first portions P11, P21, and P31 (i.e., the feature amounts associated with the positions FP1, FP2, and FP3 in the first portions P11, P21, and P31) are part of the first information (here, the feature maps fm1, fm2, and fm3 of the first partial regions sc11, sc21, and sc31). The feature amounts of the first portions P11, P21, and P31 are examples of second information that are part of the first information. The feature maps fm1, fm2, and fm3 of the second partial regions sc12, sc22, and sc32 include the feature amounts of the first portions P11, P21, and P31. Here, to acquire the second anomaly score set SS2, the processor 210 reuses the feature amounts of the first portions P11, P21, and P31 already acquired in S740d, rather than newly acquiring the feature amounts of the first portions P11, P21, and P31 using the feature extraction model 310a. In this embodiment, the number of repetitions of the same calculation using the feature extraction model 310a (e.g., the number of repetitions of calculating the feature amounts of the first portions P11, P21, and P31) is reduced, thereby speeding up the inspection process.
[0180] 18 , positions FP1, FP2, and FP3 on the overall feature maps Msc1, Msc2, and Msc3 correspond to positions PS on the captured image Ms. For example, multiple positions FP1 on the first overall feature map Msc1 correspond to multiple positions PS on the captured image Ms. In this way, the feature amounts at the multiple positions FP1 correspond to multiple positions PS in the first horizontal patch image 91s1 of the captured image Ms. In this way, the first information (here, the feature maps fm1, fm2, and fm3 of the first partial regions sc11, sc21, and sc31) includes multiple feature amounts that correspond to multiple positions PS in the first horizontal patch image 91s1 of the captured image Ms.
[0181] The first portions P11, P21, and P31 of the overall feature maps Msc1, Msc2, and Msc3 correspond to the first portion P1d of the captured image Ms. That is, the second information (here, the feature amounts of the first portions P11, P21, and P31) includes feature amounts associated with the positions PS within the first portion P1d included in the first horizontal patch image 91s1 and the second horizontal patch image 91s2.
[0182] Then, in S740d ( FIG. 17 ), the processor 210 stores data representing the second information (here, the feature amounts of the first portions P11, P21, and P31) in the storage device 215. In S890d2, the processor 210 uses the second information represented by the data stored in the storage device 215 to acquire a second anomaly score set SS2 associated with the second horizontal patch image 91s2. In this embodiment, the number of repetitions of the same calculation using the feature extraction model 310a (e.g., the number of repetitions of calculating the feature amounts of the first portions P11, P21, and P31) is reduced, thereby speeding up the inspection process.
[0183] In this embodiment, the process using the vertical patch image and the vertical model 310v is similar to the process using the horizontal patch images 91t, 91s and the horizontal model 310h, except for the direction of compression (resizing). For example, the feature extraction model 310a of the vertical model 310v is an example of a first processing unit. The Gaussian matrix GM of the vertical model 310v is an example of a second processing unit. Furthermore, the process using the vertical patch image and the vertical model 310v can provide various advantages similar to those provided by the process using the horizontal patch images 91t, 91s and the horizontal model 310h.
[0184] E. Fifth Example: FIG. 19 is a diagram showing an example of the arrangement of horizontal patch images 91s. The difference from the examples shown in FIGS. 10(A) to 10(F) is that the difference in position in the first direction Dx between two adjacent horizontal patch images 91s in the first direction Dx is smaller than the length of one horizontal patch image 91s in the first direction Dx. FIG. 19 shows the positions of four horizontal patch images 91se, 91sf, 91sg, and 91sh arranged in the first direction Dx. The total number L of block images 93s included in one horizontal patch image 91s is three. One horizontal patch image 91s is composed of three block images 93s arranged in the first direction Dx. The difference in position in the first direction Dx between two adjacent horizontal patch images 91s is the same as the size of one block image 93s. Two adjacent horizontal patch images 91s partially overlap. For example, the first portion P1e is the overlapping portion between the adjacent first horizontal patch image 91se and second horizontal patch image 91sf. The overlapping portion is composed of two block images 93s. In the drawing, six block images 93s arranged in the first direction Dx are assigned different reference symbols 93sa-93sf.
[0185] The captured image Ms in FIG. 19 shows a defect 701e extending in the first direction Dx. The defect 701e is represented by two block images 93sc and 93sd aligned in the first direction Dx. The size of the portion representing the defect 701e in the patch image 91s may vary depending on the position of the horizontal patch image 91s. In the example of FIG. 19 , the defect 701e is represented by one block image 93s in each of the first horizontal patch image 91se and the fourth horizontal patch image 91sh. The defect 701e is represented by two block images 93s in each of the second horizontal patch image 91sf and the third horizontal patch image 91sg. The defect 701e is more easily detected from the horizontal patch images 91sf and 91sg than from the horizontal patch images 91se and 91sh.
[0186] Although not shown, the arrangement of the horizontal patch images 91s in the second direction Dy is the same as the arrangement in Fig. 5A. The horizontal patch images 91s are arranged without gaps in the second direction Dy so as not to overlap with each other in the second direction Dy.
[0187] The multiple vertical patch images are arranged in the same manner as the multiple horizontal patch images 91s. Unlike the embodiment of FIG. 10(D), in this embodiment, the multiple vertical patch images 92s arranged in the second direction Dy are arranged so as to partially overlap. Although not shown, the shape of the vertical patch image 92s is the same as the shape obtained by rotating the horizontal patch image 91s (FIG. 19) by 90 degrees. The difference in position in the second direction Dy between two vertical patch images 92s adjacent to each other in the second direction Dy is the same as the difference in position in the first direction Dx between two horizontal patch images 91s adjacent to each other in the first direction Dx (here, the size of one block image 93s). The arrangement of the multiple vertical patch images 92s in the first direction Dx is the same as the arrangement in FIG. 5(B). The multiple vertical patch images 92s are arranged without gaps in the first direction Dx so as not to overlap in the first direction Dx.
[0188] Such an arrangement of the patch images 91s, 92s is applied to the embodiments of FIGS. 10 to 13. Note that defect detection is not performed in the training process. Therefore, the training process may be the same as the training process of the embodiment of FIG. 11. That is, the arrangement of the horizontal patch images 91t and the vertical patch images 92t in FIGS. 10(A) and 10(D) may be used. However, the respective arrangements of the horizontal patch images 91t and the vertical patch images 92t for training may be the same as the respective arrangements of the horizontal patch images 91s and the vertical patch images 92s for inspection.
[0189] In the inspection process ( FIG. 13 ), the processor 210 executes the following process. In S770b, the processor 210 stores feature data representing information (here, a feature matrix FM) associated with the feature quantities of the block image 93sb (here, feature maps fm1, fm2, and fm3) in the storage device 215 (e.g., the non-volatile storage device 230). The feature data includes information associated with the feature quantities of the block images 93sb included in the L block images 93s of the first horizontal patch image 91se ( FIG. 19 ). In S782b and S890b, the processor 210 acquires an abnormality score set SS associated with the second horizontal patch image 91sf. The second horizontal patch image 91sf includes the block image 93sb and a block image 93sd that is not included in the L block images 93sa, 93sb, and 93sc of the first horizontal patch image 91se. Here, the processor 210 does not newly acquire the feature quantities of the block image 93sb using the feature extraction model 310a, but instead uses information (here, the feature matrix FM of the block image 93sb) represented by the feature data stored in the nonvolatile storage device 230. This configuration reduces the number of repetitions of the same calculation using the feature extraction model 310a (for example, the number of repetitions of calculating the feature matrix FM of the block image 93sb), thereby speeding up the inspection process.
[0190] The feature data stored in the storage device 215 in S770b may represent various information associated with the feature quantities calculated by the feature extraction model 310a. The feature data may represent information generated using the feature quantities or feature quantities (e.g., feature maps fm1, fm2, fm3, feature matrix FM, or feature maps fm1, fm2r, fm3r).
[0191] In this embodiment, the arrangement of the horizontal patch image 91s and the arrangement of the vertical patch image 92s described with reference to FIG. 19 (and further with reference to FIGS. 10A and 10D)) are applied to the inspection process. In S890b (FIG. 13), the processor 210 acquires a set of abnormality scores for the horizontal patch image 91s using the horizontal feature matrix FM1s (FIG. 10B) of the horizontal patch image 91s and the horizontal model 310h. The processor 210 acquires a set of abnormality scores for each of the multiple horizontal patch images 91s. For example, the processor 210 acquires a set of abnormality scores associated with the second horizontal patch image 91sf (FIG. 19). The second horizontal patch image 91sf includes a first portion P1e that is a part of the first horizontal patch image 91se and a block image 93sd that is not included in the first horizontal patch image 91se of the captured image Ms. The first portion P1e is composed of block images 93sb and 93sd. Such overlapping horizontal patch images 91se, 91sf can increase the maximum length of a defect in one patch image 91s, thereby reducing the possibility of overlooking defects that may be formed in various positions.
[0192] In this embodiment, the horizontal model 310h (FIG. 6) includes a feature extraction model 310a and a Gaussian matrix GM. The feature extraction model 310a is an example of a first processing unit that processes an image of a reference size (here, Hs * Ws). In this embodiment, the feature extraction model 310a outputs feature maps fm1, fm2, and fm3. In S890b (FIG. 13), the processor 210 calculates the anomaly score set SS using the Gaussian matrix GM and the horizontal feature matrix FM1s (FIG. 10(B)) of the horizontal patch image 91s. The horizontal feature matrix FM1s is calculated in S770b and S782b using the feature maps fm1, fm2, and fm3 of the L block images 93s included in the horizontal patch image 91s. In this way, the Gaussian matrix GM is an example of a second processing unit that outputs a result (here, the abnormality score set SS) that corresponds to the information output from the feature extraction model 310a (here, the feature maps fm1, fm2, fm3 of each of the L block images 93s).
[0193] As described with reference to FIGS. 10B and 19 , first feature quantities (here, L feature matrices F) of L block images 93sa, 93sb, and 93sc are used to acquire a set of abnormality scores associated with the first horizontal patch image 91se. The L feature matrices F are included in the multiple feature matrices F of the multiple block images 93s calculated in S770b. The feature matrix F of each block image 93s is calculated using the feature quantities (here, feature maps Fm1, Fm2, and Fm3) of the block image 93s output from the feature extraction model 310a. In this way, the first feature quantities (i.e., L feature matrices F) of the L block images 93sa, 93sb, and 93sc are information acquired using information output from the feature extraction model 310a and are an example of first information used to acquire a set of abnormality scores associated with the first horizontal patch image 91se.
[0194] To obtain a set of anomaly scores associated with the second horizontal patch image 91sf, second feature quantities (here, L feature matrices FM) of L block images 93sb, 93sc, and 93sd included in the second horizontal patch image 91sf are used. As shown in FIG. 19 , the first horizontal patch image 91se and the second horizontal patch image 91sf include a first portion P1e. The feature quantities of the first portion P1e (here, two feature matrices FM of two block images 93sb and 93sc) are part of first information for the first horizontal patch image 91se (here, three feature matrices FM of three block images 93sa, 93sb, and 93sc). The feature quantities of the first portion P1e are an example of second information that is part of the first information. The second feature quantities for the second horizontal patch image 91sf (here, the three feature matrices FM of the three block images 93sb, 93sc, and 93sd) include the feature quantities of the first portion P1e (i.e., the second information). Here, to acquire a set of anomaly scores for the second horizontal patch image 91sf, the processor 210 reuses the feature quantities of the first portion P1e already acquired in S770b, rather than newly acquiring the feature quantities of the first portion P1e using the feature extraction model 310a. In this embodiment, the number of repetitions of the same calculation using the feature extraction model 310a (e.g., the number of repetitions of calculating the feature quantities of the first portion P1e) is reduced, thereby speeding up the inspection process.
[0195] Furthermore, the first information for the first horizontal patch image 91se ( FIG. 19 ), i.e., the three feature matrices FM for the three block images 93sa, 93sb, and 93sc, are examples of multiple pieces of feature information associated with multiple positions within the first horizontal patch image 91se. As described above, the second information, which is part of the first information, includes feature amounts for the first portion P1e, i.e., the two feature matrices FM for the two block images 93sb and 93sc. These two feature matrices FM are examples of feature information associated with positions within the first portion P1e included in the first horizontal patch image 91se and the second horizontal patch image 91sf. Then, in S770b ( FIG. 13 ), the processor 210 stores data representing the second information in the storage device 215. In S782b and S890b, the processor 210 uses the second information represented by the data stored in the storage device 215 to obtain a set of anomaly scores associated with the second horizontal patch image 91sf. In this embodiment, the number of repetitions of the same operation using the feature extraction model 310a (e.g., the number of repetitions of calculating the feature amounts of the first part P1e) is reduced, thereby speeding up the inspection process.
[0196] As shown in FIG. 19 , the first horizontal patch image 91se and the second horizontal patch image 91sf each include L block images 93s (where L is an integer equal to or greater than 2) (L=3 in FIG. 19 ). Each of the L block images 93s has a reference size (Hs*Ws ( FIG. 6 )). The L block images 93s include two or more consecutive block images 93s in the long direction of the horizontal patch image 91s (the first direction Dx). The first portion P1e includes M block images (M is an integer equal to or greater than 1 and less than L) (M=2 in FIG. 19 ). The first information includes L feature information pieces (here, L feature matrices FM) associated with the L block images 93sa, 93sb, and 93sc in the first horizontal patch image 91se. The second information includes M pieces of feature information (here, M pieces of F feature matrices FM) associated with the M block images 93sb, 93sc in the first portion P1e. In this way, when the horizontal patch image 91s includes two or more block images 93s that are consecutive in the longitudinal direction of the horizontal patch image 91s and the feature matrices FM of each of the multiple block images 93s are used for inspection, the inspection process is speeded up.
[0197] In this embodiment, the process using the vertical patch images 92t, 92s and the vertical model 310v is similar to the process using the horizontal patch images 91t, 91s and the horizontal model 310h, except that the long directions of the patch images are different. For example, the feature extraction model 310a of the vertical model 310v is an example of a first processing unit, similar to the feature extraction model 310a of the horizontal detection model 310h. The Gaussian matrix G M of the vertical model 310v is an example of a second processing unit, similar to the Gaussian matrix G M of the horizontal model 310h. The process using the vertical patch images 92t, 92s and the vertical model 310v can provide various advantages similar to those provided by the process using the horizontal patch images 91t, 91s and the horizontal model 310h.
[0198] F. Sixth Example: FIG. 20 shows another example of a horizontal patch image. The figure shows a horizontal patch image 94s for inspection. In the figure, the horizontal patch image 94s represents a portion of the fabric 700 having a defect 701. The shape of the horizontal patch image 94t for training is the same as the shape of the horizontal patch image 94s for inspection. The only difference between the horizontal patch images 94t and 94s of this example and the horizontal patch images 91t and 91s of FIGS. 5(C) and 5(D) is that the height H4 in the second direction Dy of the horizontal patch images 94t and 94s of this example is smaller than the reference height Hs.
[0199] The horizontal patch images 94t and 94s of this embodiment may be applied to the embodiments of FIGS. 4 to 8, the embodiments of FIGS. 14 to 16, or the embodiments of FIGS. 17 and 18. Here, in S142 (FIG. 4) and S730 (FIGS. 8, 15, and 17), the processor 210 performs vertical enlargement in addition to horizontal compression. As shown in FIG. 20, the processor 210 generates intermediate images 94sc and 94tc from the horizontal patch images 94s and 94t by horizontal compression (i.e., resizing in the first direction Dx). The width of the intermediate images 94sc and 94tc is the same as the reference width Ws. The height of the intermediate images 94sc and 94tc remains at height H4. The processor 210 generates processed images 94sce and 94tce from the intermediate images 94sc and 94tc by vertical enlargement (i.e., resizing in the second direction Dy). The height and width of the processed images 94sce and 94tce are the same as the reference height Hs and reference width Ws, respectively, and are used in place of the compressed horizontal patch images (e.g., compressed horizontal patch images 91sc and 91tc) in the above embodiments.
[0200] 20, vertical expansion enlarges the portion of the processed image 94sce that represents the defect 701. As a result, it is easier to detect the defect 701 in the processed image 94sce.
[0201] Although not shown, the shape of the vertical patch image is the same as the shape obtained by rotating the horizontal patch images 94s and 94t by 90 degrees. In S146 ( FIG. 4 ) and S750 ( FIGS. 8 , 15 , and 17 ), the processor 210 performs horizontal enlargement in addition to vertical compression. The vertical compression is the same as the process obtained by changing the compression direction of the horizontal compression ( FIG. 20 ) from the first direction Dx to the second direction Dy. The horizontal enlargement is the same as the process obtained by changing the enlargement direction of the vertical enlargement ( FIG. 20 ) from the second direction Dy to the first direction Dx.
[0202] G. Seventh Embodiment: In each of the above embodiments, the configuration of the anomaly detection model may be any configuration capable of detecting anomalies in an object, instead of the configurations of detection models 310h and 310c described in FIG. 6 and elsewhere. FIG. 21 is a block diagram showing another embodiment of the anomaly detection model. Anomaly detection model 310F is configured using an anomaly detection technology called "FastFlow." "FastFlow" is disclosed, for example, in the paper "Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, Liwei Wu, "FastFlow: Unsupervised anomaly detection and localization via 2d normalizing flows," arXiv:2111.07677, http: / / arxiv.org / abs / 2111.07677."
[0203] In the "FastFlow" technology, visual features are extracted from an image of an object using a feature extractor, and the extracted features are input into FastFlow to estimate a probability density. During training, FastFlow is trained using normal images to convert the original distribution into a standard normal distribution. During inspection, the probability value of each two-dimensional position of the features of the object being inspected is used as an anomaly score. If the object being inspected has a defect, the features of the defect will deviate from the normal distribution of the normal image. Therefore, the probability density of the position representing the defect will be small (i.e., the anomaly score will be large).
[0204] As shown in FIG. 21, in this embodiment, the anomaly detection model 310F includes a feature extractor FE and a fast flow model FF.
[0205] The feature extractor FE generates feature data Fd from the input image data Id. The feature data Fd represents the visual features of the image of the input image data Id. The size of the feature data Fd is Hf*Wf*Cf. "Hf*Wf" represents the resolution of the two-dimensional position. "Cf" represents the number of channels. The feature data Fd represents the "Cf"-dimensional feature amount at each two-dimensional position.
[0206] The feature extractor FE can be any of various machine learning models that extract features from images. The feature extractor FE may be selected from, for example, a convolutional neural network, ResNet, a Vision Transformer (ViT), Class-Attention in Image Transformers (CaiT), and a Swine Transformer. The feature data Fd may be data output by a specific layer (e.g., a convolutional layer) among multiple layers included in the feature extractor FE. For example, CaiT is a type of Vision Transformer and is also used in the above-mentioned FastFlow paper. When CaiT is used, the feature data Fd may be data output by a specific layer (e.g., the 40th layer) among multiple layers (e.g., 48 layers) of CaiT. Data of various trained feature extraction models is publicly available on the Internet. In this embodiment, data of the publicly available trained models is used as the feature extractor FE.
[0207] The fast flow model FF generates output data Odf from feature data Fd using a technique called "two-dimensional normalizing flow." The size of the output data Odf is the same as the size of the data input to the fast flow model FF (e.g., the size of the feature data Fd). The fast flow model FF is configured as disclosed in the above-mentioned paper on "FastFlow." Inside the fast flow model FF, "3*3 flows" and "1*1 flows" are alternately stacked. Although not shown, the "3*3 flows" are configured using a "3*3" convolutional layer, and the "1*1 flows" are configured using a "1*1" convolutional layer. The output data Odf output from the trained fast flow model FF represents Cf probability densities at each position of "Hf*Wf" (also called a probability map).
[0208] The anomaly detection model 310F of this embodiment may be applied to the horizontal model 310h ( FIG. 6 ) and the vertical model 310v of each of the above embodiments. The feature data Fd is an example of a feature of the input image data Id. The feature extractor FE is an example of a feature calculation unit or a first processing unit, similar to the feature extraction model 310a. The output data Od is an example of a result output through the anomaly detection model 310F, similar to the anomaly score set SS. The fast flow model FF is an example of an output unit or a second processing unit, similar to the Gaussian matrix GM.
[0209] Here, the training process (FIGS. 4 and 11) may be modified as follows: In S144, S148 (FIG. 4), and S160b (FIG. 11), the processor 210 obtains feature data Fd using the feature extractor FE. In S290 (FIG. 4), the fast flow model FF is trained according to a training method suitable for "FastFlow" (e.g., the training method described in the above-mentioned paper on "FastFlow"). In S170b of FIG. 11, the processor 210 generates data to be input to the fast flow model FF by concatenating L pieces of feature data Fd of L block images. In this case, the number of channels of data processed by the fast flow model FF is "L*Cf". In S290b, the fast flow model FF is trained according to a training method suitable for "FastFlow".
[0210] The inspection process may be modified as follows. When this embodiment is applied to the embodiment of FIG. 8 , in S740 and S760, the processor 210 performs feature calculation processing on the entire compressed captured image using the feature extractor FE. This calculates feature data Fd for each compressed patch image. In S890 and S892, the processor 210 obtains output data Odf by performing calculations of the fast flow model FF using the feature data Fd. Here, the processor 210 may obtain a probability map from the output data Odf using the method disclosed in the above-mentioned "FastFlow" paper. Specifically, the processor 210 obtains a final probability map by summing the probabilities of each channel. The processor 210 obtains a final probability map with the same resolution as the input image data Id by upsampling the probability map (e.g., bilinear interpolation) (a smaller probability indicates a larger anomaly score). The final probability map may represent anomaly maps similar to anomaly maps 91Ha, 91Hb, 92Hc of FIGS. 9(A)-9(C).
[0211] When this embodiment is applied to the embodiment of FIG. 13 , in S770b, the processor 210 performs feature calculation processing on the entire captured image using the feature extractor FE. As a result, feature data Fd for each block image 93s is calculated. In S782b and S784b, the processor 210 generates data to be input to the fast flow model FF by concatenating L pieces of feature data Fd for L block images (the number of channels of data processed by the fast flow model FF is "L*Cf"). In S890b and S892b, the processor 210 executes the fast flow model FF's calculations using the concatenated data to obtain output data Odf (and thus the anomaly map).
[0212] 15, in steps S740c and S760c, the processor 210 performs feature calculation processing on the compressed patch images using the feature extractor FE. This calculates feature data Fd for each compressed patch image. Steps S890 and S892 are the same as steps S890 and S892, respectively, in the embodiment of FIG. 8 when this embodiment is applied.
[0213] 17 , in S740d and S760d, the processor 210 performs feature calculation processing on the entire compressed captured image using the feature extractor FE. As a result, feature data Fd for each compressed patch image is calculated. In S890d and S892d, the processor 210 executes calculations of the fast flow model FF using the feature data Fd of the compressed patch images to obtain output data Odf (and thus an anomaly map).
[0214] The feature data Fd (FIG. 21) represents the feature values for each of Hf*Wf positions arranged at a density expressed by Hf / Hs and Wf / Ws. When this embodiment is applied to the embodiment of FIGS. 17 and 18, the feature extractor FE performs a feature calculation process on the entire larger horizontally compressed image Msc, thereby calculating the feature values for each of many more positions arranged at the same density. Here, feature data Fd corresponding to each of the multiple partially overlapping compressed patches 91sc1 and 91sc2 is extracted. The feature value corresponding to the first position PC1 within the overlapping first portion P1cd is commonly used for the multiple compressed patches 91sc1 and 91sc2. In this case, it is preferable that the feature values of the overlapping portions be calculated independently of the visual features of the other portions. Let us assume that the feature value of the first position PC1 is used to acquire the feature data Fd for the first compressed patch 91sc1. Suppose that the features at the first position PC1 are affected by visual features at a position outside the first portion P1cd (e.g., at a second position PC2 in the remaining portion of the first compressed patch 91sc1). In this case, it is not appropriate to reuse the features at the first position PC1 for the second compressed patch 91sc2, which does not include the second position PC2.
[0215] In this way, when the same feature is used to acquire feature data Fd for multiple partial regions (e.g., compressed patches 91sc1 and 91sc2), it is preferable that the feature extractor FE calculates the feature for each local region smaller than the size of the overlapping portion. As such a feature extractor, a convolutional neural network using a small convolution filter may be adopted. Alternatively, a "Swin Transformer" may be adopted. The "Swin Transformer" is disclosed, for example, in the paper "Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo, "Swin transformer: Hierarchical vision transformer using shifted windows", arXiv:2103.14030, http: / / arxiv.org / abs / 2103.14030."
[0216] FIG. 22 illustrates the architecture of the Swin Transformer (hereinafter, a model incorporating this architecture will also be referred to as the feature extractor FEa). The feature extractor FEa includes a patch division module PP and four stages ST1-ST4. The patch division module PP divides the input image data Id into multiple patches. The first stage ST1 includes a linear embedding layer ST11 and a Swin transform block ST12. The linear embedding layer ST11 projects features, each consisting of multiple color values obtained from multiple pixels in a patch, into a processing dimension (i.e., the number of channels). The Swin transform block ST12 divides the multiple patches into multiple groups (called windows) and calculates the feature values of each patch for each window, taking into account the relationships between the multiple patches. The relationships between the multiple patches are calculated using so-called self-attention. The Swin transform block ST12 may also include multiple modules that calculate features using self-attention.
[0217] 23(A)-23(C) are diagrams showing examples of information processed in this embodiment. FIG. 23(A) shows an overview of the feature maps obtained by the "Swin Transformer." The figure shows feature maps FMS1, FMS2, and FMS3, which are processed in order. The first feature map FMS1 represents the feature map generated by the first stage ST1. The small square PT1 represents a patch. A group LW1 consisting of multiple patches PT1 represents a window. The multiple windows LW1 are separated from each other. This indicates that calculations of associations between multiple patches are performed within each window LW1.
[0218] The subsequent stages ST2, ST3, and ST4 (FIG. 22) each include a patch merging layer ST21, ST31, or ST41 and a Swin transform block ST22, ST32, or ST42. The patch merging layers ST21, ST31, or ST41 merge consecutive patches to reduce the total number of patches (i.e., reduce the resolution). Similar to the Swin transform block ST12 in the first stage ST1, the Swin transform blocks ST22, ST32, and ST42 calculate features for each patch by considering the correlation between multiple patches for each window. The Swin transform blocks ST22, ST32, and ST42 may include multiple modules that calculate features using self-attention.
[0219] Feature maps FMS2 and FMS3 (FIG. 23A) show the feature maps generated by stages ST2 and ST3, respectively. Feature maps FMS2 and FMS3 show patches PT2 and PT3 and windows LW2 and LW3, respectively. As shown in the figure, the resolution of the multiple patches PT1, PT2, and PT3 gradually decreases.
[0220] Although not shown, a technique called a shifted window may be applied to the Swin transform blocks ST12, ST22, ST32, and ST42 ( FIG. 22 ). The shifted window shifts the position of the window used in a new process from the position of the window used in the previous process. This allows multiple patches distributed across multiple windows to be included in a single new window. The shifted window can consider the relationship between multiple patches distributed across multiple windows while avoiding global self-attention calculations. The shifted window may be omitted.
[0221] Various feature maps may be used as the feature data Fd. For example, the second feature map FMS2 obtained from the second stage ST2 or the third feature map FMS3 obtained from the third stage ST3 may be used as the feature data Fd.
[0222] 23B shows an overview of the compressed horizontal patch image 91sc and the block image 93s when the anomaly detection model 310F of FIG. 21 (where the feature extractor FE is the same as the feature extractor FEa (FIG. 22)) is applied to each of the above-described embodiments. The compressed horizontal patch image 91sc and the block image 93s include multiple windows LW. Note that the number and arrangement of the windows LW may differ between the compressed horizontal patch image 91sc and the block image 93s.
[0223] 23(C) shows an overview of the horizontally compressed image Msc and the captured image Ms when the anomaly detection model 310F of FIG. 21 (where the feature extractor FE is the same as the feature extractor FEa (FIG. 22)) is applied to each of the above-mentioned embodiments. The horizontally compressed image Msc and the captured image Ms include more windows LW than the compressed horizontal patch image 91sc and the block image 93s of FIG. 23(B).
[0224] As described above, in this embodiment, the mutual influence of the multiple windows LW is reduced. Therefore, when this embodiment is applied to the embodiment of FIGS. 17 and 18, the feature extractor FEa having the "Swin Transformer" architecture can generate an appropriate feature map corresponding to the horizontally compressed image Msc (FIG. 18). Here, it is preferable that the size of the window LW is smaller than the size of the first portion P1cd where the multiple compressed patches 91sc1 and 91sc2 overlap.
[0225] H. Modifications: (1) In the examples of FIGS. 10(A)-10(F) and 19, the number L of block images 93t, 93s included in one patch image 91t, 91s, 92t, 92s may be various values equal to or greater than 2. Preferably, the L block images 93t, 93s of one patch image include K consecutive block images 93t, 93s (where K is an integer between 2 and L) in the longer direction of the patch image, either the vertical or horizontal direction. In FIGS. 10(A)-10(F) and 19, K=L. Here, the number K may be 2, 3, 4, 5, 6, 7, 8, or 9. If the number K is excessively large, the proportion of portions of the patch images 91s, 92s that represent defects may decrease, and the proportion of portions without defects may increase. It may not be easy to detect defects from such patch images 91s, 92s. Therefore, the number K may be limited to ensure appropriate defect detection. The upper limit of the number K may be experimentally determined to ensure appropriate defect detection. For example, the number K may be 30 or less, preferably 25 or less, and more preferably 20 or less. In the example of FIG. 19 , the total number M of block images 93t, 93s included in the overlapping portion of two horizontal patch images 91t, 91s adjacent in the first direction Dx may be any number greater than or equal to 1 and less than L. The difference in position in the first direction Dx between two horizontal patch images 91t, 91s adjacent in the first direction Dx may be a value greater than or equal to 1 and less than K, corresponding to the number of block images 93t, 93s. The same applies to the difference in position between two vertical patch images 92t, 92s adjacent in the second direction Dy. As in the above examples, L = K. In this case, the appropriate range of L may be the same as the appropriate range of K. L may be greater than K. For example, the horizontal patch images 91t, 91s may include K block images 93t, 93s aligned in the first direction Dx, as well as a plurality of block images 93t, 93s aligned in the second direction Dy. The same applies to the vertical patch images 92t, 92s.
[0226] (2) In embodiments in which patch images are compressed (e.g., FIGS. 4-8 , 14-16 , and 17-18 ), the patch images may have a size smaller than the reference size. For example, the width of a horizontal patch image may be equal to the reference width Ws, and the height of the horizontal patch image may be smaller than the reference height Hs. In this case, the processor 210 may vertically expand the horizontal patch image to generate an image of the reference size for input to the anomaly detection model. As described in FIG. 20 , vertical expansion makes it easier to detect streak-like defects extending in the horizontal direction (here, the first direction Dx).
[0227] (3) A feature extraction model included in the anomaly detection model (e.g., feature extraction model 310a ( FIG. 6 ), feature extractor FE ( FIG. 21 )) may be trained using images of the inspection target, thereby improving the accuracy of anomaly detection.
[0228] (4) The anomaly detection model may have various configurations for detecting anomalies in an object, instead of the configurations shown in Figures 6, 21, and 22. For example, the anomaly detection model may be an image generation model (e.g., an autoencoder) that generates an image of an object without anomalies from an image of the same object with an anomaly. A difference image representing the difference between an image input to the image generation model and an image generated by the image generation model is an example of a result related to an anomaly, similar to the anomaly score sets in the above examples.
[0229] (5) The reference size of the image input to the anomaly detection model may be various sizes. For example, the reference height Hs may be different from the reference width Ws.
[0230] (6) The training process and the inspection process may be various processes instead of the above processes. For example, in the embodiment of FIG. 8, the processor 210 may execute the feature calculation process for each compressed patch image. Furthermore, the block images 93t and 93s (FIGS. 10A-10F and 19) may be acquired from compressed captured images. That is, the feature of the compressed patch image may be calculated using multiple feature of multiple block images included in the compressed patch image. Furthermore, either the inspection of defects extending in the first direction Dx or the inspection of defects extending in the second direction Dy may be omitted.
[0231] (7) The captured image used for inspection may be an image obtained by performing a correction process on the captured image generated by the digital camera 110. The correction process may correct one or more of the rotation angle, skew angle, and distortion of the object to be inspected (here, the textile 700) in the captured image.
[0232] (8) The target image used for the inspection is not limited to a photographed image, but may be various images representing the inspection target that have been optically read. For example, the target image may be generated by a stationary reading device (e.g., a scanner) instead of a portable reading device such as the digital camera 110.
[0233] (9) The inspection object may be a knitted fabric instead of a woven fabric. In this case, the identification process (S898 (e.g., FIG. 8)) may include stopping the transport of the inspection object. Furthermore, the inspection object is not limited to woven fabrics and knitted fabrics, but may also be various other objects, such as vinyl sheets, airbag fabrics, paper, and label sheets. In any case, the above-described embodiments and modifications can appropriately inspect the inspection object for streak-like defects (scratches, stains, elongated debris, etc.). Furthermore, the identification process is not limited to stopping the transport of the inspection object, but may include various processes appropriate for the inspection object. For example, the identification process may include a process of notifying an operator of an abnormality. The processor 210 may execute the identification process when the results obtained using a machine learning model (e.g., an anomaly detection model) satisfy an abnormality condition. The abnormality condition may be various conditions indicating that the inspection object has a defect.
[0234] (10) In the above embodiment and modified examples, the processor 210 may cause the GPU 260 to execute various calculations. For example, the processor 210 may cause the GPU 260 to execute some or all of the calculations of a machine learning model (e.g., the lateral detection models 310h and 310v ( FIG. 6 ) or the anomaly detection model 310F ( FIG. 21 )). Note that the GPU 260 may be omitted.
[0235] (11) In the above embodiment and modified examples, the training process may be executed by a device (e.g., a computer) different from the inspection process (e.g., the data processing device 200) that performs the inspection. Furthermore, the inspection device may be a type of device different from a personal computer (e.g., a digital camera, a scanner, a smartphone, or a tablet computer). Furthermore, multiple devices (e.g., computers) that can communicate with each other via a network may share some of the functions of the inspection process performed by the inspection device and collectively provide the inspection process functions (a system including these devices corresponds to an inspection device).
[0236] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the functions of the horizontal model 310h (FIG. 6) and the vertical model 310v may be realized by dedicated hardware circuits.
[0237] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used while stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.
[0238] The above-described examples and modifications can be combined as appropriate. The above-described examples and modifications are provided to facilitate understanding of the present disclosure and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof.
[0239] 110...digital camera, 120...light source, 200...data processing device, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 232...training program, 233...inspection program, 240...display unit, 250...operation unit, 260...graphics processing unit, 270...communication interface, 310...anomaly detection model, 310h...lateral detection model, 310v...vertical detection model, 310F...anomaly detection model, 310a...feature extraction model, 700...textile, 900...conveyor device, 910...first roller, 912...rotary encoder, 920...second roller, 922...drive motor
Claims
1. A program that causes a computer to realize the following: a first acquisition function that acquires a target image representing an optically read object to be inspected; a second acquisition function that uses a machine learning model to acquire a result that corresponds to a patch image that is a part of the target image, wherein the patch image has an aspect ratio that is different from an aspect ratio of a reference size of an image input to the machine learning model and an aspect ratio of 1:1, and the machine learning model is trained to output different results depending on the presence or absence of a streak-like defect extending in the longer of the vertical and horizontal directions of the patch image in the object to be inspected represented by the patch image; and an execution function that executes a specific process based on the result.
2. A program according to claim 1, wherein the second acquisition function uses an image of the reference size representing the patch image compressed in the long direction to acquire the result associated with the patch image.
3. The program according to claim 1, wherein the machine learning model includes a feature calculation unit trained to calculate features of an image of the standard size, and an output unit trained to output a result associated with L features of L images of the standard size (L is an integer greater than or equal to 2), the result being associated with all of the L images of the standard size; the patch image includes L block images, each of the L block images having the standard size, and the L block images include two or more block images that are consecutive in the long direction; and the second acquisition function uses the feature calculation unit to acquire L features of the L block images of the patch image, and uses the output unit and the L features of the L block images to acquire the result associated with the patch image.
4. A program as described in claim 3, wherein the second acquisition function stores in a storage device data representing information associated with features of a first block image included in the L block images of a first patch image which is the patch image, and uses the information represented by the data stored in the storage device, without newly acquiring the features of the first block image using the feature calculation unit, in order to obtain a result associated with the second patch image which is a part of the target image and includes the first block image and a second block image not included in the L block images.
5. The program according to claim 1, wherein the second acquisition function uses the machine learning model to obtain a result corresponding to a second patch image that is a part of the target image and is different from a first patch image that is the patch image, the second patch image having the same shape as a shape of the first patch image, and the second patch image including a first portion that is a part of the first patch image and a second portion of the target image that is not included in the first patch image.
6. A program as described in claim 5, wherein the machine learning model includes a first processing unit that processes an image of the standard size and a second processing unit that outputs a result corresponding to information output from the first processing unit, and the second acquisition function reuses second information, which is part of the first information used for acquiring the result corresponding to the first patch image and is acquired using information output from the first processing unit of the machine learning model, for acquiring the result corresponding to the second patch image without newly acquiring it using the first processing unit.
7. A program as described in claim 6, wherein the first information includes a plurality of pieces of feature information each corresponding to a plurality of positions within the first patch image, and the second information includes, among the plurality of pieces of feature information, feature information corresponding to a position within the first portion included in the first patch image and the second patch image, and the second acquisition function stores data representing the second information in a storage device, and uses the second information represented by the data stored in the storage device to acquire the result corresponding to the second patch image.
8. A program as described in claim 7, wherein the first patch image and the second patch image each include L block images (L is an integer equal to or greater than 2), each of the L block images having the reference size and the L block images including two or more block images consecutive in the long direction, the first portion included in the first patch image and the second patch image includes M block images (M is an integer equal to or greater than 1 and less than L), the plurality of feature information included in the first information includes L pieces of feature information respectively associated with the L block images in the first patch image, and the second information includes M pieces of feature information associated with the M block images in the first portion included in the first patch image and the second patch image.
9. A program according to any one of claims 1 to 8, wherein the ratio of the length of the long side to the length of the short side of the patch image represented by the aspect ratio is 2 or greater.
10. A program according to any one of claims 1 to 8, wherein the object to be inspected is a woven fabric or a knitted fabric, and the specific process includes stopping the transportation of the object to be inspected.
11. An inspection device comprising: a first acquisition unit that acquires an optically read target image representing an inspection object; a second acquisition unit that uses a machine learning model to acquire a result that corresponds to a patch image that is a part of the target image, the patch image having an aspect ratio different from an aspect ratio of a reference size of an image input to the machine learning model and an aspect ratio of 1:1, the machine learning model being trained to output different results depending on the presence or absence of a streak-like defect extending in the longer of the vertical and horizontal directions of the patch image in the inspection object represented by the patch image; and an execution unit that executes a specific process based on the result.
Citation Information
Patent Citations
Method and equipment for inspecting roll film
JP1997133638A
Abnormality detection device and abnormality detection method
JP2022115156A
Generation method and apparatus of surface defect determination model of metal band, success / failure determination method and apparatus of metal band, and manufacturing method of metal band
JP2023067750A
