Program and data processor

Abstract

To detect a defect of an object using a read image of the object.SOLUTION: Each of a plurality of read images is used to detect a defective portion that is a portion representing a defect in an object. When a detected first defective portion and second defective portion satisfy a merging condition, merging processing for merging the first defective portion and the second defective portion as one defective portion is executed. The merging processing includes processing of merging the first defective portion and the second defective portion when the first defective portion and the second defective portion satisfy a first merging condition. The first merging condition includes that an angle formed by a first defect direction, which is a direction in which a first defect represented by the first defective portion extends, and a second defect direction, which is a direction in which a second defect represented by the second defective portion extends, is equal to or less than a first angle threshold, and that a distance between the first defective portion and the second defective portion is equal to or less than a first distance threshold.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] This specification relates to a technique for detecting defects of an object using a read image of the object.

Background Art

[0002] Various techniques for inspecting an object have been proposed. Patent Document 1 discloses a technique for inspecting the surface of a web such as an aluminum sheet or a plastic sheet. In this technique, a surface defect detector detects a defective portion existing on the surface of the web and outputs a timing signal. The stop control means stops the defective portion at a visual inspection position preset in the conveyance path of the web after decelerating the conveyance of the web based on the timing signal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is room for improvement in detecting defects of an object using a read image of the object.

[0005] This specification discloses a technique for detecting defects of an object using a read image of the object.

Means for Solving the Problems

[0006] The technique disclosed in this specification can be realized as the following application examples.

[0007] [Application Example 1] A program that executes a defect detection process, which is a process of detecting a defective part, which is a part representing a defect in the object, using each of a plurality of captured images obtained by having a reading device read different parts of the object from each other, and a merging process of merging the first defective part and the second defective part as one defective part when the first defective part and the second defective part detected by the defect detection process satisfy a merging condition. The merging process includes a process of merging the first defective part and the second defective part when the first defective part and the second defective part satisfy a first merging condition. The first merging condition includes that the angle formed by a first defect direction, which is the direction in which a first defect represented by the first defective part extends, and a second defect direction, which is the direction in which a second defect represented by the second defective part extends, is equal to or less than a first angle threshold, and that the distance between the first defective part and the second defective part is equal to or less than a first distance threshold.

[0008] According to this configuration, since the first defective part and the second defective part representing different parts of the same defect can be merged, a defective part that appropriately represents the defect of the object can be obtained.

[0009] Note that the technology disclosed in this specification can be implemented in various forms, for example, in the form of a data processing method and a data processing device, a computer program for realizing the functions of those methods or devices, a recording medium (for example, a non-transitory recording medium) recording the computer program, and the like.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0011] A. First Embodiment: A1. Device Configuration: FIG. 1 is an explanatory diagram showing a data processing device as an example. The data processing device 200 is, for example, a personal computer. The data processing device 200 performs various data processes for inspecting the appearance of an object (for example, a fabric for sewing such as a woven fabric, a knitted fabric, or a denim fabric). Hereinafter, it is assumed that the appearance of the fabric 700 is inspected.

[0012] The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, a graphics processing unit 260 (referred to as GPU 260), and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0013] The processor 210 is a device configured to perform data processing, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile memory device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile memory device 230 is, for example, a flash memory. The non-volatile memory device 230 stores the data of each of the program 231 and the object detection model 310. The object detection model 310 is a program module that forms a trained machine learning model. The non-volatile memory device 230 further stores the result data D2 and the merged data D3. Details of the data stored in the non-volatile memory device 230 will be described later.

[0014] The display unit 240 is a device configured to display an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive an operation by a user, such as a button, a lever, or a touch panel disposed on top of the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250. The display unit 240 may display operation elements (for example, buttons, sliders, etc.), and the displayed elements may be operated through the operation of the operation unit 250.

[0015] The GPU 260 is an arithmetic device configured to execute various numerical operations such as image processing and machine learning. The GPU 260 executes various operations according to the instructions of the processor 210. Note that a driver program (not shown) for controlling the GPU 260 may be provided by the manufacturer of the GPU 260.

[0016] The communication interface 270 is an interface for communicating with other devices (for example, including one or more of a USB interface, a wired LAN interface, a wireless interface of IEEE802.11, an interface of an industrial camera (for example, CameraLink, CoaXPress, etc.)). In this embodiment, a conveying device 900, digital cameras 111 - 114, and an encoder 120 are connected to the communication interface 270. The conveying device 900 is a device for conveying the fabric 700, and the data processing device 200 can supply instructions for conveying or stopping the fabric 700 to the conveying device 900. The digital cameras 111 - 114 are used for photographing the fabric 700. The encoder 120 is used for calculating the relative position of the fabric 700 with respect to the conveying device 900 (details will be described later).

[0017] FIG. 2 is a perspective view of the digital cameras 111 - 114, the fabric 700, the conveying device 900, and the light source 130. The conveying device 900 is a device for conveying the fabric 700 for inspection (such a device is also called a fabric inspection machine). The conveying device 900 includes a plurality of rollers (including two rollers 910, 920) and a conveying motor (not shown) for driving one or more rollers to convey the fabric 700. The partial conveying path Pth in the figure shows the portion between the rollers 910, 920 of the conveying path of the fabric 700 (the partial conveying path Pth is also simply called the partial path Pth). In this embodiment, a fabric 700 longer than the partial path Pth is wound around a roller (not shown). The fabric 700 drawn from this roller is conveyed from the first roller 910 along the partial path Pth to the second roller 920 and wound around another roller (not shown). Between the rollers 910, 920 (that is, on the partial path Pth), the fabric 700 forms a flat portion 700F which is a flat part. The light source 130 irradiates light on the flat portion 700F. The forward direction Df in the figure indicates the conveying direction on the partial path Pth (the forward direction Df is also called the conveying direction Df). The reverse direction Db indicates the direction opposite to the forward direction Df, that is, the conveying direction when the fabric 700 is rewound. The orthogonal direction Dt indicates a direction parallel to the flat portion 700F and perpendicular to the partial path Pth.

[0018] The first end 700e1 and the second end 700e2 in the figure are the ends in the direction perpendicular to the partial path Pth of the fabric 700. The lines indicating the ends 700e1, 700e2 are approximately parallel to the partial path Pth. However, the fabric 700 is soft and easily deformable. The fabric 700 can be conveyed with the lines indicating the ends 700e1, 700e2 inclined with respect to the partial path Pth.

[0019] On the partial path Pth, two positions Pr and Pv are set. The first position Pr is the position for reading by the digital cameras 111-114. In the figure, the reading area Ar, which is the area read by the digital cameras 111-114, is hatched. The reading area Ar is a rectangular area having two sides Ar1, Ar2 parallel to the partial path Pth and two sides Ar3, Ar4 perpendicular to the partial path Pth. The range PRr from the third side Ar3 to the fourth side Ar4 of the partial path Pth is the reading range by the digital cameras 111-114 (the range PRr is called the reading range PRr). The first position Pr is located at the center of the reading range PRr. The reading area Ar includes the entire part of the fabric 700 within the reading range PRr. That is, the first side Ar1 and the second side Ar2 are located outside the fabric 700.

[0020] The partial areas R11-R14 in the figure respectively indicate the areas read by the digital cameras 111-114. In this embodiment, the digital cameras 111-114 (and thus the partial areas R11-R14) are arranged side by side in the orthogonal direction Dt. The entire reading area Ar is represented by the entirety of the partial areas R11-R14.

[0021] The second position Pv is the position for visual inspection. The second position Pv is arranged at a position where it is easy for the operator to observe. In this embodiment, the second position Pv is located on the downstream side of the first position Pr (that is, on the forward direction Df side with respect to the first position Pr). In this embodiment, not limited to the second position Pv, the operator can visually observe the fabric 700 over the entire range from the first position Pr to the second position Pv.

[0022] The conveying device 900 includes a control panel 980 and a control device 990. The control panel 980 includes four operation units 981 - 984. The operation units 981 - 984 are devices configured to receive operations by an operator, such as buttons, push switches, foot switches, touch panels, etc. Hereinafter, it is assumed that each of the operation units 981 - 984 is a push switch. The control device 990 is an electric circuit configured to control a conveying motor according to the operations of the control panel 980. The control device 990 includes, for example, wirings connecting the operation units 981 - 984, a conveying motor (not shown), and a power source. In this embodiment, the control device 990 performs conveying in the forward direction Df when the first operation unit 981 is pressed, and performs conveying in the reverse direction Db when the second operation unit 982 is pressed. When the operation units 981 and 982 are not pressed, the control device 990 stops the conveying. Also, the control device 990 starts conveying in the forward direction Df in response to the pressing of the third operation unit 983. Thereafter, the control device 990 continues the conveying in the forward direction Df until the fourth operation unit 984 is pressed, regardless of the state of the third operation unit 983. The conveying by the operation of the third operation unit 983 is also called automatic conveying. Also, as described above, the control device 990 can control the conveying according to instructions for conveying and stopping the fabric 700 by the data processing device 200 in addition to the operations on the control panel 980. Note that the control device 990 may be configured using a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)).

[0023] The conveying device 900 is connected to an encoder 120 that detects the direction and amount of position change due to conveyance. In this embodiment, the encoder 120 is connected to a roller (for example, the first roller 910). The configuration of the encoder 120 may be various configurations that detect the direction and amount of position change due to conveyance. For example, the encoder 120 may be an incremental encoder. From an incremental encoder, A pulses and B pulses are alternately output according to the change in position. The number of pulses output indicates the amount of movement. The phase difference (positive or negative) between the A pulse and the B pulse indicates the direction of movement. The data processing device 200 (FIG. 1) can calculate the current conveyance position of the fabric 700 conveyed by the conveying device 900 (that is, the relative position of the fabric 700 with respect to the conveying device 900) by counting the number of pulses output from the encoder 120 according to the phase difference (that is, the direction). Note that a counter for counting the number of pulses according to the direction may be connected to the encoder 120. The data processing device 200 may acquire the current relative position of the fabric 700 using the information from the counter.

[0024] A2. Inspection process: For inspection, the fabric 700 (FIG. 2) is attached to the conveying device 900. In this embodiment, an operator attaches the fabric 700 to the conveying device 900. Alternatively, a machine (for example, a robotic arm) may attach the fabric 700 to the conveying device 900. After the attachment of the fabric 700, an instruction to start the inspection process is input to the data processing device 200 (FIG. 1). In this embodiment, the operator inputs an instruction to start the inspection by operating the operation unit 250. The processor 210 starts the inspection process in response to the start instruction. In this embodiment, the processor 210 executes the inspection process according to the program 231.

[0025] FIG. 3 is a flowchart showing an example of an inspection process. In S110, the processor 210 selects a target range that is the range of the image to be processed. As will be described later, in this embodiment, the reading of the portion of the reading area Ar (FIG. 2) of the fabric 700 and the conveyance of the fabric 700 are repeated. The reading of the reading area Ar is performed for each constant conveyance amount in the forward direction Df so that no gap is generated between the plurality of portions of the fabric 700 to be read. Defects (such as linear defects and holes) are detected using the reading image (referred to as a strip fabric image) corresponding to the reading area Ar. The detection of defects is performed using the object detection model 310. The specified size, which is the size of the image that can be input to the object detection model 310, is smaller than the size of the strip fabric image. The processor 210 performs object detection using the object detection model 310 for each of the plurality of partial images representing different portions of the strip fabric image. On the other hand, the fabric 700 may have defects larger than the partial images. When one defect is represented by a plurality of partial images, the plurality of defect portions detected from the plurality of partial images may represent one defect. In order to merge the plurality of defect portions representing one defect into one defect portion, the processor 210 processes the combined image obtained by combining the plurality of partial images. In S110, the processor 210 selects a range of a predetermined size larger than such a combined image. In this embodiment, the size of the target range corresponds to three consecutive strip fabric images. In the repeated S110, the processor 210 determines a new target range by deleting the oldest strip fabric image among the three strip fabric images included in the target range and adding a new strip fabric image.

[0026] In S115, the fabric 700 is photographed by the digital cameras 111 - 114. The processor 210 supplies a reading instruction to each of the digital cameras 111 - 114. The digital cameras 111 - 114 read the fabric 700 in response to the reading instruction. The processor 210 acquires the data of the reading image representing the image read from each of the digital cameras 111 - 114.

[0027] Figs. 4(A)-4(D) are diagrams showing examples of images to be processed. Fig. 4(A) shows examples of captured images IMr1-IMr4 respectively obtained from digital cameras 111-114 (Fig. 2). The captured images IMr1-IMr4 are each rectangular images having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy perpendicular to the first direction Dx. The second direction Dy indicates a direction approximately parallel to the partial path Pth (Fig. 2). The data of each of the captured images IMr1-IMr4 is bitmap data representing the respective color values of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. The color value is represented by, for example, the respective gradation values of red R, green G, and blue B (for example, values of zero or more and 255 or less).

[0028] As described with reference to Fig. 2, the partial regions R11-R14 corresponding to the captured images IMr1-IMr4 (Fig. 4(A)) are arranged side by side in the orthogonal direction Dt. The first captured image IMr1 represents a portion including the first end 700e1 and the first ear 700L of the fabric 700 and the background BG. In the present embodiment, the fabric 700 has ears 700L and 700R. The ears of the fabric are the ends in the width direction of the fabric (the direction perpendicular to the partial path Pth in Fig. 2). The ears may have a configuration different from that of the interior of the fabric. For example, in order to reduce the possibility of fraying of the fabric, the thread density in the ears can be increased (for example, the ears can be formed using additional threads). The second captured image IMr2 and the third captured image IMr3 each represent a portion inside the fabric 700 rather than the ears 700L and 700R. The fourth captured image IMr4 represents a portion including the second end 700e2 and the second ear 700R of the fabric 700 and the background BG. The entire captured region Ar is represented by the entirety of these captured images IMr1-IMr4. Although not shown, the background BG can represent various objects located outside the fabric 700, such as a part of the conveying device 900.

[0029] In the example of FIG. 4(A), the portion represented by the third read image IMr3 of the fabric 700 has a linear defect FD. The linear defect can be formed by various causes. For example, a defect in the yarn forming the fabric 700 can form a linear defect. Also, a linear defect (e.g., a scratch, a linear drawing) can be formed by the contact between the fabric 700 and other members (e.g., a device for transporting the fabric 700, a writing instrument, etc.).

[0030] In S120, the processor 210 generates data of one read image by combining the four read images IMr1 - IMr4. FIG. 4(B) shows an example of an image generated from the read images IMr1 - IMr4. The image IMrc is a strip-shaped image representing the read area Ar (FIG. 2). This image IMrc is an example of a strip fabric image (the image IMrc is referred to as the strip fabric image IMrc).

[0031] The method for generating the strip fabric image IMrc may be various methods. For example, the sub-regions R11 - R14 (FIG. 2) may be arranged side by side in the orthogonal direction Dt so as not to overlap each other without a gap within the read area Ar. In this case, the processor 210 may generate the data of the strip fabric image IMrc by connecting the respective ends of the read images IMr1 - IMr4 arranged in the first direction Dx (FIG. 4(A)). Alternatively, the sub-regions R11 - R14 may be arranged such that two adjacent sub-regions partially overlap. In this case, the processor 210 may generate the data of the strip fabric image IMrc by combining the read images IMr1 - IMr4 in the same arrangement as the arrangement of the sub-regions R11 - R14. As the image of the overlapping portion of the two read images, the corresponding portion of one read image may be used. In any case, the arrangement of the sub-regions R11 - R14 can be adjusted by adjusting the arrangement of the digital cameras 111 - 114.

[0032] In S125 (FIG. 3), the processor 210 detects a defective portion representing a defect (e.g., a hole, a linear defect FD (FIG. 4(B)), etc.) from the belt fabric image IMrc. The method for detecting the defective portion may be various methods. In this embodiment, the processor 210 uses the trained object detection model 310 to detect the defective portion. The object detection model 310 may be various models capable of detecting the defective portion. In this embodiment, the object detection model 310 is a model called "RTMDet" disclosed in the following paper. Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang and Kai Chen. "Rtmdet: An Empirical Study of Designing Real-Time Object Detectors", arXiv.2212.07784, December 16, 2022, https: / / doi.org / 10.48550 / arXiv.2212.07784.

[0033] RTMDet is a model that detects the bounding box of an object and its category (i.e., the type of the object), and also performs region segmentation called instance segmentation. In this embodiment, the object detection model 310 is pre-trained to detect the bounding box, type, and region of a plurality of types of detection targets including defects representing holes and linear defects. As the bounding box, a rectangle composed of two sides parallel to the first direction Dx and two sides parallel to the second direction Dy is to be detected. The type of the detection target is associated with the bounding box. The region segmentation detects the region (referred to as a mask) of the detection target for each detection target. An identifier of the detection target is associated with the mask. This region segmentation determines for each pixel which mask the pixel is included in. The training method of the object detection model 310 may be various methods, for example, the training method described in the above-mentioned paper of RTMDet. The bounding box and mask of the defect are examples of the defect part detected using the image of the fabric 700. The defect part is the part of the fabric 700 that represents the defect.

[0034] In this embodiment, the specified size, which is the size of the image that can be input to the object detection model 310, is smaller than the size of the belt fabric image IMrc (FIG. 4(B)). Therefore, the processor 210 performs object detection using the object detection model 310 for each of a plurality of partial images representing different parts of the belt fabric image IMrc. Each partial image has the specified size. The entire belt fabric image IMrc is represented by the entirety of the plurality of partial images.

[0035] FIG. 4(C) is a diagram showing an example of a plurality of partial images. In this embodiment, a plurality of partial images IMa1 - IMak are extracted from the belt fabric image IMrc (FIG. 4(B)). The arrangement of the plurality of partial images IMa1 - IMak on the belt fabric image IMrc is determined in advance. The partial image IMa2 represents the first ear 700L, the partial image IMai represents the defect FD, and the partial image IMak represents the second ear 700R.

[0036] FIG. 4(B) shows the ranges of the partial images IMa1-IMak on the belt fabric image IMrc. In this embodiment, the plurality of partial images IMa1-IMak are arranged in the first direction Dx on the belt fabric image IMrc. The range of the first direction Dx is different among the partial images IMa1-IMak. The size of the partial images IMa1-IMak in the second direction Dy is the same as the size of the belt fabric image IMrc in the second direction Dy. Also, in this embodiment, two adjacent partial images partially overlap. This is to reduce the possibility of non-detection of a defect when the defect is located at the boundary between two adjacent partial images. Note that the size of the partial image in the second direction Dy may be smaller than the size of the belt fabric image IMrc in the second direction Dy. In this case, a plurality of partial images arranged along the first direction Dx and the second direction Dy may be used.

[0037] FIG. 4(D) is a diagram showing an example of a mask and a bounding box detected from the partial images IMa1-IMak (FIG. 4(C)). The partial mask images IMm1-IMmk respectively represent the mask images corresponding to the partial images IMa1-IMak. The mask MD1 and the bounding box BBD1 on the partial mask image IMmi indicate the defect FD.

[0038] In S130 (FIG. 3), the processor 210 stores the data representing the detection result in the storage device 215 (for example, the non-volatile storage device 230). In this embodiment, the processor 210 adds the data representing the bounding box and the partial mask image acquired in S125 to the result data D2.

[0039] In S135, it is determined whether an image of the target range has been acquired. For example, when S115 is executed for the first time after the start of the inspection process, two of the three strip fabric images included in the target range have not been acquired. If the target range includes an unacquired strip fabric image (S135: No), in S138, the processor 210 supplies an instruction to the transport device 900 to transport the fabric 700 to the relative position of the fabric 700 for reading the next strip fabric image. The control device 990 of the transport device 900 transports the fabric 700 according to the instruction. Then, the process proceeds to S115. As a result, S115 - S130 for a new strip fabric image is executed.

[0040] If an image of the target range has been acquired (S135: Yes), in S150, the processor 210 executes a merging process for defective portions. The merging process for defective portions merges a plurality of defective portions in order to process the whole of the plurality of defective portions as one defective portion when a plurality of portions of one defect on the fabric 700 are detected as a plurality of defective portions.

[0041] FIG. 5 is a flowchart showing an example of the merging process of defective portions. In S510, the processor 210 acquires an image of the merging range from the target range. FIGS. 6(A)-6(F) are diagrams showing examples of the merging process. FIG. 6(A) shows an example of the merging range image. In the present embodiment, the processor 210 acquires, as the merging range image IMi, a 3-row and 3-column partial mask image IMmn-IMmv centered on the target portion mask image IMmr. As the partial mask image, the partial mask image generated in S125 (FIG. 3) may be used. As described in S110 (FIG. 3), the target range shows three consecutive belt fabric images. That is, the target range includes a plurality of partial mask images arranged along the first direction Dx and the second direction Dy. In S510 (FIG. 5), the processor 210 selects an unprocessed partial mask image as the target portion mask image from the plurality of partial mask images in the target range. The processor 210 selects the target portion mask image and eight partial mask images surrounding the target portion mask image as the merging range image. Further, the processor 210 acquires the bounding box of each defective portion included in the merging range image IMi by referring to the result data D2 (FIG. 1). The merging range image IMi in FIG. 6(A) includes two masks MDa, MDb indicating two linear defects FDa, FDb and two bounding boxes BBa, BBb.

[0042] Note that when the target portion mask image is located at the edge of the target range, the positions of one or more of the surrounding eight partial mask images are outside the target range. Acquisition of the partial mask image from outside the target range may be omitted.

[0043] In S515 (FIG. 5), the processor 210 selects a pair of defect portions of interest. In this embodiment, a pair of bounding boxes is selected as the pair of interest. When the merged range image IMi includes a plurality of bounding boxes of a plurality of defect portions, the processor 210 selects, as the pair of interest, an unprocessed pair from all pairs of the bounding boxes formed by the plurality of bounding boxes of the plurality of defect portions. The condition for two bounding boxes to be selected as the pair of interest may include that the two bounding boxes represent the same type of defect.

[0044] In the example of FIG. 6(A), the first defect FDa and the first bounding box BBa are included in the partial mask image IMmr, and the second defect FDb and the second bounding box BBb are included in the adjacent partial mask image IMmu. Assume that the defects FDa and FDb represent a single long linear defect extending from the partial mask image IMmr to the partial mask image IMmu. When a defect is long, since the defect is represented by a plurality of partial mask images, a plurality of portions of the defect can be detected as a plurality of defect portions. Hereinafter, the description will be made assuming that the bounding boxes BBa and BBb are the pair of interest. Although not shown, when the total number of the bounding boxes included in the merged range image IMi is 1 or less, the processor 210 cannot select a pair of bounding boxes and thus proceeds to S575 (FIG. 5).

[0045] In S520, the processor 210 calculates parameters used for determining the parallel condition described later. The parallel condition is a condition indicating that two defective portions are in a parallel relationship. The parallel relationship indicates a positional relationship in which two different defects are arranged in parallel. Two defects in a parallel relationship are arranged in a direction perpendicular to the direction in which the defects extend. FIGS. 6(B) and 6(C) are diagrams showing examples of the positional relationship between two defects. FIG. 6(B) shows the positional relationship between the two defects FDa and FDb in FIG. 6(A). Here, it is assumed that the defects FDa and FDb indicate the same linear defect on the fabric 700. In this case, the defects FDa and FDb are located near a single straight line and are not arranged in parallel. Such defects FDa and FDb do not have a parallel relationship. FIG. 6(C) shows the positional relationship between the two defects FDc and FDd. Here, it is assumed that the defects FDc and FDd indicate two different linear defects on the fabric 700. Here, it is assumed that the defects FDc and FDd are arranged approximately in parallel. Such a positional relationship between the defects FDc and FDd is an example of a parallel relationship.

[0046] There may be various ways to distinguish a parallel relationship such as that in FIG. 6(C) from other positional relationships. As will be described later, in this embodiment, the processor 210 uses the distance DBb between two bounding boxes and the distance DLb between the long sides of the two bounding boxes to determine whether the positional relationship is a parallel relationship. Hereinafter, the distance DBb will be referred to as the box distance DBb, and the distance DLb will be referred to as the long side distance DLb. The box distance DBb is the shortest distance between two bounding boxes. The long side distance DLb is the distance in the direction perpendicular to the long sides between the two long sides. For example, in the example of FIG. 6(B), the distance in the direction perpendicular to the long sides BBaL and BBbL between the long side BBaL of the first bounding box BBa and the long side BBbL of the second bounding box BBb corresponds to the long side distance DLb. In the example of FIG. 6(C), the distance in the direction perpendicular to the long sides BBcL and BBdL between the long side BBcL of the bounding box BBc indicating the defect FDc and the long side BBdL of the bounding box BBd indicating the defect FDd corresponds to the long side distance DLb. Note that a plurality of combinations of two long sides can be selected from two bounding boxes. As the long side distance DLb, the shortest distance among the plurality of distances of the plurality of combinations is adopted.

[0047] In S520 (FIG. 5), the processor 210 calculates the box distance DBb and the long side distance DLb of two bounding boxes of the target pair.

[0048] In S525, the processor 210 determines whether a parallel condition indicating that the target pair has a predetermined parallel relationship is satisfied. In this embodiment, the parallel condition is that the box distance DBb is the same as the long side distance DLb. For example, in the example of FIG. 6(B), since the box distance DBb is different from the long side distance DLb, it is determined that the parallel condition is not satisfied. In the example of FIG. 6(C), since the box distance DBb is the same as the long side distance DLb, it is determined that the parallel condition is satisfied.

[0049] Note that depending on the shape of the defect, it may not be possible to calculate the long-side distance DLb. For example, the two long sides of the two bounding boxes of the target pair may be orthogonal to each other. In such a case, the processor 210 may determine that the parallel condition is not satisfied.

[0050] When the parallel condition is not satisfied (S525: No), in S530, the processor 210 calculates the box distance DBb between the two bounding boxes of the target pair. In this embodiment, since the box distance DBb has already been calculated in S520, recalculating the box distance DBb is omitted.

[0051] In S535, the processor 210 determines whether the box distance DBb is less than or equal to the third threshold Th3. The third threshold Th3 is experimentally determined in advance such that when the two bounding boxes are respectively associated with two different defects on the fabric 700, the distance DBb is greater than the third threshold Th3.

[0052] When the box distance DBb is less than or equal to the third threshold Th3 (S535: Yes), in S565, the processor 210 merges the target pair. FIG. 6(D) is a diagram showing an example of merging the target pair. In this embodiment, the processor 210 generates a minimum rectangle including the two bounding boxes BBa and BBb as a new bounding box BBab.

[0053] The processor 210 may add a connection mask MDab connecting the two masks MDa and NDb to the mask image (for example, a partial mask image). The connection mask MDab may be, for example, a line segment connecting the two masks MDa and NDb at the shortest distance. Note that the addition of the connection mask MDab may be omitted.

[0054] In S570 (Fig. 5), the processor 210 stores the data of the merged defective part in the storage device 215 (for example, the non-volatile storage device 230). In this embodiment, the processor 210 stores the merged data D3 representing the information of the merged defective part (for example, including the two bounding boxes before merging, the bounding box generated by the merging, and the information associating them) in the non-volatile storage device 230. Then, the processor 210 proceeds to S575.

[0055] When the box distance DBb is greater than the third threshold Th3 (S535: No), in S540, the processor 210 calculates the mask distance Dm. Fig. 6(E) is a diagram showing the mask distance Dm. The mask distance Dm is the shortest distance between the two masks MDa and MDb.

[0056] In S545, the processor 210 determines whether the mask distance Dm is less than or equal to the second threshold Th2. The second threshold Th2 is determined experimentally in advance such that when the two masks are respectively associated with two different defects on the fabric 700, the mask distance Dm is greater than the second threshold Th2. Usually, the mask is located inside the bounding box. The mask distance Dm is likely to be greater than the box distance DBb. Therefore, the second threshold Th2 may be set to a value greater than the third threshold Th3.

[0057] When the mask distance Dm is less than or equal to the second threshold Th2 (S545: Yes), the processor 210 proceeds to S565. The processor 210 merges the target pair (S565), adds the data of the merged defective part to the merged data D3 (S570), and proceeds to S575.

[0058] When the mask distance Dm is greater than the second threshold Th2 (S545: No), at S550, the processor 210 determines whether the box distance DBb is less than or equal to the first threshold Th1. The first threshold Th1 is experimentally determined in advance such that when two bounding boxes are respectively associated with two different defects on the fabric 700, the box distance DBb is greater than the first threshold Th1.

[0059] When the distance DBb is less than or equal to the first threshold Th1 (S550: Yes), at S555, the processor 210 calculates the direction in which the defect extends. FIG. 6(F) is a diagram showing an example of the direction in which the defect extends. The method for calculating the extending directions DDa and DDb of the defects FDa and FDb may be various methods. For example, the processor 210 may calculate a regression line using the positions of each pixel of the first mask MDa, and adopt the extending direction of the regression line as the extending direction DDa of the first defect FDa. Alternatively, the processor 210 may adopt the extending direction of the diagonal line of the first bounding box BBa as the extending direction DDa of the first defect FDa. The extending direction DDb of the second defect FDb is also determined in the same way. Each direction DDa and DDb may be represented by angles AGa and AGb with respect to a reference direction (for example, the first direction Dx).

[0060] At S560 (FIG. 5), the processor 210 determines whether the angle Ad formed by the two directions is less than or equal to the angle threshold Adth. As shown in FIG. 6(F), the angle Ad is represented by the absolute value of the difference between the angles AGa and AGb of the two directions DDa and DDb. The angle threshold Adth is experimentally determined in advance such that when the two defective portions respectively indicate two different defects on the fabric 700, the angle Ad is greater than the angle threshold Adth.

[0061] When the angle Ad is less than or equal to the angle threshold Adth (S560: Yes), the processor 210 proceeds to S565. The processor 210 merges the target pair (S565), adds the data of the merged defective portion to the merged data D3 (S570), and proceeds to S575.

[0062] In this way, whether to merge the target pair is determined by the combination of the conditions of S550 and the conditions of S560. Therefore, the first threshold Th1 may be a value larger than either of the thresholds Th2 and Th3 (S535, S545). As described with reference to FIG. 2, the fabric 700 can be conveyed in a state where the lines indicating the edges 700e1 and 700e2 of the fabric 700 are inclined with respect to the partial path Pth. In this case, the position of the fabric 700 in the first direction Dx can shift between two belt fabric images acquired by two consecutive readings of the reading area Ar. For example, between two partial mask images IMmr and IMmu arranged in the second direction Dy of the merged range image IMi (FIG. 6(A)), the positions of the defects FDa and FDb representing the same linear defect in the first direction Dx can shift. When the third threshold Th3 is large, even if such a position shift occurs, the processor 210 can merge the target pairs representing the same defect.

[0063] When the determination result of S525 is Yes, when the determination result of S550 is No, and when the determination result of S560 is No, the processor 210 proceeds to S575.

[0064] In S575, the processor 210 determines whether all pairs have been processed. If there are unprocessed pairs (S575: No), the processor 210 proceeds to S515 to process a new target pair. If all pairs have been processed (S575: Yes), in S580, the processor 210 determines whether all ranges of the target range have been processed. In the present embodiment, if there is a partial mask image that has not been selected as the target partial mask image remaining within the target range, the determination result of S580 is No. If all partial mask images of the target range have been processed as the target partial mask image, the determination result of S580 is Yes. If the determination result is No, the processor 210 proceeds to S510 to process a new merged range. If the determination result is Yes, the processor 210 ends the process of FIG. 5, that is, the process of S150 (FIG. 3).

[0065] Note that after the two defective portions are merged in S565, the processor 210 selects a target pair using the merged defective portion in place of the defective portions before merging in S515. The defective portion that is at least partially included in the merged range image (S510) is used as the defective portion that forms the target pair. Therefore, three or more defective portions representing a long defect can be merged into one defective portion.

[0066] Also, although not shown, one partial mask image (for example, the partial mask image IMmr in FIG. 6(A)) may include a plurality of defective portions. In this embodiment, two defective portions included in one partial mask image can be merged.

[0067] In this way, the processor 210 can merge a plurality of defective portions representing one defect into one defective portion. Therefore, the possibility of an incorrect interpretation of the inspection result (for example, an error in the total number of defects) is reduced.

[0068] After the merging process (FIG. 3: S150), in S155, the processor 210 determines whether a defective portion is detected. If one or more defective portions are detected, the determination result is Yes. In this case, in S165, if the fabric 700 is being conveyed, the processor 210 stops the conveyance of the fabric 700. In S170, the processor 210 displays the detection result on the display unit 240 (FIG. 1). Then, the processor 210 ends the inspection process.

[0069] FIG. 7 shows an example of the screen displayed at S170. The screen DP2 represents an image area AWt that is an image of the fabric image IMrN which is the image of the target range, a progress button Bt21 for proceeding with the process, and a button Bt22 for editing information representing a defect. The processor 210 generates the fabric image IMrN by combining three strip fabric images of the target range. In the example of FIG. 7, the fabric image IMrN represents a linear defect FDp. The image area AWt represents information regarding the defect in addition to the fabric image IMrN (here, defect information DFDp indicating the defect FDp). The defect information DFDp represents the entire bounding box surrounding the defect, a character string representing the type of the defect, and a character string representing the length of the defect. At S150 (FIG. 3), when a plurality of defect portions are merged, the displayed defect information includes information on the merged defect portions (for example, the bounding box generated by the merging). The method for calculating the length of the defect portion may be various methods. For example, the diameter of the minimum circle circumscribing the mask representing the defect may be used as the length of the defect portion. Alternatively, the length of the diagonal of the minimum rectangle circumscribing the mask representing the defect may be used as the length of the defect portion.

[0070] The operator can recognize the detected defect by observing the screen DP2. Thereafter, the operator can examine the state of the defect by visually observing the fabric 700 (FIG. 2). If an error in the detection result is found by investigating the defect, the operator can edit the detection result by operating the button Bt22 on the screen DP2 (FIG. 7). For example, the processor 210 edits the result data D2 and the merge data D3 according to the instruction input to the operation unit 250.

[0071] When the investigation of the defect is completed, the operator may resume the inspection process. For example, the processor 210 may resume the inspection process in response to the operation of the progress button Bt21 (FIG. 7).

[0072] When no defective part is detected (S155: No), in S160, the processor 210 determines whether the entire range of the portion of the fabric 700 to be inspected has been processed. The determination method in S160 may be various methods. For example, when the current relative position of the fabric 700 calculated using the encoder 120 reaches the end position specified in advance by the operator, the processor 210 may determine that the entire range has been processed.

[0073] When an unprocessed portion remains (S160: No), in S163, the processor 210 supplies an instruction to the transport device 900 to transport the fabric 700 to the relative position of the fabric 700 for reading the next strip fabric image. The control device 990 of the transport device 900 transports the fabric 700 according to the instruction. The process of S163 is the same as the process of S138. After S163, the process proceeds to S110. Thereby, the process for the new target range is executed.

[0074] When the entire range has been processed (S160: Yes), the processor 210 ends the inspection process.

[0075] As described above, in this embodiment, the digital cameras 111 - 114 (Figure 2) are examples of reading devices configured to read different portions of the fabric 700, which is an example of an object. The processor 210 (Figure 1) executes the following processes according to the program 231. In S125 (Figure 3), the processor 210 executes a defect detection process, which is a process of detecting a defective portion that is a portion representing a defect (for example, the defect FD in Figure 4(D)) in the fabric 700 (the process of S125 is referred to as the defect detection process S125). The bounding box and the mask detected in the defect detection process S125 are examples of defective portions (for example, the bounding box BBD1 and the mask MD1 in Figure 4(D)). The processor 210 executes the defect detection process S125 using each of the plurality of read images obtained using the digital cameras 111 - 114 (here, the plurality of strip fabric images IMrc (Figure 4(B))).

[0076] In S150, when the first defective part and the second defective part detected by the defect detection process S125 satisfy the merging condition, the processor 210 executes a merging process of merging the first defective part and the second defective part as one defective part (the process of S150 is also called the merging process S150). As described in S565 (FIG. 5) and FIG. 6(D), in this embodiment, one bounding box is formed by merging two bounding boxes (for example, by merging the bounding boxes BBa and BBb in FIG. 6(D), the bounding box BBab is formed).

[0077] The merging process S150 includes a process of merging the target pair when a condition Cm1 (referred to as the first merging condition Cm1) including the conditions of S550 and S560 in FIG. 5 is satisfied (referred to as the first merging process Sm1). The condition of S550 is that the distance DBb between the bounding boxes BBa and BBb (FIG. 6(C)) is equal to or less than the first threshold Th1. That is, the condition of S550 is that the distance between the first defective part and the second defective part is equal to or less than the first threshold Th1. The first bounding box BBa is an example of the first defective part, and the second bounding box BBb is an example of the second defective part. The condition of S560 is that the angle Ad formed by the extending directions DDa and DDb of the defects FDa and FDb represented by the masks MDa and MDb (FIG. 6(F)) is equal to or less than the angle threshold Adth. That is, the condition of S560 is that the angle formed by the first defect direction, which is the extending direction of the first defect represented by the first defective part, and the second defect direction, which is the extending direction of the second defect represented by the second defective part, is equal to or less than the angle threshold Adth. The first mask MDa is an example of the first defective part, and the direction DDa is an example of the first defect direction, which is the extending direction of the first defect (here, the first defect FDa). The second mask MDb is an example of the second defective part, and the direction DDb is an example of the second defect direction, which is the extending direction of the second defect (here, the second defect FDb).

[0078] According to this configuration, the processor 210 can merge a first defective part and a second defective part that represent different parts of the same defect, so that a defective part that appropriately represents the defect of the fabric 700 can be obtained.

[0079] Note that the merging process may be various processes for generating merging information for treating a plurality of different defective parts as one defective part. The merging information is not limited to a new bounding box, and may be various information indicating that the whole of a plurality of defective parts should be treated as one defective part. The merging information may be, for example, information for associating a plurality of defective parts to be merged. The processor 210 can determine a defective part representing the whole of a plurality of defective parts by referring to such merging information (for example, the smallest rectangle including a plurality of defective parts). The defective part representing the whole of a plurality of defective parts may be used not only for display but also for various processes (for example, calculation of the length of a defect). In addition to the process of generating merging information, the merging process may include an image correction process of filling gaps between a plurality of defects in the read image to form a continuous defect. The corrected image may be used for various processes such as display. However, the image correction process may be omitted.

[0080] Also, in this embodiment, as shown in FIG. 5, when the determination result of S550 is No (that is, DBb > Th1), the processor 210 does not execute S555, and when the determination result of S550 is Yes (the distance DBb is less than or equal to the first threshold Th1), the processor 210 executes S555. The process of S555 is an example of a direction calculation process for calculating a first defective direction and a second defective direction (for example, the extending direction DDa of the first defect FDa and the extending direction DDb of the second defect FDb in FIG. 6(F)). Such a process often has a larger calculation load than the process of calculating the distance DBb. In this embodiment, when the determination result of S550 is No, the process of S555 is omitted, so the calculation load of the merging process can be reduced.

[0081] In addition, in this embodiment, the defective part detected by the defect detection process S125 (FIG. 3) includes a bounding box associated with the defect represented by the defective part (for example, the bounding box BBD1 in FIG. 4(D)). The merging process S150 includes S565 in FIG. 5. In S565, the processor 210 merges the bounding boxes of the first defective part and the second defective part to be merged into one bounding box (for example, by merging the bounding boxes BBa and BBb in FIG. 6(D), the bounding box BBab is formed). According to this configuration, the processor 210 can appropriately represent the result of the merging by the bounding box.

[0082] In addition, in this embodiment, the defective part detected by the defect detection process S125 (FIG. 3) includes a mask which is the area of the defect represented by the defective part (for example, the mask MD1 in FIG. 4(D)). The merging process S150 includes a process (referred to as the second merging process Sm2) of merging the target pair when a condition Cm2 (referred to as the second merging condition Cm2) including the condition of S545 in FIG. 5 is satisfied. The condition of S545 is that the mask distance Dm is equal to or less than the second threshold Th2 regardless of the direction in which the defect extends. That is, the condition of S545 is that the distance between the mask of the first defective part and the mask of the second defective part is equal to or less than the second threshold Th2. In FIG. 6(F), the first mask MDa is an example of the mask of the first defective part, the second mask MDb is an example of the mask of the second defective part, and the mask distance Dm is the distance between these masks MDa and MDb. The condition of S545 is determined regardless of the angle (for example, the angle Ad in FIG. 6(F)) formed by the extending direction of the first defect of the first defective part and the extending direction of the second defect of the second defective part. According to this configuration, since the processor 210 can merge the first defective part and the second defective part representing different parts of the same defect, a defective part that appropriately represents the defect of the fabric 700 can be obtained.

[0083] In addition, in this embodiment, the defective portion detected by the defect detection process S125 (FIG. 3) includes a bounding box associated with the defect represented by the defective portion (for example, the bounding box BBD1 in FIG. 4(D)). The merging process S150 includes a process of merging the target pair when a condition Cm3 (referred to as the third merging condition Cm3) including the condition of S535 in FIG. 5 is satisfied (referred to as the third merging process Sm3). The condition of S535 is that the box distance DBb is equal to or less than the third threshold Th3 regardless of the direction in which the defect extends. That is, the condition of S535 is that the distance between the bounding box of the first defective portion and the bounding box of the second defective portion is equal to or less than the third threshold Th3 (the box distance DBb between the first bounding box BBa and the second bounding box BBb in FIG. 6(B) is an example of the distance between two bounding boxes). This condition is determined regardless of the angle (for example, the angle Ad in FIG. 6(F)) formed by the extending direction of the first defect of the first defective portion and the extending direction of the second defect of the second defective portion. According to this configuration, the processor 210 can merge the first defective portion and the second defective portion representing different portions of the same defect, so that a defective portion that appropriately represents the defect of the fabric 700 can be obtained.

[0084] Also, in this embodiment, the merging process S150 (FIG. 3) includes a first merging process Sm1, a second merging process Sm2, and a third merging process Sm3 (FIG. 5). As shown in FIG. 5, the processor 210 makes judgments in the order of the third merging condition Cm3 (including S535), the second merging condition Cm2 (including S545), and the first merging condition Cm1 (including S550 and S560). When any of the conditions is satisfied, the processor 210 merges the first defective portion and the second defective portion according to the satisfied condition without judging the conditions after the satisfied condition (S565). The process of calculating the direction of the defect (S555) often has a higher computational load than the processes of calculating the distances DBb and Dm (S520, S530, S540). In this embodiment, the determination of using a process with a lower computational load is executed prior to the determination of using a process with a higher computational load. When the condition for merging is satisfied, the conditions after the satisfied condition are not judged. Therefore, the computational load of the merging process can be reduced. For example, when the third merging condition Cm3 or the second merging condition Cm2 is satisfied, the calculation of the parameters used in the first merging condition Cm1 after the satisfied condition (e.g., S555) is omitted. Note that the second merging condition Cm2 may be judged prior to the third merging condition Cm3.

[0085] Also, in this embodiment, the conditions Cm1, Cm2, and Cm3 (FIG. 5) for merging the first defective portion and the second defective portion each include that the parallel condition is not satisfied (S525: No). The parallel condition is a condition indicating that two defective portions are in a parallel relationship, as described with reference to FIGS. 6(B) and 6(C). According to this configuration, the possibility that two defective portions respectively associated with two different defects on the fabric 700 are erroneously merged is reduced.

[0086] Note that the processor 210 may determine whether two defective portions are in a parallel relationship regardless of whether the two defects are actually arranged in parallel. In other words, the processor 210 may determine that two defective portions are in a parallel relationship when the two defects can be arranged in parallel. The parallel condition may be various conditions for such a determination (other embodiments of the parallel condition will be described later).

[0087] Note that in this embodiment, the third merging condition Cm3 includes S525: No, S535: Yes. The second merging condition Cm2 includes S525: No, S535: No, S545: Yes. The first merging condition Cm1 includes S525: No, S535: No, S545: No, S550: Yes, S560: Yes.

[0088] B. Second Embodiment: FIG. 8 is a flowchart showing another embodiment of the inspection process. There are two differences from the inspection process of FIG. 3. The first difference is that when a defect is detected (S155: Yes), instead of S165 and S170, S175 is executed. The second difference is that after S175, the process proceeds to S160. The processing of other parts of the inspection process is the same as the processing of the corresponding parts in FIG. 3.

[0089] In S175, the processor 210 stores data representing defect information related to the detected defective portion in the storage device 215 (for example, the non-volatile storage device 230). The defect information may be various information related to the defective portion. For example, the defect information may represent the defective portion (for example, the bounding box and the mask) and its position, the length of each defective portion, and the total number of defective portions.

[0090] After S175, the processor 210 transitions to S160. Then, without stopping the conveyance, the processor 210 inspects the entire range and finishes the inspection process. After the inspection process ends, an operator can recognize the inspection result by referring to the data stored in the storage device 215 (e.g., data of defect information, result data D2, merged data D3). Thus, in this embodiment, after starting the inspection process, the processor 210 can complete the inspection of the entire range without receiving an instruction from the operator.

[0091] C. Third Embodiment: FIGS. 9(A) and 9(B) are diagrams showing another embodiment of the parallel condition determined in S525 (FIG. 5). In this embodiment, the parallel condition is that the line segment connecting the respective centroids of two bounding boxes intersects any one of the four long sides of the two bounding boxes.

[0092] The defects FDa and FDb in FIG. 9(A) are the same as the defects FDa and FDb in FIG. 6(B), respectively. The line segment LBa is a line segment connecting the centroid BBac of the first bounding box BBa and the centroid BBbc of the second bounding box BBb. The line segment LBa does not intersect the long sides of the bounding box BBa and the long side of the second bounding box BBb, but intersects the short side. Therefore, the parallel condition is not satisfied.

[0093] The defects FDc and FDd in FIG. 9(B) are the same as the defects FDc and FDd in FIG. 6(C), respectively. The line segment LBc is a line segment connecting the centroid BBcc of the bounding box BBc and the centroid BBdc of the bounding box BBd. The line segment LBc intersects the long side BBcL of the bounding box BBc and the long side BBdL of the bounding box BBd. Therefore, the parallel condition is satisfied.

[0094] In S520 (Fig. 5), the processor 210 calculates a line segment connecting the centroids of each of the two bounding boxes. In S525, the processor 210 determines that the parallel condition is satisfied when the calculated line segment intersects any of the four long sides of the two bounding boxes. Similar to the parallel condition in the above embodiment, the parallel condition in this embodiment can also reduce the possibility that two defective parts respectively associated with two different defects on the fabric 700 are erroneously merged.

[0095] D. Fourth Embodiment: FIG. 10 is a flowchart showing another embodiment of the merging process. The difference from the merging process in Fig. 5 is only that S518 is added between S515 and S520. The processing of other parts of the merging process is the same as the processing of the corresponding parts in Fig. 5. Note that this embodiment is applicable to each of the above embodiments.

[0096] After S515 (selection of the pair of interest), in S518, the processor 210 determines whether the two defective parts of the pair of interest indicate linear defects. In this embodiment, the object detection model 310 detects a plurality of types of detection objects including defective parts representing holes and defective parts representing linear defects.

[0097] If one or both of the two defective parts constituting the pair of interest are different from the defective parts representing linear defects, the determination result of S518 is No. In this case, the processor 210 does not proceed to S520 but proceeds to S575. That is, the pair of interest is not merged.

[0098] If both of the two defective parts constituting the pair of interest represent linear defects, the determination result of S518 is Yes. In this case, the processor 210 proceeds to S520. That is, the pair of interest can be merged.

[0099] As described above, in this embodiment, when each of the first defective portion and the second defective portion constituting the target pair is determined to represent a linear defect (S518: Yes), the first defective portion and the second defective portion can be merged. That is, in this embodiment, the condition of S518: Yes is added to each of the first merging condition Cm1, the second merging condition Cm2, and the third merging condition Cm3 described with reference to FIG. 5.

[0100] One linear defect can be long. One long linear defect can be represented by being divided into a plurality of partial images (for example, a plurality of partial mask images (FIG. 6(A))). That is, a plurality of defective portions suitable for merging can be detected from one long linear defect. When the condition for merging includes the condition of S518, the processor 210 can merge the plurality of defective portions detected from one long linear defect. In addition, defects different from linear defects (for example, holes) are often small. The possibility of detecting a plurality of defective portions suitable for merging from small defects is low. When the condition for merging includes the condition of S518, the possibility of erroneously merging two defective portions respectively indicating two different defects is reduced.

[0101] In general, as a technique for merging a plurality of bounding boxes, a technique using so-called IoU (Intersection over Union) can be used. IoU is an index value representing the ratio of the overlapping portion of two bounding boxes. For example, when two bounding boxes show a large IoU, it is highly likely that those two bounding boxes represent the same object. Such two bounding boxes are suitable for merging. When a plurality of defective portions are detected from a single long linear defect, each defective portion may have a bounding box representing an elongated region. When a plurality of defective portions have bounding boxes representing elongated regions, the overlapping portion between the plurality of bounding boxes tends to be small, and the IoU tends to be a small value. Therefore, the merging using IoU may not be able to merge a plurality of defective portions detected from a single long linear defect. In this embodiment, when each of the first defective portion and the second defective portion constituting the target pair is determined to represent a linear defect (S518: Yes), the processor 210 uses the distances DBb, Dm and the angle Ad to determine whether the conditions for merging are satisfied. Therefore, the processor 210 can appropriately merge a plurality of defective portions detected from a single long linear defect.

[0102] E. Modification example: (1) The process of merging two defective parts may be various processes for generating merging information for treating a plurality of different defective parts as one defective part. The merging information is not limited to a new bounding box and may be various information indicating that the whole of a plurality of defective parts should be treated as one defective part. The merging information may be, for example, information for associating a plurality of defective parts to be merged. By referring to such merging information, the processor 210 can determine a defective part representing the whole of a plurality of defective parts (for example, the smallest rectangle including a plurality of defective parts). The defective part representing the whole of a plurality of defective parts may be used not only for display but also for various processes (for example, calculation of the length of a defect). Further, in addition to the process of generating merging information, the merging process may include an image correction process for filling gaps between a plurality of defects in the read image to form continuous defects. The corrected image may be used for various processes such as display. However, the image correction process may be omitted.

[0103] Note that in the example of FIG. 6(A), the processor 210 selects a pair of defective parts of interest from the defective part of interest mask image IMmr and the adjacent partial mask image of the defective part of interest mask image (here, eight partial mask images surrounding the defective part of interest mask image IMmr). The processor 210 may select a pair of defective parts of interest from a wider range. For example, the merging range image IMi may be composed of partial mask images of 5 rows and 5 columns centered on the defective part of interest mask image.

[0104] (2) The object detection model 310 used in the defect detection process S125 (Fig. 3) may be various other object detection models instead of RTMDet (for example, YOLO (You only look once), Mask R-CNN, etc.). Also, the defect detection process may be a process of detecting defects without using a machine learning model. For example, the defect detection process may be a process of detecting defective parts by template matching using a template image representing a defect. The defective parts detected by the defect detection process are not limited to holes and linear defects, and may include parts representing various defects such as dirty parts. The defective parts detected by the defect detection process may include one or more types of defective parts including, for example, a defective part representing a linear defect.

[0105] (3) In S555 (Fig. 3), the method of calculating the direction in which the defect extends may be various other methods instead of the method of calculating the direction in which the regression line extends as the direction in which the defect extends. For example, the processor 210 calculates the smallest rectangle circumscribing the mask representing the defect. Here, the rotation of the rectangle is considered. That is, the sides of the smallest circumscribing rectangle can be inclined with respect to the first direction Dx and the second direction Dy. The processor 210 may adopt the direction in which the long side of the smallest circumscribing rectangle extends as the direction in which the defect extends.

[0106] (4) In the embodiment of Fig. 10, the method of determining whether the defective part represents a linear defect may be various other methods instead of the method of using the type of defect predicted by the object detection model 310 (Fig. 3: S125). For example, when the aspect ratio of the bounding box (here, the ratio of the length of the long side to the length of the short side) is equal to or greater than a ratio threshold, the processor 210 may determine that the defective part represents a linear defect. The ratio threshold is determined experimentally in advance such that the aspect ratio of the bounding box representing a type of defect different from the linear defect (for example, a hole) is less than the ratio threshold. Such a determination (referred to as a linear determination) may be made at various timings before S518. For example, the processor 210 may execute the linear determination between S515 and S518.

[0107] The linear determination may be executed according to another program different from program 231. The linear determination may be executed by another device different from data processing device 200. In these cases, processor 210 may acquire the result of the linear determination at various timings before S518 (for example, between S515 and S518).

[0108] (5) The parallel condition determined in S525 (FIG. 5) may be various conditions indicating that the first defective portion and the second defective portion are in a predetermined parallel relationship, instead of the conditions described in FIGS. 6(B), 6(C), 9(A), and 9(B). For example, the parallel condition may be that the following two conditions using two regression lines indicating two masks of two defective portions are satisfied. Condition C1: The angle formed by the two regression lines is equal to or less than a second angle threshold value. Condition C2: The distance between the intersection point of the two regression lines and the two masks is equal to or greater than a fourth distance threshold value. Note that the second angle threshold value and the fourth distance threshold value are determined experimentally in advance so as to obtain an appropriate determination result. The second angle threshold value may be the same as the angle threshold value Adth in S560 (FIG. 5).

[0109] With reference to the examples in FIGS. 6(B) and 6(C), the determination result of the parallel condition will be described. As in the example of FIG. 6(B), when the defects FDa and FDb indicate the same linear defect on the fabric 700, the angle formed by the two regression lines is approximately zero, and the possibility that the condition C1 is satisfied is high. Also, when the defects FDa and FDb are located on one linear defect, the intersection of the two regression lines can be formed at a position close to the defects FDa and FDb. Therefore, the possibility that the condition C2 is satisfied is low. As in the example of FIG. 6(C), when the defects FDc and FDd are arranged approximately parallel to each other, the angle formed by the two regression lines is approximately zero, and the possibility that the condition C1 is satisfied is high. When the defects FDc and FDd are arranged at positions separated from each other, the intersection of the two regression lines can be formed at a position far from the defects FDc and FDd. Therefore, the possibility that the condition C2 is satisfied is high. Thus, in the example of FIG. 6(C), compared with the example of FIG. 6(B), the possibility that the parallel condition is satisfied is high. Therefore, the possibility that two defective portions with the arrangement as in FIG. 6(C) are erroneously merged is reduced.

[0110] The processor 210 calculates the regression line of each of the two masks in S520 (FIG. 5). The processor 210 calculates the angle formed by the two regression lines in the same manner as the angle Ad in FIG. 6(F). The processor 210 calculates the position of the intersection of the two regression lines. In S525, the processor 210 determines whether the parallel condition is satisfied using the calculated parameters.

[0111] Thus, the parallel condition defined by the conditions C1 and C2 can reduce the possibility that two defective portions respectively associated with two different defects on the fabric 700 are erroneously merged.

[0112] Also, the parallel condition may be that the following condition C3 is satisfied. Condition C3: The short side of the bounding box of the second defective portion is located between the two short sides of the bounding box of the first defective portion. Here, the position of each short side is compared using the position in the direction perpendicular to the short side of the bounding box of the first defective portion.

[0113] In the examples of FIGS. 9(A) and 9(B), the parallel condition is determined using the positions in the second direction Dy perpendicular to the short sides SLa1, SLa2, SLc1, and SLc2 of the bounding boxes BBa and BBc. In the example of FIG. 9(A), neither of the short sides SLb1 and SLb2 of the second bounding box BBb is located between the short sides SLa1 and SLa2 of the first bounding box BBa. Therefore, the parallel condition is not satisfied. In the example of FIG. 9(B), the short side SLd2 of the bounding box BBd is located between the short sides SLc1 and SLc2 of the bounding box BBc. Therefore, the parallel condition is satisfied.

[0114] The processor 210 obtains the position of each short side of each bounding box in S520 (FIG. 5). In S525, the processor 210 determines whether the parallel condition is satisfied using the positions of the short sides.

[0115] In this way, the parallel condition defined by condition C3 can reduce the possibility that two defective portions respectively associated with two different defects on the fabric 700 are erroneously merged.

[0116] Note that each of conditions C1, C2, and C3 may be combined with the other parallel conditions described above. In any case, conditions C1, C2, and C3 may be used when the long sides are parallel (i.e., the short sides are also parallel) between two bounding boxes. When the long sides are not parallel (i.e., the short sides are not parallel either) between two bounding boxes, it may be determined that conditions C1, C2, and C3 are not satisfied.

[0117] (6) The merging conditions for merging the first defective portion and the second defective portion may be various other conditions instead of the conditions of the embodiments of FIGS. 5, 9(A), 9(B), and 10 and the conditions of each of the above-described modified examples. For example, the order of determination of the first merging condition Cm1 (including S550 and S560), the second merging condition Cm2 (including S545), and the third merging condition Cm3 (including S535) may be any order. S555 may be executed before S550. S525 may be omitted. One or two conditions arbitrarily selected from the conditions Cm1, Cm2, and Cm3 may be omitted.

[0118] (7) The processing for inspection is not limited to the above-described embodiments and modified examples, and may be various types of processing. For example, after S170 (FIG. 3), the processor 210 may execute an inspection process on the remaining portion of the fabric 700.

[0119] (8) The total number of digital cameras used for reading the fabric 700 is not limited to 4, and may be various numbers of 1 or more. Also, the reading device used for reading the fabric 700 may include a line sensor instead of an area sensor such as a digital camera. In this case, the processor 210 may obtain data of a read image by causing the reading device to read the fabric 700 while transporting the fabric 700 by the transport device 900. In any case, the processor 210 may detect a defect using the read image obtained using the reading device. For example, when the reading device includes one sensor, the read image obtained from the one sensor may be used as it is. Also, a plurality of sensors may be arranged to read different portions of the fabric 700. In this case, one read image may be generated by combining a plurality of read images obtained from the plurality of sensors. The combination of the read images may be performed by a device different from the data processing device 200 (for example, the reading device).

[0120] (9)In S138 and S163 (Fig. 3), the processor 210 supplied an instruction to convey to the conveying device 900. Instead, in at least one of S138 and S163, an operator may convey the fabric 700 by operating the control panel 980 (Fig. 2). Also, the conveying device 900 may continue to convey (automatic conveyance). In this case, in S138 and S163, the processor 210 may wait, referring to the information from the encoder 120, until the relative position of the fabric 700 becomes the relative position for reading the next strip fabric image. Here, in response to the current relative position becoming the relative position for reading the next strip fabric image, the processor 210 may stop the conveyance of the fabric 700.

[0121] (10) The object to be processed, which is the object for the defect detection process, may be various fabrics for sewing (such as woven fabrics, knitted fabrics, denim fabrics, etc.). The fabric may be a fabric without ears. The object to be processed is not limited to fabrics and may be various sheet-like objects (for example, paper, resin film, etc.). The object to be processed is not limited to sheet-like objects and may be various objects such as an automobile body. The configuration of the conveying device for conveying the object to be processed may be various configurations suitable for conveying the object to be processed. The configuration of the reading device may be various configurations suitable for the object to be processed and the conveying device.

[0122] (11) In the above-described embodiment and the above-described modification, the processor 210 may cause the GPU 260 to execute various operations. For example, the processor 210 may cause the GPU 260 to execute part or all of the operations by the object detection model 310. Note that the GPU 260 may be omitted.

[0123] (12) The data processing device 200 in Fig. 1 may be a type of device different from a personal computer (for example, a digital camera, a scanner, a smartphone). Also, a plurality of devices (for example, computers) that can communicate with each other via a network may share part of the data processing function by the data processing device and provide the data processing function as a whole (a system including these devices corresponds to the data processing device).

[0124] In each of the above embodiments, part of the configuration realized by hardware may be replaced with software, and conversely, part or all of the configuration realized by software may be replaced with hardware. For example, the processing by the object detection model 310 (FIG. 1) may be executed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0125] Also, when part or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored in a computer-readable recording medium (for example, a non-transitory recording medium). The program can be used in a state stored in the same or a different recording medium (a computer-readable recording medium) at the time of provision. The "computer-readable recording medium" includes not only portable recording media such as memory cards and CD-ROMs, but also internal storage devices in a computer such as various ROMs and external storage devices connected to a computer such as a hard disk drive.

[0126] The above embodiments and modifications can be combined as appropriate. Also, the above embodiments and modifications are for facilitating the understanding of the present disclosure and do not limit the present invention. The present invention can be changed and improved without departing from its gist, and equivalents thereof are included in the present invention.

Description of Reference Numerals

[0127] 111 - 114…Digital camera, 120…Encoder, 130…Light source, 200…Data processing device, 210…Processor, 215…Memory device, 220…Volatile memory device, 230…Non - volatile memory device, 231…Program, 240…Display unit, 250…Operation unit, 260…Graphics processing unit (GPU), 270…Communication interface, 310…Object detection model, 700…Fabric, 900…Conveyor, 910…First roller, 920…Second roller, 980…Control panel, 981…First operation unit, 982…Second operation unit, 983…Third operation unit, 984…Fourth operation unit, 990…Control device, Cm1…First merging condition, Cm2…Second merging condition, Cm3…Third merging condition, S125…Defect detection process, S150…Merging process

Claims

1. A program, using each of a plurality of read images obtained by reading different parts of an object by a reading device, to execute a defect detection process which is a process of detecting a defective part which is a part representing a defect in the object; a function of executing a merging process of merging the first defective part and the second defective part as one defective part when the first defective part and the second defective part detected by the defect detection process satisfy a merging condition; is realized by a computer, the merging process includes a process of merging the first defective part and the second defective part when the first defective part and the second defective part satisfy a first merging condition, the first merging condition is that an angle formed by a first defect direction which is a direction in which a first defect represented by the first defective part extends and a second defect direction which is a direction in which a second defect represented by the second defective part extends is equal to or less than a first angle threshold; that a distance between the first defective part and the second defective part is equal to or less than a first distance threshold; including, a program.

2. The program according to claim 1, wherein the process of merging the first defective part and the second defective part when the first merging condition is satisfied includes a process of not executing a direction calculation process for calculating the first defect direction and the second defect direction when the distance between the first defective part and the second defective part is greater than the first distance threshold, and executing the direction calculation process when the distance between the first defective part and the second defective part is equal to or less than the first distance threshold. A program.

3. The program according to claim 1 or 2, wherein the first merging condition includes that each of the first defective part and the second defective part is determined to represent a linear defect. A program.

4. The program according to claim 1 or 2, wherein the defective part detected by the defect detection process includes a bounding box associated with the defect represented by the defective part, and the merging process includes a process of merging the bounding boxes of the first defective part and the second defective part to be merged as one bounding box. A program.

5. The program according to claim 1 or 2, The defective part detected by the defect detection process includes a mask that is the area of the defect represented by the defective part. The merging process includes a process of merging the first defective part and the second defective part when the first defective part and the second defective part satisfy a second merging condition. The second merging condition includes that the distance between the mask of the first defective part and the mask of the second defective part is equal to or less than a second distance threshold regardless of the angle formed by the extending direction of the first defect of the first defective part and the extending direction of the second defect of the second defective part. Program.

6. The program according to claim 1 or 2, The defective part detected by the defect detection process includes a bounding box associated with the defect represented by the defective part. The merging process includes a process of merging the first defective part and the second defective part when the first defective part and the second defective part satisfy a third merging condition. The third merging condition includes that the distance between the bounding box of the first defective part and the bounding box of the second defective part is equal to or less than a third distance threshold regardless of the angle formed by the extending direction of the first defect of the first defective part and the extending direction of the second defect of the second defective part. Program.

7. The program according to claim 1 or 2, The defective part detected by the defect detection process is a bounding box associated with the defect represented by the defective part, and a mask that is the area of the defect represented by the defective part, and includes The merging process is a process of merging the first defective part and the second defective part when the first defective part and the second defective part satisfy a second merging condition, and a process of merging the first defective part and the second defective part when the first defective part and the second defective part satisfy a third merging condition, and includes The second merging condition includes that the distance between the mask of the first defective part and the mask of the second defective part is equal to or less than a second distance threshold regardless of the angle formed by the extending direction of the first defect of the first defective part and the extending direction of the second defect of the second defective part. The third merging condition includes that the distance between the bounding box of the first defect part and the bounding box of the second defect part is equal to or less than a third distance threshold regardless of the angle formed by the extending direction of the first defect of the first defect part and the extending direction of the second defect of the second defect part. The merging process is a process that determines the first merging condition, the second merging condition, and the third merging condition in the order of the third merging condition, the second merging condition, and the first merging condition. is a process that, when any one of the first merging condition, the second merging condition, and the third merging condition is satisfied, merges the first defect part and the second defect part according to the satisfied condition without determining the conditions after the satisfied condition. A program including the above.

8. A program according to claim 1 or 2, wherein the condition for merging the first defect part and the second defect part includes that a parallel condition indicating that the first defect part and the second defect part are in a predetermined parallel relationship is not satisfied.

9. A data processing device, comprising a detection unit that executes a defect detection process, which is a process of detecting a defect part, which is a part representing a defect in the object, using each of a plurality of read images obtained by reading different parts of the object by a reading device; and a merging unit that executes a merging process of merging the first defect part and the second defect part detected by the defect detection process into one defect part when the first defect part and the second defect part satisfy a merging condition. The data processing device is provided with wherein the merging process includes a process of merging the first defect part and the second defect part when the first defect part and the second defect part satisfy a first merging condition. The first merging condition is that the angle formed by a first defect direction, which is the extending direction of the first defect represented by the first defect part, and a second defect direction, which is the extending direction of the second defect represented by the second defect part, is equal to or less than a first angle threshold; and that the distance between the first defect part and the second defect part is equal to or less than a first distance threshold. A data processing device including the above.

Images