Computer program and data processing device

Abstract

To record information on a defect of an object by using a machine learning model.SOLUTION: A computer program allows a computer to realize a function of acquiring an object image showing an object, a function of acquiring a recording instruction for recording a defect of an object shown in the object image from a user, a function of displaying a candidate of a defect of a recording object when the recording instruction is acquired, a function of acquiring a selection instruction for selecting a defect of the recording object from candidates of defects from a user, and a function of recording specific information of the defect of the recording object selected by the selection instruction. The object image is an image generated by using an image sensor. The candidates of defects are determined by determination processing executed to an object image by using a machine learning model. The specific information includes information that is information generated by using a machine learning model and is related to at least one of the position and shape of a defect of a recording object.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present specification relates to a technology for recording information about defects in an object using a machine learning model. [Background technology]

[0002] The inspection device disclosed in Patent Document 1 uses a trained estimator to determine whether a product is good or bad. When image data showing a product is input, the estimator outputs an output value indicating the result of estimating the type and range of defects contained in the product. If the output value indicates that the product contains a defect, the inspection device determines that the product is not good, and if the output value indicates that the product does not contain a defect, the inspection device determines that the product is good. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-60879 Summary of the Invention [Problem to be solved by the invention]

[0004] This specification discloses a new technique for recording information about defects in an object using a machine learning model. [Means for solving the problem]

[0005] The techniques disclosed in this specification can be implemented in the following application examples.

[0006] [Application Example 1] A computer program that causes a computer to realize the following: an image acquisition function that acquires a target image showing an object, the target image being an image generated using an image sensor; a recording instruction acquisition function that acquires from a user a recording instruction to record a defect of the object shown in the target image; a candidate display function that displays candidate defects to be recorded when the recording instruction is acquired, the candidate defects being determined by a decision process executed on the target image using a machine learning model; a selection instruction acquisition function that acquires from a user a selection instruction to select a defect to be recorded from the candidate defects; and a recording function that records specific information of the defect to be recorded selected by the selection instruction, the specific information being information generated using the machine learning model and including information regarding at least one of the position and shape of the defect to be recorded.

[0007] According to the above configuration, specific information regarding the position and shape of defects in the object shown in the target image can be recorded based on a decision process including a process performed using a machine learning model and user instructions.

[0008] The technology disclosed in this specification can be realized in various forms, such as a method for recording information about defects in an object, a data processing device for recording information about defects in an object, a computer program for realizing these methods and devices, a recording medium on which the computer program is recorded, etc. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the configuration of an inspection system 1000. [Figure 2] FIG. 1 is a perspective view showing a schematic configuration of an inspection device 10. [Figure 3] 4 is a flowchart of an inspection process according to the first embodiment. [Figure 4] FIG. 10 is a diagram showing an example of an image used in the inspection process. [Figure 5]FIG. 10 is a diagram showing an example of a threshold value table TT. [Figure 6] FIG. 11 is a diagram showing an example of a result display screen W11. [Figure 7] FIG. 2 is a diagram showing an example of a defect database DD. [Figure 8] FIG. [Figure 9] FIG. 10 is a diagram conceptually illustrating the difference in detection results at multiple thresholds. [Figure 10] FIG. 10 is a diagram showing an example of a screen displayed in additional recording processing. [Figure 11] 10 is a flowchart of an inspection process according to a second embodiment. [Figure 12] 10 is a flowchart of additional recording processing according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] A. First Example A1. Inspection system configuration FIG. 1 is a diagram showing the configuration of an inspection system 1000. The inspection system 1000 is a system for inspecting a cloth 700. The cloth to be inspected is a known fabric such as a woven fabric, a knitted fabric, or a nonwoven fabric. The inspection system 1000 in FIG. 1 includes a data processing device 200 and an inspection device 10. The data processing device 200 and the inspection device 10 are communicatively connected. This allows the data processing device 200 to receive image data generated by digital cameras 111-114 (described later), control signals from a control unit 990 of a conveying device 900, and the like.

[0011] The data processing device 200 is, for example, a personal computer. The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, a graphics processing unit (GPU) 260, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to perform data processing, such as a central processing unit (CPU) or a system on a chip (SoC). The volatile storage device 220 is, for example, a dynamic random access memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory.

[0013] The non-volatile storage device 230 stores a computer program PG, a defect database DD, and a threshold table TT. The computer program PG includes an object detection model MD as a program module. The object detection model MD will be described later. The processor 210 executes the computer program PG to perform the inspection process, which will be described later. The defect database DD is a database for recording information about defects in the fabric detected in the inspection process, which will be described later. The threshold table TT is a table for recording thresholds used to determine valid detection results in the detection process, which will be described later. The defect database DD and the threshold table TT will be described later.

[0014] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel overlaid on the display unit 240. The display unit 240 and the operation unit 250 may form a so-called touch screen. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250. The display unit 240 may display operation elements (e.g., buttons, sliders, etc.), and the displayed elements may be operated through operation of the operation unit 250.

[0015] The GPU 260 is a computing device configured to perform various numerical operations such as image processing and machine learning. The GPU 260 performs various operations in accordance with instructions from the processor 210. In this embodiment, the GPU 260 performs operations for the object detection model MD in accordance with instructions from the processor 210.

[0016] The communication interface 270 is an interface for communicating with the inspection device 10 (for example, it includes one or more of a USB interface, a wired LAN interface, an IEEE802.11 wireless interface, and an industrial camera interface (for example, CameraLink, CoaXPress, etc.)). In this embodiment, the communication interface 270 is connected to the inspection device 10.

[0017] The inspection device 10 is a device that performs inspection of the fabric, which is the object of inspection in this embodiment, in cooperation with a data processing device 200. The inspection device 10 includes digital cameras 111-114, a rotary encoder 120, a light source 130, and a conveying device 900.

[0018] The digital cameras 111-114 are devices that use image sensors including imaging elements such as CCDs and CMOSs ​​to capture images of objects (subjects) and generate and acquire captured images of the objects. The digital cameras 111-114 are used to capture images of the fabric to be inspected. The rotary encoder 120 is used to calculate the relative position of the fabric with respect to the conveying device 900 (details will be described later). The light source 130 irradiates light onto the fabric to be inspected in order to capture a clear image of the fabric.

[0019] The conveying device 900 is a device that conveys the fabric to be inspected, and includes a conveying mechanism 950, an operation unit 980 that receives operations by the user, and a control unit 990.

[0020] FIG. 2 is a perspective view showing a schematic configuration of the inspection device 10. The conveying mechanism 950 includes a plurality of rollers including rollers 951 and 952, a support plate 953, and a conveying motor (not shown). At least a portion of the plurality of rollers is driven by the conveying motor to convey the cloth 700 to be inspected. The support plate 953 is a flat plate disposed between the upstream roller 951 and the downstream roller 952, and supports a portion of the cloth 700 to be inspected that is to be photographed by the digital cameras 111-114. In FIG. 2, the rotary encoder 120 and the operation unit 980 and control unit 990 of the conveying device 900 are not shown.

[0021] The partial path Pth in the figure indicates the portion of the conveyance path of the cloth 700 between the rollers 951 and 952, i.e., the portion along the support plate 953. The direction Df in the figure indicates the conveyance direction on the partial path Pth (direction Df is also referred to as the conveyance direction Df). The orthogonal direction Dt indicates a direction parallel to the flat portion 700F and perpendicular to the partial path Pth. Hereinafter, the upstream side of the conveyance direction Df of the conveyance path of the cloth 700 will be simply referred to as the upstream side, and the downstream side of the conveyance direction Df of the conveyance path of the cloth 700 will be simply referred to as the downstream side.

[0022] The cloth 700 to be inspected is a long piece of cloth that is longer than the partial path Pth. The cloth 700 is wound around a core 31 to form an upstream roll 710. The upstream roll 710 is disposed in a mounting section (not shown) upstream of the partial path Pth. The upstream mounting section includes, for example, a holding member including two or more rollers, and is configured to rotatably hold the upstream roll 710. The cloth 700 pulled out from the upstream roll 710 is transported from the position of the upstream roller 951 to the position of the downstream roller 952 along the partial path Pth.

[0023] The fabric 700 conveyed to the downstream roller 952 is wound around the core 32 downstream of the partial path Pth. The fabric 700 wound around the core 32 forms the downstream roll 720. The downstream roll 720 is disposed in a mounting section (not shown) downstream of the partial path Pth. The downstream mounting section includes, for example, a holding member including two or more rollers, and is configured to rotatably hold the downstream roll 720.

[0024] Between the rollers 951 and 952, the cloth 700 is supported by the support plate 953 to form a flat portion, that is, a flat portion 700F. The light source 130 is disposed at a position where it irradiates the flat portion 700F with light.

[0025] 2, the imaging range Ar, which is the area captured by the digital cameras 111-114, is hatched. The imaging range Ar is a rectangular area having two sides Ar1 and Ar2 parallel to the partial path Pth and two sides Ar3 and Ar4 perpendicular to the partial path Pth. The first side Ar1 and the second side Ar2 of the imaging range Ar are located outside the cloth 700.

[0026] In the figure, partial areas R11-R14 indicate areas photographed by digital cameras 111-114, respectively. In this embodiment, the digital cameras 111-114 are arranged side by side in the orthogonal direction Dt, and therefore the partial areas R11-R14 are also arranged side by side in the orthogonal direction Dt. The photographing range Ar is the entire partial areas R11-R14.

[0027] The operation unit 980 (FIG. 1) of the conveying device 900 includes a plurality of switches (not shown) and is configured to receive operations from a user (specifically, an operator who inspects the cloth 700). In this embodiment, the plurality of switches are foot switches operated by the operator with their feet, and include at least a conveyance start switch and a stop switch. Alternatively, the plurality of switches may be push switches operated by the operator with their hands, or buttons displayed on a touch panel.

[0028] The control unit 990 (FIG. 1) of the conveying device 900 is connected to, for example, the operation unit 980 and a conveying motor and power supply (not shown). The control unit 990 is an electric circuit configured to control the conveying motor in response to the operation of the operation unit 980 or a signal from the data processing device 200. The control unit 990 may be configured using a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). For example, when a conveyance start switch is pressed while the cloth 700 is not being conveyed, the control unit 990 starts conveying the cloth 700. When a stop switch is pressed while the cloth 700 is being conveyed, the conveying device 900 stops conveying the cloth 700. When the control unit 990 receives a conveyance stop signal from the data processing device 200 requesting that conveyance be stopped while the cloth 700 is being conveyed, the control unit 990 stops conveying the cloth 700.

[0029] A rotary encoder 120 is connected to the conveying device 900 to detect the direction and amount of position change due to conveyance. For example, the rotary encoder 120 is connected to a roller (e.g., the upstream roller 951). The rotary encoder 120 may have various configurations for detecting the direction and amount of position change due to conveyance. For example, the rotary encoder 120 may be an incremental encoder. An incremental encoder alternately outputs A pulses and B pulses in response to position changes. The number of output pulses indicates the amount of movement. The phase difference (positive or negative) between the A pulse and the B pulse indicates the direction of movement. The data processing device 200 (FIG. 1) can calculate the current conveyance position of the cloth 700 conveyed by the conveying device 900 (i.e., the relative position of the cloth 700 with respect to the conveying device 900) by counting the number of pulses output from the rotary encoder 120 according to the phase difference (i.e., the direction). Note that a counter that counts the number of pulses according to the direction may be connected to the rotary encoder 120. The data processing device 200 may use the information from the counter to obtain the current relative position of the fabric 700 .

[0030] A2.Inspection process The inspection process for the cloth 700 is performed, for example, after the cloth manufacturing process or before manufacturing products such as clothing using the cloth. In the inspection process for the cloth 700, an operator mounts the roll of cloth 700 to be inspected as the upstream roll 710 on an upstream mounting portion (not shown) of the conveying mechanism 950. The operator pulls out the downstream end of the cloth 700 from the upstream roll 710. Specifically, the downstream end of the cloth 700 passes through a partial path Pth ( FIG. 2 ) that passes through the imaging range Ar on the support plate 953, is pulled out until it reaches a position downstream of the support plate 953, and is wound around the interlining 32. The operator mounts the interlining 32 around which the downstream end of the cloth 700 is wound on an downstream mounting portion (not shown) of the conveying mechanism 950. As a result, the cloth 700 to be inspected is set in the inspection device 10 so that it can be inspected, as shown in FIG. 2. At the start of the inspection, the portion of the cloth 700 to be inspected that is located downstream of the shooting range Ar in the conveying direction Df cannot be inspected by the inspection process (described later) by the data processing device 200, so it is inspected visually by an operator, for example.

[0031] When the operator sets the cloth 700 to be inspected on the inspection device 10, he or she operates the operation unit 250 of the data processing device 200 to start the computer program PG and inputs an instruction to start the inspection process. This causes the data processing device 200 to start the inspection process. The inspection process is a process for inspecting the cloth 700 using captured images of the cloth 700 in synchronization with the conveyance of the cloth 700 by the conveying device 900. Details of the inspection process will be described later.

[0032] When the data processing device 200 starts the inspection process, the operator presses the conveyance start switch on the operation unit 980 of the conveyance device 900. In response to the pressing of the conveyance start switch, the control unit 990 of the conveyance device 900 drives the conveyance motor to start conveying the cloth 700 in the conveyance direction Df.

[0033] When the conveyance of the cloth 700 begins, the cloth 700 is inspected by an inspection process for each portion that passes through the imaging range Ar. FIG. 3 is a flowchart of the inspection process of the first embodiment. In S100, the processor 210 of the data processing device 200 determines whether the imaging timing has arrived. For example, the first imaging timing is the start of the inspection process. The nth (n is an integer greater than or equal to 2) imaging timing is the timing at which the inspected cloth has been conveyed from the (n-1)th imaging timing by a distance AH (FIG. 2) of the imaging range Ar in the conveying direction Df. Therefore, in FIG. 2, when a portion of the cloth 700 located within the imaging range Ar is imaged at the nth imaging timing, the portion of the cloth 700 imaged at the (n-1)th imaging timing is a portion located within the adjacent range Arb adjacent to the imaging range Ar on the downstream side in the conveying direction Df in FIG. 2. This makes it possible to sequentially generate multiple inspection images IMs (described below) showing portions of the conveyed cloth 700 that are positioned at different positions in the conveying direction Df. In a modified example, in order to inspect the cloth 700 more reliably and without omission, the timing of the nth photograph may be set to the timing when the cloth to be inspected has been transported a length slightly shorter than length AH (Figure 2) from the timing of the (n-1)th photograph.

[0034] The processor 210 can recognize the amount of transport of the cloth 700 based on a count value, which is the number of pulses obtained from the rotary encoder 120. For this reason, the processor 210 determines that the timing for photographing has arrived based on the count value each time the cloth 700 is transported by the length AH of the photographing range Ar in the transport direction Df.

[0035] If the timing to take a photograph has not arrived (S100: NO), the processor 210 proceeds to S150. If the timing to take a photograph has arrived (S100: YES), the processor 210 acquires a photographed image using the digital cameras 111-114 in S105. Specifically, the processor 210 supplies a photographing instruction to each of the digital cameras 111-114. In response to the photographing instruction, the digital cameras 111-114 photograph the cloth 700 and generate photographed images IM1-IM4. The processor 210 acquires data of the photographed images IM1-IM4 from each of the digital cameras 111-114.

[0036] FIG. 4 is a diagram showing an example of an image used in the inspection process. FIG. 4(A) shows examples of captured images IM1-IM4 obtained from the digital cameras 111-114 (FIG. 1), respectively. Each of the captured images IM1-IM4 is a rectangular image having two sides parallel to the first direction Dx and two sides parallel to a second direction Dy perpendicular to the first direction Dx. The second direction Dy is a direction roughly parallel to the conveying direction Df (FIG. 1). The data of each of the captured images IM1-IM4 is bitmap data representing the color values ​​of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. The color values ​​are, for example, RGB values ​​including the respective gradation values ​​of red R, green G, and blue B (for example, values ​​greater than or equal to zero and less than or equal to 255).

[0037] As shown in FIG. 2, partial regions R11-R14 corresponding to captured images IM1-IM4 (FIG. 4(A)) are arranged side by side in the orthogonal direction Dt. The first captured image IM1 and the fourth captured image IM4 represent the cloth 700 and the background BG, respectively. The second captured image IM2 and the third captured image IM3 represent the cloth 700, respectively. These captured images IM1-IM4 as a whole represent the entire capturing range Ar. Although not shown, the background BG may represent various objects located outside the cloth 700, such as a part of the conveying device 900.

[0038] In the example of FIG. 4(A), the captured image IM3 includes a defect FD1 (e.g., a linear scratch) on the cloth 700. The defect FD1 can be formed due to various causes. For example, a defect in the thread forming the cloth 700 can form the linear defect. Furthermore, a linear defect (e.g., a scratch or a linear drawing) can be formed due to contact between the cloth 700 and another member (e.g., a device for carrying the cloth 700, a writing implement, etc.).

[0039] In S110, the processor 210 combines the four captured images IM1-IM4 to generate data for a single inspection image IMs. Fig. 4(B) shows the inspection image IMs generated from the captured images IM1-IM4 of Fig. 4(A). The inspection image IMs is a strip-shaped image that represents the portion of the cloth 700 within the imaging range Ar (Fig. 2).

[0040] Various methods may be used to generate the inspection image IMs. For example, the partial regions R11-R14 (FIG. 2) may be arranged side by side in the orthogonal direction Dt within the imaging range Ar without any gaps and without overlapping each other. In this case, the processor 210 may generate data for the inspection image IMs by connecting the ends of the captured images IM1-IM4 (FIG. 4(A)) arranged side by side in the first direction Dx. Alternatively, the partial regions R11-R14 may be arranged so that two adjacent partial regions partially overlap each other. In this case, two adjacent images among the captured images IM1-IM4 include overlapping portions. In this case, the processor 210 combines the captured images IM1-IM4 by using the corresponding portion of one of the two images as the image of the overlapping portion of the two images.

[0041] In S115, the processor 210 divides the inspection image IMs into images of a size (also referred to as the input size) that can be input to the object detection model MD. As a result, k (k is an integer of 2 or greater) partial images PI are generated from one inspection image IMs. To distinguish each of the k partial images PI, an identification number i (i is an integer of 1 or greater and k or less) is assigned to the partial image PI. For example, the first partial image PI is expressed as partial image PI1, and the i-th partial image PI is expressed as partial image PIi.

[0042] FIG. 4C shows an example of k partial images PI (PI1-PIk). The arrangement of the partial images PI1-PIk on the inspection image IMs is predetermined. FIG. 4B shows the range of the partial images PI1-PIk on the inspection image IMs. In this embodiment, the partial images PI1-PIk are aligned in the first direction Dx on the inspection image IMs. Among the partial images PI1-PIk, two adjacent partial images partially overlap. For example, as shown in FIGS. 4B and 4C, the right edge of partial image PI1 and the left edge of partial image PI2 represent the same area IA1 (also referred to as the overlap area IA1) on the inspection image IMs. All portions of the inspection image IMs are included in at least one of the k partial images PI. As shown in FIG. 4C, partial image PIi includes defect FD1 included in the inspection image IMs.

[0043] In S120, the processor 210 inputs each of the k partial images PI to the object detection model MD to generate output data indicating a plurality of detection results.

[0044] The object detection model MD is a well-known machine learning model including a convolutional neural network (CNN). In this example, the object detection model MD is a model called "RTMDet," which is disclosed in the following paper: Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, and Kai Chen. "Rtmdet: An Empirical Study of Designing Real-Time Object Detectors," arXiv.2212.07784, December 16, 2022, https: / / doi.org / 10.48550 / arXiv.2212.07784.

[0045] The RTMDet is a model that detects the bounding box and category (i.e., object type) of an object and performs pixel-level region division called instance segmentation. In this embodiment, the object detection model MD is pre-trained to output multiple detection results for multiple types of defects that may exist in the cloth 700, specifically, linear scratches, holes, chalk marks, etc., as detection targets. The object detection model MD may be trained using various methods, such as the training method described in the above-mentioned paper by the RTMDet.

[0046] One detection result includes box information indicating a bounding box (rectangular region) surrounding the defect, a reliability (also called a certainty factor) for each defect type, and mask data indicating the pixel-level region where the defect is located. The bounding box and mask data are examples of region information indicating the region where the defect is located.

[0047] The bounding box includes information indicating, for example, the coordinates of the center of the rectangular area, the length in the first direction Dx (height of the rectangle), and the length in the second direction Dy (width of the rectangle). The reliability is, for example, a numerical value greater than 0 and less than 1. The higher the reliability, the more likely the detection result is to be correct. The mask data is, for example, binary image data indicating one or more defective pixels that constitute a defect among the multiple pixels included in the bounding box (rectangular area). The mask is an area indicated by one or more defective pixels.

[0048] By inputting one partial image PI into the object detection model MD, multiple (for example, several tens of) detection results are output.

[0049] In S125, the processor 210 determines, from among the multiple detection results for each partial image PI, a detection result having a reliability equal to or greater than a normal detection threshold THn as a valid detection result. The normal detection threshold THn is recorded in a threshold table TT (FIG. 1).

[0050] Fig. 5 is a diagram showing an example of the threshold value table TT. The threshold value table TT records two types of threshold values, namely, a threshold value THn for normal detection and an initial value THc0 of a threshold value THc for candidate detection, which will be described later, for each type of defect (in the example of Fig. 5, linear scratches, holes, and chalk). The threshold value THn for normal detection is a value experimentally determined in advance so that defects in fabrics can be detected.

[0051] The processor 210 determines a detection result as valid when at least one of the reliability values ​​for each defect type included in the detection result is greater than the threshold value THn for normal detection recorded in the threshold table TT. Note that, typically, the reliability value for one defect type is significantly greater than the other reliability values, so that only one of the reliability values ​​for each defect type is greater than the threshold value THn recorded in the threshold table TT. In this embodiment, it is not assumed that multiple reliability values ​​for each defect type are greater than the threshold value THn recorded in the threshold table TT. In a valid detection result, the defect type corresponding to a reliability value greater than the threshold value THn indicates the type of defect detected. Therefore, it can be said that a valid detection result includes type information indicating the type of defect detected.

[0052] For example, if the valid detection result obtained by inputting the partial image PI into the object detection model MD is zero, it means that no defect was detected in the partial image PI. If the valid detection result obtained by inputting the partial image PI into the object detection model MD is one or more, it means that a defect was detected in the partial image PI. As can be seen from the above explanation, the normal detection threshold THn can be said to be a threshold used to determine the presence or absence of a defect.

[0053] In S127, the processor 210 uses the valid detection results to generate a detection result image DI. First, the processor 210 uses the valid detection results to generate mask images MI1-MIk that indicate the defect detection results.

[0054] FIG. 4(D) shows examples of mask images MI1-MIk that indicate valid detection results for partial images PI1-PIk in FIG. 4(C). Mask images MI1-MIk are mask images corresponding to partial images PI1-PIk. Mask image MIi corresponding to partial image PIi that includes defect FD1 includes a mask MS1 (a region consisting of one or more defective pixels) that indicates the position and shape of defect FD1. Mask images MI1, MI2, MIi-1, MIi+1, MIk corresponding to other partial images PI1, PI2, PIi-1, PIi+1, PIk that do not include defects do not include a mask.

[0055] In Fig. 4(D), the dashed rectangle indicates a bounding box. In Fig. 4(D), a bounding box BB1 that surrounds the mask MS1 (defect FD1) is associated with the mask MS1 of the mask image MIi.

[0056] The processor 210 combines the generated mask images MI1-MIk to generate a detection result image DI. FIG. 4(E) shows the detection result image DI corresponding to the inspection image IMs in FIG. 4(B). The detection result image DI is, for example, an image of the same size as the inspection image IMs. The detection result image DI describes the mask and bounding box of a valid detection result. In the example of FIG. 4(E), the detection result image DI includes a mask MS1 and a bounding box BB1 associated with the mask MS1.

[0057] In S130, the processor 210 updates the result display screen W11 using the generated detection result image DI. The result display screen W11 is a screen showing the detection results, and is displayed on the display unit 240. FIG. 6 is a diagram showing an example of the result display screen W11. The result display screen W11 of FIG. 6(A) includes a message MG1 notifying the defect detection status, an inspection image IMs, and a detection result image DI superimposed on the inspection image IMs. The result display screen W11 also includes text TX indicating the type of the detected defect, the reliability of the detection, and the size of the defect (for example, the length of a linear scratch), and an addition instruction button BTa.

[0058] The result display screen W11 in FIG. 6A shows an example in which a defect has been detected, and the message MG1 indicates that a defect has been detected. The result display screen W11 includes the three most recent inspection images IMs (IMs1-IMs3) generated in steps S105 and S110, which are executed multiple times. The result display screen W11 also includes, on each of the three inspection images IMs1-IMs3, a detection result image DI (DI1-DI3) generated in steps S115-S127 using each inspection image. As shown in FIG. 2, the flat portion 700F of the cloth 700 positioned on the support plate 953 includes areas Arb, Arc, and Ard equivalent to three sheets of the imaging range Ar downstream of the imaging range Ar. Therefore, the three inspection images IMs1-IMs3 displayed on the result display screen W11 represent the portion of the cloth 700 that has been inspected at that time and is located on the support plate 953. Therefore, the worker can visually compare the result display screen W11 with the portion of the cloth 700 located on the support plate 953, thereby monitoring for any missed defects.

[0059] The defect type information included in the text TX is information based on the type information included in the valid detection result determined in S125. The detection reliability included in the text TX is the reliability included in the valid detection result. The size of the defect included in the text TX is calculated based on, for example, the size (number of pixels in the vertical or horizontal direction) of the mask MS1 or bounding box BB1 included in the detection result image DI.

[0060] In S135, the processor 210 determines whether a defect is detected in the inspection image IMs. If there is one or more valid detection results, i.e., a detection result having a reliability equal to or greater than the threshold THn, the processor 210 determines that a defect is detected. If there is no valid detection result, the processor 210 determines that a defect is not detected.

[0061] If no defect is detected (S135: NO), the processor 210 proceeds to S150. If a defect is detected (S135: YES), the processor 210 records information about the detected defect (i.e., information about the valid detection result) in the defect database DD in S140.

[0062] FIG. 7 is a diagram showing an example of the defect database DD. The defect database DD records information about each detected defect. In the defect database DD of FIG. 7, one record RC (one line of information) is information about one detected defect, i.e., one valid detection result. One record RC includes an identification number for identifying the defect and the various types of information included in the defect detection result, namely, type information, reliability, box information, and mask data. In the box information and mask data, the coordinates of the bounding box or mask in the second direction Dy are converted into coordinates in the conveyance direction Df of the cloth 700 based on the detection value of the rotary encoder 120. This makes it possible to identify the position of the detected defect on the cloth 700.

[0063] In S145, the processor 210 transmits a conveyance stop signal requesting the stop of conveyance to the conveyance device 900. Upon receiving the conveyance stop signal, the control unit 990 of the conveyance device 900 stops conveyance of the fabric 700. In this way, when a defect is detected (YES in S135), conveyance of the fabric 700 is stopped.

[0064] If a defect is detected in the inspection process, the conveyance device 900 stops conveying the cloth 700, and the portion of the cloth 700 containing the detected defect remains stationary on the support plate 953. For example, after the worker looks at the result display screen W11 and determines the location of the defect, the worker confirms the defect on the actual cloth 700 and adds a marker to the cloth 700 for later identification. The worker then presses the conveyance start switch on the operation unit 980 of the conveyance device 900. In response to the pressing of the conveyance start switch, the control unit 990 of the conveyance device 900 drives the conveyance motor to resume conveying the cloth 700 in the conveyance direction Df. The inspection process in FIG. 3 is performed synchronously with the conveyance of the cloth 700 (S100 in FIG. 3), and therefore, when the conveyance of the cloth 700 is resumed, the inspection process is also resumed.

[0065] In S150, the processor 210 determines whether an additional recording instruction has been received. The additional recording instruction is an instruction to additionally record defect information of the cloth 700 in the defect database DD based on a user operation. For example, an operator monitors for missed detection of defects by visually comparing the result display screen W11 with the portion of the cloth 700 located on the support plate 953. For example, the detection result of a specific portion photographed within the photographing range Ar in FIG. 2 is displayed on the result display screen W11 while the specific portion is located in the upstream half of the support plate 953. For this reason, for example, if there is a defect that is not displayed on the result display screen W11 even after the cloth 700 is transported to the downstream half of the support plate 953, it is considered that the defect was not detected due to a missed detection. When the operator visually finds a missed detection defect, he or she presses the stop switch on the operation unit 980 to stop the transport of the cloth 700 by the transport device 900. For example, when the worker checks a defect in the cloth 700 on the support plate 953 and determines that information about the defect should be additionally recorded, the worker presses the additional instruction button BTa on the result display screen W11 to input an additional recording instruction to the data processing device 200. As a result, the processor 210 of the data processing device 200 acquires the additional recording instruction.

[0066] If the processor 210 acquires an additional recording instruction (S150: YES), the processor 210 executes additional recording processing in S155. The additional recording processing is processing for adding information about defects in the cloth 700 to the defect database DD based on an operation by the worker. The additional recording processing will be described later. When the additional recording processing is completed, the processor 210 returns the processing to S100. If the processor 210 does not acquire an additional recording instruction (S150: NO), the processor 210 returns the processing to S100.

[0067] As can be seen from the above description, the processor 210 repeatedly acquires multiple inspection images IMs showing different portions of the cloth 700 (S110-S115), detects defects in the cloth 700 shown in each inspection image IMs (S120-S125), and displays the detection results (S127, S130). When the processor 210 acquires an additional recording instruction (S150: YES), it interrupts the repeated process and executes additional recording processing.

[0068] A3. Additional recording process Next, the additional recording process of S155 in FIG. 3 will be described. FIG. 8 is an explanatory diagram of the additional recording process. FIG. 8A shows a flowchart of the additional recording process of the first embodiment. In S210, the processor 210 acquires a designation of an area where the defect to be recorded is located. Specifically, upon acquiring the additional recording instruction, the processor 210 displays the area designation screen W12 of FIG. 6B on the display unit 240, instead of the result display screen W11 of FIG. 6A. Like the result display screen W11, the area designation screen W12 includes the three most recent inspection images IMs (IMs1-IMs3) and the detection result images DI (DI1-DI3) superimposed on each of these three inspection images IMs1-IMs3. The area designation screen W12 further includes a message MG2 prompting the user to designate an area where the defect to be recorded is located, a rectangular frame FL for designating the area, a confirm button BTd, and a back button BTr. In the example of Fig. 6(B), inspection image IMs3 displayed on region designation screen W12 includes defect FD2, which is the target of recording, near the right edge. In the inspection process S120-S125 using inspection image IMs3, defect FD2 and other defects are not detected. In other words, although defect FD2 appears in inspection image IMs3, it is not detected due to a detection omission in the inspection process S120-S125 described above.

[0069] The operator uses a pointing device such as a mouse to adjust the position and size of a rectangular frame FL on the area designation screen W12 to designate an area of ​​the inspection image IMs that includes the defect FD2. The operator presses the confirm button BTd with the frame FL designating the area that includes the defect FD2. At this point, the processor 210 recognizes the area surrounded by the frame FL as the designated area SA. Note that when the back button BTr is pressed, the additional recording process is interrupted, but this interruption is not shown in the flowchart of FIG. 8(A).

[0070] In S215, the processor 210 generates a partial image PIs including a specified area SA. For example, the processor 210 generates the partial image PIs by cutting out an image that includes the specified area SA at the center and has the input size of the object detection model MD from the inspection image IMs. For example, in the example of FIG. 6(B), since the specified area SA is included in the inspection image IMs3, the processor 210 cuts out the partial image PIs from the inspection image IMs3 so that the center of gravity of the specified area SA is located at the center in the first direction Dx. Note that since defects that cause detection omissions are often small, the specified area SA is often smaller than the input size of the object detection model MD as in the example of FIG. 6(B). When the specified area SA is larger than the input size of the object detection model MD, for example, the partial image PIs may be generated by reducing the image including the specified area SA to the input size. Alternatively, two or more partial images PIs may be generated by dividing the image including the specified area SA.

[0071] In S220, the processor 210 inputs the partial image PIs generated in S215 into the object detection model MD to generate a plurality of detection results.

[0072] In S225, the processor 210 determines, as valid detection results, the detection results including a confidence level equal to or higher than a threshold THc for candidate detection from the plurality of detection results of the partial image PIs. The threshold THc for candidate detection is smaller than the above-described normal detection threshold THn used in S125 of FIG. 3 (THn < THc). Therefore, it can be said that the threshold THc for candidate detection is a threshold set so that the probability of being determined as valid, in other words, the probability of determining that there is a defect, is higher compared to the normal detection threshold THn.

[0073] As shown in Fig. 5, the threshold value table TT stores an initial value THc0 of the threshold value THc for candidate detection for each defect type (in the example of Fig. 5, a linear scratch, a hole, or a chalk), similar to the threshold value THn for normal detection. As will be described later, the process of S225 can be performed multiple times (NO in S240, which will be described later). In the first S225, the initial value THc0 stored in the threshold value table TT is used as the threshold value THc for candidate detection. The threshold value THc used in the process of S225 from the second time onwards will be described later. When at least one reliability of each defect type included in one detection result is greater than the threshold value THc for candidate detection, the processor 210 determines that the one detection result is valid.

[0074] 9A and 9B are diagrams conceptually illustrating differences in detection results using multiple thresholds. FIG. 9A shows mask images representing masks of defects detected in the partial image PIs generated in S215 (i.e., masks included in valid detection results). Each mask image shows the above-described designated area SA designated by the operator for reference. The mask image MIh in FIG. 9A is an example in which the normal detection threshold THn is used. The mask image MIh in FIG. 9A does not include a mask. In other words, when the normal detection threshold THn is used, no defect candidates are detected.

[0075] The mask image MIm in FIG. 9(B) is an example of a case where an initial value THc0 of the candidate detection threshold THc, which is smaller than the normal detection threshold THn, is used. The mask image MIm in FIG. 9(B) includes two masks CM1 and CM2. That is, when the initial value THc0 of the candidate detection threshold THc is used, two defect candidates are detected. Neither of the two defect candidate masks CM1 and CM2 indicates the defect FD2 that the operator wants to add. The two defect candidate masks CM1 and CM2 indicate, for example, something that is not a defect (hereinafter also referred to as a non-defective object, such as a normal weave or shading in fabric). That is, a detection result including the masks CM1 and CM2 is a so-called false positive, in which a non-defective object is mistakenly detected as a defect candidate. The normal detection threshold THn is set to detect only defects and prevent false positives. However, when a candidate detection threshold THc smaller than the normal detection threshold THn is used, false positives are likely to be included in the effectively detected defects. The initial value THc0 of the threshold value THc for candidate detection recorded in the threshold value table TT of Figure 5 is a value experimentally determined in advance so that, for example, an appropriate number of defect candidates are detected that are not detected when the threshold value THn for normal detection is used, but are detected when the threshold value THc for candidate detection is used.

[0076] In S230, the processor 210 displays the valid detection results as defect candidates. Specifically, the processor 210 displays a candidate display screen W21 on the display unit 240. The candidate display screen W21 is displayed, for example, as a pop-up screen superimposed on the area designation screen W12.

[0077] 10A and 10B are diagrams showing examples of screens displayed during the additional recording process. The candidate display screen W21 in FIG. 10A includes a message MG3 prompting the user to select a defect to be recorded, a candidate display area CA, a list LTa, a confirm button BTd, and a back button BTr. The candidate display area CA is an area having the same shape as the specified area SA and includes masks CM1 and CM2 of defect candidates detected within the specified area SA. The list LTa is a list showing the identification numbers of selectable defect candidates. In addition to the identification numbers, the list LTa includes options for inputting an instruction to display additional defect candidates (also called a candidate increase instruction) as items.

[0078] The operator can recognize the position and shape of the defect candidate by looking at the masks CM1 and CM2 displayed in the candidate display area CA. The operator can select an item in the list by operating a cursor CS in the list LTa using a pointing device or the like. If the candidate display area CA contains a mask of the defect to be recorded, the operator selects the identification number of the mask of the defect from the list LTa and presses the confirm button BTd. This allows the operator to input an instruction to select the defect to be recorded (also called a defect selection instruction) to the data processing device 200. If the candidate display area CA does not contain a mask of the defect to be recorded, the operator selects an option for inputting an instruction to increase the number of candidates from the list LTa and presses the confirm button BTd. This allows the operator to input an instruction to increase the number of candidates to the data processing device 200. Note that pressing the back button BTr returns to a state in which the specified area SA can be re-specified (a state in which only the area specification screen W12 is displayed), but this is not illustrated in the flowchart of FIG. 8(A).

[0079] In S235, processor 210 acquires a defect selection instruction or a candidate increase instruction input by the operator in response to the operator's operation. When candidate display screen W21 of FIG. 10(A) is displayed, the mask of defect FD2 that the operator wants to add is not in candidate display area CA, so the operator inputs a candidate increase instruction. In S240, it is determined whether the instruction acquired in S235 is a defect selection instruction or a candidate increase instruction.

[0080] If a candidate increase instruction is received (S240: NO), in S245, the processor 210 changes the candidate detection threshold THc to a smaller value than the current value. For example, the threshold table TT of FIG. 5 records an increment value ΔTH for each defect type. The processor 210 calculates a new threshold THc for each defect type by subtracting the increment value from the current threshold THc. For example, if the current threshold is the initial value THc0, the new threshold THc is set to (THc0-ΔTH). Since the threshold (THc0-ΔTH) is even smaller than the initial value THc0, it can be said that the threshold (THc0-ΔTH) is set to a value that increases the probability of determining that a defect exists compared to the initial value THc0. After S245, the processor 210 returns the process to S225.

[0081] In the second iteration of S225, a valid detection result is determined from the multiple detection results for the partial image PIs using the new threshold value THc (THc0-ΔTH). The mask image MI1 in FIG. 9C is an example in which (THc0-ΔTH) is used as the threshold value THc for candidate detection. The mask image MI1 in FIG. 9C includes four masks CM1-CM4. That is, in this case, four defect candidates are detected. The masks CM1 and CM2 for the two defect candidates are the same as the masks CM1 and CM2 in the mask image MIm in FIG. 9B. The masks CM3 and CM4 for the two defect candidates are not determined to be valid when the initial value THc0 is used as the threshold value THc, but are included in the detection results determined to be valid when a smaller threshold value (THc0-ΔTH) is used as the threshold value THc. The mask CM3 is a mask indicating the defect FD2 that the operator wants to add. The mask CM4 is a mask indicating a false detection that indicates a non-defective object.

[0082] In the second S230, for example, processor 210 may display a larger number of valid detection results as defect candidates than in the first S230. For example, in the examples of Figures 9 and 10, in the first S230, the candidate display screen W21 in Figure 10(A) described above, which includes two defect candidates, is displayed, but in the second S230, the candidate display screen W22 in Figure 10(B) which includes three defect candidates is displayed.

[0083] 10(B), the candidate display area CA includes masks CM1-CM3 of defect candidates detected within the specified area SA in the second S225. The list LTa is a list showing the identification numbers of selectable defect candidates. The list LTa includes, as items, the identification numbers of the defect candidates (masks CM1-CM3) and options for inputting an instruction to increase the number of candidates.

[0084] 10(B) is displayed, the mask CM3 of the defect FD2 that the operator wants to add is in the candidate display area CA. To do this, the operator selects the mask CM3 of the defect FD2 that the operator wants to add in the list LTa with the cursor CS and then presses the confirm button BTd. This causes the processor 210 to acquire a defect selection instruction to select the defect FD2 that the operator wants to add.

[0085] If a defect selection instruction is acquired (S240: YES), the processor 210 acquires type information of the selected defect from the operator in S250. Specifically, the processor 210 displays a type input screen W23 of FIG. 10(C) on the display unit 240. The type input screen W23 is displayed as a pop-up screen, for example, similar to the candidate display screen W21.

[0086] The type input screen W23 of FIG. 10(C) includes a message MG4 prompting the user to select a defect type, a list LTb showing a plurality of selectable defect types, a decision button BTd, and a back button BTr.

[0087] The operator operates the cursor CS to select the type of defect FD2 that has been visually confirmed and is to be recorded from the list LTb, and then presses the decision button BTd. This allows the operator to input the type of defect to be recorded into the data processing device 200. Note that when the back button BTr is pressed, the state returns to a state where the specified area SA can be re-specified (a state where only the area specification screen W12 is displayed), but illustration of this is omitted in the flowchart of FIG. 8(A).

[0088] In S260, the processor 210 records information about the defect selected via the candidate display screen W21 in the defect database DD (FIG. 5). FIG. 8B shows a record AD recorded in the defect database DD in S260. For example, as shown in FIG. 8B, the processor 210 generates an identification number ("3" in FIG. 8B) of the defect FD2 to be recorded, and records type information "DT_3," reliability "CV_3," box information "BI_3," and mask data "MD_3" in association with the identification number. The type information "DT_3" is information indicating the type of defect FD2 acquired by the user via the type input screen W23 in S250. The reliability "CV_3" is a value indicating the maximum reliability ("1" in this embodiment). The box information "BI_3" is information indicating the position and size of a bounding box included in the detection result (i.e., the detection result including the mask CM3) of the defect FD2 selected via the candidate display screen W22. The mask data "MD_2" is data of the mask CM3 included in the detection result of the defect FD2 selected via the candidate display screen W22. When the defect information is added to the defect database DD, the processor 210 ends the additional recording process.

[0089] According to the present embodiment described above, the processor 210 acquires an inspection image IMs showing the fabric 700 (S105 and S110 in FIG. 3). The processor 210 acquires an additional recording instruction from the operator to record a defect FD2 shown in the inspection image IMs (S150 in FIG. 3). When the additional recording instruction is acquired, the processor 210 displays defect candidates to be recorded (S230 in FIG. 8A). The defect candidates are determined by a determination process (S215-S225 in FIG. 8A) that is executed on a partial image PIs, which is a part of the inspection image IMs, using the object detection model MD. The processor 210 acquires a defect selection instruction from the operator to select a defect to be recorded from the defect candidates (S235 in FIG. 8A), and records information about the selected defect to be recorded (S260 in FIG. 8). The recorded information includes mask data and box information included in the detection result generated using the object detection model MD (FIG. 8B). It can be said that the mask data (e.g., data on the mask CM3) and box information are information indicating the position and shape of the defect FD2 to be recorded. As a result, based on the process of determining defect candidates using the object detection model MD and the selection instruction of the operator, information regarding the position and shape of the defect FD2 in the fabric 700 shown in the inspection image IMs can be recorded in the defect database DD.

[0090] This will be explained in more detail. Even when defects are automatically detected using a machine learning model such as the object detection model MD, defects may be missed from detection. When a detection miss occurs, it is preferable to record information about the defects missed from automatic detection. This is because recording information about the defects missed from automatic detection allows for proper inspection of the cloth 700 to be completed. Furthermore, information about the defects missed from automatic detection can be used not only as an inspection result for the cloth 700, but also to further improve the detection accuracy of the object detection model MD. For example, it can be used to generate image data or training data for training the object detection model MD.

[0091] For this reason, when an operator visually finds a defect that was missed by automatic detection, it is desirable to accurately record information about the position and shape of the defect. For example, if an operator were to manually record the position and shape of the defect, the burden on the operator would be heavy. Furthermore, efficiently and accurately manually recording the position and shape of the defect may require the operator to have advanced skills. According to this embodiment, candidates for defects that were missed are determined and displayed using the object detection model MD, so the operator only needs to select the defect to be recorded from the candidates. Therefore, the burden on the operator of recording information about the position and shape of the defects that were missed can be reduced, and the information can be recorded efficiently and accurately.

[0092] Furthermore, according to this embodiment, the processor 210 acquires an additional recording instruction (S150 in FIG. 3) after acquiring the inspection image IMs (S105, S110 in FIG. 3). As a result, it is possible to determine candidates for defects to be recorded more quickly than when, for example, candidates for defects to be recorded are determined using an image obtained by photographing the fabric 700 after a defect that was not detected is found.

[0093] Furthermore, according to this embodiment, before an additional recording instruction is received (i.e., before S150 in FIG. 3 ), the processor 210 executes a defect detection process using the object detection model MD on the inspection image IMs, using the normal detection threshold THn, to detect a defect (e.g., defect FD1) in the fabric 700 shown in the inspection image IMs (S120-S125 in FIG. 3 ). Then, the process of determining defect candidates to be recorded in the additional recording process is a process of determining defect candidates by executing the defect detection process using the candidate detection threshold THc (S220-S225 in FIG. 8A ). As described above, the candidate detection threshold THc is set so that the probability of determining that a defect exists is higher than with the normal detection threshold THn. As a result, information on defects that cannot be detected using the normal detection threshold THn, i.e., defects that may be missed when detected, can be recorded based on the detection process using the candidate detection threshold THc and the operator's selection instruction.

[0094] Furthermore, according to this embodiment, the object detection model MD is a model that outputs multiple detection results, each including information on a bounding box or mask indicating the area of ​​a detected defect and information indicating the reliability of the defect detection. The normal detection threshold THn and the candidate detection threshold THc are thresholds for the reliability of defect detection. As a result, defect candidates that may include detection results with low reliability that could result in false detection are displayed, and the operator is prompted to select the defect to be recorded. This makes it possible to record information on defects that cannot be detected when the normal detection threshold THn is used in the defect database DD.

[0095] Furthermore, according to this embodiment, after an additional recording instruction is received (i.e., after S150 in FIG. 3), processor 210 executes processing using object detection model MD to cause object detection model MD to generate output data (detection results) (S220 in FIG. 8A), and determines defect candidates using the detection results output by object detection model MD and threshold value THc for candidate detection (S225 in FIG. 8A). As a result, there is no need to store the detection results generated in the detection processing using threshold value THn for normal detection (S120, S125 in FIG. 3), and the required memory amount can be reduced.

[0096] Furthermore, according to this embodiment, the object detection model MD for the detection process using the normal detection threshold THn (S120, S125 in FIG. 3) and the object detection model MD for the detection process using the candidate detection threshold THc (S220, S225 in FIG. 8A) are the same model. As a result, by using a single object detection model MD and simply changing the threshold, it is possible to both automatically detect defects and determine defect candidates that can be selected by an operator.

[0097] Furthermore, according to this embodiment, the processor 210 receives an instruction to designate a designated area SA, which is a partial area of ​​the inspection image IMs (S210 in FIG. 8A). The processor 210 displays defect candidates located in an area that includes the designated area SA but does not include the entire inspection image IMs (FIGS. 10A and 10B). As a result, only defect candidates located within a limited area are displayed, allowing the operator to easily select defects to be recorded. Furthermore, the processor 210 can determine defect candidates using the partial image PIs that includes the designated area SA, thereby reducing the processing load of the defect candidate determination process (S215-S225 in FIG. 8A).

[0098] Furthermore, according to this embodiment, the defect information recorded in the defect database DD in the additional recording process includes area information (box information and mask data in this embodiment) indicating the area of ​​the detected defect and type information indicating the type of the detected defect ( FIG. 8(B)). As a result, it is possible to record in the defect database DD defect information that can be used to prepare training data (image data or teacher data) used to train an object detection model MD that outputs the defect type and defect area, for example.

[0099] Furthermore, according to this embodiment, of the defect information recorded in the defect database DD in the additional recording process, area information indicating the defect area (box information and mask data in this embodiment) is information generated using the object detection model MD, and type information indicating the defect type is information input by the worker (S250 in Figure 8(A) and Figure 10(C)). It places a heavy burden on the worker to input area information accurately. The defect type information is information that the object detection model MD is likely to make errors with, but is information that the worker can easily input. Furthermore, the worker can accurately determine the defect type by visual inspection. According to this embodiment, accurate defect information can be recorded in the defect database DD while reducing the burden on the worker.

[0100] Furthermore, according to this embodiment, the defect information recorded in the defect database DD in the additional recording process is information indicating the highest reliability of detection. The defect information recorded in the additional recording process is information about defects that have been visually confirmed by an operator, and therefore can be considered to be correct information (information with extremely high reliability). Furthermore, when the defect information recorded in the additional recording process is used as training data for training the object detection model MD, it is preferable that the defect information recorded in the additional recording process be information with the highest reliability. In this way, according to the additional recording process of this embodiment, defect information with appropriate reliability can be recorded in the defect database DD.

[0101] Furthermore, according to this embodiment, the threshold value THn for normal detection and the threshold value THc for candidate detection are values ​​that are set for each type of defect (FIG. 5). Since the reliability output by the object detection model MD differs depending on the type of defect, the threshold values ​​THn and THc to be applied to the reliability are also set for each type of defect, making it possible to appropriately determine defects and defect candidates.

[0102] Furthermore, according to this embodiment, the candidate detection threshold value THc used to determine defect candidates includes an initial value THc0 and a value (THc0-ΔHT) obtained by subtracting the step value ΔTH from the initial value THc0 (FIGS. 5 and S245 in FIG. 8A). The processor 210 displays the first defect candidate (masks CM1 and CM2 in FIG. 10A) determined using the initial value THc0 (the first S230 in FIG. 8A). Thereafter, when an instruction to increase the number of candidates is received from the operator (NO in S240 in FIG. 8A), the processor 210 displays the second defect candidate (mask CM3 in FIG. 10B) determined using the threshold value (THc0-ΔHT) (the second S230 in FIG. 8A). As a result, the number of displayed defect candidates can be gradually increased while decreasing the candidate detection threshold value THc in response to the operator's instruction. Therefore, it is possible to avoid the candidate display screens W21 and W22 becoming cluttered due to the display of an excessively large number of defect candidates, while reducing the inconvenience of defect candidates not being displayed.

[0103] As can be seen from the above explanation, the threshold value THn for normal detection is an example of the first threshold value, the initial value THc0 of the threshold value for candidate detection is an example of the second threshold value, and the threshold value (THc0-ΔTH) for candidate detection is an example of the third threshold value.

[0104] B. Second Example FIG. 11 is a flowchart of the inspection process of the second embodiment. FIG. 12 is a flowchart of the additional recording process of the second embodiment. In the flowcharts of FIGS. 11 and 12, steps that are the same as those in the flowcharts of FIGS. 3 and 8(A) are given the same reference numerals as those in FIGS. 3 and 8(A), and steps that differ from those in the flowcharts of FIGS. 3 and 8(A) are given a "B" at the end of the reference numeral. The following describes the differences between the inspection process of the second embodiment and the inspection process of the first embodiment.

[0105] In the inspection process of the second embodiment, the process of S122B is added between S120 and S125. In S122B, the processor 210 stores a plurality of detection results for each partial image PI generated in S120 in the non-volatile storage device 230. In S120, for example, several tens of detection results are generated for each of the k partial images PI, and all of these detection results are stored in the non-volatile storage device 230.

[0106] In the inspection process of the second embodiment, the additional recording process of Fig. 12 is executed as the additional recording process of S155B in Fig. 11 instead of the additional recording process of Fig. 8(A). In the additional recording process of Fig. 12, the process of S220B in Fig. 12 is executed instead of S215 and S220 in Fig. 8(A).

[0107] In S220B, the processor 210 acquires a plurality of detection results of the partial image PI including the designated area SA designated by the operator in S210 from the non-volatile storage device 230. The plurality of detection results acquired here are a portion of the plurality of detection results of each of the k partial images PI saved in S122B of FIG.

[0108] In S225, a valid detection result is determined from the multiple detection results acquired in S220B using a threshold value THc for candidate detection. Other processing in the inspection process of the second embodiment is the same as the inspection process of the first embodiment.

[0109] According to the second embodiment described above, the detection results (output data) output by the object detection model MD in the detection process using the normal detection threshold THc (S120, S125 in FIG. 11) are stored in the non-volatile storage device 230 (S122B in FIG. 11). The processor 210 determines defect candidates using the stored detection results and the candidate detection threshold THc (225 in FIG. 12). According to this configuration, defect candidates can be determined using the detection results generated in the detection process using the normal detection threshold THc. As a result, since the calculation process of the object detection model MD is not executed in the additional recording process, the amount of processing required to determine defect candidates can be reduced.

[0110] C. Variations (1) In the above embodiment, the additional recording process of FIG. 8(A) is executed in S155 of FIG. 3 to record information about defects that were not detected in the detection process using the normal detection threshold THn (S115-S127 of FIG. 3). Alternatively, for example, the detection process using the normal detection threshold THn does not need to be executed. In this case, the defect information is recorded in the defect database DD simply by executing the recording process of FIG. 8(A) in response to a recording instruction from an operator. For example, such a process may be executed not for inspecting the cloth 700, but for visually searching for defects in the cloth 700 and collecting defect information for training the object detection model MD.

[0111] In this case, for example, after an operator visually discovers a defect that should be recorded and inputs a recording instruction into the data processing device 200, the cloth 700 is photographed and an inspection image IMs is generated, and the recording process of Figure 8(A) may be performed using the inspection image IMs.

[0112] (2) In each of the above embodiments, the process of detecting defects contained in each partial image PI using each partial image PI is a process using an object detection model MD. The defect detection method is not limited to this, and other methods may be used. For example, defect detection may be performed based on an anomaly detection mechanism using a machine learning model called PaDiM (a Patch Distribution Modeling Framework for Anomaly Detection and Localization). Furthermore, defect detection may be performed based on a method using an autoencoder, which is a machine learning model including an encoder and a decoder. In this method, for example, the partial image PI is input to the autoencoder, and an output image corresponding to the partial image PI is generated. Then, the partial image PI is compared with the output image to detect defects contained in the partial image PI.

[0113] When these detection methods are used, for example, an abnormality degree is calculated for each pixel constituting the partial image PI, and pixels with an abnormality degree equal to or greater than a threshold value are detected as abnormal pixels. For example, the processor 210 labels a plurality of detected abnormal pixels to identify a pixel group consisting of one or more consecutive abnormal pixels as a single mask. In this case, for example, when automatically detecting defects, the processor 210 uses a normal detection threshold as a threshold for the abnormality degree used to determine whether a pixel is abnormal. When detecting defect candidates in the additional recording process of FIG. 8A, the processor 210 uses a candidate detection threshold that is smaller than the normal detection threshold as a threshold for the abnormality degree used to determine whether a pixel is abnormal. As a result, similar to the above embodiment, defects that cannot be detected using the normal detection threshold can be detected as candidates for defects to be recorded by using the candidate detection threshold.

[0114] (3) In the first embodiment, the object detection model MD used in S115 of Fig. 3 and the object detection model MD used in S220 of Fig. 8(A) are the same machine learning model. Alternatively, the object detection model used in S115 of Fig. 3 and the object detection model MD used in S220 of Fig. 8(A) may be different machine learning models. For example, the object detection model used in S220 may be a model that has been trained to have a higher probability of falsely detecting structures other than defects, such as shading in fabric, as defects, but a lower probability of missing a defect, compared to the object detection model used in S115. The same applies to the second embodiment.

[0115] (4) The defect information recorded in the defect database DD shown in FIG. 7 and FIG. 8(B) is merely an example and is not limited thereto. For example, the recorded defect information does not have to include defect type information or certainty. The recorded defect information may include only one of box information and mask data. Furthermore, the recorded defect information may include other information, such as a captured image of the defective portion.

[0116] (5) In the additional recording process of Fig. 8(A) in the first embodiment, if the defect to be recorded is not included in the defect candidates displayed on the candidate display screen W21, the processor 210 may generate mask data and box information based on an operator's operation and record the mask data and box information in the defect database DD without using the object detection model MD. The same applies to the second embodiment.

[0117] (6) In each of the above embodiments, the processor 210 divides the inspection image IMs obtained by previously combining the captured images IM1-IM4 to generate multiple partial images PI having the input size of the object detection model MD. Alternatively, the size of the captured images IM1-IM4 may be matched to the input size of the object detection model MD in advance. In this case, the processor 210 may input each of the captured images IM1-IM4 to the object detection model MD and detect defects and defect candidates in each captured image.

[0118] (7) The object detection model MD used in each of the above embodiments is a machine learning model called RTMDet that realizes instance segmentation, but may be another object detection model. For example, the object detection model MD may be a machine learning model that realizes other instance segmentation, such as Mask R-CNN, or a machine learning model that realizes semantic segmentation, such as YOLO (You only look once).

[0119] Furthermore, the object detection model MD in the above embodiment is trained to determine multiple types of defects. Alternatively, the object detection model MD may be trained to detect only one type of defect. In other words, the object detection model MD may be trained to determine only the presence or absence of a defect, without determining the type of defect.

[0120] (8) In the above embodiment, the object to be inspected is cloth 700, but it may be another object. The other object may be, for example, a sheet-like object, such as a metal foil, a resin film, or paper. The object to be inspected may also be an object having a shape other than a sheet, such as various products or parts thereof, such as automobiles or electrical appliances. Depending on the object to be inspected, the defects to be detected may take various forms.

[0121] (9) The total number of digital cameras used to photograph the cloth 700 is not limited to four and may be any number greater than or equal to one. Furthermore, the device used to photograph the cloth 700 may include a line sensor as an image sensor instead of an area sensor such as a digital camera. In either case, the processor 210 uses the target images acquired using these devices to detect defects in the cloth 700. For example, if there is one digital camera, the captured image generated by the single digital camera may be used as the inspection image IMs as is.

[0122] (10) In the above embodiment, the processor 210 may cause the GPU 260 to execute various calculations. For example, the processor 210 may cause the GPU 260 to execute some or all of the calculations based on the object detection model MD. Note that the GPU 260 may be omitted.

[0123] (11) Data processing device 200 in Fig. 1 may be a device of a type different from a personal computer (e.g., a digital camera, a scanner, or a smartphone). Furthermore, multiple devices (e.g., computers) that can communicate with each other via a network may share some of the data processing functions of the data processing device and collectively provide the data processing functions (a system including these devices corresponds to a data processing device).

[0124] (12) In each of the above embodiments, a part of the hardware configuration may be replaced with software, and conversely, a part or all of the software configuration may be replaced with hardware. For example, the processing by the object detection model MD (FIG. 2) may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0125] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used in a state stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0126] The above-described examples and modifications can be combined as appropriate. The above-described examples and modifications are provided to facilitate understanding of the present disclosure and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof. [Explanation of symbols]

[0127] 1000... inspection system, 10... inspection device, 111-114... digital camera, 120... rotary encoder, 130... light source, 200... data processing device, 210... processor, 215... storage device, 220... volatile storage device, 230... non-volatile storage device, 240... display unit, 250... operation unit, 270... communication interface, 31, 32... core material, 700... cloth, 700F... flat portion, 710... upstream roll, 720... downstream roll, 900... conveying device, 950... Conveyance mechanism, 951...upstream roller, 952...downstream roller, 953...support plate, 980...operation unit, 990...control unit, DD...defect database, DI...detection result image, IM1-IM4...captured image, IMs...inspection image, MD...object detection model, PG...computer program, PI...partial image, MI...mask image, PI...partial image, TT...threshold table, W11...result display screen, W12...area designation screen, W21, W22...candidate display screen, W23...type input screen

Claims

1. A computer program comprising: an image capture function for capturing an image of an object, the image of the object being generated using an image sensor; a recording instruction acquisition function that acquires, from a user, a recording instruction to record a defect of the object shown in the target image; a candidate display function that displays candidates for defects to be recorded when the recording instruction is acquired, the candidate defects being determined by a determination process that is executed on the target image using a machine learning model; and a selection instruction acquisition function for acquiring, from a user, a selection instruction for selecting a defect to be recorded from among the defect candidates; a recording function that records specific information of the defect to be recorded selected by the selection instruction, the specific information being information generated using the machine learning model and including information regarding at least one of the position and shape of the defect to be recorded; A computer program that enables a computer to realize the above.

2. 2. The computer program of claim 1, The recording instruction acquisition function acquires the recording instruction after the acquisition function acquires the target image.

3. 3. The computer program of claim 2, further comprising: a defect detection function that, before the recording instruction is acquired, executes a defect detection process, including a process that is executed using the machine learning model, on the target image using a first threshold value to detect a defect in the object shown in the target image, wherein the first threshold value is a threshold value used to determine the presence or absence of the defect, and the determination process is a process of determining the defect candidate by performing the defect detection process using a second threshold value; The second threshold is a threshold used to determine whether or not there is a defect, and is set so as to increase the probability that a defect will be determined to exist, compared to the first threshold.

4. 4. The computer program of claim 3, further comprising: the machine learning model is an object detection model that outputs a plurality of detection results, each of which includes information indicating the area of ​​a detected defect and a reliability of the defect detection; The first threshold and the second threshold are thresholds for the reliability.

5. 4. The computer program of claim 3, further comprising: A computer program that causes a computer to realize a first candidate determination function that determines the defect candidate using output data output by the machine learning model and the second threshold in the defect detection process that uses the first threshold.

6. 4. The computer program of claim 3, further comprising: A computer program that, after the recording instruction is obtained, causes a computer to perform a process using the machine learning model, causing the machine learning model to generate output data, and causes a computer to realize a second candidate determination function that determines a defect candidate using the output data output by the machine learning model and the second threshold value.

7. 4. A computer program according to claim 3, comprising: a machine learning model for the defect detection process using the first threshold and a machine learning model for the defect detection process using the second threshold, the machine learning model being the same model;

8. 2. The computer program of claim 1, the recording instruction acquisition function acquires an instruction to designate a target area that is a part of the target image; The candidate display function displays candidates for the defect located in an area that includes the specified target area but does not include the entire target image.

9. 2. The computer program of claim 1, A computer program, wherein the recorded specific information includes at least one of area information indicating an area of ​​the detected defect and type information indicating a type of the detected defect.

10. 10. A computer program according to claim 9, comprising: A computer program in which, among the recorded specific information, the area information is information generated using the machine learning model, and the type information is information input by a user.

11. 10. A computer program according to claim 9, comprising: A computer program, wherein the recorded specific information further includes information indicating the reliability of detection, the information indicating the maximum reliability.

12. 5. A computer program according to claim 4, the object detection model further outputs a plurality of detection results, each of which includes type information indicating a type of defect detected; The second threshold value is a value set for each type of defect.

13. 4. A computer program according to claim 3, comprising: the determination process further includes a process of determining the defect candidate by performing the defect detection process using a third threshold value; the third threshold is a threshold used to determine whether or not there is a defect, and is set so as to increase the probability of determining that there is a defect compared to the second threshold; The candidate display function is displaying the first candidate defect determined using the second threshold; a computer program that, after displaying the first candidate, displays a second candidate for the defect determined using the third threshold value when a display instruction is obtained from a user;

14. 1. A data processing device, comprising: an image acquisition unit that acquires a target image showing an object, the target image being an image generated using an image sensor; a recording instruction acquisition unit that acquires, from a user, a recording instruction to record a defect of the object shown in the target image; a candidate display unit that displays candidates for defects to be recorded when the recording instruction is acquired, the candidate defects being determined by a determination process that is executed on the target image using a machine learning model; and a selection instruction acquisition unit that acquires a selection instruction from a user to select a defect to be recorded from the defect candidates; a recording unit that records specific information of the defect to be recorded selected by the selection instruction, the specific information being information generated using the machine learning model and including information regarding at least one of the position and shape of the defect to be recorded; and A data processing device comprising:

Images