Computer program, image processing device, and image processing method

Abstract

Efficiently acquire training images for training machine learning models to detect defects in objects. [Solution] The computer program implements the following functions: an image acquisition function that acquires an image showing a defect in the object to be processed, wherein the image is generated by optically reading the object having the defect to be processed; and an acquisition function that acquires a training image from the image by cutting out a partial image from the image that includes a part of the defect to be processed but does not include the other part of the defect to be processed, wherein the training image is used to train a machine learning model for detecting defects in the object.

Description

Technical Field

[0001] This specification relates to a technique for obtaining training images for training a machine learning model for detecting defects in an object.

Background Art

[0002] Patent Document 1 discloses a technique for performing discrimination such as defects included in a discrimination target based on an image of the discrimination target. In this technique, when conditions such as the discrimination accuracy by a classification learner being below a threshold value are satisfied, a teacher data collection process is executed. In the teacher data collection process, a snapshot including an image of the discrimination target and a list of labels are displayed on a display device, and selection of a label by an operator is received. Then, new teacher data composed of a combination of the image of the discrimination target and the selected label is added to a storage unit. Relearning of the classification learner is performed using the newly added teacher data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above technique, defects necessary as teacher data are not always detected as discrimination targets, so there is a possibility that teacher data including images of necessary defects cannot be obtained.

[0005] This specification discloses a technique capable of efficiently obtaining training images for training a machine learning model for detecting defects in an object.

Means for Solving the Problems

[0006] The technologies disclosed herein have been made to solve at least some of the problems described above and can be realized in the following examples of applications.

[0007] [Application Example 1] A computer program that enables a computer to implement an image acquisition function that acquires an image showing a defect in a target object, wherein the image is generated by optically reading an object having the defect; and an acquisition function that acquires a training image from the target image by cutting out a partial image from the target image that includes a part of the defect in the target object but does not include the other part of the defect, wherein the training image is used to train a machine learning model for detecting defects in the object.

[0008] According to the above configuration, a training image is obtained using a target image that shows the defect to be processed, and which includes a portion of the defect but excludes other parts. As a result, training images containing the defect can be obtained efficiently.

[0009] According to the above configuration, multiple defective pixels constituting the defects to be processed in the target image are identified based on the user's input, so that a training image can be appropriately extracted from the target image.

[0010] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, in the form of an image processing method, an image processing apparatus, a computer program for realizing the functions of such methods or apparatuses, a recording medium (e.g., a non-temporary recording medium) on which such computer programs are recorded, and so on. [Brief explanation of the drawing]

[0011] [Figure 1] A diagram showing the configuration of inspection system 1000. [Figure 2] A perspective view showing the schematic configuration of the inspection device 10. [Figure 3]Flowchart of the inspection process in the first embodiment. [Figure 4] A diagram showing an example of an image used in the inspection process. [Figure 5] A diagram showing an example of the results display screen W11. [Figure 6] Flowchart of the training image acquisition process in the first embodiment. [Figure 7] A diagram showing an example of a training image acquisition table TT. [Figure 8] Diagram illustrating the extraction location of the training image TI. [Figure 9] Flowchart of the training image acquisition process in the second embodiment. [Figure 10] Diagram illustrating the cutting pattern. [Modes for carrying out the invention]

[0012] A. First Example A1. Configuration of the inspection system Figure 1 shows the configuration of the inspection system 1000. The inspection system 1000 is a system for inspecting cloth 700. The cloth to be inspected is a known fabric such as woven fabric, knitted fabric, or nonwoven fabric. The inspection system 1000 in Figure 1 comprises a data processing device 200 and an inspection device 10. The data processing device 200 and the inspection device 10 are connected in a communicative manner. This allows the data processing device 200 to receive image data generated by the digital cameras 111-114 (described later) and control signals from the control unit 990 of the transport device 900.

[0013] The data processing device 200 is, for example, a personal computer. The data processing device 200 comprises a processor 210, a storage device 215, a display unit 240, an operation unit 250, a Graphics Processing Unit (GPU) 260, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0014] The processor 210 is a device configured to perform data processing, for example, a Central Processing Unit (CPU) or a System on a chip (SoC). The volatile memory device 220 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile memory device 230 is, for example, a flash memory.

[0015] The non-volatile memory device 230 stores a computer program PG and a database DD. The computer program PG includes an object detection model MD as a program module. The object detection model MD will be described later. The processor 210 executes the inspection process described later by executing the computer program PG. The database DD is used, for example, to record data such as information on detected defects and generated training images, as will be described later.

[0016] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by the user. The operation unit of this embodiment includes a touch panel disposed over the display unit 240. The operation unit 250 may further include a keyboard, buttons, a lever, and the like. The user can input various requests and instructions to the data processing device 200 by operating the operation unit 250. The display unit 240 may display operation elements (for example, buttons, sliders, etc.), and the displayed elements may be operated through the operation of the operation unit 250.

[0017] The GPU 260 is an arithmetic device configured to execute various numerical operations such as image processing and machine learning. The GPU (Graphics Processing Unit) 260 executes various operations according to the instructions of the processor 210. In this embodiment, the GPU 260 executes operations for the object detection model MD according to the instructions of the processor 210.

[0018] The communication interface 270 is an interface for communicating with the inspection device 10 (for example, including one or more of a USB interface, a wired LAN interface, a wireless interface of IEEE 802.11, an interface of an industrial camera (for example, CameraLink, CoaXPress, etc.)). In this embodiment, the communication interface 270 is connected to the inspection device 10.

[0019] The inspection device 10 is a device that cooperates with the data processing device 200 to perform an inspection of the cloth, which is the object of inspection in this embodiment. The inspection device 10 includes digital cameras 111 - 114, a rotary encoder 120, a light source 130, and a transport device 900.

[0020] The digital cameras 111 - 114 are devices that generate and acquire a read image indicating the object by reading the object (subject) using an image sensor including an imaging element such as a CCD or a CMOS. The digital cameras 111 - 114 are used for reading the cloth to be inspected. The rotary encoder 120 is used for calculating the relative position of the cloth with respect to the transport device 900 (details will be described later). The light source 130 irradiates the cloth with light in order to clearly read the cloth to be inspected.

[0021] The transport device 900 is a device that transports the cloth to be inspected, and includes a transport mechanism 950, an operation unit 980 that receives an operation by the user, and a control unit 990.

[0022] FIG. 2 is a perspective view showing a schematic configuration of the inspection device 10. The transport mechanism 950 includes a plurality of rollers including rollers 951 and 952, a support plate 953, and a transport motor (not shown). By driving at least a part of the plurality of rollers by the transport motor, the cloth 700 to be inspected is transported. The support plate 953 is disposed between the upstream roller 951 and the downstream roller 952, and is a flat plate that supports the portion of the cloth 700 to be read by the digital cameras 111 - 114. In FIG. 2, the illustration of the rotary encoder 120, the operation unit 980, and the control unit 990 of the transport device 900 is omitted.

[0023] The partial path Pth in the figure represents the portion of the fabric 700's transport path between rollers 951 and 952, i.e., the portion along the support plate 953. The direction Df in the figure represents the transport direction on the partial path Pth (direction Df is also referred to as the transport direction Df). The orthogonal direction Dt represents the direction parallel to the flat portion 700F and perpendicular to the partial path Pth. Hereafter, the upstream side of the transport direction Df of the fabric 700's transport path will simply be referred to as the upstream side, and the downstream side of the transport direction Df of the fabric 700's transport path will simply be referred to as the downstream side.

[0024] The cloth 700 to be inspected is a long piece of cloth that is longer than the partial path Pth. The cloth 700 is wound around a core material 31 to form an upstream roll 710. The upstream roll 710 is positioned at a mounting section (not shown) upstream of the partial path Pth. The upstream mounting section includes, for example, a holding member including two or more rollers, and is configured to rotatably hold the upstream roll 710. The cloth 700 drawn from the upstream roll 710 is transported from the position of the upstream roller 951 along the partial path Pth to the position of the downstream roller 952.

[0025] The fabric 700, conveyed to the downstream roller 952, is wound onto the core material 32 downstream of the partial path Pth. The fabric 700 wound onto the core material 32 forms the downstream roll 720. The downstream roll 720 is positioned in a mounting section (not shown) downstream of the partial path Pth. The downstream mounting section includes, for example, a holding member including two or more rollers, and is configured to rotatably hold the downstream roll 720.

[0026] Between rollers 951 and 952, the cloth 700 is supported by the support plate 953 to form a flat portion 700F. The light source 130 is positioned to illuminate the flat portion 700F.

[0027] In Figure 2, the reading area Ar, which is the region read by the digital cameras 111-114, is hatched. The reading area Ar is a rectangular region having two sides Ar1 and Ar2 parallel to the partial path Pth, and two sides Ar3 and Ar4 perpendicular to the partial path Pth. The first side Ar1 and the second side Ar2 of the reading area Ar are located outside the cloth 700.

[0028] The subregions R11-R14 in the figure represent the areas read by the digital cameras 111-114, respectively. In this embodiment, the digital cameras 111-114 are arranged in the orthogonal direction Dt, so the subregions R11-R14 are also arranged in the orthogonal direction Dt. The reading range Ar is the entire subregion R11-R14.

[0029] The operating unit 980 (Figure 1) of the conveying device 900 is equipped with a plurality of switches (not shown) and is configured to receive operations from an operator. The plurality of switches are, for example, foot switches operated by the operator with their feet, and include a conveying start switch and a conveying stop switch. Alternatively, the plurality of switches may be push switches operated by the operator with their hands, or buttons displayed on a touch panel.

[0030] The control unit 990 (Figure 1) of the transport device 900 is connected, for example, to the operation unit 980 and a transport motor and power supply (not shown). The control unit 990 is an electrical circuit configured to control the transport motor in response to operations on the operation unit 980 and signals from the data processing device 200. The control unit 990 may be configured using a computer or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC)). For example, if the transport start switch is pressed when the cloth 700 is not being transported, the control unit 990 starts transporting the cloth 700. If the transport start signal is received from the data processing device 200 requesting the start of transport while the cloth 700 is not being transported, the transport unit 990 starts transporting the cloth 700. If the stop switch is pressed while the cloth 700 is being transported, the transport device 900 stops transporting the cloth 700. If the control unit 990 receives a transport stop signal from the data processing device 200 requesting the stop of transport while the cloth 700 is being transported, it stops transporting the cloth 700.

[0031] A rotary encoder 120 is connected to the conveying device 900 to detect the direction and amount of position change due to conveying. For example, the rotary encoder 120 is connected to a roller (e.g., an upstream roller 951). The configuration of the rotary encoder 120 can be various configurations for detecting the direction and amount of position change due to conveying. For example, the rotary encoder 120 may be an incremental encoder. An incremental encoder outputs A pulses and B pulses alternately according to the change in position. The number of pulses output indicates the amount of movement. The phase difference (positive or negative) between the A pulse and the B pulse indicates the direction of movement. The data processing device 200 (Figure 1) can calculate the current conveying position of the cloth 700 being conveyed by the conveying device 900 (i.e., the relative position of the cloth 700 with respect to the conveying device 900) by counting the number of pulses output from the rotary encoder 120 according to the phase difference (i.e., direction). A counter that counts the number of pulses according to the direction may be connected to the rotary encoder 120. The data processing device 200 may use information from the counter to obtain the current relative position of the cloth 700.

[0032] A2.Inspection process The inspection process for the fabric 700 is performed, for example, after the fabric manufacturing process or before manufacturing products such as clothing using the fabric. In the inspection process for the fabric 700, the operator attaches the roll of fabric 700 to be inspected as the upstream roll 710 to the upstream mounting section (not shown) of the conveying mechanism 950. The operator pulls out the downstream end of the fabric 700 from the upstream roll 710. Specifically, the downstream end of the fabric 700 is pulled out through a partial path Pth (Figure 2) that passes through the reading range Ar on the support plate 953, until it reaches a point downstream of the support plate 953, and is wound around the core material 32. The operator attaches the core material 32, to which the downstream end of the fabric 700 is wound, to the downstream mounting section (not shown) of the conveying mechanism 950. As a result, as shown in Figure 2, the fabric 700 to be inspected is set up for inspection by the inspection device 10. At the start of the inspection, the portion of the fabric 700 to be inspected that is located downstream of the reading range Ar in the transport direction Df cannot be inspected by the inspection process performed by the data processing device 200 (described later). Therefore, for example, an operator may visually inspect this portion.

[0033] When the operator places the cloth 700 to be inspected into the inspection device 10, they operate the control panel 250 of the data processing device 200 to start the computer program PG and input an instruction to start the inspection process. This causes the data processing device 200 to begin the inspection process. The inspection process involves inspecting the cloth 700 using a read image of the cloth, synchronized with the transport of the cloth 700 by the transport device 900.

[0034] When the data processing device 200 starts the inspection process, the operator presses the transport start switch on the operation unit 980 of the transport device 900. In response to the press of the transport start switch, the control unit 990 of the transport device 900 drives the transport motor to start transporting the cloth 700 in the transport direction Df.

[0035] When the transport of the cloth 700 begins, the cloth 700 is inspected by an inspection process for each portion that has passed through the reading range Ar. Figure 3 is a flowchart of the inspection process in the first embodiment. In S100, the processor 210 of the data processing device 200 determines whether or not a reading timing has arrived. For example, the first reading timing is at the start of the inspection process. The nth reading timing (where n is an integer of 2 or more) is the timing when the cloth to be inspected has been transported by a length AH (Figure 2) in the transport direction Df of the reading range Ar from the (n-1)th reading timing. For this reason, in Figure 2, if the portion of the cloth 700 located in the reading range Ar is read at the nth reading timing, the portion of the cloth 700 read at the (n-1)th reading timing is the portion located in the adjacent range Arb, which is adjacent to the downstream side of the transport direction Df with respect to the reading range Ar in Figure 2. This makes it possible to sequentially generate multiple inspection images IMs (described later) that show different positions in the transport direction Df of the transported cloth 700. In the modified example, in order to more reliably inspect the cloth 700 without omission, the nth reading timing may be set to the timing when the cloth to be inspected has been transported by a length slightly shorter than length AH (Figure 2) from the (n-1)th reading timing.

[0036] The processor 210 can recognize the amount of cloth 700 being transported based on a count value, which is the number of pulses obtained from the rotary encoder 120. To this end, the processor 210 determines, based on the count value, that a reading timing has arrived each time the cloth 700 has been transported by a length AH in the transport direction Df of the reading range Ar.

[0037] If the reading timing has not yet arrived (S100: NO), the processor 210 waits until the reading timing arrives. If the reading timing has arrived (S100: YES), the processor 210 acquires the read image using the digital cameras 111-114 in S105. Specifically, the processor 210 supplies a reading instruction to each of the digital cameras 111-114. The digital cameras 111-114 read the cloth 700 in response to the reading instruction and generate read images IM1-IM4. The processor 210 acquires the data of the read images IM1-IM4 from each of the digital cameras 111-114.

[0038] Figure 4 shows an example of an image used in the inspection process. Figure 4(A) shows examples of read images IM1-IM4 obtained from digital cameras 111-114 (Figure 1), respectively. Read images IM1-IM4 are rectangular images having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy, which is perpendicular to the first direction Dx. The second direction Dy is approximately parallel to the transport direction Df (Figure 1). The data for each of the read images IM1-IM4 is bitmap data representing the color values ​​of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. The color values ​​are RGB values ​​that include, for example, the gradation values ​​of red R, green G, and blue B (for example, values ​​between zero and 255).

[0039] As shown in Figure 2, the subregions R11-R14 corresponding to the read images IM1-IM4 (Figure 4(A)) are arranged in a line in the orthogonal direction Dt. The first read image IM1 and the fourth read image IM4 represent the cloth 700 and the background BG, respectively. The second read image IM2 and the third read image IM3 also represent the cloth 700, respectively. These read images IM1-IM4 together represent the entire reading range Ar. Although not shown in the illustration, the background BG may represent various objects located outside the cloth 700, such as a part of the conveying device 900.

[0040] In the example in Figure 4(A), the read image IM3 contains a defect FD in the cloth 700. Defect FD can be formed by various causes (e.g., contact between the cloth 700 and other materials). Defect FD includes, for example, linear defects and defects of non-linear shapes. Linear defects may include, for example, scratches and chalk marks. Chalk marks refer to lines made by a worker using chalk. Defects of non-linear shapes may include holes, stains (dirt), and tears. The defect FD in Figure 4(A) is a linear scratch.

[0041] In S110, the processor 210 combines the four read images IM1-IM4 to generate data for a single inspection image IMs. Figure 4(B) shows the inspection image IMs generated from the read images IM1-IM4 in Figure 4(A). The inspection image IMs is a strip-shaped image representing the portion of the cloth 700 within the reading range Ar (Figure 2).

[0042] There are various methods for generating the inspection images IMs. For example, the subregions R11-R14 (Figure 2) may be arranged in a line in the orthogonal direction Dt without gaps and without overlapping within the reading range Ar. In this case, the processor 210 may generate the inspection image IMs data by connecting the respective ends of the read images IM1-IM4 (Figure 4(A)) arranged in the first direction Dx. Alternatively, the subregions R11-R14 may be arranged such that two adjacent subregions partially overlap. In this case, two adjacent images of the read images IM1-IM4 include overlapping portions. In this case, the processor 210 combines the read images IM1-IM4 by using the corresponding portion of one of the two images as the image of the overlapping portion.

[0043] In S115, the processor 210 divides the inspection images IMs into images of a size (also called the input size) that can be input to the object detection model MD. This generates m divided images PI (where m is an integer greater than or equal to 2) from a single inspection image IMs. In Figure 4(B), the m divided images PI are shown as dashed lines. In the example in Figure 4(B), the m divided images PI are arranged in a matrix on the inspection image IMs. Note that two adjacent divided images among the m divided images PI may partially overlap. Every part of the inspection image IMs is contained in at least one of the m divided images PI. As shown in Figure 4(B), the defect FD contained in the inspection image IMs spans multiple divided images PI1-PI3.

[0044] The size of the segmented image PI (number of pixels in the vertical and horizontal directions) is the size of the input image for the object detection model MD, which in this embodiment is 480 pixels vertically × 680 pixels horizontally. The size of the training image TI, described later, is also the size of the segmented image PI.

[0045] In S120, the processor 210 inputs each of the m segmented images PI into the object detection model MD to generate multiple detection results.

[0046] The object detection model MD is a well-known machine learning model, including CNNs (Convolutional Neural Networks). In this example, the object detection model MD is called "RTMDet," which is disclosed in the following paper: Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang and Kai Chen. "Rtmdet: An Empirical Study of Designing Real-Time Object Detectors", arXiv.2212.07784, December 16, 2022, https: / / doi.org / 10.48550 / arXiv.2212.07784.

[0047] RTMDet is a model that detects the bounding box and category (i.e., the type of object) of an object, and performs pixel-level region segmentation called instance segmentation. In this embodiment, the object detection model MD is pre-trained to output multiple detection results for multiple types of defects that may be present in the cloth 700, specifically linear scratches, holes, chalk marks, etc. The training method for the object detection model MD can be various, for example, the training method described in the above-mentioned paper on RTMDet is employed. For example, the training image TI described later is used in training the object detection model MD.

[0048] A single detection result includes box information indicating the bounding box (rectangular area) surrounding the defect, confidence level (also called confidence score) for each type of defect, and mask data indicating the pixel-level area where the defect is located.

[0049] The bounding box includes information such as the coordinates of the center of the rectangular area, the length in the first direction Dx (height of the rectangle), and the length in the second direction Dy (width of the rectangle). The confidence level is, for example, a number greater than 0 and less than 1. A higher confidence level indicates a higher probability that the detection result is correct. The mask data is, for example, binary image data showing one or more defective pixels that constitute the defect among multiple pixels included in the bounding box (rectangular area). The mask is the area indicated by one or more defective pixels.

[0050] By inputting a single segmented image PI into the object detection model MD, multiple detection results (e.g., dozens of results) are output. Among the multiple detection results for each segmented image PI, those with a confidence level equal to or greater than a predetermined threshold THn are considered valid detection results.

[0051] Figure 4(C) shows the detection results for three segmented images PI1-PI3 containing the defective FD shown in Figure 4(B). The detection result for segmented image PI1 includes a mask MS1 and bounding box BB1 indicating the left edge of the defective FD. The detection result for segmented image PI2 includes a mask MS2 and bounding box BB2 indicating the central part of the defective FD. The detection result for segmented image PI3 includes a mask MS3 and bounding box BB3 indicating the right edge of the defective FD.

[0052] In S125, a detection result image DI is generated using the detection results of multiple segmented images PI. For example, the detection result image DI is a binary image of the same size as the inspection images IMs, and is an image on which masks MS indicating the detected defects are placed. Figure 4(D) shows an example of the generated detection result image DI. For example, if multiple masks MS indicating defects detected in multiple segmented images PI are consecutive, the consecutive masks MS are concatenated into a single mask. Similarly, multiple bounding boxes BB surrounding the consecutive masks MS are also concatenated into a single bounding box. In the example in Figure 4(D), masks MS1-MS3 and bounding boxes BB1-BB3 in Figure 4(C) are concatenated to form a single mask MS and a single bounding box BB. In the detection result image DI, it can be seen that the linear defect FD of the inspection images IMs in Figure 4(B) has been detected.

[0053] In S130, the processor 210 updates the result display screen W11 using the generated detection result image DI. The result display screen W11 is a screen that shows the detection results and is displayed on the display unit 240. Figure 5 is a diagram showing an example of the result display screen W11. The result display screen W11 in Figure 5 includes messages MG1 and MG2, inspection images IMs, and a detection result image DI superimposed on the inspection images IMs. Message MG1 is a message that notifies the status of defect detection. Message MG2 is a message that notifies tracing the defect. The result display screen W11 also includes text TX indicating the type of defect detected, the confidence level of the detection, and the size of the defect (e.g., the length of a linear scratch), and an OK button BTa.

[0054] The result display screen W11 in Figure 5 shows an example where a defect is detected, so message MG1 indicates that a defect has been detected. The defect type information contained in text TX is based on the type information contained in the detection result generated in S120. The detection confidence level contained in text TX is the confidence level contained in the detection result.

[0055] In S135, the processor 210 determines whether or not a defect has been detected in the inspection images IMs. If there is one or more valid detection results, i.e., detection results with a confidence level of THn or higher, the processor 210 determines that a defect has been detected. If there are no valid detection results, the processor 210 determines that no defect has been detected.

[0056] If no defect is detected (S135: NO), the processor 210 returns to S100. If a defect is detected (S135: YES), the processor 210 records the information of the detected defect (i.e., valid detection result information) in the database DD in S140. The information recorded includes the various types of information included in the defect detection result, namely, type information, confidence level, box information, and mask data. In the box information and mask data, the coordinates of the bounding box and mask in the second direction Dy are converted to the coordinates of the fabric 700 transport direction Df based on the detected value of the rotary encoder 120. This makes it possible to identify the position of the detected defect on the fabric 700.

[0057] In S145, the processor 210 sends a transport stop signal to the transport device 900 requesting that transport be stopped. When the control unit 990 of the transport device 900 receives the transport stop signal, it stops transporting the cloth 700. In this way, if a defect is detected (YES in S135), the transport of the cloth 700 is stopped.

[0058] When a defect is detected during the inspection process, the transport of the cloth 700 by the transport device 900 is stopped, and the portion of the cloth 700 containing the detected defect is stopped on the support plate 953. For example, an operator can look at the result display screen W11 to understand the location of the defect, then check the actual cloth 700 for the defect and add a marker to the cloth 700 to identify the defect later.

[0059] In S150, the processor 210 performs training image acquisition processing. Training image acquisition processing involves generating training images TI using images of detected defective FDs and storing them in the database DD. Details of the training image acquisition processing will be described later.

[0060] In S155, after the training image acquisition process is completed, the processor 210 sends a transport start signal to the transport device 900 requesting the start of transport. Upon receiving the transport start signal, the control unit 990 of the transport device 900 resumes transporting the cloth 700. After the processor 210 sends the transport start signal, it returns to S100. This resumes the inspection process synchronized with the transport of the cloth 700.

[0061] As can be seen from the above explanation, the processor 210 repeatedly acquires multiple inspection images IMs that show different parts of the cloth 700 sequentially (S110-S115), detects defects in the cloth 700 shown in each inspection image IMs (S120-S125), and displays the results of the detection (S130). When a defect is detected (S150:YES), the processor 210 temporarily suspends the above repeated processing and executes the training image acquisition process.

[0062] A3. Training image acquisition process Next, the training image acquisition process at S150 in Figure 3 will be described. Figure 6 is a flowchart of the training image acquisition process in the first embodiment. At S210, the processor 210 identifies multiple pixels that constitute the defective FD to be processed (hereinafter referred to as defective pixels) from among multiple pixels of the inspection image IMs based on the user's input.

[0063] Specifically, the user confirms the detected defects by visually inspecting the inspection images IMs on the results display screen W11 and the cloth 700. Then, following the message MG2 on the results display screen W11, the user traces the defective FD with their finger on the inspection images IMs displayed on the results display screen W11. The processor 210 uses the touch panel function included in the operation unit 250 to recognize the position traced by the finger on the inspection images IMs. Once the user has finished tracing the defective FD on the inspection images IMs with their finger, they press the OK button BTa on the results display screen W11. Between the display on the results display screen W11 and the pressing of the OK button BTa, the processor 210 identifies the position traced by the finger on the inspection images IMs and identifies multiple pixels at the identified position as defective pixels. For example, if the user traces a linear defective FD with their finger, multiple defective pixels that form a line of a certain thickness along the path traced by the finger are identified. This allows for the correct recognition of multiple defective pixels constituting the defective FD in the inspection images (IMs), even if the entire defective FD is not correctly detected in the detection result image (DI).

[0064] In S215, the processor 210 calculates the size of the defective FD. In this embodiment, the length traced by the user's finger is calculated as the size of the defective FD. In this embodiment, the length traced by the user's finger is proportional to the number of defective pixels identified in S210, so the processor 210 calculates the size of the defective FD based on the number of multiple defective pixels identified in S210. The correspondence between the number of defective pixels and the length of the defective FD is determined experimentally in advance. If the defective FD is a linear defect (scratch or chalk), the size of the defective FD is the length of the linear defect. In this embodiment, it is also assumed that a point-like defect, such as a "hole," may be detected in addition to linear defects. In this embodiment, since holes are not classified according to their size (described later), the size calculation may be omitted if the defect to be processed is a "hole."

[0065] In S220, the processor 210 refers to the training image acquisition table TT to determine the size category of defects to be acquired as training images TI. Figure 7 shows an example of the training image acquisition table TT. The training image acquisition table TT is stored in the database DD. The training image acquisition table TT is a table that classifies and aggregates the defects FD acquired as training images TI into categories according to the type of defect and the size of the defect. In this embodiment, as shown in Figure 7, defects are classified into three categories according to the type of defect (scratch, chalk, hole). Furthermore, scratches and chalk are further classified into multiple categories according to the size of the defect. In this embodiment, as shown in Figure 7, there are four size categories: 3 mm or more and less than 10 mm (size 1), 10 mm or more and less than 50 mm (size 2), 50 mm or more and less than 100 mm (size 3), and 100 mm or more and less than 150 mm (size 4). Note that in this embodiment, it is not assumed that holes of 10 mm or more will occur, so there is only one size category for holes. In Figure 7, the integers listed for each category represent the number of training images TI (also called the number acquired) obtained for defects belonging to that category. For example, in the example in Figure 7, 5 training images TI showing scratches between 3 mm and 10 mm have been acquired, and 10 training images TI showing scratches between 5 mm and 50 mm have been acquired. Note that defects smaller than 3 mm are not targeted for detection in this embodiment, so the size of the defect FD to be processed is always 3 mm or larger.

[0066] For example, if the type of defective FD to be processed is a linear defect (scratch or chalk), the processor 210 determines the category of defect size to be processed as the category with the fewest acquired values ​​among the categories that can be acquired from the defective FD to be processed.

[0067] The categories obtainable from the defective FD to be processed are those of the same type of defect as the defective FD to be processed, and are of a size equal to or less than the category to which the defective FD belongs. For example, if the defective FD to be processed is a scratch and its size (length) is 45 mm, the obtainable categories are scratch size 1 and scratch size 2. The number of copies obtained for scratch size 1 and scratch size 2 is 5 and 10, respectively. Therefore, in this case, the defect size category to be obtained is determined to be scratch size 1 (3 mm to 10 mm).

[0068] For example, if the defective FD to be processed is a scratch and its size (length) is 200 mm, the available categories are scratch size 1 to size 4. The number of scratch sizes 1 to 4 that have already been processed are 5, 10, 2, and 15, respectively. Therefore, in this case, the defect size category to be processed is determined to be scratch size 3 (50 mm or more and less than 100 mm).

[0069] If the type of defective FD to be processed is a hole, there is only one size category for the hole (size 1). Therefore, regardless of the size of the defective FD (hole) to be processed, the processor 210 determines the size category of the defect to be acquired to be hole size 1 (3mm or more and less than 10mm).

[0070] In S225, the processor 210 determines the cropping position of the training image TI based on the size category of the defect to be acquired. Figure 8 is an explanatory diagram of the cropping position of the training image TI. Referring to Figure 8, the case where the defect FD to be processed is a linear defect (scratch or chalk) will be explained. Figures 8(A)-(D) show a portion of the inspection images IMs that include the defect FD (linear scratch). Also, Figures 8(A)-(D) show the cropping position of one or more training image TIs. The size and shape of the training image TI are the size and shape of the input image of the object detection model MD. In this embodiment, the training image TI is a rectangular image with 480 pixels vertically and 680 pixels horizontally.

[0071] Of the four size categories mentioned above (size 1-size 4), even the largest size 4, i.e., defects between 100mm and 150mm in length, can fit within a single training image TI. The defect FD in the inspection images IMs in Figure 8 is a scratch that is approximately 2 to 3 times longer than a size 4 equivalent (100mm to 150mm).

[0072] The processor 210 determines the length of the defect to be acquired within the range of the category determined in the preceding S220. For example, if the category of the size to be acquired is size 3, the length of the defect to be acquired is randomly determined within the range of size 3, i.e., between 50 mm and less than 100 mm.

[0073] In the example shown in Figure 8, during the initial S225, the processor 210 determines the cutting position of the first training image TI1 so as to cut out the first partial defect PF1 having a determined length from the left end TP1 side of the defect FD to be processed (Figure 8(A)).

[0074] Specifically, the processor 210 determines the rightmost edge TP2 of the first partial defect PF1 such that the first partial defect PF1 consists of a number of defective pixels corresponding to the determined length (Figure 8(A)). The processor 210 determines the horizontal cropping position of the training image TI1 such that the horizontal position (first direction Dx) of the rightmost edge RT1 of the training image TI1 coincides with the horizontal position of the rightmost edge TP2 of the partial defect PF1. In the initial S225, the vertical position (second direction Dy) of the cropping position has degrees of freedom. Therefore, the processor 210 randomly determines the vertical cropping position of the training image TI1 within the range that can be cropped to include the entire first partial defect PF1. In this way, the cropping position of the training image TI1 is determined such that the rightmost edge RT1 of the training image TI1 intersects with the defect FD to be processed.

[0075] Furthermore, when the defective FD to be processed is a hole, the expected size is always small, between 3mm and 10mm, and fits within a single training image TI. For this reason, when the defective FD to be processed is a hole, the cropping position of the training image TI is determined randomly within the range that includes the entire defective FD to be processed (not shown in the diagram).

[0076] In S230, the processor 210 extracts the training image TI from the examination images IMs at the determined extraction position. This generates the training image TI. For example, in the example in Figure 8, in the first S230, the processor 210 generates the training image TI1 shown in Figure 8(A).

[0077] In S235, the processor 210 generates training data to be associated with the generated training image TI. In this embodiment, the training data includes type information indicating the type of defect contained in the training image TI, and position information indicating the location of the defect contained in the training image TI. In this embodiment, the position information includes information defining a bounding box BB surrounding the defect, and mask data indicating the position and shape of the defect within the bounding box BB at the pixel level. The type information used is the type information included in the defect detection result FD in the inspection process (Figure 3). The position information is generated based on a plurality of defect pixels that constitute a defect (e.g., first partial defect PF1) contained in the training image TI.

[0078] In S240, the processor 210 associates and stores the training images TI generated in S230 with the teacher data generated in S235. For example, the training images TI and the teacher data are associated with each other and stored in the database DD of the non-volatile storage device 230.

[0079] In S245, the processor 210 updates the training image acquisition table TT. Specifically, in the training image acquisition table TT, 1 is added to the number of acquired images associated with the category to which a defect (e.g., partial defect PF1) contained in the newly generated training image TI belongs.

[0080] In S250, the processor 210 determines whether the length of the remaining defects after removing the already acquired defects (e.g., partial defect PF1) from the defect FD to be processed is less than the threshold THm. In this embodiment, the threshold THm is 3 mm, which is the lower limit length of the defects to be detected. For example, in the example in Figure 8, the length of the remaining defects after removing the partial defect PF1 from the defect FD to be processed is sufficiently long, so in the first S250, it is determined that the length of the remaining defects is greater than or equal to the threshold THm. Also, if the defect FD to be processed is a hole, the training image TI including the entire defect FD (hole) to be processed is extracted in the first extraction, so there are no remaining defects. For this reason, in this case, in the first S250, it is determined that the length of the remaining defects is less than the threshold THm.

[0081] If the length of the remaining defects is less than the threshold THm (S250:YES), the processor 210 terminates the training image acquisition process. If the length of the remaining defects is greater than or equal to the threshold THm (S250:YES), the processor 210 returns to S220. The nth S220-S245 (where n is an integer greater than or equal to 2) is executed with the remaining defects, after removing the defects already acquired in the (n-1)th process, as the new defect FD to be processed. In other words, in the second and subsequent S220-S245, a new training image TI is extracted from the portion of the inspection image IMs excluding the already acquired training image TI. For example, in the example in Figure 8, the second S220-S245 is executed with the remaining defects, after removing the partial defect PF1, from the defect FD to be processed as the new defect FD to be processed.

[0082] For example, in the example in Figure 8, in the second S220, the category with the fewest acquired size among the categories obtainable from the defect FD to be processed, excluding the first partial defect PF1, is determined as the defect size category to be acquired. Then, in the second S225, the processor 210 determines the extraction position of the second training image TI2 so as to cut out the second partial defect PF2, which has the determined length, from the defect FD to be processed, excluding the first partial defect PF1 (Figure 8(B)).

[0083] Specifically, the processor 210 determines the rightmost TP3 of the second partial defect PF2 such that it consists of a number of defective pixels corresponding to the determined length (Figure 8(B)). The leftmost end of the second partial defect PF2 is determined to be the rightmost TP2 of the first partial defect PF1. In this way, the second partial defect PF2 is determined so that the first partial defect PF1 and the second partial defect PF2 do not overlap with each other.

[0084] The processor 210 determines the horizontal cropping position of the training image TI2 such that the horizontal position of the left edge LT2 of the training image TI2 coincides with the horizontal position of the left edge TP2 of the second partial defect PF2. The processor 210 also determines the vertical cropping position of the training image TI2 such that the vertical position of the top edge UT2 of the training image TI2 coincides with the vertical position of the right edge TP3 of the second partial defect PF2. In this way, the cropping position of the training image TI2 is determined so that the left edge LT2 and the top edge UT2 of the training image TI2 intersect with the defect FD to be processed.

[0085] Furthermore, in the example in Figure 8, in the third S220, the category with the fewest acquired values ​​among the categories obtainable from the defect FD to be processed, excluding the partial defects PF1 and PF2, is determined as the category of defect size to be acquired. Then, in the third S225, the processor 210 determines the extraction position of the third training image TI3 so as to cut out the third partial defect PF3, which has the determined length, from the defect FD to be processed, excluding the partial defects PF1 and PF2 (Figure 8(C)).

[0086] Specifically, the processor 210 determines the rightmost TP4 of the third partial defect PF3 such that it consists of a number of defective pixels corresponding to the determined length (Figure 8(C)). The leftmost end of the third partial defect PF3 is determined to be the rightmost TP3 of the second partial defect PF2. In this way, the third partial defect PF3 is determined so that the second partial defect PF2 and the third partial defect PF3 do not overlap with each other.

[0087] The processor 210 determines the vertical cropping position of the training image TI3 such that the vertical position of the lower edge BT3 of the training image TI3 coincides with the vertical position of the left edge TP3 of the third partial defect PF3. The processor 210 also determines the horizontal cropping position of the training image TI3 such that the horizontal position of the right edge RT3 of the training image TI3 coincides with the horizontal position of the right edge TP4 of the third partial defect PF3. In this way, the cropping position of the training image TI3 is determined so that the lower edge BT3 and the right edge RT3 of the training image TI3 intersect with the defect FD to be processed.

[0088] Furthermore, in the example in Figure 8, in the fourth S220, the category with the fewest acquired values ​​among the categories obtainable from the defect FD to be processed, excluding the partial defects PF1-PF3, is determined as the defect size category to be acquired. Then, in the fourth S225, the processor 210 determines the extraction position of the fourth training image TI4 so as to cut out the fourth partial defect PF4, which has the determined length, from the defect FD to be processed, excluding the partial defects PF1-PF3 (Figure 8(D)).

[0089] Specifically, the processor 210 determines the rightmost TP5 of the fourth partial defect PF4 such that it consists of a number of defective pixels corresponding to the determined length (Figure 8(D)). The leftmost end of the fourth partial defect PF4 is determined to be the rightmost TP4 of the third partial defect PF3. In this way, the fourth partial defect PF4 is determined so that the third partial defect PF3 and the fourth partial defect PF4 do not overlap with each other.

[0090] The processor 210 determines the horizontal cropping position of the training image TI4 such that the horizontal position of the left edge LT4 of the training image TI4 coincides with the horizontal position of the left edge TP4 of the fourth partial defect PF4. The processor 210 also determines the vertical cropping position of the training image TI4 such that the vertical position of the top edge UT4 ​​of the training image TI4 coincides with the vertical position of the right edge TP5 of the fourth partial defect PF4. In this way, the cropping position of the training image TI4 is determined so that the left edge LT4 and the top edge UT4 ​​of the training image TI4 intersect with the defect FD to be processed.

[0091] In the example in Figure 8, once the four training images TI1-TI4 are extracted, the length of the remaining defect FDx (Figure 8(B)) becomes less than the threshold THm. For this reason, in the example in Figure 8, at the fourth S250, it is determined that the length of the remaining defect FDx is less than the threshold THm, and the training image acquisition process is terminated. As can be seen from the above explanation, in the example in Figure 8, the training image acquisition process generates training images TI1-TI4 that show linear partial defects PF1-PF4 that are shorter than the defect FD being processed.

[0092] Furthermore, during the training image acquisition process in the inspection process, the training images TI stored in the database DD are used after the inspection process to further train the already trained object detection model MD used in the inspection process. For example, the additional training may be performed using the training image dataset used when training the object detection model MD before the inspection process, in addition to the training images newly generated by the training image acquisition process. Alternatively, the additional training may be performed using only the training images newly generated by the training image acquisition process, without using the training image dataset used when training the object detection model MD before the inspection process. In this embodiment, the specific method of additional training is the same as the method used when training the object detection model MD before the inspection process, except that the training images used are newly generated training images from the training image acquisition process.

[0093] According to the first embodiment described above, the processor 210 acquires inspection images IMs showing the defective FD to be processed (S105-S110 in Figure 3). The inspection images IMs are generated by optically reading the cloth 700 having the defective FD to be processed using digital cameras 111-114 (S105 in Figure 3). The processor 210 acquires training images TI from the inspection images IMs by extracting partial images from the inspection images IMs that include a portion of the defective FD to be processed (e.g., the first partial defect PF1 and the second partial defect PF2) but do not include the rest of the defective FD to be processed, as training images TI (e.g., training image TI1 and training image TI2 in Figure 8) (S230 in Figure 6). With the above configuration, training images TI that include a portion of the defective FD to be processed but do not include the rest are acquired using the inspection images IMs showing the defective FD to be processed. As a result, training images TI containing defects can be acquired efficiently.

[0094] Since defects only occur in abnormal situations, collecting a sufficient number of defect samples is often difficult. Furthermore, because the size of possible defects (e.g., the length of a linear defect) varies, it is preferable to collect images of defects of various sizes as training images (TI). However, since defects of various sizes do not always occur in a balanced manner, collecting the required number of defects of various sizes is extremely difficult. Moreover, in order to perform additional training to improve the detection accuracy of the object detection model (MD), it is necessary to collect images of defects of sizes that are currently poorly detected by the object detection model (MD). However, when collecting defects detected during the detection process, defects of sizes with low detection accuracy are difficult to detect, making it even more difficult to collect images of defects of low detection accuracy. According to this embodiment, images of defects of various sizes (e.g., partial defects PF1-PF4) can be obtained as training images (TI) from inspection images (IMs) showing a defect FD of a single target object. As a result, various training images (TI) can be efficiently obtained. Therefore, the burden of acquiring training images (TI), and consequently the burden of preparing a trained object detection model (MD) capable of accurately detecting various defects, can be reduced.

[0095] Furthermore, according to this embodiment, training images TI showing partial defects PF1-PF4, which are linear defects shorter than the target defect FD, are generated. As a result, training images TI showing shorter linear defects are obtained from inspection images IMs showing linear defects FD, thus enabling efficient acquisition of training images TI showing linear defects of various lengths.

[0096] Furthermore, according to this embodiment, the processor 210 further extracts multiple training images TI1-TI4 from the inspection images IMs, each containing a different portion (partial defect PF1-PF4) of the linear defect FD to be processed (Figure 8). As a result, multiple training images TI can be acquired efficiently.

[0097] Furthermore, in this embodiment, the defects (partial defects PF1-PF4) contained in the multiple training images TI1-TI4 extracted from a single inspection image IMs do not overlap with each other (Figure 8). As a result, since two or more training images containing the same part are not generated, training images TI showing various variations of defects can be efficiently obtained.

[0098] More specifically, the processor 210 extracts a training image TI1 containing the first partial defect PF1 of the linear defect FD to be processed from the inspection images IMs. After extracting the training image TI1, it extracts a training image TI2 containing the second partial defect PF2 of the linear defect FD to be processed from the remaining images of the inspection images IMs (Figures 8(A), (B), and S220-S250 in Figure 6). As a result, with this configuration, by extracting multiple training images TI from the inspection images IMs in a way that does not overlap with each other, multiple training images TI (e.g., TI1, IT2) can be obtained, each containing a non-overlapping portion (e.g., partial defects PF1, PF2) of the defect to be processed.

[0099] Furthermore, in this embodiment, the processor 210 acquires a training image TI1 that includes the end TP1 of the linear defect FD to be processed, and then acquires a training image TI2 that does not include the end TP1 of the linear defect FD to be processed. The training image TI1, which is cropped to include the end TP1 of the defect FD, can crop partial defects PF1 of various lengths, and as described above, the vertical cropping position can be determined at any position. For this reason, the training image TI1, which is cropped to include the end TP1 of the defect FD, has a high degree of freedom in terms of cropping position. For this reason, by acquiring the training image TI1 that includes the end TP1 of the defect FD before the training images TI1 and TI2 that do not include the end TP1 of the defect FD, it is easier to acquire various variations of training image TI.

[0100] Furthermore, in this embodiment, training image TI2 is cropped so that its left edge LT2 and top edge UT2 intersect with the defect FD to be processed (Figure 8(B)). Similarly, training image TI3 is cropped so that its bottom edge BT3 and right edge RT3 intersect with the defect FD to be processed (Figure 8(C)). Training image TI4 is cropped so that its left edge LT4 and top edge UT4 ​​intersect with the defect FD to be processed (Figure 8(D)). In this way, training image TI that does not include the edges of the defect FD to be processed is cropped so that its first edge (e.g., top edge or bottom edge) and a second edge perpendicular to the first edge (e.g., left edge or right edge) intersect with the defect FD to be processed. As a result, even training image TI that does not include the edges of the defect FD to be processed can be cropped to include defects of any length. Therefore, it is easier to obtain training images that include images of defects of a wider range of lengths.

[0101] Furthermore, according to this embodiment, the processor 210 classifies the acquired training images TI into multiple categories according to the size of the defects contained in the training images TI (Figure 7, S245 in Figure 6). Based on the number of training images TI belonging to the multiple categories, the processor 210 determines which category to acquire from among the multiple categories (S220 in Figure 6). The processor 210 preferentially acquires training images TI containing defects of a size belonging to the determined category (S225-S230 in Figure 6). As a result, since training images TI containing defects in the missing categories are acquired preferentially, the necessary training images TI can be acquired in a balanced manner.

[0102] Furthermore, according to this embodiment, the processor 210 inputs inspection images IMs, which are read images of the cloth 700 that are the object to be inspected, as input images to the trained object detection model MD, thereby performing a detection process to detect defects in the cloth 700 (Figure 3). When a defect is detected during the detection process, the processor 210 acquires a training image TI using the inspection images IMs, which are read images, that were input to the trained object detection model MD (YES at S135 in Figure 3, S150 in Figure 3). The acquired training image TI is then used to further train the trained object detection model MD. As a result, by acquiring a training image TI from the inspection images IMs used when performing the detection process with the trained object detection model MD, and further training the trained object detection model MD, the performance of the trained object detection model MD (e.g., detection accuracy) can be effectively improved.

[0103] Furthermore, according to this embodiment, the processor 210 identifies multiple defective pixels that constitute the defective FD to be processed from among multiple pixels of the inspection image IMs based on user input (S210 in Figure 6, Figure 5). The processor 210 uses the results of identifying the multiple defective pixels to extract the training image TI from the inspection image IMs (S215, S225 in Figure 6).

[0104] As can be seen from the above explanation, the inspection images IMs in this embodiment are examples of target images, the training image TI1 is an example of the first training image, and the training image TI2 is an example of the second training image.

[0105] B. Second Example In the second embodiment, the content of the training image acquisition process differs from that of the first embodiment. The other configurations of the second embodiment are the same as those of the first embodiment. Figure 9 is a flowchart of the training image acquisition process in the second embodiment.

[0106] In S310, similar to S210 in Figure 6, the processor 210 identifies multiple defective pixels that constitute the defective FD to be processed from among multiple pixels of the inspection image IMs, based on the user's input. In S315, similar to S215 in Figure 6, the processor 210 calculates the size of the defective FD.

[0107] In S330, the processor 210, similar to S220 in Figure 6, refers to the training image acquisition table and determines the category of defect size to be acquired as training image TI. However, in the second embodiment, for linear defects (scratches and chalk), it is assumed that the acquired training images have been aggregated into two categories: short defects (e.g., 3 mm or more and less than 100 mm) and long defects (e.g., 100 mm or more). The processor 210 then determines that short defects should be acquired as the defect size category if the number of acquired short defects is less than the number of acquired long defects. The processor 210 also determines that long defects should be acquired as the defect size category if the number of acquired long defects is less than the number of acquired short defects. However, if the size of the defect FD to be processed is less than 100 mm and it is not possible to acquire training images containing long defects of 100 mm or more, the processor determines that short defects should be acquired as the defect size category regardless of the number of acquired images in that category.

[0108] In S340, the processor 210 determines the extraction pattern based on the category of the defect to be acquired. Figure 10 is an explanatory diagram of the extraction patterns. If the category of the defect to be acquired is determined to be a long defect, the extraction pattern is determined to be the long defect acquisition pattern shown in Figure 10(A). If the category of the defect to be acquired is determined to be a short defect, the extraction pattern is determined to be the short defect acquisition pattern shown in Figure 10(B).

[0109] In Figure 10(A), points Pa1 and Pa4 indicate the ends of the defect FD to be processed, and points Pa2 and Pa3 indicate the positions where the edges of the training image TIa intersect with the defect FD to be processed. The long defect acquisition pattern in Figure 10(A) is a pattern that extracts K (K is an integer greater than or equal to 1) training image TIa so as to include a long defect that is longer than half the horizontal length of the training image TIa and approximately the same length as the horizontal length of the training image TI. In the long defect acquisition pattern, the extraction position of the training image TIa that does not include the ends Pa1 and Pa4 of the defect FD to be processed (training image TIa2 in the example of Figure 10(A)) is determined so that the right and left edges of the training image TIa intersect with the defect FD to be processed. In addition, the extraction positions of at least one, and preferably both, of the two training image TIa that include the ends Pa1 and Pa4 of the defect FD to be processed (training images TIa1 and TIa3 in the example of Figure 10(A)) are determined so as to include the long defect described above.

[0110] In FIG. 10(B), points Pb1 and Pb7 indicate both ends of the defect FD to be processed, and points Pb2 - Pb6 indicate the positions where the sides of the training image TIb and the defect FD to be processed intersect. The defect acquisition pattern in FIG. 10(B) is a pattern for cutting out L (L is an integer satisfying K < L) training images TIb so as to include defects shorter than half of the horizontal length of the training image TIb or about half the length. In the short defect acquisition pattern, the cutting positions of the training images TIb (training images TIb2 - TIb5 in the example of FIG. 10(B)) that do not include both ends Pb1 and Pb6 of the defect FD to be processed are determined such that the first side (for example, the right side or the left side) of the training image TIb and the second side (the upper side or the lower side) orthogonal to the first side intersect the defect FD to be processed. In the example of FIG. 10(B), the cutting positions of the training images TIb2 and TIb4 are determined such that the left side and the upper side of the training images TIb2 and TIb4 intersect the defect FD to be processed. Also, the cutting positions of the training images TIb3 and TIb5 are determined such that the lower side and the right side of the training images TIb3 and TIb5 intersect the defect FD to be processed. Also, the cutting positions of at least one, and if possible both, of the two training images TIb (training images TIb1 and TIb6 in the example of FIG. 10) that include both ends Pb1 and Pb6 of the defect FD to be processed are determined so as to include the short defects described above.

[0111] Thus, the average length of the defects included in one training image TIb in the short defect acquisition pattern is shorter than the average length of the defects included in one training image TIa in the long defect acquisition pattern. And the number L of the training images TIb obtained from one inspection image IMs in the short defect acquisition pattern is larger than the number K of the training images TIa obtained from one inspection image IMs in the long defect acquisition pattern.

[0112] In S350, the processor 210 extracts training images from the inspection images IMs at the determined extraction position. If the extraction pattern is a long defect acquisition pattern, K training images TIa (3 in the example of Figure 10(A)) are generated from the inspection images IMs (TIa1-TIa3 in the example of Figure 10(A)). If the extraction pattern is a short defect acquisition pattern, L training images TIb (6 in the example of Figure 10(A)) are generated from the inspection images IMs (TIb1-TIb6 in the example of Figure 10(B)).

[0113] In S360, the processor 210 generates training data to be associated with each of the multiple training images that have been generated. The training data includes type information indicating the type of defect contained in the training image, and location information indicating the location of the defect contained in the training image, similar to the first embodiment.

[0114] In S370, the processor 210 associates and stores the training images generated in S350 with the teacher data generated in S360. For example, each training image and the teacher data for each training image are associated with each other and stored in the database DD of the non-volatile storage device 230.

[0115] In S380, the processor 210 updates the training image acquisition table (not shown). Specifically, in the training image acquisition table, if the extraction pattern is a long defect acquisition pattern, K is added to the number of long defects acquired, and if the extraction pattern is a short defect acquisition pattern, L is added to the number of short defects acquired. After updating the training image acquisition table, the processor 210 terminates the training image acquisition process.

[0116] According to the second embodiment described above, the processor 210 determines an extraction pattern according to the category of defect size to be acquired (S340 in Figure 9), and generates multiple training images by extracting multiple training images from the inspection images IMs based on the extraction pattern (S350 in Figure 9). As a result, training images showing defects of the size to be acquired can be efficiently obtained.

[0117] Furthermore, according to this embodiment, the category of defect size to be acquired is determined by referring to a training image collection table that aggregates the number of training images acquired for each category. Therefore, for example, training images containing defects of sizes for which samples are insufficient can be acquired efficiently.

[0118] C. Variations (1) In each of the above embodiments, the processor 210 obtains a training image TI that includes a portion of the linear defect FD from the linear defect FD to be processed. However, the processor 210 may also use defects with shapes other than linear, such as dirt or tears, as the defects to be processed. In this case, for example, the processor 210 may cut out two or four training images TI so as to divide the non-linear defect to be processed into two or four parts along the edges of the training image TI. As a result, multiple training images TI showing dirt or tears can be efficiently obtained from one inspection image IMs showing dirt or tears.

[0119] (2) In each of the above embodiments, training images TI1-TI4 are obtained from a single inspection image IMs, each containing non-overlapping partial defects PF1-PF4. Alternatively, the processor 210 may obtain two or more training images TI from the defect FD to be processed, each containing two or more partial defects that partially overlap but do not completely overlap. For example, in Figure 8(B), the left edge of the second partial defect PF2 included in training image TI2 may overlap with the right edge of the first partial defect PF1 included in training image TI1.

[0120] (3) In the first embodiment described above, the processor 210 sequentially cuts out training images TI containing partial defects PF from the left side of the defect FD to be processed in the inspection images IMs. However, the processor 210 may also sequentially cut out training images TI containing partial defects PF from the right end of the defect FD to be processed. Furthermore, the processor 210 may cut out training images TI containing the right end and training images TI containing the left end of the defect FD to be processed, and then cut out one or more training images TI containing partial defects from the remaining defects. Since there is a high degree of freedom in cutting out training images TI containing the edges of the defect FD to be processed, cutting out training images TI containing the edges first makes it easier to obtain training images TI with defects of various lengths placed in various positions.

[0121] (4) In each of the above embodiments, the processor 210 identifies multiple defective pixels that constitute the defective FD to be processed in the inspection image IMs based on user input (specifically, input of tracing the defective FD with a finger) input via the result display screen W11. Alternatively, the processor 210 may identify multiple defective pixels that constitute the defective FD to be processed based on mask data included in the detection results using the object detection model MD. Or, the processor 210 may have the user confirm the results of identifying the defective FD based on the mask data included in the detection results, and may use a combination of identifying multiple defective pixels using the mask data based on the user's instructions and identifying multiple defective pixels based on the user's input.

[0122] (5) In each of the above embodiments, defects detected during the inspection process of the cloth 700 using the object detection model MD are designated as defects FD to be processed, and multiple training images TI are obtained from inspection images IMs showing the defects FD to be processed. However, if a defect to be acquired by a method that does not use a machine learning model is discovered, the cloth 700 may be photographed and inspection images IMs may be generated, and the data processing device 200 may use the inspection images IMs to perform the training image acquisition process shown in Figures 6 and 9. The method that does not use a machine learning model may be, for example, a visual inspection by an operator. Alternatively, the method that does not use a machine learning model may be an analysis process that does not use a machine learning model. For example, defects may be detected by searching for defects of a predetermined shape, such as linear scratches, in the inspection images IMs using pattern matching.

[0123] (6) In each of the above embodiments, the processor 210 classifies the training images TI into several categories according to the size of the defects contained in the acquired training images TI (Figure 7), but such classification may be omitted. In this case, for example, the processor 210 may randomly and unbiasedly determine the size of the defects to be acquired within a predetermined size range (e.g., 3 mm - 150 mm).

[0124] (7) In the above embodiment, the training image TI generated in the training image acquisition process is used to further train the pre-trained object detection model MD used in the detection process. However, the training image TI generated in the training image acquisition process may also be used to train a machine learning model different from the pre-trained object detection model MD used in the detection process. The machine learning model trained using the training image TI1 is not limited to an object detection model, but may be of any other type. For example, an image recognition model may be trained using the training image TI1. The image recognition model is trained to identify the type of defect contained in a read image when a read image showing an object containing a defect is input. For example, known CNNs (Convolutional Neural Networks), such as ResNet, VGG16, and VGG19, can be used for the image recognition model. Alternatively, an autoencoder may be trained using the training image TI. The autoencoder is trained to generate an output image showing an object without defects when a read image showing an object containing a defect is input. The autoencoder can be a standard autoencoder, a VQ-VAE (Vector Quantized Variational Auto Encoder), a VAE (Variational Autoencoder), or an autoencoder included in so-called GANs (Generative Adversarial Networks).

[0125] (8) The object detection model MD used in each of the above embodiments is a machine learning model that implements instance segmentation called RTMDet, but other machine learning models may be used. For example, the object detection model MD may be a machine learning model that implements other instance segmentation such as Mask R-CNN, or a machine learning model that implements semantic segmentation such as YOLO (You only look once).

[0126] (9) In the above embodiment, the object to be inspected is cloth 700, but it may be other objects. Other objects may be other sheet-like objects, such as metal foil, resin film, or paper. Also, the object to be inspected may be an object with a shape different from a sheet, such as various products or parts of automobiles or electrical appliances. Depending on the object to be inspected, the defects to be detected may take on various forms.

[0127] (10) The total number of digital cameras used to read the cloth 700 is not limited to 4, but may be any number of 1 or more. Also, the device used to read the cloth 700 may include a line sensor as an image sensor instead of an area sensor such as a digital camera. In any case, the processor 210 uses the target images acquired using these devices to detect defects in the cloth 700. For example, if there is only one digital camera, the read image generated by that single digital camera may be used as the inspection image IMs.

[0128] (11) In the above embodiment, the processor 210 may have the GPU 260 perform various calculations. For example, the processor 210 may have the GPU 260 perform some or all of the calculations performed by the object detection model MD. The GPU 260 may be omitted.

[0129] (12) The data processing device 200 in Figure 1 may be a different type of device from a personal computer (for example, a digital camera, scanner, or smartphone). Alternatively, multiple devices (for example, computers) that can communicate with each other via a network may each share a portion of the data processing function performed by the data processing device, and together they may provide the data processing function (a system comprising these devices corresponds to the data processing device).

[0130] (13) In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, the processing by the object detection model MD (Figure 2) may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0131] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0132] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of symbols]

[0133] 1000…Inspection system, 10…Inspection device, 111-114…Digital camera, 120…Rotary encoder, 130…Light source, 200…Data processing device, 210…Processor, 215…Storage device, 220…Volatile storage device, 230…Non-volatile storage device, 240…Display unit, 250…Operation unit, 270…Communication interface, 700…Cloth, 900…Conveying device, 950…Conveying mechanism, 980…Operation unit, 990…Control unit, DD…Database, DI…Detection result image, IM1-IM4…Reading image, IMs…Inspection image, MD…Object detection model, PF…Partial defect, PG…Computer program, TI…Training image, TT…Training image acquisition table, W11…Result display screen

Claims

1. It is a computer program, An image acquisition function that acquires an image showing a defect in the object to be processed, wherein the image acquisition function generates the image by optically reading the object having the defect in the object to be processed, An acquisition function that obtains a training image from the target image by extracting a partial image from the target image that includes a part of the defect to be processed but does not include the other part of the defect to be processed, wherein the training image is used to train a machine learning model for detecting defects in the target object, and the acquisition function A computer program that enables a computer to realize something.

2. A computer program according to claim 1, The defect to be processed is a linear defect. A computer program in which a portion of the defects to be processed is a linear defect shorter than the defects to be processed.

3. A computer program according to claim 2, The acquisition function further includes a computer program that extracts a plurality of training images from the target image, each containing a different portion of the linear defect to be processed.

4. A computer program according to claim 3, A computer program in which defects contained in multiple training images extracted from the aforementioned target image do not overlap with each other.

5. A computer program according to claim 4, The aforementioned acquisition function is, From the aforementioned target image, a first training image is extracted that includes the first portion of the linear defect to be processed. A computer program that, after extracting the first training image, extracts a second training image from the target image, excluding the first training image, which includes the second portion of the linear defect to be processed.

6. A computer program according to claim 5, The aforementioned acquisition function is, The first training image is an image that includes the end of the linear defect to be processed, A computer program in which the second training image is an image that does not include the edges of the linear defect to be processed.

7. A computer program according to claim 6, The aforementioned training image is a rectangular image, The acquisition function is a computer program that crops the second training image such that the first edge of the second training image and the second edge perpendicular to the first edge intersect with the linear defect to be processed.

8. A computer program according to claim 1, further, A computer implements a decision function that classifies multiple training images into multiple categories according to the size of defects contained in the training images, and determines which category to acquire from among the multiple categories based on the number of training images belonging to each of the multiple categories. The acquisition function is a computer program that prioritizes acquiring the training images containing defects of a size belonging to the determined acquisition target category.

9. A computer program according to claim 1, The machine learning model is a trained model that has been trained to detect defects in the object shown in the input image. The aforementioned computer program, further, A detection function is implemented in a computer that performs a detection process to detect defects in the object by inputting the read image, which is generated by optically reading the object to be detected, as the input image to the trained model. The image acquisition function, when a defect is detected in the detection process, acquires the read image input to the trained model as the target image. The training images obtained using the aforementioned read images are used in a computer program to further train the previously trained model.

10. A computer program according to claim 1, further, Based on user input, the computer implements a defect image identification function that identifies multiple defective pixels constituting the defect to be processed from among multiple pixels of the target image. The acquisition function is a computer program that extracts the training image from the target image using the results of identifying the multiple defective pixels.

11. An image processing device, An image acquisition unit that acquires an image showing a defect in the object to be processed, wherein the image acquisition unit generates the image by optically reading the object having the defect in the object to be processed, An acquisition unit that acquires a training image from the target image by extracting a partial image from the target image that includes a part of the defect to be processed but does not include the other part of the defect to be processed, wherein the training image is used to train a machine learning model for detecting defects in the target object, An image processing device equipped with the following features.

12. An image processing method, An image acquisition step in which an image showing a defect in the object to be processed is obtained, wherein the image is generated by optically reading the object having the defect in the object to be processed, An acquisition step of obtaining the training image from the target image by extracting a partial image from the target image that includes a part of the defect to be processed but does not include the other part of the defect to be processed, wherein the training image is used to train a machine learning model for detecting defects in the target object, An image processing method comprising:

Images