Computer programs and methods for training object detection models.

Abstract

Train an object detection model. [Solution] In the control of a training process in which training input images are input to an object detection model to generate training output data and multiple parameters of the object detection model are adjusted using loss values ​​calculated using the training output data, and an evaluation process in which evaluation input images are input to an object detection model to generate evaluation output data and an index value indicating the detection accuracy of object regions by the object detection model is calculated based on the evaluation output data, the training process is executed repeatedly for a specific number of epochs, and the evaluation process is executed. If the index value calculated in the evaluation process shows a detection accuracy of a standard or higher, the training is terminated, and if the index value shows a detection accuracy of a standard or lower, the training is continued.

Description

Technical Field

[0006] , , , ,

[0005] , , ,

[0007] , , ,

[0001] This specification relates to the training of an object detection model for detecting objects in an image.

Background Art

[0002] The object detection model disclosed in Non-Patent Document 1 performs the detection of an object in an image and the identification (class classification) of the object. The object detection result is output as information indicating a rectangular region where the object exists.

[0003] The training of the object detection model is executed using a training image including the object to be detected and teacher data indicating the correct answer of the object detection result so that the loss calculated using a predetermined calculation formula becomes small.

Prior Art Documents

Patent Documents

[0004] <​​​​​​​​​​​​​​​​​​​​​​​[Application Example 1] A training process for training an object detection model, which is a computer program that performs an arithmetic process using multiple parameters on an input image containing an object to generate output data that indicates at least the object region in the input image, wherein the training process involves inputting the input image for training to the object detection model, generating the output data for training corresponding to the input image for training, and adjusting the multiple parameters of the object detection model using a loss value calculated using the output data for training; and an evaluation process for evaluating the object detection model, wherein the evaluation process involves inputting the input image for evaluation to the object detection model, generating the output data for evaluation corresponding to the input image for evaluation, and A computer program that causes a computer to implement the following: an evaluation process that calculates an index value indicating the detection accuracy of the object region by the object detection model based on the output data; and a control process that controls the execution of the training process and the evaluation process, wherein after repeatedly executing the training process for a specific number of epochs, the evaluation process is executed, and if the index value calculated in the evaluation process indicates the detection accuracy to be above a standard, the training of the object detection model is terminated without executing the training process for the next epoch, and if the index value calculated in the evaluation process indicates the detection accuracy to be below a standard, the training process for the next epoch is executed to continue training the object detection model.

[0008] For example, even if the loss value falls below a threshold, the accuracy of object detection may not be sufficient. Conversely, even if the loss value is greater than the threshold, the accuracy of object detection may still be sufficient. With the above configuration, after repeatedly performing the training process for a specific number of epochs, if the index value calculated in the evaluation process shows a detection accuracy above the standard, the training of the object detection model is terminated. This prevents excessive training when the object detection accuracy is sufficient. Furthermore, if the index value calculated in the evaluation process shows a detection accuracy below the standard, the training of the object detection model is continued. This prevents training from being terminated when the object detection accuracy is insufficient.

[0009] Furthermore, the technologies disclosed herein can be implemented in various forms, including methods for training object detection models, apparatus for training object detection models, systems for training object detection models, computer programs for implementing these methods, apparatuses, and systems, object detection models trained by these methods, apparatuses, and systems, recording media storing the computer programs and object detection models, and so on. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing the configuration of the inspection system 1000 in this embodiment. [Figure 2] Diagram illustrating product 300. [Figure 3] A block diagram showing the configuration of the object detection model AN. [Figure 4] A diagram illustrating the operation of the object detection model AN. [Figure 5] A flowchart for training the object detection model AN. [Figure 6] A diagram showing an example of training image LI and evaluation image EI. [Figure 7] Flowchart of the model evaluation process in the first embodiment. [Figure 8] A diagram showing an example of an evaluation record table RT. [Figure 9] A figure showing an example of label TL and label area LA in the first embodiment. [Figure 10] Flowchart of the inspection process. [Figure 11] Diagram illustrating the inspection process. [Figure 12] Flowchart of the model evaluation process in the second embodiment. [Figure 13] A figure showing an example of label TL and label region LA in the second embodiment. [Modes for carrying out the invention]

[0011] A. First Example A-1. Configuration of the inspection device Next, embodiments will be described based on examples. Figure 1 is a block diagram showing the configuration of the inspection system 1000 of this embodiment. The inspection system 1000 includes a processing device 100 and an imaging device 400. The processing device 100 and the imaging device 400 are connected in a communication manner.

[0012] The processing unit 100 is, for example, a computer such as a personal computer. The processing unit 100 includes a CPU 110 as the controller of the processing unit 100, a GPU 115, a volatile storage device 120 such as RAM, a non-volatile storage device 130 such as a hard disk drive, an operating unit 150 such as a mouse or keyboard, a display unit 140 such as a liquid crystal display, and a communication unit 170. The communication unit 170 includes a wired or wireless interface for communicating with an external device, such as a camera 400.

[0013] The GPU (Graphics Processing Unit) 115 is a processor that performs computational processing for image processing such as 3D graphics, in accordance with the control of the CPU 110. In this embodiment, it is used to perform computational processing for the object detection model AN and the image generation model GN, which will be described later.

[0014] The volatile storage device 120 provides a buffer area for temporarily storing various intermediate data generated when the CPU 110 performs processing. The non-volatile storage device 130 stores the computer program PG for the inspection device, the training image data group LG, the teacher data group TG, the evaluation image data group EG, and the evaluation record table RT. The training image data group LG contains data for multiple training images (described later) (e.g., several hundred to tens of thousands). The teacher data group TG contains data for multiple teacher data corresponding to multiple training images. The evaluation image data group EG contains data for multiple evaluation images (described later) (e.g., several tens to several hundred). These data groups LG, TG, and EG are used in training the object detection model AN, which will be described later. The evaluation record table RT is a table used to record the evaluation of the object detection model AN in the model evaluation process, which will be described later.

[0015] The computer program PG includes, as a module, a computer program that causes the CPU 110 and the GPU 115 to cooperate to realize the functions of the object detection model AN and the image generation model GN described later. The computer program PG is provided, for example, by the manufacturer of the processing device 100. The computer program PG may be provided, for example, in a form downloaded from a server or in a form stored in a DVD-ROM or the like. The CPU 110 executes the computer program PG to execute the inspection process and the training of the object detection model AN described later.

[0016] The imaging device 400 is a digital camera that generates data of a captured image representing a subject by capturing the subject using a two-dimensional image sensor. The data of the captured image is bitmap data indicating an image including a plurality of pixels, and specifically, RGB image data representing the color of each pixel by RGB values. The RGB values are color values in the RGB color system including the gradation values of three color components (hereinafter also referred to as component values), that is, the R value, the G value, and the B value. The R value, the G value, and the B value are, for example, gradation values of a predetermined number of gradations (for example, 256). The data of the captured image may be data representing the luminance of each pixel.

[0017] The imaging device 400 generates data of a captured image in accordance with the control of the processing device 100 and transmits it to the processing device 100. In the present embodiment, the imaging device 400 is used to capture a product 300 with a label L attached thereto, which is an inspection object of the inspection process, and generate data of the captured image.

[0018] FIG. 2 is an explanatory diagram of the product 300. FIG. 2(A) shows a perspective view of the product 300. In the present embodiment, the product 300 is a printer having a substantially rectangular parallelepiped housing 30. In the manufacturing process, a rectangular label L is attached to a predetermined attachment position on the front surface 31 (+Y side surface) of the housing 30.

[0019] Figure 2(B) shows label L. Label L includes, for example, background B and letters TX and marks MK that indicate various information such as the manufacturer's or product's brand logo, model number, lot number, etc.

[0020] A2. Configuration of the object detection model AN Figure 3 is a block diagram showing the configuration of the object detection model AN. Various object detection models can be used as the object detection model AN. In this embodiment, the object detection model AN is an object detection model called YOLO (You Only Look Once). YOLO is disclosed, for example, in the paper "Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788". The YOLO model uses a convolutional neural network to predict the region in an image where an object is located and the type of object located in that region.

[0021] As shown in Figure 3, the object detection model AN has m convolutional layers CV11-CV1m (where m is an integer greater than or equal to 1) and n fully connected layers CN11-CN1n (where n is an integer greater than or equal to 1) following the convolutional layers CV11-CV1m (where m is, for example, 24; where n is, for example, 2). A pooling layer is provided immediately after one or more of the m convolutional layers CV11-CV1m.

[0022] The convolutional layers CV11-CV1m perform processing on the input data, including convolution and bias addition. The convolution process applies t filters sequentially to the input data and calculates a correlation value that shows the correlation between the input data and the filters (t is an integer greater than or equal to 1). In the filter application process, multiple correlation values ​​are calculated sequentially while sliding the filters. The bias addition process adds a bias to the calculated correlation value. One bias is prepared for each filter. The dimension of the filter and the number of filters t usually differ among the m convolutional layers CV11-CV1m. Each convolutional layer CV11-CV1m has a parameter set that includes multiple weights and multiple biases for multiple filters.

[0023] The pooling layer performs a process to reduce the dimensionality of the data input from the preceding convolutional layer. Various pooling processes are available, such as average pooling and max pooling. In this embodiment, the pooling layer performs max pooling. Max pooling reduces the dimensionality by sliding a predetermined size window (e.g., 2x2) with a predetermined stride (e.g., 2) and selecting the maximum value within the window.

[0024] The fully connected layers CN11-CN1n take f-dimensional data (i.e., f values, where f is an integer greater than or equal to 2) input from the previous layer and output g-dimensional data (i.e., g values, where g is an integer greater than or equal to 2). Each of the g output values ​​is the inner product of a vector composed of the f input values ​​and a vector composed of f weights, with a bias added. The dimensions f of the input data and g of the output data usually differ among the n fully connected layers CN11-CN1n. Each fully connected layer CN11-CN1n has a parameter set that includes multiple weights and multiple biases.

[0025] The data generated by the convolutional layers CV11-CV1m and the fully connected layers CN11-CN1n are input to an activation function and transformed. Various functions can be used as activation functions. In this embodiment, a linear activation function is used for the last layer (here, the fully connected layer CN1n), and leaky-normalized linear units (LReLU) are used for the other layers.

[0026] Figure 4 is an explanatory diagram of the operation of object detection model AN. The object detection model AN receives input image IIa. In this embodiment, the captured image, which will be described in detail later, is input as input image IIa. Figure 4(A) shows input image IIa. Input image IIa contains multiple pixels arranged in a matrix along a first direction Dx and a second direction Dy perpendicular to the first direction Dx. The value of each pixel is, for example, a color value (RGB value) that includes the values ​​of each component of RGB.

[0027] When the object detection model AN receives the input image IIa, it performs calculations on the input image IIa using the parameter set described above to generate output data OD. The output data OD is data containing S × S × (B × 5 + C) predicted values. Each predicted value includes predicted region information indicating the predicted region BB (also called a bounding box) where an object (label in this embodiment) in the image is predicted to be located, and class information indicating the type (also called a class) of the object present in the predicted region BB.

[0028] The prediction region BB is a rectangular area. Prediction regions BB are set for each (S × S) cell CL obtained by dividing the input image IIa into S × S (S is an integer greater than or equal to 2; in the example in Figure 4(A), S=5), with B (B is an integer greater than or equal to 1, e.g., 2) being set. Figure 4(A) shows an example of one prediction region BB set for a hatched cell CL. The center Cb of the prediction region BB is located within the cell CL to which the prediction region BB is set.

[0029] Figure 4(B) shows an enlarged view of the prediction region BB in Figure 4(A). Each prediction region information includes five values: the center coordinates (Xb, Yb) of the prediction region BB relative to cell CL, the width (length in direction Dx) Wb, the height (length in direction Dy) Hb, and the confidence level Vc of the prediction region. The confidence level Vc indicates the probability that an object exists in the prediction region BB. The class information indicates the type of object present in cell CL, expressed as a probability for each type. When classifying the object into C types (C is an integer greater than or equal to 1), the class information includes C values ​​indicating probabilities. In this embodiment, C=1, and it is identified whether the object is a label or not. For this reason, the output data OD includes S×S×(B×5+C) prediction values ​​as described above.

[0030] A-3. Training of the object detection model AN Figure 5 is a flowchart of the training of the object detection model AN. Figure 6 is a diagram showing an example of the training image LI and the evaluation image EI. The training image LI is the input image IIa used for training, and the evaluation image EI is the input image IIa used for evaluation. In training the object detection model AN, the object detection model AN is trained to achieve the following objectives (1) to (3). (1) If the input image IIa contains a specific type of label as an object, output data OD is output, which includes prediction region information of the prediction region BB that indicates the label region where the label is located. (2) The confidence level Vc of the prediction region BB that represents the label region is set to a confidence level close to the maximum value (e.g., 1), and the confidence level Vc of the prediction region that represents a region different from the label region is set to a confidence level close to the minimum value (e.g., 0). (3) The class information of the cell CL to which the prediction region BB, which indicates the label region, is associated, indicates that the object is a specific type of label.

[0031] As a result, the object detection model AN is trained so that the output data OD indicates the appropriate label regions of the input image (e.g., the training image LI in Figure 6(A)) and the types of labels contained within those label regions. Training adjusts several computational parameters used in the calculations of the object detection model AN (including several computational parameters used in the calculations of each of the multiple layers CV11-CV1m and CN11-CN1n). Before training, the multiple computational parameters are set to initial values ​​such as random values.

[0032] In S110, the CPU 110 obtains multiple training image data (for example, several hundred) and multiple training data corresponding to the multiple training image data LI from the training image data group LG and the training data group TG stored in the non-volatile storage device 130.

[0033] As shown in Figure 6(A), the training image LI includes a background BI and a label TL. To distinguish the labels shown in the training image LI and evaluation image EI (described later) from the actual label L (Figure 2), a code with a "T" prefix is ​​used to designate the label TL. The training image LI in Figure 6(A) is image data generated by photographing the label L attached to product 300 using a digital camera. For this reason, the background BI is an image showing a part of product 300. However, multiple training image LIs may also include images obtained by compositing a photographed image (e.g., a landscape or interior image) with an image showing the label TL as the foreground. Multiple training image LIs may include images in which label TLs of various sizes are arranged at various positions and angles. Multiple training image LIs may also include images obtained by performing known augmentation processing (e.g., noise addition, scaling, inversion) on the above-mentioned images.

[0034] The training data represents the ideal output data OD that should be output when the corresponding synthetic image data is input to the object detection model AN. For example, the training data includes, among S × S × (B × 5 + C) predicted values, the ideal predicted value corresponding to the cell CL where the center (centroid) Ct of label TL in the training image LI (Figure 6(A)) is located, including ground truth region information, the maximum confidence level Vc (e.g., 1), and class information indicating the type of label. The training data also includes the minimum confidence level Vc (e.g., 0) as the predicted value corresponding to the cell CL where the center Ct of label TL is not located.

[0035] The ground truth region information is information that indicates the ground truth region TB (Figure 6(A)). The ground truth region TB is the ideal label region that the output data OD of the object detection model AN should indicate when the training image LI is input to the object detection model AN. The ground truth region TB is the region in the training image LI where the label TL is located, for example, a rectangular region circumscribing the label TL. Note that the ground truth region TB of Figure 6(A) is also illustrated in Figures 4(A) and (B). The ground truth region TB is associated with the cell CL where the center Ct of the ground truth region TB is located. The ground truth region information is information that defines the ground truth region TB and includes the coordinates (Xt, Yt) of the center Ct of the ground truth region TB relative to the cell CL corresponding to the ground truth region TB, the width (length in direction Dx) Wt, and the height (length in direction Dy) Ht.

[0036] Training data is created in advance, for example, as follows. For example, the processing unit 100 displays the training image LI on the display unit 140 and accepts the designation of the correct region TB. The operator identifies the label TL within the training image LI while looking at the training image LI and designates the correct region TB that is circumscribing the label TL. The designation of the correct region TB is performed, for example, by using a mouse pointing device. Based on the designated correct region TB, the processing unit 100 generates the above-described training data and stores it in the non-volatile storage device 130 in association with the training image data showing the training image LI.

[0037] In S120, the CPU 110 inputs multiple training images LI into the object detection model AN and generates multiple output data OD corresponding to the multiple training images LI.

[0038] In S130, the loss value is calculated using multiple output data ODs and multiple training data corresponding to the multiple output data ODs.

[0039] The loss value is calculated using a loss function that calculates a loss value corresponding to the difference between the output data OD and the training data. In this embodiment, the loss function calculates a smaller loss value the smaller the difference between the output data OD and the training data. The formula for calculating the loss function is disclosed in the YOLO paper mentioned above.

[0040] In S140, the CPU 110 adjusts several computational parameters of the object detection model AN using the calculated loss value. Specifically, the CPU 110 adjusts the computational parameters according to a predetermined algorithm so that the sum of the loss values ​​calculated for each training image data is small. As a predetermined algorithm, for example, an algorithm using backpropagation and gradient descent is used. This adjusts the computational parameters of the object detection model AN so that the difference between the output data and the training data is small.

[0041] In S150, the CPU 110 determines whether a specific number of epochs of training have been completed. The number of epochs is the number of times training has been repeated using the entire training dataset (the entire training image data set LG). For example, if the number of training images LI included in the training image data set LG is 1000 and the batch size is 100, then one epoch of training will be completed when the process in S110-S140 in Figure 5 is repeated 10 times. The specific number is, for example, 5 to 20, and in this embodiment, it is 10. In this step, the number of epochs is counted from the start of training the first time, and from the second time onward, it is counted from after the previous model evaluation process.

[0042] If a specific number of epochs of training have not been completed (S150: NO), CPU 110 returns to S110 and continues training. If a specific number of epochs of training have been completed (S150: YES), CPU 110 performs the model evaluation process in S160.

[0043] The model evaluation process evaluates the object detection model AN by calculating the accuracy pass rate Rap (described later), which is an index value indicating the accuracy with which the object detection model AN detects the label region of the input image IIa. Details of the model evaluation process will be described later.

[0044] In S170, the CPU 110 determines whether the accuracy pass rate Rap calculated in the model evaluation process is equal to or greater than the threshold THr. The threshold THr is the value of the accuracy pass rate Rap required for the object detection model AN, and in this embodiment, it is set to 90%.

[0045] If the accuracy pass rate Rap is greater than or equal to the threshold Thr (S170: YES), the CPU 110 terminates training of the object detection model AN. The trained object detection model AN data (i.e., the set of adjusted computation parameters) is stored in the non-volatile memory device 130.

[0046] If the accuracy pass rate Rap is less than the threshold Thr (S170: NO), the CPU 110 determines in S180 whether a predetermined number of epochs of training have been completed. In this embodiment, the predetermined number is 500. It is preferable that the predetermined number be set to a value of 500 or more. It is even more preferable that the predetermined number be set to a value of 500 or more but less than 1000. If the predetermined number of epochs of training have not been completed (S180: NO), the CPU 110 returns to S110 and continues training the object detection model AN. If the predetermined number of epochs of training have been completed (S180: YES), the CPU 110 terminates training the object detection model AN. This is because, empirically, no further improvement in detection accuracy can be expected once 500 or more epochs of training have been completed. In this case, for example, the configuration of the object detection model AN itself or the training image data set LG may be reviewed, and training of the object detection model AN may be restarted from the beginning.

[0047] A-4. Model Evaluation Process Next, the model evaluation process in S160 of Figure 5 will be explained. Figure 7 is a flowchart of the model evaluation process in the first embodiment. In S205, the CPU 110 acquires multiple (for example, tens to hundreds) evaluation image EIs. In this embodiment, the evaluation image data group EG (Figure 1) stored in the non-volatile storage device 130 contains data for 100 evaluation image EIs. The CPU 110 acquires all 100 evaluation image EIs. As shown in Figure 5, the model evaluation process in S160 is performed multiple times, but in this embodiment, the same 100 evaluation image EIs are always acquired. That is, the CPU 110 performs the nth model evaluation process (where n is an integer greater than or equal to 1), and the (n+1)th model evaluation process, which is performed after multiple training processes that are performed after the nth model evaluation process. In this embodiment, the multiple evaluation image EIs used in the nth model evaluation process and the multiple evaluation image EIs used in the (n+1)th model evaluation process are the same image. Furthermore, it is possible to use a portion of the same image for the multiple evaluation images EI used in the nth model evaluation process and the multiple evaluation images EI used in the (n+1)th model evaluation process, and use a separate image for the remaining portion.

[0048] As shown in Figure 6(B), the evaluation image EI, like the training image LI, includes a background BI and a label TL. The evaluation image EI in Figure 6(B) is an image obtained by compositing an image of the label TL as a foreground to an image captured with a digital camera. However, the evaluation image EI may also include an image generated by photographing a label L attached to product 300 using a digital camera. Multiple evaluation image EIs include images in which label TLs of various sizes are placed in various positions. Furthermore, multiple evaluation image EIs may also include images obtained by performing known augmentation processing on the above-mentioned images.

[0049] In this embodiment, the pre-generated evaluation image EI is obtained from the non-volatile storage device 130. However, the CPU 110 may also obtain the evaluation image EI by generating it in this step. For example, in this case, the non-volatile storage device 130 stores an image of the label TL (for example, a design drawing of the label TL or a print-ready image used for printing the label TL) and multiple captured images. The CPU 110 then combines the captured images and the image of the label TL to generate the evaluation image EI.

[0050] In S210, the CPU 110 selects one of several evaluation images EI to focus on. In S215, the CPU 110 inputs the selected evaluation image to the object detection model AN and generates multiple output data OD corresponding to the evaluation image.

[0051] In S220, the CPU 110 determines whether the detection of label TL was successful based on the output data OD. Specifically, the CPU 110 determines whether the output data OD contains a confidence level Vc equal to or greater than a predetermined threshold THc. Specifically, the CPU 110 determines whether the following (1) and (2) are satisfied.

[0052] (1) The output data OD includes a confidence level Vc that is equal to or greater than a predetermined threshold THc. (2) Class information corresponding to a confidence level Vc that is equal to or greater than the judgment threshold THc indicates that the object is a label TL included in the evaluation image of interest.

[0053] The CPU 110 determines that the detection of the label TL was successful if both (1) and (2) are satisfied. The CPU 110 determines that the detection of the label TL was unsuccessful if either (1) or (2) is not satisfied. If (1) is not satisfied, it means that the label TL could not be detected, i.e., it was not detected. If (1) is satisfied but (2) is not satisfied, it means that the existence of a label was detected, but the type of label detected was incorrect, i.e., it was a false positive. If there are multiple types of label TLs to be detected, the evaluation image EI is associated with information indicating the type of label TL, so the CPU 110 determines whether or not (2) is satisfied based on this information.

[0054] If the detection of label TL fails (S220:NO), CPU110 records in S225 that the detection failed (either not detected or falsely detected) in the evaluation record table RT.

[0055] Figure 8 shows an example of an evaluation record table RT. In the evaluation record table RT, the detection result of the label TL of the evaluation image EI is recorded in association with the ID of the evaluation image EI. Row R1 in Figure 8 records that the detection result of the label TL of evaluation image EI with ID "1" was either not detected or a false detection.

[0056] If the detection of label TL is successful (S220:YES), CPU 110 performs pattern matching on the detected label region LA in S230. The detected label region LA is the region output by the object detection model AN as the detection result of label TL, and is therefore also called the output region.

[0057] Specifically, the CPU 110 identifies the predicted region BB as the label region LA among the (S × S × B) predicted regions BB indicated by the output data OD, where the corresponding confidence level Vc is greater than or equal to a predetermined judgment threshold THc, and where the object within the region is predicted to be the label TL based on the corresponding class information. The CPU 110 then performs pattern matching (also called template matching) on ​​the label region LA among the evaluation images of interest, using a reference image representing the label TL as a template. The reference image may be, for example, a design drawing of the label TL or a print-ready image used for printing the label TL.

[0058] Pattern matching is a process that, for example, searches for the position, size, and angle of a template relative to its label region LA by brute force, while varying its position, size, and angle. The matching region MA is determined by pattern matching. The matching region MA is the region where the template is located when the match rate between the template and the label region LA is maximized during pattern matching. In other words, the matching region MA is the region in the label region LA where the label TL found by pattern matching is located.

[0059] Furthermore, the matching region MA explored by pattern matching and the template are obtained as the matching similarity MP. In this embodiment, the matching similarity MP takes a value between 0 and less than 1, and a larger value indicates a higher similarity between the template and the matching region MA.

[0060] In S240, the CPU 110 determines whether the matching similarity MP obtained by performing pattern matching is greater than or equal to the first threshold THm. The first threshold THm is a predetermined value, which in this embodiment is 0.9.

[0061] The identified label region LA is exemplified on the evaluation image EI in Figure 6(B). As shown in Figure 6(B), the label TL and the label region LA often do not perfectly coincide. Figure 9 shows an example of the label TL and label region LA in the first embodiment.

[0062] In the example in Figure 9(A), the left-hand side LS of label region LA is located to the left of the left-hand side LL of label TL. The top-hand side US of label region LA is located above the top-hand side LU of label TL. The bottom-hand side BS of label region LA is located below the bottom-hand side LB of label TL. In other words, the left-hand side LS, top-hand side US, and bottom-hand side BS of label region LA are all located outside label TL. On the other hand, the right-hand side RS of label region LA is located to the left of the right-hand side LR of label TL. In other words, the right-hand side RS of label region LA passes through the interior of label TL. For this reason, the rightmost part of label TL (the hatched area in Figure 9(A)) is not included in label region LA. Pattern matching is performed only on label region LA. For this reason, in the example in Figure 9(A), since part of label TL is not included in label region LA, the matching region MA is the region of only part of label TL, i.e., the region of label TL excluding the hatched area in Figure 9(A). Therefore, the matching similarity MP decreases as the area of ​​the label TL that is not included in the label region LA (the hatched area in Figures 9(A) and (B)) increases. For example, in the example in Figure 9(A), the area of ​​the label TL that is not included in the label region LA is larger than the standard, so the matching similarity MP is judged to be less than the first threshold THm.

[0063] In the example in Figure 9(B), the bottom edge BS and the right edge RS of label region LA are located outside label TL, respectively. The left edge LS and the top edge LU of label region LA pass through the interior of label TL, respectively. For this reason, the left and top edges of label TL (the hatched areas in Figure 9(B)) are not included in label region LA. Therefore, in the example in Figure 9(B), the matching region MA is only a portion of label TL, i.e., the area of ​​label TL excluding the hatched areas in Figure 9(B). In the example in Figure 9(B), because the area of ​​the portion of label TL not included in label region LA is larger than the criterion, the matching similarity MP is judged to be less than the first threshold THm.

[0064] In the examples in Figures 9(C) and (D), the top edge US, bottom edge BS, right edge RS, and left edge LS of the label region LA are located outside the label region TL. For this reason, the entire label region TL is contained within the label region LA. Therefore, in the examples in Figures 9(C) and (D), the matching region MA is the region where the entire label region TL is located. In this case, the matching similarity MP becomes sufficiently large. Therefore, in the examples in Figures 9(C) and (D), the matching similarity MP is judged to be greater than or equal to the first threshold THm.

[0065] If the matching similarity MP is less than the first threshold THm, the label region LA is not detected to include most of the label TL, and the label region LA is detected as excessively small. In other words, in this case, the accuracy of the label region LA shown by the output data OD (hereinafter also referred to as label region accuracy) is considered to be lower than the standard. For this reason, if the matching similarity MP is less than the first threshold THm (S240:NO), the CPU 110 records in S260 that the label region accuracy is lower than the standard in the evaluation record table RT. Row R2 in Figure 8 records that the label region accuracy is lower than the standard as the detection result of the label TL of the evaluation image EI with ID "99".

[0066] If the matching similarity MP is equal to or greater than the first threshold THm (S240: YES), the CPU 110 calculates the distance D between each edge of the detected label region LA and the corresponding edge of the matching region MA in S250.

[0067] Figures 9(C) and (D) illustrate the distances D(Du, Db, Dl, Dr) between the top edge US, bottom edge BS, left edge LS, and right edge RS of the label region LA and the corresponding edges of the matching region MA. When the accuracy of pattern matching is sufficiently high, it can be said that the distance D is the distance between each detected edge of the label region LA and the corresponding edge of the label TL within the label region LA. In this embodiment, the accuracy of pattern matching is sufficiently high. For this reason, in Figures 9(C) and (D), the matching region MA is illustrated as the region of label TL within the label region LA.

[0068] As shown in Figures 9(C) and (D), the CPU 110 identifies the representative points Pl of the top edge US, bottom edge BS, left edge LS, and right edge RS of the label region LA, and the representative points Pm of the corresponding edges of the matching region MA. The CPU 110 calculates the distance between the representative points Pl of the top edge US, bottom edge BS, left edge LS, and right edge RS and the representative points Pm of the corresponding edges as the distance D(Du, Db, Dl, Dr). In this embodiment, the representative points Pm of each edge of the matching region MA are the midpoints of each edge (points that bisect each edge). The representative points Pl of each edge of the label region LA are the intersection points of the line that passes through the representative point Pm of the corresponding edge of the matching region MA and is perpendicular to the corresponding edge of the matching region MA, and each edge of the label region LA.

[0069] In S255, the CPU 110 determines whether the distance D for all sides of the label area LA is less than the second threshold THd. The second threshold THd is a predetermined value, for example, set to a length of about 1% to 10% of the size of the label TL to be detected (length of the long side or short side).

[0070] In the example in Figure 9(C), the distances Du, Db, and Dl between the top edge US, bottom edge BS, and left edge LS of the label region LA and the corresponding edges of the matching region MA are less than the second threshold THd. In the example in Figure 9(C), the distance Dr between the right edge RS of the label region LA and the corresponding edge of the matching region MA is greater than or equal to the second threshold THd. For this reason, in the example in Figure 9(C), it is determined that the distance D for at least one edge of the label region LA is greater than or equal to the second threshold THd.

[0071] In the example in Figure 9(D), the distances Du, Dl, Db, and Dr between the top edge US, bottom edge BS, left edge LS, and right edge RS of the label region LA and the corresponding edges of the matching region MA are all less than the second threshold THd. For this reason, in the example in Figure 9(D), it is determined that the distance D for all edges of the label region LA is less than the second threshold THd.

[0072] If the distance D for at least one side of the label region LA is greater than or equal to the second threshold THd, the label region LA is detected as excessively larger than the label TL, and the label region accuracy is considered to be lower than the standard. For this reason, if the distance D for at least one side of the label region LA is greater than or equal to the second threshold THd (S255:NO), the CPU 110 records in S260 that the label region accuracy is lower than the standard in the evaluation record table RT.

[0073] If the distance D for all sides of the label region LA is less than the second threshold THd, then the label region LA has been detected to appropriately represent the area where the label TL is located, and the label region accuracy is considered to be above the standard. For this reason, if the distance D for all sides of the label region LA is less than the second threshold THd (S255: YES), the CPU 110 records in S265 that the label region accuracy is above the standard in the evaluation record table RT. A label region accuracy above the standard means that the label region accuracy is satisfactory. Row R3 in Figure 8 records that the detection result of the label TL in the evaluation image EI with ID "100" is satisfactory, meaning that the label region accuracy is satisfactory and above the standard.

[0074] In S270, the CPU 110 determines whether all evaluation image EIs have been processed. If not all evaluation image EIs have been processed (S270: NO), the CPU 110 returns to S210 and selects the next evaluation image EI as the evaluation image of interest. If all evaluation image EIs have been processed (S270: YES), the CPU 110 proceeds to S275.

[0075] In S275, CPU110 calculates the accuracy pass rate Rap for the object detection model AN by referring to the evaluation record table RT. The accuracy pass rate Rap is the value obtained by dividing the number of evaluation images EIs Mg that were judged to have label area accuracy above the standard during the model evaluation process by the total number of evaluation images EIs Mt used in the model evaluation process (Rap = (Mg / Mt)). In other words, the accuracy pass rate Rap is the percentage of evaluation images that were judged to have acceptable label area accuracy. Once the accuracy pass rate Rap is calculated, the model evaluation process is terminated.

[0076] A-5. Configuration and training of the image generation model GN The image generation model GN is a so-called autoencoder, comprising an encoder and a decoder (not shown in the diagram). The encoder performs dimensionality reduction on the input image to extract features from the input image and generate feature data. The decoder performs dimensionality restoration on the feature data to generate an output image. The size of the output image is the same as the size of the input image.

[0077] The generative model (GN) is trained to produce reconstructed images that represent the features of the input image that indicate the label. For this reason, the output image generated by a trained GN is also called a reconstructed image. The reconstructed image is almost identical to the input image. Furthermore, a trained GN is trained to reconstruct only the features of normal labels. For this reason, for example, if an input image with a label containing defects is input to a trained GN, the generated reconstructed image can be expected to be an image with a normal label without defects. In other words, whether an image with a normal label or an image with a label containing defects is input to the GN, the reconstructed image will be an image that reproduces a normal label.

[0078] A-6. Inspection Processing Figure 10 is a flowchart of the inspection process. Figure 11 is an explanatory diagram of the inspection process. The inspection process is the process of checking whether the label L to be inspected is an abnormal product containing defects, etc., or a normal product that does not contain defects, etc. The inspection process is performed for each label L. The inspection process is started when a user (for example, an inspection worker) inputs a start command to the processing device 100 via the operation unit 150. For example, the user inputs a start command for the inspection process with the product 300 to which the label L to be inspected is attached placed in a predetermined position for photography using the imaging device 400.

[0079] In S300, the CPU 110 acquires image data showing a captured image containing the label L to be inspected (hereinafter also referred to as the inspected item). For example, the CPU 110 sends a shooting instruction to the shooting device 400, causing the shooting device 400 to generate a captured image, and acquires the image data from the shooting device 400. As a result, for example, the data of the captured image FI in Figure 11(A) is acquired. The captured image FI is an image showing the front surface F31 of the product and the label FL attached to the front surface F31. In order to distinguish the front surface and label of the product shown in the captured image FI from the actual front surface 31 and label L (Figure 2), the letters "F" are added to the beginning of the symbols, and these are referred to as front surface F31 and label FL. The label FL in the captured image FI may contain defects such as scratches.

[0080] In S305, the CPU 110 inputs the acquired captured image FI into the object detection model AN to identify a label region LA, which is a part of the captured image FI and contains the label FL. Specifically, the CPU 110 inputs the captured image FI into the object detection model AN as input image IIa (Figure 3) to generate output data OD (Figure 3) corresponding to the captured image FI. From the (S × S × B) predicted regions BB indicated by the output data OD, the CPU 110 identifies a predicted region BB in which the corresponding confidence level Vc is greater than or equal to a predetermined threshold THa, and in which the object within the region is predicted to be a label based on the corresponding class information. The CPU 110 identifies this predicted region BB as the label region LA. For example, in the example in Figure 11(A), a label region LA is identified within the captured image FI that includes the entire label FL and is almost circumscribing the label FL.

[0081] In S310, the CPU 110 generates a verification image SI using the captured image FI. Specifically, the CPU 110 extracts the label region LA from the captured image FI to generate the verification image SI. The CPU 110 performs a size adjustment process to enlarge or reduce the verification image SI as needed, adjusting its size to match the size of the input image of the image generation model GN. The verification images SI in Figures 11(B) and (C) show the image within the label region LA (i.e., the image of the label FL). Note that the label FLa in verification image SIa in Figure 11(B) is a normal product and does not contain defects such as scratches. The label FLb in verification image SIb in Figure 11(C) is an abnormal product and contains a linear scratch df.

[0082] In S315, CPU110 inputs the verification image SI into the trained image generation model GN and generates a reproduced image corresponding to the verification image SI. As described above, the reproduced image is an image that reproduces the label FL of the input verification image SI. For example, regardless of whether the input verification image SI is verification image SIa or SIb in Figure 11(B) or (C), the generated reproduced image will be a defect-free image like verification image SIa in Figure 11(B).

[0083] In S320, the CPU 110 generates a difference image DI using the verification image SI and the reconstructed image. For example, the CPU 110 calculates the difference value (v1-v2) between the component value v1 of a pixel in the verification image SI and the component value v2 of the corresponding pixel in the reconstructed image, and normalizes this difference value to a value in the range of 0 to 1. The CPU 110 calculates this difference value for each pixel and each color component, and generates a difference image DI using this difference value as the color value of the pixel.

[0084] Figures 11(D) and (E) show examples of differential image DIs. Differential image DIa in Figure 11(D) is a differential image generated when the input image is verification image SIa, which shows a normal product in Figure 11(B). Differential image DIa does not contain defects such as scratches. Differential image DIb in Figure 11(E) is a differential image DI generated when the input image is verification image SIb, which shows a defective product in Figure 11(C). Differential image DIb contains scratches dfd corresponding to scratches df contained in verification image SIb. Therefore, by referring to differential image DI, it is possible to identify, for example, the presence, location, size, and shape of defects contained in verification image SI.

[0085] In S325, the CPU 110 uses the difference image DI to identify abnormal pixels contained in the difference image DI. An abnormal pixel is, for example, a pixel among several pixels contained in the difference image DI whose RGB value of at least one component is greater than or equal to the threshold TH1. For example, if the difference image DIa in Figure 11(D) is the target of processing, no abnormal pixels are identified. If the difference image DIb in Figure 11(E) is the target of processing, several pixels constituting the defect dfd are identified as abnormal pixels.

[0086] In S330, the CPU 110 determines whether the number of abnormal pixels identified in the differential image DI is equal to or greater than the threshold TH2. If the number of abnormal pixels is less than the threshold TH2 (S330: NO), in S340, the CPU 110 determines that the label for the inspected product is normal. If the number of abnormal pixels is equal to or greater than the threshold TH2 (S330: YES), in S335, the CPU 110 determines that the label for the inspected product is defective. In S350, the CPU 110 displays the inspection result on the display unit 140 and terminates the inspection process. In this way, it is possible to determine whether an inspected product is normal or defective using the object detection model AN and the image generation model GN.

[0087] According to the first embodiment described above, the CPU 110 performs a training process (S110-S140 in Figure 5) and a model evaluation process (S160 in Figure 5) to evaluate the object detection model AN during training (Figure 5). In the training process, the CPU 110 inputs a training image IIa, which is the training image LI (Figure 6(A)), to the object detection model AN, generates output data OD corresponding to the training image LI (S120 in Figure 5), and adjusts the multiple parameters of the object detection model using the loss value calculated using the output data for training (S130 in Figure 5). In the model evaluation process, the CPU 110 inputs the evaluation image EI (Figure 6(B)), which is the evaluation input image IIa, to the object detection model AN and generates output data OD corresponding to the evaluation image EI (Figure 7S215). Based on the output data OD, the CPU 110 calculates the accuracy pass rate Rap, which is an index value indicating the detection accuracy of the label region LA by the object detection model AN (also called the region detection accuracy of the object detection model AN) (Figure 7S220-S275).

[0088] The CPU 110 controls the execution of the training process (S110-S140 in Figure 5) and the model evaluation process (S160 in Figure 5) as follows. The CPU 110 repeatedly executes the training process for a specified number of epochs, and then executes the model evaluation process (YES in S150, S160). If the accuracy pass rate Rap calculated in the model evaluation process is greater than or equal to the threshold Thr, that is, if the accuracy pass rate Rap shows a detection accuracy of a standard or higher (YES in S170 in Figure 5), the CPU 110 terminates the training of the object detection model AN without executing the training process for the next epoch. If the accuracy pass rate Rap is less than the threshold Thr, that is, if the accuracy pass rate Rap shows a detection accuracy of a standard or lower (NO in S170 in Figure 5), the CPU 110 continues training the object detection model AN by executing the training process for the next epoch, provided that it does not exceed a predetermined number of epochs of 500 or more (NO in S180 in Figure 5).

[0089] The object detection model AN may require, for example, high region detection accuracy. For instance, suppose that in the inspection process of this embodiment, the region detection accuracy of the object detection model AN is low, and in S305 of Figure 10, a label region LA that does not include part of label FL is detected. In this case, part of label FL is not included in the verification image SI (Figures 11(B) and (C)). Therefore, if a defect such as a scratch is included in part of label FL that is not included in the verification image SI, that defect will not be detected as an anomaly. Alternatively, suppose that in the inspection process of this embodiment, the region detection accuracy of the object detection model AN is low, and in S305 of Figure 10, a label region LA that is excessively large compared to label FL is detected. In this case, the proportion of parts other than label FL (e.g., background) in the verification image SI (Figures 11(B) and (C)) becomes excessively high. The image generation model GN is assumed to receive a verification image SI showing only label FL as input. For this reason, if a validation image SI with an excessively high proportion of parts different from the label FL is input to the image generation model GN, the validation image SI may not be properly reproduced in the regenerated image, which can lead to a decrease in the accuracy of anomaly detection.

[0090] On the other hand, during training of the object detection model AN, even if the loss value falls below a predetermined threshold, the region detection accuracy of the object detection model AN may not be sufficient. Therefore, if the decision to terminate training of the object detection model AN is based on the loss value, there is a possibility that training may be terminated when the region detection accuracy of the object detection model AN is insufficient. Also, even if the loss value is greater than a predetermined threshold, the region detection accuracy of the object detection model AN may be sufficient. Therefore, if the decision to terminate training of the object detection model AN is based on the loss value, there is a possibility that training will continue even though the region detection accuracy of the object detection model AN is sufficient, leading to problems such as overfitting and unnecessary training. According to this embodiment, after repeatedly executing the training process for a specific number of epochs, the training of the object detection model is terminated when the accuracy pass rate Rap calculated in the model evaluation process shows a region detection accuracy of a standard or higher. This prevents excessive training from being performed even when the object detection accuracy is sufficient. Furthermore, if the accuracy pass rate Rap calculated during the model evaluation process indicates an area detection accuracy below a certain threshold, training of the object detection model continues. This prevents training from being terminated prematurely when the area detection accuracy is insufficient.

[0091] Furthermore, according to this embodiment, if the CPU 110 has executed training for a predetermined number of epochs (500 or more, indicated as YES in S180 of Figure 5), it terminates training of the object detection model AN regardless of the accuracy pass rate Rap.

[0092] It has been empirically observed that once training has been completed for more than 500 epochs, further improvement in region detection accuracy is unlikely. This embodiment helps to suppress unnecessary training.

[0093] Furthermore, according to this embodiment, the multiple evaluation images EI used in the nth model evaluation process and the multiple evaluation images EI used in the (n+1)th model evaluation process are the same image (S205 in Figure 7). In this way, the evaluation images EI are reused in multiple model evaluation processes, so the number of evaluation images EI required can be reduced.

[0094] Furthermore, according to this embodiment, in the model evaluation process, the CPU 110 performs pattern matching on the label region LA indicated by the output data OD corresponding to the evaluation image EI to determine the matching region MA (S230 in Figure 7), and calculates the accuracy pass rate Rap based on the matching region MA (S240-S265, S275 in Figure 7). As a result, by determining the matching region MA through pattern matching, the accuracy of the label region can be evaluated based on the matching region MA. Therefore, for example, even if the evaluation image EI does not have corresponding ground truth region information for the label region LA, the accuracy of the label region LA indicated by the output data OD can be evaluated.

[0095] More specifically, the CPU 110 determines the matching similarity MP, which indicates the similarity between the reference image (e.g., layout image) showing the label TL and the matching region MA, and the distance D between each edge of the label region LA and the corresponding edge of the matching region MA (S230, S250 in Figure 7). The CPU 110 determines that the label region accuracy for one evaluation image EI is acceptable if the matching similarity MP is greater than or equal to the first threshold THm and the distance D is less than the second threshold THd (YES in S240, YES in S255, and S265 in Figure 7). As a result, the label region accuracy for each evaluation image EI can be appropriately evaluated based on the matching region MA.

[0096] Furthermore, the CPU 110 calculates the distance D using a representative point Pl on an edge of the label region LA and a representative point Pm on the corresponding edge of the matching region MA. As a result, the distance D between each edge of the label region LA and the corresponding edge of the matching region MA can be easily calculated.

[0097] As can be seen from the above explanation, the label in this embodiment is an example of an object, and the label area LA is an example of the object area and output area. Also, the accuracy pass rate Rap in this embodiment is an example of the application of the index value 1.

[0098] B. Second Example In the second embodiment, the content of the model evaluation process S160 in Figure 5 differs from that of the first embodiment. The other configurations of the second embodiment are the same as those of the first embodiment. The model evaluation process of the second embodiment will be described below.

[0099] Figure 12 is a flowchart of the model evaluation process in the second embodiment. In Figure 12, processes identical to those in Figure 7 are denoted by the same reference numerals, while processes different from those in Figure 7 have a "B" appended to the end of their reference numerals. In the model evaluation process in Figure 12, S205B is executed instead of S205 in Figure 7, and S230B, S240B, S250B, and S255B are executed instead of S230, S240, S250, and S255 in Figure 7. Other processes in the model evaluation process in Figure 12 are identical to those in Figure 7. The following describes the differences between the model evaluation process in Figure 12 and Figure 7.

[0100] In S205B, CPU 110 acquires multiple evaluation images EI (for example, tens to hundreds of them) and ground truth region information associated with each evaluation image EI. Ground truth region information indicates the position and size of the ground truth region TBb (Figure 6(B)) within the evaluation image EI. The ground truth region TBb is the area within the evaluation image EI where the label TL is placed, for example, a rectangular area circumscribing the label TL. In other words, it is the ideal label area that the output data OD of the object detection model AN should indicate when the evaluation image EI is input to the object detection model AN. The ground truth region information defines the ground truth region TBb and includes the coordinates (Xtb, Ytb) of the center (centroid) Ctb of the ground truth region TBb, the width (length in direction Dx) Wtb, and the height (length in direction Dy) Htb.

[0101] The ground truth region information for the evaluation image EI is, for example, created in advance and recorded in the non-volatile storage device 130 in association with the evaluation image EI. The ground truth region information for the evaluation image EI may also be created based on operator operations, similar to the ground truth region information included in the training data of the training image LI described above. If the evaluation image EI is created by compositing an image of label TL as a foreground to an image captured by a digital camera, the ground truth region information may be created by recording information indicating the region in which the label TL image is composited as ground truth region information when compositing the label TL image.

[0102] In S230B, which is executed if the detection of label TL is successful (S220:YES), the CPU 110 identifies each edge of the ground truth region TBb based on the ground truth region information associated with the evaluation image EI.

[0103] Figure 13 shows an example of the label TL and label region LA in the second embodiment. Figure 13 shows the same positional relationship as the label TL and label region LA in Figure 9. In the first embodiment, the region where the label TL is located in the evaluation image EI is recognized by determining the matching region MA through a matching process, but in the second embodiment, the region where the label TL is located is recognized by identifying each side of the correct region TBb based on the correct region information. That is, in the second embodiment, the correct region TBb correctly indicates the region where the label TL is located in the evaluation image EI, so identifying each side of the correct region TBb means identifying the top side LU, bottom side LB, right side LR, and left side LL of the label TL in the evaluation image EI.

[0104] In S240B, the CPU 110 identifies the label region LA detected by the object detection model AN based on the output data OD. For example, as described above, among the (S × S × B) predicted regions BB indicated by the output data OD, the predicted region BB in which the corresponding confidence level Vc is greater than or equal to a predetermined decision threshold THc, and in which the object within the region is predicted to be labeled TL based on the corresponding class information, is identified as the label region LA.

[0105] In S250B, CPU110 calculates the distance Db between each edge of the detected label region LA and the corresponding edge of the ground truth region TBb.

[0106] Figures 13(A)-(D) illustrate the distances Db (Dbu, Dbb, Dbl, Dbr) between the top edge US, bottom edge BS, left edge LS, and right edge RS of the label region LA and the corresponding edges of the ground truth region TBb. If the ground truth region TBb accurately represents the region where the label TL is located, then the distance Db can be said to be the distance between each edge of the detected label region LA and the corresponding edge of the label TL in the evaluation image EI. In this embodiment, the ground truth region TBb accurately represents the region where the label TL is located. For this reason, in Figures 13(A)-(D), the ground truth region TBb is illustrated as the region of the label TL.

[0107] As shown in Figures 13(A)-(D), the CPU 110 identifies the representative points Pl of the top edge US, bottom edge BS, left edge LS, and right edge RS of the label region LA, and the representative points Pb of the corresponding edges of the correct answer region TBb. The CPU 110 calculates the distance between the representative points Pl of the top edge US, bottom edge BS, left edge LS, and right edge RS and the representative points Pb of the corresponding edges as the distance D(Dbu, Dbb, Dbl, Dbr). In this embodiment, the representative points Pb of each edge of the correct answer region TBb are the midpoints of each edge (points that bisect each edge). The representative points Pl of each edge of the label region LA are the intersection points of the line that passes through the representative point Pb of the corresponding edge of the correct answer region TBb and is perpendicular to the corresponding edge of the correct answer region TBb, and each edge of the label region LA.

[0108] In S255B, the CPU 110 determines whether the distance Db for all sides of the label region LA is less than the threshold THb. The threshold THb is set to a length of approximately 1% to 10% of the size of the label TL to be detected (length of the long side or short side), similar to the second threshold THd in the first embodiment.

[0109] In the examples in Figures 13(A) and (C), the distances Dbu, Dbb, and Dbl between the top edge US, bottom edge BS, and left edge LS of the label region LA and the corresponding edges of the correct answer region TBb are less than the threshold THb. In the examples in Figures 13(A) and (C), the distance Dbr between the right edge RS of the label region LA and the corresponding edge of the correct answer region TBb is greater than or equal to the threshold THb. For this reason, in the examples in Figures 13(A) and (C), it is determined that the distance Db for at least one edge of the label region LA is greater than or equal to the threshold THb.

[0110] In the example in Figure 13(B), the distance Dbb between the bottom edge BS of the label region LA and the corresponding edge in the correct answer region TBb is less than the threshold THb. In the example in Figure 13(B), the distances Dbu, Dbr, and Dbl between the top edge US, right edge RS, and left edge LS of the label region LA and the corresponding edges in the correct answer region TBb are greater than or equal to the threshold THb. For this reason, in the example in Figure 9(B), it is determined that the distance Db for at least one edge of the label region LA is greater than or equal to the threshold THb.

[0111] In the example in Figure 13(D), the distances Dbu, Dbl, Dbb, and Dbr between the top edge US, bottom edge BS, left edge LS, and right edge RS of the label region LA and the corresponding edges of the correct answer region TBb are less than the threshold THb. For this reason, in the example in Figure 13(D), it is determined that the distance Db for all edges of the label region LA is less than the threshold THd.

[0112] If the distance Db for at least one side of the label region LA is greater than or equal to the threshold THb, then the label region LA cannot be said to accurately represent the region where the label TL is located. For example, the label region LA may be detected as excessively larger than the label TL, or it may be detected as excessively smaller than the label TL, in which case the label region accuracy is considered to be lower than the standard. For this reason, if the distance Db for at least one side of the label region LA is greater than or equal to the threshold THb (S255B:NO), the CPU 110 records in S260 that the label region accuracy is lower than the standard in the evaluation record table RT.

[0113] If the distance Db for all edges of the label region LA is less than the threshold THb, then the label region LA has been detected to appropriately represent the region where the label TL is located, and the label region accuracy is considered to be above the standard. For this reason, if the distance Db for all edges of the label region LA is less than the threshold THb (S255B: YES), the CPU 110 records in S265 that the label region accuracy is above the standard, in other words, that the label region accuracy is satisfactory, in the evaluation record table RT.

[0114] According to the second embodiment described above, during training of the object detection model AN, the CPU 110 performs a comparison process (S250B, S255B, S260, S265 in Figure 12) that compares the label region LA indicated by the output data corresponding to the evaluation image EI with the ground truth region TBb indicated by the ground truth region information, and calculates the accuracy pass rate Rap based on the results of the comparison process (S275 in Figure 12). As a result, by using the ground truth region information, the label region accuracy in the evaluation image EI and the region detection accuracy by the object detection model AN can be easily evaluated.

[0115] Specifically, the comparison process includes calculating the distance Db between each edge of the label region LA and the corresponding edge of the ground truth region TB (S250B in Figure 12), and determining that the accuracy of the label region LA indicated by the output data OD corresponding to the evaluation image EI is acceptable if the distance Db is less than the threshold THb (YES in S250B in Figure 12) (S265 in Figure 12). As a result, the accuracy of the label region LA indicated by the output data corresponding to the evaluation image EI can be appropriately evaluated using the ground truth region information.

[0116] Furthermore, in this embodiment, the distance Db is calculated using a representative point Pl on an edge of the label region LA and a representative point Pb on the corresponding edge of the ground truth region TBb (Figure 13). As a result, with this configuration, the distance between each edge of the output region and the corresponding edge of the ground truth region can be easily calculated.

[0117] C. Variations (1) In each of the above embodiments, the method for evaluating the region detection accuracy of the object detection model AN based on the output data OD corresponding to the evaluation image EI is an example and is not limited thereto. For example, the CPU 110 may calculate the distance between each of the four vertices of the matching region MA and the corresponding vertex of the detected label region LA, and calculate the sum or average of these distances as the evaluation value for the label region accuracy of each evaluation image EI. Alternatively, the CPU 110 may calculate the distance between each of the four vertices of the ground truth region TBb and the corresponding vertex of the detected label region LA, and calculate the sum or average of these distances as the evaluation value for the label region accuracy of each evaluation image EI. The CPU 110 may also calculate the average value of the evaluation values ​​calculated for each of the multiple evaluation images EI as an index value for the region detection accuracy of the object detection model AN. In this case, the CPU 110 may terminate the training of the object detection model AN if the index value is less than or equal to a predetermined threshold, and may continue training the object detection model AN if the index value is greater than a predetermined threshold.

[0118] Furthermore, the CPU 110 may calculate, for example, the area S1 where the correct region TBb and the detected label region LA overlap, and use this area S1 to determine whether the label region accuracy of each evaluation image EI is satisfactory. For example, the CPU 110 calculates the ratio Rtb of the area S1 to the area Stb of the correct region TBb, and the ratio Rla of the area S1 to the area Sla of the detected label region LA. The CPU 110 may then determine that the label region accuracy of the evaluation image EI is satisfactory if the ratio Rtb is greater than or equal to a predetermined threshold (e.g., 0,9) and the ratio Rla is greater than or equal to a predetermined threshold (e.g., 0,9).

[0119] (2) In the training of the object detection model AN in each of the above embodiments, if training for a predetermined number of epochs of 500 or more is completed, the training of the object detection model AN is terminated even if the accuracy pass rate Rap is less than the threshold Thr. However, this is not limited to this, for example, if the trend of change in the loss value changes from a decreasing trend to an increasing trend, the training of the object detection model AN may be terminated even if the accuracy pass rate Rap is less than the threshold Thr.

[0120] (3) In each of the above embodiments, the same evaluation image EI is reused in multiple model evaluation processes, but a different evaluation image EI may be used each time the model evaluation process is executed. For example, the CPU 110 may generate and use an evaluation image EI by combining the captured image and the label TL layout image each time the model evaluation process is executed.

[0121] (4) In the first embodiment described above, the distance D is the distance between a representative point Pl on an edge of the label region LA and a representative point Pm on the corresponding edge of the matching region MA. Alternatively, the distance D may be the average of the distances between multiple points on an edge of the label region LA and the corresponding edge of the matching region MA. Alternatively, the distance D may be the average of the distances between multiple points on an edge of the matching region MA and the corresponding edge of the label region LA. The same applies to the distance Db in the second embodiment.

[0122] (5) The configuration of the object detection model AN used in each of the above embodiments is an example, and other models may be used. For example, any other model may be used instead of the object detection model AN or the YOLO model. The object detection model may be an improved YOLO model such as "YOLO v3", "YOLO v4", or "YOLO v5". In addition, other models such as SSD, R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN may be used.

[0123] (6) The prediction region BB (label region LA) and the ground truth regions TB and TBb used in each of the above embodiments are rectangular regions, but are not limited to these, and may be circular or polygonal regions other than rectangles (for example, hexagons or octagons).

[0124] (7) The objects to be detected by the object detection model AN are not limited to labels attached to products (e.g., multifunction printers, sewing machines, cutting machines, mobile terminals, etc.), but may be any object. The objects to be detected may be, for example, the label image printed on the product. The objects to be detected may also be any part of the product, such as tags, accessories, parts, markings, nameplates, etc. that are attached to the product. Furthermore, the objects to be detected by the object detection model AN may be living things such as people, animals, and plants, or man-made things such as cars, buildings, and industrial products, or natural things such as stones and mountains.

[0125] (8) In each of the above embodiments, the object detection model AN is used to identify the label region LA in the inspection process of label L. However, the object detection model AN may be used for other purposes. For example, the object detection model AN may be used to identify the face region in a captured image in order to perform a predetermined image processing (e.g., skin tone correction) on the face region in the captured image. Even in this case, according to this embodiment, the object detection model AN can be trained to identify the face region that accurately encloses the entire face portion in the captured image, so that the predetermined image processing can be applied to the entire face portion.

[0126] (9) In the above embodiment, the training of the object detection model AN and the inspection process using the object detection model AN are performed by the processing unit 100 shown in Figure 1. Alternatively, the training and inspection processes may be performed by separate devices. In this case, for example, the object detection model AN trained by the processing unit 100 is stored in the memory of the device that performs the inspection process. Furthermore, all or part of the training and inspection processes may be performed by multiple computers (for example, so-called cloud servers) that can communicate with each other via a network. Also, the computer program that performs the inspection process and the computer program that trains the object detection model AN may be different computer programs.

[0127] (10) In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, all or part of the training and inspection process of the object detection model AN may be performed by a hardware circuit such as an ASIC (Application Specific Integrated Circuit).

[0128] The present invention has been described above based on examples and modifications. However, the embodiments of the invention described above are for the purpose of facilitating understanding of the present invention and do not limit it. The present invention can be modified and improved without departing from its spirit and claims, and the present invention includes equivalents thereof. [Explanation of Symbols]

[0129] 1000…Inspection system, 100…Processing device, 110…CPU, 120…Volatile memory device, 130…Non-volatile memory device, 140…Display unit, 150…Operation unit, 170…Communication unit, 30…Enclosure, 300…Product, 400…Imaging device, AN…Object detection model, BB…Prediction region, EG…Evaluation image data set, EI…Evaluation image, FI…Captured image, L, FL, TL…Label, GN…Image generation model, IIa…Input image, LA…Label region, LG…Training image data set, LI…Training image, MA…Matching region, MP…Matching similarity, OD…Output data, PG…Computer program, RT…Evaluation record table, Rap…Accuracy pass rate, SI…Verification image, TB, TBb…Correct answer region, TG…Training data set

Claims

1. It is a computer program, A training process for training an object detection model that performs an arithmetic operation using multiple parameters on an input image containing an object to generate output data that indicates at least the object region within the input image, The object detection model is given the input image for training, and the output data for training corresponding to the input image for training is generated. The multiple parameters of the object detection model are adjusted using the loss values ​​calculated using the output data for training. The aforementioned training process, An evaluation process for evaluating the object detection model, The object detection model is input to the input image for evaluation, and the output data for evaluation corresponding to the input image for evaluation is generated. Based on the output data for evaluation, an index value indicating the detection accuracy of the object region by the object detection model is calculated. The evaluation process described above, A control process that controls the execution of the training process and the evaluation process, After repeatedly executing the training process for a specific number of epochs, the evaluation process is executed. If the index value calculated in the evaluation process indicates a detection accuracy equal to or greater than the standard, the training of the object detection model is terminated without executing the training process for the next epoch. If the index value calculated in the evaluation process indicates a detection accuracy below the standard, the training process for the next epoch is executed to continue training the object detection model. The control process described above, A computer program that enables a computer to realize something.

2. A computer program according to claim 1, The control process is a computer program that, when it has executed the training process for a predetermined number of epochs (500 or more), terminates the training of the object detection model regardless of the index value.

3. A computer program according to claim 1, The control process executes the nth evaluation process (where n is an integer of 1 or more), and the (n+1)th evaluation process executed after multiple training processes executed after the nth evaluation process, A computer program in which one or more input images for evaluation used in the nth evaluation process and one or more input images for evaluation used in the (n+1)th evaluation process include the same image.

4. A computer program according to claim 1, The input image for evaluation is associated with ground truth region information indicating the position and size of the object region within the input image for evaluation. The evaluation process includes a comparison process that compares the output region, which is the object region indicated by the output data for evaluation, with the correct region, which is the object region indicated by the correct region information. The evaluation process is a computer program that calculates the index value based on the results of the comparison process.

5. A computer program according to claim 4, The aforementioned object region is a rectangular region, The aforementioned comparison process is, A process for calculating the distance between each edge of the output region and the corresponding edge of the correct answer region, If the distance is less than a threshold, the process determines that the accuracy of the object region indicated by the output data for evaluation is acceptable. Includes, The aforementioned index value is the accuracy pass rate, which is the percentage of cases in which the accuracy of the object region is judged to be acceptable. The control process is a computer program that terminates training of the object detection model when the accuracy pass rate is above a threshold, and continues training of the object detection model when the accuracy pass rate is below a threshold.

6. A computer program according to claim 5, The distance is calculated using a representative point on the edge of the output region and a representative point on the corresponding edge of the correct answer region, in a computer program.

7. A computer program according to claim 1, The evaluation process includes a process to determine a matching region by performing pattern matching using a reference image representing the object on the output region, which is the object region indicated by the output data for evaluation, The evaluation process is a computer program that calculates the index value based on the matching region.

8. A computer program according to claim 7, The aforementioned object region is a rectangular region, The aforementioned evaluation process is, A process for determining the similarity between the reference image and the matching region, and the distance between each side of the output region and the corresponding side of the matching region. A process to determine that the accuracy of the object region indicated by the output data for evaluation is acceptable when the similarity is equal to or greater than a first threshold and the distance is less than a second threshold, A computer program that includes [this].

9. A computer program according to claim 8, The distance is calculated using a representative point on the edge of the output region and a representative point on the corresponding edge of the matching region, in a computer program.

10. A training step for training an object detection model that performs an arithmetic process using multiple parameters on an input image containing an object to generate output data that indicates at least the object region within the input image, The object detection model is given the input image for training, and the output data for training corresponding to the input image for training is generated. The multiple parameters of the object detection model are adjusted using the loss values ​​calculated using the output data for training. The aforementioned training process, An evaluation step for evaluating the object detection model, The object detection model is input to the input image for evaluation, and the output data for evaluation corresponding to the input image for evaluation is generated. Based on the output data for evaluation, an index value indicating the detection accuracy of the object region by the object detection model is calculated. The evaluation process described above, Equipped with, After repeatedly performing the training process for a specific number of epochs, the evaluation process is performed. If the index value calculated in the evaluation step indicates a detection accuracy equal to or greater than the standard, the training of the object detection model is terminated without performing the training step for the next epoch. A method for training an object detection model, wherein if the index value calculated in the evaluation step indicates a detection accuracy below a standard, the training step for the next epoch is executed to continue training the object detection model.

Images