Computer programs and methods for training object detection models.

Abstract

This invention provides a computer program for training a machine learning model that detects specific objects that have a glossy surface, and a method for training such a machine learning model. [Solution] A computer program for training a machine learning model that receives a read image obtained by optically reading an object as input includes an image acquisition function to acquire an image of the target object, an image processing function that generates a processed image by performing image processing including masking, which replaces a part of the image showing a specific object with a monochrome mask image when the image shows a specific object that has gloss, and a training function that trains the machine learning model using the training image including the processed image.

Description

Technical Field

[0001] This specification relates to the training of a machine learning model into which a read image obtained by optically reading an object having gloss is inputted.

Background Art

[0002] Non-Patent Document 1 discloses an object detection model that executes detection of an object in an image and identification (classification) of the object as a machine learning model.

[0003] The training of the object detection model is executed so that the loss calculated using a predetermined calculation formula becomes small, using a training image including the object to be detected and teacher data indicating the correct answer of the detection result of the object.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] This specification discloses a new technique for training a machine learning model (for example, an object detection model).

Means for Solving the Problems

[0006] The technique disclosed in this specification is made to solve at least a part of the above-described problems, and can be realized as the following application examples.

[0007] [Application Example 1] A computer program for training a machine learning model that receives a read image obtained by optically reading an object as input, the computer program comprising: an image acquisition function for acquiring an object image; an image processing function that generates a processed image by performing image processing including masking, which replaces a portion of the object showing a specific object with a monochrome mask image when the object image shows a specific object that has gloss; and a training function that trains the machine learning model using a training image including the processed image.

[0008] For example, certain glossy objects reflect light specularly, so when an object is a specific object, the object shown in the read image is prone to overexposure. In this embodiment, a machine learning model is trained using training images that include processed images in which overexposure is simulated by replacing a part of the object with a monochrome mask image. As a result, the machine learning model can learn the characteristics of specific glossy objects, so that when a read image obtained by optically reading a specific object is input, the machine learning model can be trained to perform the desired function (e.g., object detection or object classification).

[0009] Furthermore, the technologies disclosed herein can be implemented in various forms, including methods for training machine learning models, apparatus for training machine learning models, systems for training machine learning models, computer programs for implementing these methods, apparatuses, and systems, object detection models trained by these methods, apparatuses, and systems, recording media storing the computer programs and object detection models, and so on. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing the configuration of the inspection system 1000 of the embodiment. [Figure 2] Diagram illustrating product 300. [Figure 3] A block diagram showing the configuration of the object detection model AN. [Figure 4] Flowchart for training the object detection model AN in the first embodiment. [Figure 5] A figure showing an example of an image used in the first embodiment. [Figure 6] Flowchart for generating glossy training images. [Figure 7] A figure showing an example of background image BI and training image LI. [Figure 8] Flowchart of the normal training image generation process. [Figure 9] Flowchart of the inspection process. [Figure 10] Diagram illustrating the inspection process. [Figure 11] A flowchart for training the object detection model AN in the second embodiment. [Figure 12] A figure showing an example of an image used in the third embodiment. [Figure 13] Flowchart of the gloss treatment in the third embodiment. [Figure 14] A figure showing an example of an image used in the fourth embodiment. [Figure 15] A flowchart for training the object detection model AN in the fourth embodiment. [Figure 16] A flowchart for hologram processing. [Modes for carrying out the invention]

[0011] A. First Example A-1. Configuration of the inspection device Next, embodiments will be described based on examples. Figure 1 is a block diagram showing the configuration of the inspection system 1000 of the embodiment. The inspection system 1000 includes an inspection device 100 and a digital camera 400. The inspection device 100 and the digital camera 400 are connected in a communication manner.

[0012] The inspection device 100 is a computer such as a personal computer, for example. The inspection device 100 includes a CPU 110 as a controller of the inspection device 100, a GPU 115, a volatile memory device 120 such as a RAM, a non-volatile memory device 130 such as a hard disk drive, an operation unit 150 such as a mouse and a keyboard, a display unit 140 such as a liquid crystal display, and a communication unit 170. The communication unit 170 includes a wired or wireless interface for communicably connecting to an external device, for example, a digital camera 400.

[0013] The GPU (Graphics Processing Unit) 115 is a processor that performs computational processing for image processing such as three-dimensional graphics according to the control of the CPU 110. In this embodiment, it is used to execute the arithmetic processing of the machine learning model.

[0014] The volatile memory device 120 provides a buffer area for temporarily storing various intermediate data generated when the CPU 110 performs processing. The non-volatile memory device 130 stores a computer program PG for the inspection device, a background image data group BG, and an underplate image data group OG. The background image data group BG and the underplate image data group OG will be described later.

[0015] The computer program PG includes, as modules, computer programs that enable the CPU 110 and the GPU 115 to cooperate to realize the functions of a plurality of machine learning models. The computer program PG is provided, for example, by the manufacturer of the inspection device 100. The computer program PG may be provided, for example, in a form downloaded from a server, or may be provided in a form stored in a DVD-ROM or the like. The CPU 110 executes the inspection process and the training of the object detection model AN, which will be described later, by executing the computer program PG.

[0016] The multiple machine learning models include two object detection models AN(AN1, AN2) and an image generation model GN. Object detection model AN1 is an object detection model for glossy labels. Object detection model AN2 is an object detection model for normal labels. Object detection models AN1 and AN2 have the same network configuration. Object detection models AN1 and AN2 are trained using different training images. These models will be discussed later.

[0017] The digital camera 400 is a digital camera that generates data of a read image representing a subject by optically reading the subject using a two-dimensional image sensor. The read image data is bitmap data that shows an image containing multiple pixels, and specifically, it is RGB image data that represents the color of each pixel by RGB values. The RGB values ​​are color values ​​of the RGB color system that include the gradation values ​​of three color components (hereinafter also called component values), namely R, G, and B values. The R, G, and B values ​​are values ​​of a predetermined number of gradations. In this embodiment, the R, G, and B values ​​are 256 gradation values ​​that take values ​​between 0 and 255. The read image data may also be data that represents the brightness of each pixel.

[0018] The digital camera 400 generates data of the read image according to the control of the inspection device 100 and transmits it to the inspection device 100. In this embodiment, the digital camera 400 is used to capture images of the product 300 to which the label L, which is the object of inspection for the inspection process, is attached, and to generate data of the read image.

[0019] Figure 2 is an explanatory diagram of product 300. Figure 2(A) shows a perspective view of product 300. In this embodiment, product 300 is a printer having a roughly rectangular prism-shaped housing 30. During the manufacturing process, a rectangular label L is affixed to a predetermined position on the front surface 31 (+Y side) of the housing 30.

[0020] Figure 2(B) shows two types of labels L1 and L2 as examples of labels L. Label L1 includes, for example, a background B1 and characters X1 and a mark M1 that indicate various information such as the manufacturer's or product's brand logo, model number, lot number, etc. Similarly, label L2 includes, for example, a background B2, characters X2, and a mark M2. The two types of labels L1 and L2 are, for example, labels affixed to different products, and at least some of the characters and marks are different from each other. In this embodiment, multiple labels, including the two types of labels L1 and L2, are subject to inspection.

[0021] Here, label L1 is a label that is glossy overall (also called a glossy label). Label L2 is a label that is not glossy overall (also called a normal label). Here, a specific object (for example, label L) being glossy means that the 60-degree specular gloss of the specific object is within the reference range RR. In this example, the reference range RR is 10 or more. The reference range RR is more preferably 50 or more, and even more preferably 70 or more. The 60-degree specular gloss is the gloss measured using the 60-degree specular gloss measurement method specified in JIS Z8741:1997 (Specular gloss - Measurement method). Hereafter, when simply referring to specular gloss, it means the 60-degree specular gloss described here.

[0022] A2. Configuration of the object detection model AN Figure 3 is a block diagram showing the configuration of the object detection model AN. Various object detection models can be used as the object detection model AN. In this embodiment, the object detection model AN is an object detection model called YOLO (You Only Look Once). YOLO is disclosed, for example, in the paper "Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788". The YOLO model uses a convolutional neural network to predict the region in an image where an object is located and the type of object located in that region.

[0023] As shown in Figure 3, the object detection model AN has m convolutional layers CV11-CV1m (where m is an integer greater than or equal to 1) and n fully connected layers CN11-CN1n (where n is an integer greater than or equal to 1) following the convolutional layers CV11-CV1m (where m is, for example, 24, and n is, for example, 2). A pooling layer is provided immediately after one or more of the m convolutional layers CV11-CV1m.

[0024] The convolutional layers CV11-CV1m perform processing on the input data, including convolution and bias addition. The convolution process applies t filters sequentially to the input data and calculates a correlation value that shows the correlation between the input data and the filters (t is an integer greater than or equal to 1). In the filter application process, multiple correlation values ​​are calculated sequentially while sliding the filters. The bias addition process adds a bias to the calculated correlation value. One bias is prepared for each filter. The dimension of the filter and the number of filters t usually differ among the m convolutional layers CV11-CV1m. Each convolutional layer CV11-CV1m has a parameter set that includes multiple weights and multiple biases for multiple filters.

[0025] The pooling layer performs a process to reduce the dimensionality of the data input from the preceding convolutional layer. Various pooling processes are available, such as average pooling and max pooling. In this embodiment, the pooling layer performs max pooling. Max pooling reduces the dimensionality by sliding a predetermined size window (e.g., 2x2) with a predetermined stride (e.g., 2) and selecting the maximum value within the window.

[0026] The fully connected layers CN11-CN1n take f-dimensional data (i.e., f values, where f is an integer greater than or equal to 2) input from the previous layer and output g-dimensional data (i.e., g values, where g is an integer greater than or equal to 2). Each of the g output values ​​is the inner product of a vector composed of the f input values ​​and a vector composed of f weights, with a bias added. The dimensions f of the input data and g of the output data usually differ among the n fully connected layers CN11-CN1n. Each fully connected layer CN11-CN1n has a parameter set that includes multiple weights and multiple biases.

[0027] The data generated by the convolutional layers CV11-CV1m and the fully connected layers CN11-CN1n are input to an activation function and transformed. Various functions can be used as activation functions. In this embodiment, a linear activation function is used for the last layer (here, the fully connected layer CN1n), and leaky-normalized linear units (LReLU) are used for the other layers.

[0028] This section outlines the operation of the object detection model AN. The object detection model AN receives data from the input image IIa. The object detection model AN performs calculations on the data from the input image IIa using the parameter set described above to generate output data OD. Output data OD is data containing multiple predicted values. Each predicted value includes predicted region information indicating the predicted region (also called a bounding box) where an object (label in this embodiment) is predicted to be located, and class information indicating the type (also called a class) of the object present in the predicted region.

[0029] The prediction region information includes information indicating the location and size of the prediction region (e.g., center coordinates, width, and height), and a confidence score Vc. The confidence score Vc indicates the probability that an object exists in the prediction region. The class information indicates the type of object present in the prediction region, expressed as a probability for each type. The class information includes C probability values ​​when classifying the object types into C types (where C is an integer greater than or equal to 1).

[0030] A-3. Training of the object detection model AN Figure 4 is a flowchart of the training of the object detection model AN in the first embodiment. In S110, the CPU 110 of the inspection device 100 acquires data for N layout images OI, each representing one of the N labels L that should be detected by the object detection model AN. N is an integer of 2 or more, for example, an integer of about 5 to 20. In this embodiment, the data for the N layout images OI is included in the layout image data group OG already stored in the non-volatile storage device 130, and is therefore acquired from the non-volatile storage device 130. The layout image data group OG is stored in the non-volatile storage device 130, for example, by the user.

[0031] Figure 5 shows an example of an image used in the first embodiment. Figure 5(A) shows the print-ready image OI1 of the glossy label L1 in Figure 2(B) as an example of a print-ready image OI. The data of the print-ready image OI is the data used to create the label L. For example, the glossy label L1 is created by printing the print-ready image OI1 on a glossy sheet for labels using the data of the print-ready image OI1. The print-ready image OI is bitmap data that represents an image containing multiple pixels, similar to the read image described above, and in this embodiment, it is RGB image data. For example, the print-ready image OI contains multiple pixels arranged in a matrix along a first direction Dx and a second direction Dy perpendicular to the first direction Dx.

[0032] In S115, the CPU 110 displays an input screen (not shown) on the display unit 140 and obtains information about the labels shown in N layout images OI from the user. The obtained label information includes information indicating whether the label shown in each of the N layout images OI is a glossy label or a normal label. If the label shown in the layout image OI is a glossy label, the obtained label information includes the specular gloss of the glossy label.

[0033] The input screen includes, for example, N layout images (OI) (i.e., label images), and N checkboxes and numerical input fields associated with the N layout images (OI). A check box is entered, for example, if the label shown in the corresponding layout image (OI) is a glossy label. The numerical input field is an input field that becomes available when the corresponding checkbox is checked, and is used to input the specular gloss level.

[0034] The user inputs information indicating a glossy label by checking the checkbox corresponding to the print-ready image OI that represents the glossy label, out of N checkboxes. When the user checks a checkbox, they also input the specular gloss level in the corresponding numerical input field. The CPU 110 obtains the label information entered by the user via the input screen.

[0035] In S120, the CPU 110 classifies N print-ready images OI into glossy label images and normal label images based on the label information input by the user. A glossy label image is an image that represents a glossy label (e.g., L1). A normal label image is an image that represents a normal label (e.g., L2). Whether a print-ready image OI represents a glossy label or a normal label may depend on, for example, the material of the sheet used for the label, and therefore may not be apparent from the print-ready image OI data alone. For this reason, the CPU 110 classifies the N print-ready images OI based on the label information input by the user. For example, let's assume that N print-ready images OI are classified into n1 glossy label images and n2 normal label images (n1 + n2 = N).

[0036] In S130, CPU110 executes the glossy training image generation process. The glossy training image generation process generates training images used to train the object detection model AN1 for glossy labels. The object detection model AN1 for glossy labels is assumed to receive a read image showing glossy labels as input image IIa, and in that case, it is an object detection model that detects glossy labels in the read image.

[0037] Figure 6 is a flowchart of the glossy training image generation process. The processes S210-S250 in Figure 6 are performed for each of the n1 glossy label images. Below, we will explain the process performed for one glossy label image OI (a print-ready image OI showing the glossy label).

[0038] In S210, the CPU 110 performs a pixel value inversion process on the glossy label image OI to generate an inverted glossy label image RI. The pixel value inversion process inverts the values ​​of pixels with a specific percentage Rr. The CPU 110 determines the specific percentage Rr of pixels to be inverted. In this embodiment, the specific percentage Rr is a predetermined fixed value of 50%. The specific percentage Rr is not limited to 50% and may be other values, preferably between 40% and 60%. The specific percentage Rr may be determined randomly within a predetermined range (for example, between 40% and 60%). The CPU 110 randomly selects pixels with a specific percentage Rr to be inverted from all the pixels constituting the glossy label image OI1. The CPU 110 inverts the pixel values ​​of each of the selected pixels to be inverted.

[0039] Pixel value inversion is performed by inverting each of the one or more component values ​​of the pixel according to the following equation (1). Vb = Vmax - Va …(1)

[0040] In equation (1), Vb is the value after inversion, and Va is the value before inversion. Vmax is the maximum value that it can take. In this embodiment, the value of each pixel in the glossy label image OI1 is an RGB value, and each RGB value can take a value between 0 and 255. For this reason, if the values ​​of each RGB component before inversion are Ra, Ga, and Ba, then the values ​​of each RGB component after inversion, Rb, Gb, and Bb, are given by the following equations (2)-(4).

[0041] Rb = 255 - Ra …(2) Gb = 255 - Ga …(3) Bb = 255 - Ba …(4)

[0042] Furthermore, let's assume that the glossy label image OI is a grayscale image, and that the luminance value Y of each pixel is between 0 and 255. In this case, if the luminance value before inversion is Ya, then the luminance value Yb after inversion is given by the following equation (5). Vb = 255 - Va …(5)

[0043] Through inversion, for example, black is converted to white. Also, chromatic colors are converted to their complementary colors, which have a hue that is 180 degrees different.

[0044] Figure 5(B) shows an example of a glossy label image RI that has undergone inversion processing. In the glossy label image RI, the pixel value inversion process reduces the average density of areas with high density (e.g., character X1) and increases the average density of areas with low density (e.g., background B1).

[0045] In S220-S240, CPU110 performs masking. Masking is the process of replacing a portion of the inverted glossy label image RI with a monochrome mask image WM. In this embodiment, the shape of the mask image WM is rectangular, and the color of the mask image WM is white. In this embodiment, white is represented by the RGB value where all component values ​​(R, G, B) are at their maximum values. In this embodiment, the values ​​of each component of the RGB value can be between 0 and 255, so the RGB value for white is (255, 255, 255). However, the RGB value for white may also be an RGB value where all component values ​​(R, G, B) are within 2% of the maximum value (for example, (250, 250, 250) or (251, 252, 252)).

[0046] In S220, the CPU 110 determines the area Sm of the mask image WM according to the specular gloss of the gloss label shown in the gloss label image RI. The specular gloss has already been obtained from the user via the input screen in S115. For example, information showing the correspondence between specular gloss and area Sm is pre-programmed into the computer program PG, and the CPU 110 determines the area Sm according to this correspondence. In this embodiment, if the specular gloss is 10 or more and less than 70, the area Sm is randomly determined from a value of (1 / 5) or more and less than (1 / 4) of the area of ​​the gloss label image RI. If the specular gloss is 70 or more, the area Sm is randomly determined from a value of (1 / 4) or more and less than (1 / 3) of the area of ​​the gloss label image RI.

[0047] Thus, in this embodiment, the higher the specular gloss, the larger the area Sm tends to be set. That is, if the specular gloss is a first value (for example, a value of 10 or more and less than 70), the CPU 110 determines the area Sm of the mask image WM to be a first area (for example, an area of ​​(1 / 5) or more and less than (1 / 4) of the glossy label image RI). If the specular gloss is a second value greater than the first value (for example, a value of 70 or more), the CPU 110 determines the area Sm of the mask image WM to be a second area greater than the first area (for example, an area of ​​(1 / 4) or more and less than (1 / 3) of the glossy label image RI).

[0048] In S230, CPU110 determines the aspect ratio and placement position (for example, the coordinates of the top-left corner of the mask image WM) of the rectangular mask image WM. The aspect ratio and placement position are determined randomly, for example, so that the entire mask image WM with a predetermined area Sm fits within the gloss label image RI.

[0049] In S240, CPU 110 replaces a portion of the glossy label image RI with a mask image WM. The aspect ratio, position, and area Sm of the mask image WM are set to the values ​​determined in S220 and S230. Figure 5(C) shows the masked glossy label image MI. In the example in Figure 5(C), the mask image WM is positioned near the center of the glossy label image MI in the second direction Dy, and on the positive side of the first direction Dx.

[0050] In S250, CPU110 performs a resizing process on the masked glossy label image MI. The resizing process enlarges or reduces the image by a specific magnification factor. This enlargement or reduction is performed using known methods such as the nearest neighbor method, bilinear method, or bicubic method. The specific magnification factor is determined randomly within a predetermined range (e.g., 0.8 to 1.2). The resizing process is performed to introduce variation into the generated training image LI.

[0051] In S260, the CPU 110 generates a training image LI by combining a resized glossy label image MI with a background image BI. First, the CPU 110 obtains the background image BI data to be used from the background image data group BG stored in the non-volatile storage device 130.

[0052] Figure 7 shows an example of background image BI and training image LI. The acquired background image BI (Figure 7(A)) is selected randomly, for example. Background image BI is an image obtained by capturing various subjects (e.g., landscapes, rooms, devices such as printers) using a digital camera. However, it is not limited to this and may also include images obtained by scanning original documents such as pictures or photographs using a scanner. The number of background image BI data points included in the background image data set BG is, for example, several tens to several thousand. The size of the background image BI (number of pixels in the Dx and Dy directions) is adjusted to the size of the input image IIa of the object detection model AN (AN1, AN2). Also, the size of the background image BI is sufficiently larger than the glossy label image MI.

[0053] CPU110 uses n1 glossy label images MI as the foreground and a background image BI as the background, and generates a training image LI by combining the n1 glossy label images MI and the background image BI. In the training image LI in Figure 7(B), three glossy label images MI (MI1-MI3) are combined with one background image BI. The position in which the glossy label images MI are combined with the background image BI is randomly determined within the range where the entire glossy label image MI is located within the background image BI.

[0054] If the number of glossy label images (MI) n1 is greater than a certain threshold (for example, n1 > 5), the n1 glossy label images (MI) may be combined in small quantities with each of the two or more background images (BI) to generate two or more training images (LI). For example, one glossy label image (MI) may be combined with one background image (BI).

[0055] In S270, the CPU 110 performs augmentation processing on the generated training image LI. Augmentation processing is a process that applies various image processing to the training image LI in order to increase the variations of the training image LI. In this embodiment, the augmentation processing is performed in the following order: brightness correction processing, shear processing, rotation processing, smoothing processing, and noise addition processing. In the modified example, the augmentation processing may consist of only a part of these processes, or it may include other processes (for example, sharpening processing), and the execution order of these processes may be changed as appropriate.

[0056] Brightness correction is a process that corrects the brightness of an image. Brightness correction is performed, for example, by transforming each of the three component values ​​(R, G, and B values) of the RGB values ​​of each pixel using a gamma curve. Shearing is a process that transforms a rectangle into a parallelogram. Shearing is achieved, for example, by transforming the coordinates of each pixel in the image using a known shear matrix. Rotation is a process that rotates an image. Rotation is achieved by transforming the coordinates of each pixel on the image using a known rotation matrix. Smoothing is a process that smooths (blurs) an image. Smoothing is achieved, for example, by transforming the values ​​of each pixel in the image using a known smoothing filter such as a Gaussian filter, Laplacian filter, or median filter. Noise addition is a process that adds noise to an image, for example, by adding noise that follows a normal distribution. The level of these processes (for example, the degree of brightness correction, the size of the rotation angle, the amount of noise added) is determined randomly within a predetermined range.

[0057] In S275, the CPU 110 generates training data for each training image LI. The training data includes region information indicating the label region where the glossy label image MI is located in the training image LI, and class information indicating the type of glossy label image MI located in the label region. Region information can be easily generated, for example, by recording information indicating the region where the glossy label image MI is synthesized when the glossy label image MI is synthesized with the background image BI.

[0058] In S280, the CPU 110 stores the augmented training image LI data in association with the target data generated in S275. For example, the training image LI data and the target data are stored in the non-volatile storage device 130.

[0059] In S290, the CPU 110 determines whether a predetermined number of training images LI have been generated. If the predetermined number of training images LI (for example, several thousand to several hundred thousand) has not been generated (S290: NO), the CPU 110 returns to S210. If the predetermined number of training images LI have been generated (S290: YES), the CPU 110 terminates the glossy training image generation process.

[0060] Once the glossy training image generation process is complete, in step S140 of Figure 4, the CPU 110 executes a training process to train the object detection model AN using the generated training images LI for glossy labels.

[0061] During the training process, the object detection model AN is trained so that the output data OD indicates the appropriate label regions and appropriate label types in the input image (e.g., the training image LI) that contains glossy labels. The training process adjusts several computational parameters used in the calculations of the object detection model AN (including several computational parameters used in the calculations of each of the multiple layers CV11-CV1m and CN11-CN1n). Before the training process, several computational parameters are set to initial values ​​such as random values.

[0062] First, the CPU 110 obtains multiple training image LI data in batch size from the multiple pre-generated training image LI for gloss labels. The CPU 110 inputs the multiple training image LI as input image IIa to the object detection model AN and generates multiple output data OD corresponding to the multiple training image LI.

[0063] The CPU 110 calculates the loss value using multiple output data ODs and multiple training data corresponding to the multiple output data ODs. Here, the training data corresponding to the output data ODs refers to the training data stored in association with the training images LI corresponding to the output data ODs. The loss value is calculated for each training image LI.

[0064] The loss value is calculated using a loss function that calculates a loss value corresponding to the difference between the output data OD and the training data. In this embodiment, the loss function calculates a smaller loss value the smaller the difference between the output data OD and the training data. The formula for calculating the loss function is disclosed in the YOLO paper mentioned above.

[0065] The CPU 110 adjusts several computational parameters of the object detection model AN using the calculated loss values. Specifically, the CPU 110 adjusts the computational parameters according to a predetermined algorithm so that the sum of the loss values ​​calculated for each synthesized image data is reduced. As a predetermined algorithm, for example, an algorithm using backpropagation and gradient descent may be used.

[0066] After the CPU 110 repeats the series of processes described above a predetermined number of times, for example, several epochs, it terminates the training process and proceeds to S150.

[0067] In S150, CPU110 determines whether or not to complete the training. For example, CPU110 may decide to complete the training if the loss value falls below a certain threshold or if the change in the loss value falls below a certain threshold. CPU110 may also decide to complete the training if the number of training epochs exceeds a predetermined threshold.

[0068] If CPU110 does not complete the training (S150:NO), it returns to S130. If CPU110 completes the training (S150:YES), in S160, it stores the data of the trained object detection model AN, including the adjusted computation parameters, in the non-volatile memory device 130 as object detection model AN1 for glossy labels.

[0069] In S170, CPU110 performs the normal training image generation process. The normal training image generation process generates training images used to train the object detection model AN2 for normal labels. The object detection model AN2 for training labels is assumed to receive a read image showing a normal label as input image IIa, and in that case, it is an object detection model that detects the normal label within the read image.

[0070] Figure 8 is a flowchart of the normal training image generation process. As shown in Figure 8, the normal training image generation process in Figure 8 does not perform the processing for glossy labels in S210-S240 of Figure 6. In the normal training image generation process in Figure 8, the remaining processing excluding S210-S240 of Figure 6, i.e., the processing in S250-S290 of Figure 6, is performed. That is, the CPU 110 performs the processing in S250-S290 of Figure 6 described above on n2 normal label images (print-ready images OI that show normal labels). This generates a predetermined number of training images LI for normal labels (for example, several thousand to several hundred thousand).

[0071] When the normal training image generation process is completed, in S180 of Figure 4, the CPU 110 executes a training process to train the object detection model AN using the generated normal label training image LI.

[0072] The training process in S180 is identical to that in S140, except that the training image LI for normal labels is used instead of the training image LI for glossy labels. For this reason, the explanation of the training process in S180 is omitted.

[0073] In S190, CPU110 determines whether or not to complete the training. For example, similar to S150, CPU110 decides to complete the training if the loss value falls below a certain threshold or if the change in the loss value falls below a certain threshold.

[0074] If CPU 110 does not complete the training (S190: NO), it returns to S170. If CPU 110 completes the training (S190: YES), in S195, it stores the data of the trained object detection model AN, including the adjusted computation parameters, in the non-volatile memory device 130 as the object detection model AN2 for normal labeling.

[0075] A-4. Configuration and training of the image generation model GN The image generation model GN is a so-called autoencoder, comprising an encoder and a decoder (not shown in the diagram). The encoder performs dimensionality reduction on the input image to extract features from the input image and generate feature data. The decoder performs dimensionality restoration on the feature data to generate an output image. The size of the output image is the same as the size of the input image.

[0076] The generative model (GN) is trained to produce reconstructed images that represent the features of the input image that indicate the label. For this reason, the output image generated by a trained GN is also called a reconstructed image. The reconstructed image is almost identical to the input image. Furthermore, a trained GN is trained to reconstruct only the features of normal labels. For this reason, for example, if an input image with a label containing defects is input to a trained GN, the generated reconstructed image can be expected to be an image with a normal label without defects. In other words, whether an image with a normal label or an image with a label containing defects is input to the GN, the reconstructed image will be an image that reproduces a normal label.

[0077] A-5. Inspection Processing Figure 9 is a flowchart of the inspection process. Figure 10 is an explanatory diagram of the inspection process. The inspection process is the process of checking whether the label L to be inspected is an abnormal product containing defects, etc., or a normal product that does not contain defects, etc. The inspection process is started when a user (for example, an inspection worker) inputs a start command to the inspection device 100 via the operation unit 150. The label L to be inspected is either a glossy label or a normal label. The user inputs either a start command for the inspection process of a glossy label or a start command for the inspection process of a normal label. When a start command for the inspection process of a glossy label is input, the object detection model AN1 for glossy labels is used as the object detection model AN described later. When a start command for the inspection process of a normal label is input, the object detection model AN2 for normal labels is used as the object detection model AN described later. For example, the user inputs a start command for the inspection process with the product 300 to which the label L to be inspected is attached placed in a predetermined position for photography using a digital camera 400.

[0078] In S300, the CPU 110 acquires data of a read image FI that includes the label L to be inspected (hereinafter also referred to as the inspected item). For example, the CPU 110 sends a shooting instruction to the digital camera 400, causing the digital camera 400 to generate a read image FI, and acquires data of the read image FI from the digital camera 400. As a result, for example, the data of the read image FI shown in Figure 10(A) is acquired. The read image FI is an image showing the front surface F31 of the product and the label FL attached to the front surface F31. In order to distinguish the front surface and label of the product shown in the read image FI from the actual front surface 31 and label L (Figure 2), the letters "F" are added to the beginning of the codes, and these are referred to as front surface F31 and label FL. The label FL in the read image FI may contain defects such as scratches.

[0079] In S305, the CPU 110 inputs the acquired read image FI to the object detection model AN to identify the label region LA, which is a part of the read image FI and contains the label FL. As described above, if the label FL is a glossy label, the object detection model AN1 for glossy labels is used as the object detection model AN. If the label FL is a normal label, the object detection model AN2 for normal labels is used as the object detection model AN.

[0080] Specifically, the CPU 110 inputs the read image FI as input image IIa (Figure 3) to the object detection model AN and generates output data OD (Figure 3) corresponding to the read image FI. From among the multiple predicted regions (also called bounding boxes) indicated by the output data OD, the CPU 110 identifies a predicted region in which the corresponding confidence level Vc is greater than or equal to a predetermined threshold THa, and in which the object within the region is predicted to be the target label based on the corresponding class information. The CPU 110 identifies this predicted region as the label region LA. For example, in the example in Figure 10(A), within the read image FI, a label region LA is identified that includes the entire label FL and is almost circumscribing the label FL.

[0081] In S310, the CPU 110 generates a verification image SI using the read image FI. Specifically, the CPU 110 extracts the label region LA from the read image FI to generate the verification image SI. The CPU 110 performs a size adjustment process to enlarge or reduce the verification image SI as needed, adjusting its size to match the size of the input image of the image generation model GN. The verification images SI in Figures 10(B) and (C) show the image within the label region LA (i.e., the image of the label FL). Note that the label FLa in verification image SIa in Figure 10(B) is a normal product and does not contain defects such as scratches. The label FLb in verification image SIb in Figure 10(C) is an abnormal product and contains a linear scratch df.

[0082] In S315, CPU110 inputs the verification image SI into the trained image generation model GN and generates a reproduced image corresponding to the verification image SI. As described above, the reproduced image is an image that reproduces the label FL of the input verification image SI. For example, regardless of whether the input verification image SI is verification image SIa or SIb in Figure 10(B) or (C), the generated reproduced image will be a defect-free image like verification image SIa in Figure 10(B).

[0083] In S320, the CPU 110 generates a difference image DI using the verification image SI and the reconstructed image. For example, the CPU 110 calculates the difference value (v1-v2) between the component value v1 of a pixel in the verification image SI and the component value v2 of the corresponding pixel in the reconstructed image, and normalizes this difference value to a value in the range of 0 to 1. The CPU 110 calculates this difference value for each pixel and each color component, and generates a difference image DI using this difference value as the color value of the pixel.

[0084] Figures 10(D) and (E) show examples of differential image DIs. Differential image DIa in Figure 10(D) is a differential image generated when the input image is verification image SIa, which shows a normal product in Figure 10(B). Differential image DIa does not contain defects such as scratches. Differential image DIb in Figure 10(E) is a differential image DI generated when the input image is verification image SIb, which shows a defective product in Figure 10(C). Differential image DIb contains scratches dfd corresponding to scratches df contained in verification image SIb. Therefore, by referring to differential image DI, it is possible to identify, for example, the presence, location, size, and shape of defects contained in verification image SI.

[0085] In S325, the CPU 110 uses the difference image DI to identify abnormal pixels included in the difference image DI. An abnormal pixel is, for example, a pixel among several pixels included in the difference image DI whose RGB value of at least one component is greater than or equal to the threshold TH1. For example, if the difference image DIa in Figure 10(D) is the target of processing, no abnormal pixels are identified. If the difference image DIb in Figure 10(E) is the target of processing, several pixels constituting the defect dfd are identified as abnormal pixels.

[0086] In S330, the CPU 110 determines whether the number of abnormal pixels identified in the differential image DI is equal to or greater than the threshold TH2. If the number of abnormal pixels is less than the threshold TH2 (S330: NO), in S340, the CPU 110 determines that the label for the inspected product is normal. If the number of abnormal pixels is equal to or greater than the threshold TH2 (S330: YES), in S335, the CPU 110 determines that the label for the inspected product is defective. In S350, the CPU 110 displays the inspection result on the display unit 140 and terminates the inspection process. In this way, it is possible to determine whether an inspected product is normal or defective using the object detection model AN and the image generation model GN.

[0087] According to the first embodiment described above, when the layout image OI is a glossy label image OI showing a glossy label L1, the CPU 110 performs image processing including masking (S220-S240 in Figure 6) to replace a portion of the glossy label image OI showing the glossy label L1 (in this embodiment, the entire glossy label image OI) with a monochrome mask image WM to generate a processed glossy label image MI (Figure 5(C)). The CPU 110 trains the object detection model AN using the training image LI (Figure 7(B)) which includes the processed glossy label image MI (S140 in Figure 4). As a result, when a read image FI (Figure 10(A)) obtained by optically reading the glossy label L1 is input, the object detection model AN can be trained to detect the glossy label L1 in the read image FI.

[0088] For example, a glossy label L1, which has a glossy surface, reflects light specularly, so if label L is a glossy label L1, the glossy label L1 shown in the read image FI is prone to overexposure. Overexposure is a phenomenon in which a part of the glossy label L1, which is originally a color other than white, becomes white. When a read image FI containing a glossy label L1 with overexposure is input to an object detection model AN, the detection accuracy of the glossy label L1 by the object detection model AN tends to decrease. In this embodiment, a processed glossy label image MI that simulates overexposure is generated by replacing a part of the glossy label image OI with a monochrome mask image WM. Then, the object detection model AN is trained using the training image LI containing the processed glossy label image MI. As a result, the object detection model AN can learn that a characteristic of a glossy label L1 is that overexposure can occur in the glossy label L1. Therefore, when a read image FI obtained by optically reading a glossy label L1 is input, the object detection model AN can be trained to detect the glossy label L1 in the read image FI. This can, for example, improve the detection accuracy of glossy labels L1 by the object detection model AN1 for glossy labels generated through training.

[0089] Furthermore, according to this embodiment, the area Sm of the mask image WM is at least 1 / 5 and less than 1 / 3 of the area Sm of the portion of the glossy label image OI that shows the glossy label L1 (in this embodiment, the total area Sm of the glossy label image OI). As a result, the overexposure of the glossy label L1 can be simulated appropriately. The range of area Sm is determined empirically.

[0090] Furthermore, according to this embodiment, the mask image WM is a white image. The areas of overexposure have maximum brightness and therefore appear white. Thus, by making the mask image WM white, the overexposure occurring in the gloss label L1 can be represented more appropriately.

[0091] Furthermore, according to this embodiment, the mask image WM is a rectangular image (Figure 5(C)). Since the shape of the mask image WM is simple, the process of replacing it with the mask image WM can be easily performed.

[0092] Furthermore, according to this embodiment, the aspect ratio of the mask image WM and its position relative to the glossy label L1 are determined randomly (S230 in Figure 6). In the glossy label L1, overexposure can appear in various positions and shapes. By randomly determining the aspect ratio and position of the mask image WM, the object detection model AN can be trained to recognize that various patterns of overexposure can occur.

[0093] Furthermore, according to this embodiment, the glossy training image generation process further includes a pixel value inversion process that inverts the values ​​of pixels of a specific proportion Rr randomly selected from a plurality of pixels constituting the glossy label image OI (S210 in Figure 6).

[0094] The glossy label L1 shown in the read image FI includes areas where specular reflection occurred and areas where specular reflection did not occur, depending on conditions such as the angle of light at the time of reading and the camera's field of view. For this reason, in the glossy label L1 shown in the read image FI, the boundary region between the area where specular reflection occurred and the area where specular reflection did not occur (for example, the area around a blown-out highlight) may contain a mixture of features from both regions. Pixel value inversion processing represents the region where both features are mixed. Therefore, by using a training image LI that includes a glossy label image MI from which pixel value inversion processing has been performed, the object detection model AN can be trained to more effectively learn the features of the glossy label L1. Thus, when a read image FI obtained by optically reading the glossy label L1 is input, the object detection model AN can be trained to detect the glossy label L1 in the read image FI with greater accuracy.

[0095] Furthermore, according to this embodiment, the CPU 110 performs the masking process (S240 in Figure 6) to place the mask image WM after the pixel value inversion process (S210 in Figure 6). If the pixel value inversion process were performed after the masking process, the pixel values ​​within the mask image WM would be inverted, potentially resulting in the mask image WM being an image that is not monochrome. According to this embodiment, such problems can be suppressed.

[0096] Furthermore, in this embodiment, the specific percentage Rr of pixels to be inverted during the pixel value inversion process is 40% or more and less than 60% (for example, 50%). As a result, the characteristics of the boundary region between the area where specular reflection occurs and the area where specular reflection does not occur on the gloss label L1 can be suitably represented. This specific percentage Rr is a value determined empirically.

[0097] Furthermore, according to this embodiment, the CPU 110 performs a synthesis process (S260 in Figure 6) to synthesize the processed glossy label image MI as the foreground with the background image BI to generate a training image LI (Figure 6). As a result, a training image LI suitable for training the object detection model AN1 to detect the glossy label L1 in the read image FI can be generated.

[0098] Furthermore, according to this embodiment, the CPU 110 determines whether the layout image OI is a glossy label image indicating a glossy label based on user input (S115, S120 in Figure 4). If the layout image OI is a glossy label image indicating a glossy label L1, the CPU 110 performs image processing including pixel value inversion and masking (S210-S240 in Figure 6) to generate a processed glossy label image MI (Figure 6). If the glossy label image OI is not an image indicating a glossy label L1, the CPU 110 does not perform image processing including pixel value inversion and masking (Figure 5). In some cases, it may not be possible to determine whether the layout image OI is a glossy label image indicating a glossy label L1 based on the layout image OI alone, but by using user input, it is possible to appropriately determine whether the layout image OI is an image indicating a glossy label. Therefore, appropriate processing can be performed according to the type of label L shown in the layout image to generate a training image LI. Image processing that includes pixel value inversion and masking (S210-S240 in Figure 6) is also called gloss-correction processing.

[0099] Furthermore, in this embodiment, the higher the specular gloss, the larger the area Sm tends to be set. That is, when the specular gloss is a first value (for example, a value between 10 and 70), the area Sm of the mask image WM is determined to be the first area, and when the specular gloss is a second value greater than the first value (for example, a value of 70 or more), the area Sm of the mask image WM is determined to be a second area greater than the first area (S220 in Figure 6). The higher the specular gloss of the glossy label L1, the larger the area of ​​overexposure in the glossy label L1 shown in the read image FI tends to be. In this embodiment, the object detection model AN can be trained using a training image LI that appropriately represents overexposure according to the specular gloss of the glossy label L1. Therefore, the detection accuracy of the glossy label L1 by the object detection model AN1 for glossy labels can be improved.

[0100] As can be seen from the above explanation, the glossy label L1 in the above embodiment is an example of a specific object. The glossy label image OI is an example of the target image, and the masked glossy label image MI is an example of the processed image.

[0101] B. Second Example In the inspection process of the first embodiment, two types of object detection models AN are used: object detection model AN1 for glossy labels and object detection model AN2 for normal labels. Instead, in the inspection process of the second embodiment, one type of object detection model AN is used that is common to both glossy and normal labels. For this reason, the training content of object detection model AN in the second embodiment differs from that of the first embodiment.

[0102] Figure 11 is a flowchart of the training of the object detection model AN in the second embodiment. In S410, similar to S110 in Figure 5, the CPU 110 of the inspection device 100 acquires data for N layout images OI, each representing one of the N labels L that should be detected by the object detection model AN.

[0103] In S415, similar to S115 in Figure 4, the CPU 110 displays an input screen (not shown) on the display unit 140 and obtains information about the labels shown in N layout images OI from the user. The obtained label information includes information indicating whether the label shown in each of the N layout images OI is a glossy label or a normal label, and the specular gloss of the glossy label.

[0104] In S420, the CPU 110 selects one target image from among N target images OI. In S430, the CPU 110 determines whether the target image is a glossy label image or a regular label image. This determination is made based on the label information obtained from the user in S415.

[0105] If the target image is a glossy label image (S430:YES), the CPU 110 executes the gloss-correction processing described above (S210-S240 in Figure 6) in S440. If the target image is a normal label image (S430:NO), the CPU 110 skips S440. In other words, in this case, the gloss-correction processing is not executed.

[0106] In S450, CPU110 executes common processing (S250-S280 in Figure 6) that is performed in common for both glossy label images and normal label images.

[0107] In S460, the CPU 110 determines whether a predetermined number of training images LI have been generated. If a predetermined number of training images LI (for example, several thousand to several hundred thousand) have not been generated (S460: NO), the CPU 110 returns to S420. If a predetermined number of training images LI have been generated (S460: YES), the CPU 110 executes the training process in S470. The training process is the same as the training process in S140 and S180 in Figure 4 of the first embodiment.

[0108] After repeating the training process for several epochs, the CPU 110 proceeds to S480. In S480, the CPU 110 determines whether or not to complete the training. For example, the CPU 110 may determine that training is complete when it has performed a predetermined number of training processes on each of the N artwork images OI as the artwork image of interest. Alternatively, the CPU 110 may determine that training is complete when it has performed a predetermined number of training processes on each of the N artwork images OI as the artwork image of interest and the loss value falls below a certain threshold.

[0109] If the CPU 110 does not complete the training (S480: NO), it returns to S420. If the CPU 110 completes the training (S480: YES), in S490, it stores the data of the trained object detection model AN, including the adjusted computation parameters, in the non-volatile memory device 130 as an object detection model AN common to both glossy labels and normal labels.

[0110] According to the second embodiment described above, a common object detection model AN can be created for both glossy labels and normal labels. As a result, for example, in the inspection process, it is not necessary to use different object detection models depending on whether the label L being inspected is a glossy label or not. This can reduce the workload of the inspection process.

[0111] In the modified version, for example, the training image LI may be generated such that a single training image LI includes both a processed glossy label image MI (Figure 5(C)) that has undergone gloss-correction processing and a layout image OI (normal label image) that has not undergone gloss-correction processing.

[0112] C. Third Example In the first embodiment, the entire glossy label L1 is a glossy portion. The glossy label L1c of the third embodiment has a glossy portion A1 and a non-glossy portion A2. Figure 12 shows an example of an image used in the third embodiment. The print-ready image OIc in Figure 12(A) is data used to create the glossy label L1c. For this reason, the print-ready image OIc is also called the glossy label image OIc. The glossy label L1c includes a background Bc, text Xc, and a mark Mc. Of the glossy label L1c, the background Bc is the glossy portion A1, and the text Xc and mark Mc are the non-glossy portion A2. For example, the glossy label L1c is created by printing the text Xc and mark Mc on a glossy sheet using a non-glossy paint.

[0113] An α channel is associated with the glossy label image OIc. The α channel is generally used to indicate the transmittance (transparency) of each pixel in an image. In this embodiment, the α channel is used to indicate glossy and non-glossy areas. The α channel in this embodiment is binary image data in which each pixel has either a first value indicating glossy pixels that make up the glossy area, or a second value indicating pixels that make up the non-glossy area. The binary image BIc in Figure 12(B) is a binary image indicated by the α channel. In the binary image BIc, the white area indicates the glossy area A1 made up of glossy pixels, and the hatched area indicates the non-glossy area A2 made up of non-glossy pixels. In the example in Figure 12(B), the binary image BIc indicates that, of the glossy label L1c, the background Bc is the glossy area A1, and the letters Xc and mark Mc are the non-glossy area A2.

[0114] In the third embodiment, the gloss treatment differs from that of the first and second embodiments in order to accommodate the fact that the gloss label L1c includes a glossy portion A1 and a non-glossy portion A2. Other components of the third embodiment may be the same as those of the first embodiment or the second embodiment.

[0115] Figure 13 is a flowchart of the gloss-response processing in the third embodiment. In S510, the CPU 110 executes the processes S210-S240 in Figure 6. That is, in S510, the gloss-response processing of the first and second embodiments is executed.

[0116] Specifically, the CPU 110 performs a pixel value inversion process on the glossy label image OIc to generate the inverted glossy label image RIc. Figure 12(C) shows the inverted glossy label image RIc. In the glossy label image RIc, the pixel value inversion process inverts the pixel values ​​throughout the entire image, including the non-glossy area A2, which consists of characters X1c and marks Mc, and the non-glossy area A2, which consists of background B1c.

[0117] Furthermore, the CPU 110 performs a masking process (S220-S240 in Figure 6) on the glossy label image RIc in Figure 12(C). Figure 12(D) shows the masked glossy label image MIc. In the example in Figure 12(D), the mask image WM is positioned near the center of the second direction Dy in the glossy label image MIc, and on the positive side of the first direction Dx.

[0118] In S520, the CPU 110 refers to the alpha channel to identify the non-glossy area A2 of the glossy label image OIc. In the example in Figure 12, the area where the characters Xc and mark Mc of the glossy label image OIc are located is identified as the non-glossy area A2.

[0119] In S530, the CPU 110 replaces multiple pixels constituting the non-glossy area A2 in the masked glossy label image MIc with corresponding pixels from the layout image OIc (the original glossy label image OIc before processing). In other words, the non-glossy area A2 of the masked glossy label image MIc is returned to its state before the pixel value inversion and masking processes were performed. This generates the glossy label image PIc, which has been processed to support glossiness according to the third embodiment.

[0120] Figure 12(E) shows the glossy label image PIc of the third embodiment after gloss-compatible processing. As shown in Figure 12(D), in the glossy label image PIc, the non-glossy areas A2, namely the characters Xc and the mark Mc, have not undergone pixel value inversion or masking processing.

[0121] According to the third embodiment described above, when the glossy label L1c includes a glossy portion A1 and a non-glossy portion A2, the CPU 110 performs masking and pixel value inversion processing on the glossy portion A1 of the glossy label L1c shown in the glossy label image OIc, but does not perform masking and pixel value inversion processing on the non-glossy portion A2.

[0122] When the label L to be detected is a glossy label L1c having a glossy portion A1 and a non-glossy portion A2, phenomena caused by specular reflection, such as overexposure, may occur in the glossy portion A1 of the glossy label L1c shown in the read image FI. On the other hand, in this case, phenomena caused by specular reflection, such as overexposure, do not occur in the non-glossy portion A2 of the glossy label L1c shown in the read image FI. In this embodiment, an appropriate training image LI corresponding to these characteristics can be generated. As a result, even when the label L to be detected is a glossy label L1c having a glossy portion A1 and a non-glossy portion A2, the characteristics of the glossy label L1c can be appropriately learned by the object detection model AN. Therefore, for example, the accuracy of detecting a glossy label L1c including a glossy portion A1 and a non-glossy portion A2 can be improved by the object detection model AN generated by training.

[0123] D. Fourth Example The label Ld of the fourth embodiment is a label with a hologram applied to its surface (hereinafter also referred to as a hologram label). Figure 14 shows an example of an image used in the fourth embodiment. The artwork image OId in Figure 14(A) is an image showing the hologram label Ld and is the data used to create the hologram label Ld. For this reason, the artwork image OId is also called the hologram label image OId. The hologram label Ld includes a background Bd, text Xd, and a mark Md. The hologram label Ld is created, for example, by applying a hologram to the surface of a label sheet on which the hologram label image OId is printed. The hologram application may be a process of laminating a hologram film onto the label sheet (also called holo-lamination). Alternatively, the hologram application may be a process of transferring a hologram pattern onto a printed label sheet (also called trans-stamping), or a process called hologram foil stamping. The hologram label image OId (print-ready image OId) in this embodiment itself does not have any special decoration or coloring, and is indistinguishable from a standard label print-ready image OI.

[0124] Due to the holographic processing applied to the surface of the hologram label Ld, light incident on the surface is reflected at different angles depending on its wavelength. Therefore, when white light containing various wavelengths of light is incident on the surface of the hologram label Ld, the white light is spectrally separated into multiple chromatic colors. Consequently, in the read image FI obtained by reading the hologram label Ld, the hologram label Ld may contain areas with various colors depending on conditions such as the angle of incidence of light and the field of view when the digital camera 400 is taking the picture.

[0125] In the fourth embodiment, to address the characteristics of the hologram label Ld, a hologram-matching process is performed instead of the gloss-matching process in the first to third embodiments. Figure 15 is a flowchart of the training of the object detection model AN in the fourth embodiment. In the example in Figure 15, S415B, S430B, and S440B are performed instead of S415, S430, and S440 in the training of the object detection model AN in the second embodiment (Figure 11). The other processes in the flowchart of Figure 15 are the same as those in the flowchart of Figure 11. In Figure 15, processes that are the same as in Figure 11 are denoted by the same reference numerals, and processes that are different from those in Figure 11 have "B" appended to the end of their reference numerals. The processes in Figure 15 will be explained focusing on the parts that differ from those in Figure 11.

[0126] In S415B, after N artwork images OI have been acquired in S410, the CPU 110 displays an input screen (not shown) on the display unit 140, similar to S415 in Figure 11, and obtains information about the labels shown in the N artwork images OI from the user. The acquired label information includes information indicating whether the label shown in each of the N artwork images OI is a hologram label or a normal label.

[0127] When a target image is selected in S420, in S430B, the CPU 110 determines whether the target image is a hologram label image or a regular label image. This determination is made based on the label information obtained from the user in S415B.

[0128] If the target image is a hologram label image (S430B:YES), the CPU 110 executes the hologram processing described later in S440B. If the target image is a regular label image (S430B:NO), the CPU 110 skips S440B. In other words, in this case, the hologram processing is not executed.

[0129] The processing from S450 onwards in Figure 15 is the same as the processing of the same reference numerals in Figure 11, so its explanation will be omitted. The hologram processing will be explained below.

[0130] Figure 16 is a flowchart of the hologram processing. In S610, the CPU 110 divides the hologram label image OId into multiple division regions DA1-DA7. Figure 14(B) shows the multiple division regions DA1-DA7 set in the hologram label image OId by dividing it. In the example in Figure 14(B), the hologram label image OId is divided by six dividing lines VL, thereby setting seven division regions DA1-DA7 in the hologram label image OId. The six dividing lines VL are straight lines parallel to each other. The dividing lines VL are parallel to the second direction Dy. In other words, the dividing lines VL are parallel to the right and left sides of the hologram label image OId. The seven division regions DA1-DA7 are each a rectangular strip extending in the second direction Dy. The spacing of the dividing line VL, i.e., the width of the seven divided regions DA1-DA7, is determined randomly within a predetermined range. This predetermined range is, for example, 5%-30% of the width of the first direction Dx (left-right direction) of the hologram label image OId.

[0131] In the S620, the CPU 110 synthesizes chromatic colors with different hues for each of the seven divided regions DA1-DA7. This generates the processed hologram label image HI. For example, red, orange, yellow, green, blue, indigo, and violet are synthesized into the seven divided regions DA1-DA7, respectively. The brightness (luminance) of these chromatic colors is assumed to be approximately the same. For example, let the RGB values ​​of the pixels in the divided region DAk (where k is an integer between 1 and 7) before synthesis be (Rak, Gak, Bak), and the RGB values ​​of the chromatic colors synthesized into the divided region DAk be (Rpk, Gpk, Bpk). The RGB values ​​of the pixels in the divided region DAk after synthesis, (Rbk, Gbk, Bbk), are given by the following equations (6)-(8).

[0132] Rbk = (Rak + Rpk) / 2 …(6) Gbk = (Gak + Gpk) / 2 …(7) Bbk = (Bak + Bpk) / 2 …(8)

[0133] Figure 14(C) shows the generated processed hologram label image HI. As shown in Figure 14(C), the S620 process adds a chromatic color with a different hue to each of the seven divided regions DA1-DA7.

[0134] As can be seen from the above explanation, according to the fourth embodiment, when the layout image OI is a hologram label image OId that shows a hologram label Ld, the CPU 110 performs hologram matching processing (S440B in Figure 15, Figure 16) on the hologram label image OId to generate a processed hologram label image HI. The CPU 110 trains the object detection model AN using the training image LI which includes the hologram label image HI (S450-S470 in Figure 15). The hologram matching processing involves dividing the hologram label image OId with multiple dividing lines VL to set up multiple divided regions (S610 in Figure 16), and then adding chromatic colors with different hues to each of the multiple divided regions (S620 in Figure 16).

[0135] As described above, when white light containing light of various wavelengths is incident on the surface of the hologram label Ld, the white light is spectrally separated into multiple chromatic colors. Therefore, in the read image FI obtained by reading the hologram label Ld, the hologram label Ld may contain areas with various chromatic colors depending on conditions such as the angle of incidence of light and the field of view when the digital camera 400 is taking a picture. According to this embodiment, multiple divided regions are set in the hologram label image OId, and chromatic colors with different hues are added to each of the multiple divided regions, thereby generating a processed hologram label image HI that simulates the above-mentioned features of the hologram label Ld. Then, the object detection model AN is trained using a training image that includes the processed hologram label image HI. As a result, the object detection model AN can learn that a feature of the hologram label Ld is that it may contain areas with various chromatic colors. Therefore, when a read image obtained by optically reading the hologram label Ld is input, the object detection model AN can be trained to detect the hologram label Ld in the read image. This can, for example, improve the detection accuracy of the hologram label Ld by the object detection model AN, which is generated through training.

[0136] As can be seen from the above explanation, the hologram processing in this embodiment is an example of coloring processing, and the processed hologram label image HI is an example of a colored image.

[0137] D. Variations (1) The gloss-responding processing in the first to third embodiments described above is an example and can be modified or omitted as appropriate. For example, the area Sm of the mask image WM in the gloss label image MI may be less than (1 / 5) or (1 / 3) or more of the area of ​​the print-ready image OI.

[0138] Furthermore, the color of the mask image WM does not have to be white; any single color will have a certain effect, so the color of the mask image WM can be, for example, gray. Also, the shape of the mask image WM is not limited to a rectangle; it can be an ellipse, a circle, or a parallelogram, for example.

[0139] Furthermore, in the first embodiment described above, the area Sm of the mask image WM is set to a different value depending on the specular gloss (S220 in Figure 6). However, the area Sm of the mask image WM may be determined independently of the specular gloss.

[0140] (3) In addition, in the gloss-responding processing, the pixel value inversion processing (S210 in Figure 6) may be omitted, and only the mask processing (S220-S240 in Figure 6) may be performed.

[0141] (4) In addition, during the training of the object detection model AN in each of the above embodiments, whether or not the artwork image OI is a glossy label image is determined based on user input (S115, S120 in Figure 4). However, this is not limited to this, and for example, if it is predetermined that all artwork images are glossy label images, S115, S120 in Figure 4 may be omitted. Alternatively, information indicating whether or not the artwork image OI is a glossy label image may be stored as metadata in the artwork image OI file. In this case, the CPU 110 may refer to the metadata to determine whether or not the artwork image OI is a glossy label image.

[0142] (5) In the pixel value inversion process for glossy finish processing (S210 in Figure 6), the specific percentage Rr of pixels whose pixel values ​​are inverted is not limited to 40% or more and less than 60%, but may be less than 40% or 60% or more.

[0143] (6) In addition, in the training of the object detection model of the first embodiment, N layout images OI are classified into glossy label images and normal label images (S120 in Figure 4). Then, a glossy training image generation process is performed on the glossy label images among the layout images OI, and a normal training image generation process is performed on the normal label images to generate training images LI. However, this is not limited to this, and for example, if the objects to be inspected are limited to glossy objects such as metal products, the glossy training image generation process may be performed on all layout images OI.

[0144] (7) The configuration of the object detection model AN used in each of the above embodiments is an example, and other models may be used. For example, any other model may be used instead of the object detection model AN or the YOLO model. The object detection model may be an improved YOLO model such as "YOLO v3", "YOLO v4", or "YOLO v5". In addition, other models such as SSD, R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN may be used.

[0145] (8) In the above embodiment, the training of the object detection model AN and the inspection process using the object detection model AN are performed by the inspection device 100 shown in Figure 1. Alternatively, the training and inspection process may be performed by separate devices. In this case, for example, the object detection model AN trained by the inspection device 100 is stored in the memory of the device that performs the inspection process. Furthermore, all or part of the training and inspection process may be performed by multiple computers (for example, so-called cloud servers) that can communicate with each other via a network. Also, the computer program that performs the inspection process and the computer program that trains the object detection model AN may be different computer programs.

[0146] (9) The machine learning model trained in the first to fourth embodiments described above is an object detection model, but is not limited thereto. For example, the CPU 110 may train an image recognition model using the masked glossy label image MI shown in Figure 5(C). The image recognition model is trained to identify the type of glossy label contained in a read image when a read image showing a glossy label is input. For example, known CNNs (Convolutional Neural Networks), such as ResNet, VGG16, and VGG19, can be used for the image recognition model. Similarly, the CPU 110 may train an image recognition model using the processed hologram label image HI shown in Figure 14(C). The image recognition model is trained to identify the type of hologram label contained in a read image when a read image showing a hologram label is input.

[0147] Alternatively, CPU 110 may train the autoencoder using the masked glossy label image MI shown in Figure 5(C). The autoencoder is trained to generate a glossy label image without overexposure (e.g., an image like a printing plate) when a read image showing a glossy label with overexposure is input. The autoencoder may be a standard autoencoder, a VQ-VAE (Vector Quantized Variational Auto Encoder), a VAE (Variational Autoencoder), or an autoencoder included in so-called GANs (Generative Adversarial Networks). Similarly, CPU 110 may train the autoencoder using the processed hologram label image HI shown in Figure 14(C). The autoencoder is trained to generate a hologram label image without chromatic coloration (e.g., an image like a printing plate) when a read image showing a hologram label with various chromatic colors due to spectral dispersion is input.

[0148] When training image recognition models or autoencoders, the masked glossy label image MI in Figure 5(C) and the processed hologram label image HI in Figure 14(C) may be used directly as training images without being combined with the background image BI.

[0149] (10) In each of the above embodiments, a label L is used as the object. The object is not limited to a label, but may be any other object. The object may be, for example, a nameplate, tag, accessory, part, or engraving attached to a product (e.g., a multifunction printer). Furthermore, the product to which the label or nameplate is attached is not limited to a multifunction printer, but may be any other product such as a sewing machine, cutting machine, or mobile terminal. Alternatively, the object may be a metallic product or part with a glossy finish.

[0150] (11) In the fourth embodiment, the dividing line VL that divides the hologram label image OId is parallel to the left and right sides of the hologram label image OId. However, the dividing line VL may also be a diagonal straight line that is inclined with respect to the left and right sides of the hologram label image OId. The dividing line VL may also be a straight line that is parallel to the top and bottom sides of the hologram label image OId.

[0151] (12) In the fourth embodiment, seven divided regions DA1-DA7 are set in the hologram label image OId (Figure 14(C)). However, the number of divided regions set in the hologram label image OId may be eight or more, or six or less. Generally, it is preferable that the number of divided regions set in the hologram label image OId be four or more.

[0152] (13) In the fourth embodiment, the chromatic colors added to the seven divided regions DA1-DA7 are red, orange, yellow, green, blue, indigo, and purple. These chromatic colors are just examples, and various other colors, such as light blue, yellow-green, pink, magenta, cyan, etc., may be used. The order of the seven chromatic colors added to the seven divided regions DA1-DA7 may be changed as appropriate.

[0153] (14) The first to third embodiments and the fourth embodiment may be combined as appropriate. For example, in the training of the object detection model AN of the first embodiment (Figure 4), if the N number of print-ready images OI include hologram label images, in S120, the CPU 110 classifies the N number of print-ready images OI into glossy label images, normal label images, and hologram label images. The CPU 110 then uses the glossy label images to execute S130-S160 in Figure 4 to generate an object detection model for glossy labels. The CPU 110 then uses the normal label images to execute the object detection model for normal labels using S170-S195 in Figure 4. The CPU 110 then uses the hologram label images to execute the hologram-compatible processing (S440B in Figure 15) and common processing (S450 in Figure 15) of the fourth embodiment to generate training images, and then uses these training images to train the object detection model, thereby generating an object detection model for hologram labels.

[0154] Furthermore, in the training of the object detection model AN of the second embodiment (Figure 11), if the N layout images OI include hologram label images, glossy label images, and normal label images, in S430, the CPU 110 determines whether the layout image of interest is a hologram label image, a glossy label image, or a normal label image. If the layout image of interest is a glossy label image, the CPU 110 performs gloss-compatible processing (S440 in Figure 11) and common processing (S450 in Figure 11) to generate a training image, as shown in Figure 11. If the layout image of interest is a hologram label image, the CPU 110 performs hologram-compatible processing (S440B in Figure 15) and common processing (S450 in Figure 15) to generate a training image, as shown in Figure 15. If the layout image of interest is a normal label image, the CPU 110 performs only common processing to generate a training image. The CPU 110 may then use the generated training images to train an object detection model, thereby generating an object detection model common to glossy labels, hologram labels, and normal labels.

[0155] (15) The technology disclosed in the fourth embodiment can be realized, for example, in the following embodiments. [Aspect 1] A computer program for training a machine learning model that receives a read image obtained by optically reading an object as input, Image acquisition function to acquire the target image, When the aforementioned target image shows a specific object that has been processed with a hologram, the generation function includes image processing, including coloring, to the portion of the target image showing the specific object to generate a colored image. A training function that trains the machine learning model using the training images including the aforementioned colored images, To make this a reality on a computer, The coloring process is a computer program that divides the target image into a number of straight lines to set up four or more divided regions, and adds chromatic colors with different hues to each of the four or more divided regions.

[0156] For example, when white light containing various wavelengths of light is incident on the surface of a specific object that has been holographically processed, the white light is spectrally separated into multiple chromatic colors. Therefore, in the read image obtained by reading the specific object, the specific object may contain parts with various chromatic colors. According to the above embodiment 1, multiple divided regions are set in the target image, and chromatic colors with different hues are added to each of the multiple divided regions, thereby generating a colored image that simulates the above-mentioned features of the specific object. Then, a machine learning model is trained using a training image that includes this colored image. As a result, the machine learning model can be taught that the specific object may contain parts with various chromatic colors as a feature of the specific object. Therefore, when a read image obtained by optically reading the specific object is input, the machine learning model can be trained to perform the desired function (e.g., object detection or object classification).

[0157] Furthermore, various configurations included in each of the above embodiments and each of the above modifications can be applied to the above embodiment 1.

[0158] (16) In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, all or part of the training and inspection process of the object detection model AN may be performed by a hardware circuit such as an ASIC (Application Specific Integrated Circuit).

[0159] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0160] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of symbols]

[0161] 1000…Inspection system, 100…Inspection device, 110…CPU, 120…Volatile memory device, 130…Non-volatile memory device, 140…Display unit, 150…Operation unit, 170…Communication unit, 30…Enclosure, 300…Product, 400…Digital camera, AN…Object detection model, FI…Read image, GN…Image generation model, L…Label, LI…Training image, OD…Output data, OG…Print-read image data group, OI, OIc, OId…Print-read image, PG…Computer program, WM…Mask image

Claims

1. A computer program for training a machine learning model that receives a read image obtained by optically reading an object as input, Image acquisition function to acquire the target image, An image processing function that generates a processed image by performing image processing including masking, which replaces a portion of the target image showing a specific object that has gloss with a monochrome mask image when the target image shows a specific object that has gloss, A training function that trains the machine learning model using training images including the processed images, A computer program that enables a computer to realize something.

2. A computer program according to claim 1, A computer program in which the area of ​​the mask image is (1 / 5) or more and (1 / 3) or less than the area of ​​the target image.

3. A computer program according to claim 2, The aforementioned mask image is a white image, representing a computer program.

4. A computer program according to claim 2, The aforementioned mask image is a rectangular image, as is the case with the computer program.

5. A computer program according to claim 1, The image processing further includes a computer program that performs an inversion process to invert the values ​​of a specific proportion of pixels randomly selected from a plurality of pixels constituting the target image.

6. A computer program according to claim 5, The aforementioned image processing is a computer program that performs the masking process after the inversion process.

7. A computer program according to claim 5, The aforementioned specific percentage is a computer program where the percentage is 40% or more and less than 60%.

8. A computer program according to claim 1, further, A training image generation function that generates the training image by performing a generation process that includes a synthesis process in which the processed image is used as the foreground and composited onto a background image larger than the processed image. A computer program that enables a computer to realize something.

9. A computer program according to claim 1, further, A computer is provided with a judgment function that determines whether or not the target image represents the specified object based on user input. The aforementioned image processing function is The judgment function, when the target image is an image representing the specific object, performs the image processing to generate the processed image. A computer program in which the judgment function does not perform the image processing if the target image is an image that does not show the specified object.

10. A computer program according to claim 1, The aforementioned image processing function is When the specular gloss of the specified object is a first value, the area of ​​the mask image is set to the first area. If the specular gloss of the specified object is a second value greater than the first value, the area of ​​the mask image is set to a second area greater than the first area. A computer program that performs the masking process, which replaces a portion of the target image showing the specified object with the mask image having a set area.

11. A computer program according to claim 1, When the specified object includes a glossy portion having a glossy finish and a non-glossy portion not having a glossy finish, the image processing function is a computer program that performs the masking process on the glossy portion of the specified object shown in the target image, but does not perform the masking process on the non-glossy portion.

12. A computer program according to claim 1, further, The image processing function, when the target image represents an object that has been subjected to holographic processing, performs processing including coloring on the target image to generate a colored image. The training function further trains the machine learning model using the image including the colored image, The coloring process is a computer program that divides the target image into a number of straight lines to set up four or more divided regions, and adds chromatic colors with different hues to each of the four or more divided regions.

13. A method for training a machine learning model in which a read image obtained by optically reading an object is input, The image acquisition process involves obtaining the target image, Image processing step to generate a processed image by performing image processing including masking, which replaces a portion of the portion of the target image showing the specific object with a monochrome mask image when the target image shows a specific object that has gloss, A training step of training the machine learning model using training images including the processed images, A method that includes [a certain feature].

Images