Acquisition method, acquisition system, and computer program

Abstract

To provide a method for appropriately acquiring defect data of the appearance of an object to be inspected, an acquisition system, and a computer program.SOLUTION: A defect data acquisition processing method acquires photographic image data of an attention printer being a printer to be inspected, corrects a color of attention photographic image data indicating a photographic image of an attention object by using a color correction parameter indicating a reference of colors, inputs the corrected attention photographic image data to a machine learning model trained to indicate a relation between the photographic image data indicating the photographic image of the object and defect data related to a defect in the appearance of the object, to acquire attention defect data related to the defect in the appearance of the attention object from the trained machine learning model, and stores the attention defect data in a storage.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present specification relates to a technique for acquiring data regarding defects in the appearance of an object. [Background technology]

[0002] Conventionally, captured images have been used for various processes. For example, Patent Document 1 proposes the following technology. A marker placed in a moving environment is captured, and the current position of the moving object is obtained using the marker. In addition, the captured image is applied to a learning device to obtain a corrected captured image as if it had been captured under a reference lighting condition. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-129241 A Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, machine learning models are also used to inspect the appearance of various products such as printers and scanners. For example, photographed image data of an object to be inspected is input to the machine learning model. The machine learning model outputs defect data regarding defects in the appearance of the object. The defect data indicates, for example, a pass / fail judgment result, an image of a detected scratch, and the like. Here, since the photographing environment of the object may vary in various ways, various photographed image data may be generated. As a result, depending on the photographed image data, there are cases where appropriate defect data cannot be obtained from the machine learning model.

[0005] This specification discloses techniques for obtaining appropriate failure data. [Means for solving the problem]

[0006] The technology disclosed in this specification can be realized in the following application examples.

[0007] [Application Example 1] A method for acquiring data regarding defects in the appearance of an object, comprising: a correction step of correcting color of target photographed image data that indicates a photographed image of the target object using color correction parameters that indicate a color standard; and an acquisition step of acquiring target defect data regarding defects in the appearance of the target object from a trained machine learning model by inputting the corrected target photographed image data into the machine learning model trained to indicate a relationship between photographed image data that indicates a photographed image of the object and defect data regarding the defects in the appearance of the object.

[0008] According to this configuration, the color of the target photographed image data is corrected using a color correction parameter that indicates a color standard, and the corrected target photographed image data is input into the machine learning model, so that appropriate defect data can be obtained when the color of the target photographed image data may deviate from the standard. [Application example 2] The acquisition method according to Application Example 1, The trained machine learning model includes a plurality of parameters; The plurality of parameters of the trained machine learning model are adjusted using a plurality of defect data acquired by inputting a plurality of learning photographed image data into the machine learning model; The color correction parameters are calculated using the plurality of learning captured image data. How to get it. [Application example 3] The acquisition method according to Application Example 2, The plurality of parameters of the trained machine learning model are adjusted so that a condition including a loss indicating a difference between a plurality of defect data acquired by inputting the plurality of confirmation photographed image data corrected using the color correction parameters into the machine learning model having a plurality of adjusted parameters and a plurality of teacher data respectively corresponding to the plurality of confirmation photographed image data is equal to or less than a reference value is satisfied. How to get it. [Application example 4] The acquisition method according to Application Example 1, The color correction parameters are predetermined, The trained machine learning model includes a plurality of parameters; The plurality of parameters of the trained machine learning model are adjusted using a plurality of defect data obtained by inputting a plurality of learning photographed image data corrected using the color correction parameters into the machine learning model. How to get it. [Application example 5] The acquisition method according to any one of Application Examples 1 to 4, the color correction parameters indicate a brightness criterion and a white balance criterion; The correction step includes: performing a brightness correction process for bringing the brightness of the target photographed image data closer to the brightness standard; a step of performing a white balance correction process for bringing the white balance of the target photographed image data closer to the white balance standard; Including, how to obtain. [Application Example 6] The acquisition method according to Application Example 5, The lightness standard indicates a standard of the L* component of the CIELAB color space, the white balance standard indicates a standard of the a* component and a standard of the b* component of the CIELAB color space, the brightness correction process includes a process of correcting the L* component of the target photographed image data using the reference for the L* component, The white balance correction process includes: a process of correcting the a* component of the target photographed image data using the reference for the a* component; a process of correcting the b* component of the target photographed image data using the reference for the b* component; Including, how to obtain. [Application Example 7] 1. An acquisition system for acquiring data relating to defects in the appearance of an object, comprising: a correction unit that corrects a color of target captured image data indicating a captured image of a target object by using a color correction parameter indicating a color reference; An acquisition unit that acquires, from a trained machine learning model that is trained to indicate a relationship between photographed image data showing a photographed image of an object and defect data related to defects in the appearance of the object, the target defect data related to defects in the appearance of the object by inputting the corrected photographed image data of interest into the trained machine learning model; An acquisition system comprising: [Application Example 8] A computer program for acquiring data relating to defects in the appearance of an object, comprising: a correction function for correcting the color of the target captured image data representing a captured image of the target object by using a color correction parameter representing a color reference; An acquisition function for acquiring defect data of interest relating to defects in the appearance of the target object from a trained machine learning model by inputting the corrected captured image data of interest into the machine learning model trained to indicate a relationship between captured image data showing an image of the target object and defect data relating to defects in the appearance of the target object; A computer program that enables a computer to realize the above.

[0009] The technology disclosed in this specification can be realized in various forms, such as a method for training a machine learning model and a training device for performing the training, a method for generating defect data using captured image data and a trained machine learning model and a generating device, a method for determining defects using captured image data and a trained machine learning model and a determination device, a computer program for realizing the functions of those methods or devices, a recording medium (e.g., a non-temporary recording medium) on which that computer program is recorded, and the like. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating an information processing device according to an embodiment; [Diagram 2] 1A and 1B are schematic diagrams showing examples of captured images, and 1C to 1E are explanatory diagrams of the capture environment and the captured images. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a machine learning model 238. [Figure 4] 13 is a flowchart illustrating an example of a training process. [Diagram 5] (A) is an explanatory diagram showing an example of a calculation formula for an average brightness value TL. (B) is an explanatory diagram showing a calculation formula for a first average white balance value Ta. (C) is an explanatory diagram showing a calculation formula for a second average white balance value Tb. (D) is an explanatory diagram showing an example of a calculation formula for a learning loss. (E) is an explanatory diagram showing an example of a calculation formula for an image brightness value PViL. (F) is an explanatory diagram showing an example of a calculation formula for an average value PVia. (G) is an explanatory diagram showing an example of a calculation formula for an average value PVib. (H) is an explanatory diagram showing an example of a calculation formula for a confirmation loss. [Figure 6] 1A is a flowchart showing an example of a correction process, FIG. 1B is a graph showing an example of a correction process for an L* value, and FIG. 1C is an explanatory diagram showing correction equations Ea and Eb. [Figure 7] 13 is a flowchart illustrating an example of an inspection process. [Figure 8] 13 is a flowchart illustrating an example of an acquisition process. [Figure 9] 13 is a flowchart showing a second embodiment of the training process. [Figure 10] 13 is a flowchart showing a second embodiment of the acquisition process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A. First Example: A1. Equipment configuration: FIG. 1 is an explanatory diagram showing an information processing device as an embodiment. In this embodiment, the information processing device 200 is, for example, a personal computer. The information processing device 200 acquires data on defects in an object to be inspected (for example, a product such as a printer) using a captured image of the object. The information processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to perform data processing, and is, for example, a CPU. The volatile storage device 220 is, for example, a DRAM, and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores programs 231 and 232, correction parameter data 237, and a machine learning model 238.

[0013] The machine learning model 238 is a prediction model that predicts data related to defects of an object using photographed image data of the object. In this embodiment, the machine learning model 238 is a program module. The first program 231 is a program for training the machine learning model 238. The second program 232 is a program for acquiring data related to defects of the object using photographed images of the object. The correction parameter data 237 indicates correction parameters for correcting the photographed images. The processor 210 temporarily stores various intermediate data used for executing the programs 231 and 232 in the storage device 215 (for example, either the volatile storage device 220 or the non-volatile storage device 230). Details of the programs 231 and 232, the correction parameter data 237, and the machine learning model 238 will be described later.

[0014] The display unit 240 is a device configured to display an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel arranged overlaid on the display unit 240. The user can input various requests and instructions to the information processing device 200 by operating the operation unit 250. The communication interface 270 is an interface for communicating with other devices (for example, a USB interface, a wired LAN interface, or a wireless interface of IEEE802.11). The digital camera 100 is connected to the communication interface 270. The digital camera 100 captures an object DV of inspection to generate captured image data of the captured image. The object DV is captured under a light source LS. Hereinafter, the object DV is assumed to be a printer (the object DV is also referred to as a printer DV).

[0015] A2.Photo: 2(A) and 2(B) are schematic diagrams showing examples of captured images. The captured images IMa and IMb are represented by the color values ​​of a plurality of pixels arranged in a matrix along a first direction Dx and a second direction Dy perpendicular to the first direction Dx. The width W1 is the number of pixels in the first direction Dx, and the height H1 is the number of pixels in the second direction Dy. In this embodiment, the color values ​​are represented by three component values ​​of R (red), G (green), and B (blue).

[0016] The first captured image IMa in Fig. 2(A) shows the first printer DVa without any defect. Specifically, a label LB is attached to the first printer DVa, and the first captured image IMa includes an image of the label LB. The second captured image IMb in Fig. 2(B) shows the second printer DVb with a defect. Specifically, a label LB is not attached to the second printer DVb, and the second captured image IMb does not include an image of the label LB. In this way, in this embodiment, it is inspected whether or not the label LB is attached to the printer.

[0017] 2(C)-2(E) are explanatory diagrams of the shooting environment and the shot images. FIG. 2(C)-2(E) respectively show shot images IMx, IMy, IMz of the same printer DV. The conditions of the light sources LSx, LSy, LSz at the time of shooting are different between these shot images IMx, IMy, IMz. As described below, when the conditions of the light source are different between the multiple shot images, the color of the printer DV in the multiple shot images may be different between the multiple shot images. For example, the types of the light sources LSx, LSy, LSz may be different from each other, such as a fluorescent lamp, an incandescent lamp, and a light emitting diode. The spectral distribution of the light of the light source differs depending on the type of the light source. Therefore, the color of the printer DV may be different between the shot images IMx, IMy, IMz. In addition, although the type of the light source is the same between the light sources LSx, LSy, LSz, the total lighting time may be different from each other. The spectral distribution of the light of the light source may change according to an increase in the total lighting time. Therefore, the color of the printer DV may be different between the shot images IMx, IMy, IMz.

[0018] If the state of the light source LS differs between when an image for training the machine learning model 238 is captured and when an image for inspecting the printer is captured, the color of the printer may differ between the image for training and the image for inspection. Such color differences may reduce the accuracy of prediction by the machine learning model 238. Therefore, in this embodiment, training of the machine learning model 238 and prediction by the machine learning model 238 are performed so as to mitigate the influence of differences in the color of the printer caused by differences in the state of the light source LS.

[0019] A3. Machine learning model configuration: 3 is a block diagram showing an example of the machine learning model 238. The machine learning model 238 receives input of captured image data Din indicating a captured image IM of the printer DV. As the captured image IM, images of the printer DV captured under various light source conditions, as shown in FIGS. 2(A) to 2(E), are used. The captured image data Din is bitmap data indicating three component values ​​(R value, G value, B value) of each of (H1×W1) pixels, i.e., data indicating (H1×W1×3) values.

[0020] The machine learning model 238 uses the input photographed image data Din to predict defect data Dout related to defects in the appearance of the printer DV. In this embodiment, the defect data Dout is binary data indicating either a first value (here, 1) indicating a pass or a second value (here, zero) indicating a fail.

[0021] In this embodiment, the machine learning model 238 is an artificial neural network called a CNN (Convolutional Neural Network). The machine learning model 238 includes a first convolution layer 510, a first pooling layer 520, a second convolution layer 530, a second pooling layer 540, a first fully connected layer 550, a second fully connected layer 560, a third fully connected layer 570, and a determination unit 580. These layers 510-580 are connected in this order.

[0022] The captured image data Din is input to the first convolutional layer 510.

[0023] The convolutional layers 510 and 530 execute processing including convolution processing and bias addition processing on input data. The convolution processing is processing in which s filters of (p×q×r) dimensions are sequentially applied to data input from the immediately preceding layer, and a correlation value indicating the correlation between the input data and the filters is calculated (p, q, r, and s are integers equal to or greater than 1). In processing for applying each filter, multiple correlation values ​​are sequentially calculated while sliding the filter. One filter includes (p×q×r) weights. The bias addition processing is processing for adding a bias to the calculated correlation value. One bias is prepared for each filter. Note that the filter dimensions (p×q×r) and the number s of filters are usually different between the multiple convolutional layers 510 and 530. The convolutional layers 510 and 530 each have a parameter set P1s, P2s including multiple weights and multiple biases of multiple filters.

[0024] The data generated by the convolution layers 510 and 530 is input to an activation function and transformed, and then input to the next layer (pooling layer 520 and 540). In this embodiment, ReLU (Rectified Linear Unit) is used as the activation function.

[0025] The pooling layers 520 and 540 perform max pooling on the data input from the immediately preceding convolutional layer to reduce the number of dimensions of the data. Max pooling is a process of reducing the number of dimensions by so-called downsampling, and reduces the number of dimensions by selecting the maximum value in a window while sliding a window of a predetermined size (e.g., 2×2) with a predetermined stride (e.g., 2). The data generated by the pooling layers 520 and 540 is input directly to the next layer (layers 530 and 550).

[0026] The fully connected layers 550, 560, and 570 output n-dimensional data (i.e., n values, n is an integer of 1 or more) using m-dimensional data (i.e., m values, m is an integer of 2 or more) input from the immediately preceding layer. Each of the n output values ​​is a value (inner product + bias) obtained by adding a bias to the inner product of a vector consisting of m input values ​​and a vector consisting of m weights. The fully connected layers 550, 560, and 570 output n-dimensional data using (m×n) weights and n biases. Note that the number of dimensions m of the input data and the number of dimensions n of the output data are usually different between the multiple fully connected layers 550, 560, and 570. In this embodiment, the number of dimensions n of the data output from the upstream fully connected layers 560 and 570 is 2 or more, and the number of dimensions n of the data output from the third fully connected layer 570 is 1. The fully connected layers 550, 560, and 570 each have a parameter set P3s, P4s, and P5s that includes a plurality of weights and a plurality of biases.

[0027] Each value generated by the upstream fully connected layers 550 and 560 is input to an activation function and transformed, and then input to the next layer (fully connected layers 560 and 570). For example, ReLU is used as the activation function.

[0028] The data generated by the third fully connected layer 570 is input to an activation function and transformed. For example, Sigmoid is used as the activation function. Sigmoid transforms all real numbers into values ​​in the range from zero to one.

[0029] The determination unit 580 compares the value input from the immediately preceding third fully connected layer 570 with a determination threshold to output defect data Dout indicating one of two classes. In this embodiment, the two classes are a first class CL1 (also called pass class CL1) indicating that the printer DV has no defect, and a second class CL2 (also called fail class CL2) indicating that the printer DV has a defect. Specifically, the pass class CL1 indicates that the image captured by the printer DV includes an image of the label LB, and the fail class CL2 indicates that the image captured by the printer DV does not include an image of the label LB. If the input value is equal to or greater than the determination threshold, the determination unit 580 outputs defect data Dout of "1", and if the input value is less than the determination threshold, the determination unit 580 outputs defect data Dout of "zero". The defect data Dout of "1" indicates the pass class CL1, and the defect data Dout of "zero" indicates the fail class CL2. The determination unit 580 has a parameter set P6s including a determination threshold.

[0030] The machine learning model 238 is trained so that the output defect data Dout indicates an appropriate class. The output value from the third fully connected layer 570 of the trained machine learning model 238 indicates the confidence that the class of the captured image IM is the pass class CL1. The confidence is correlated with the probability that the pass class CL1 is the correct answer.

[0031] In FIG. 3, the numbers on the left side of each layer (e.g., H1×W1×D1) indicate the number of dimensions of the data input to that layer. The numbers on the right side of each layer (e.g., H2×W2×D1) indicate the number of dimensions of the data output from that layer. The number of dimensions of each data may be various values. For example, H1=W1=500, D1=8, H2=W2=50, D2=16, H3=W3=25, D3=512, D4=64.

[0032] A4. Training process: 4 is a flowchart showing an example of a training process of the machine learning model 238 (FIG. 3). The machine learning model 238 is trained so that the defect data Dout indicates an appropriate class of the printer DV of the captured image IM. By the training, a plurality of calculation parameters (including parameter sets P1s-P6s) used in the calculation of each of the plurality of layers 510-580 are adjusted. The processor 210 performs the training according to the first program 231.

[0033] In S110, the processor 210 generates a labeled data set. First, the worker photographs the printer using the digital camera 100. As a result, the digital camera 100 generates photographed image data. In this embodiment, the first type of photographed image data is generated by photographing the first type of printer, which is a printer having the label LB, and the second type of photographed image data is generated by photographing the second type of printer, which is a printer not having the label LB. For appropriate training, a large number of first type photographed image data and a large number of second type photographed image data are generated. In order to suppress bias between the two classes CL1 and CL2, the same number of first type photographed image data and second type photographed image data may be generated. For example, 500 pieces of first type photographed image data and 500 pieces of second type photographed image data may be generated. Here, the worker may photograph a plurality of first type printers and a plurality of second type printers. In addition, the worker may photograph one first type printer multiple times and one second type printer multiple times.

[0034] The processor 210 acquires the photographed image data from the digital camera 100. The processor 210 also causes the worker to input class information indicating a class corresponding to the photographed image data. The processor 210 then associates the photographed image data with data indicating class information corresponding to the photographed image data (also called label data or teacher data), and stores the data in the storage device 215 (for example, the non-volatile storage device 230). In this embodiment, the label data indicates a target value (i.e., a correct answer) of the defect data Dout. Specifically, the label data associated with the first type of photographed image data is "1" indicating a pass class CL1, and the label data associated with the second type of photographed image data is "zero" indicating a fail class CL2. Hereinafter, the photographed image data and the label data associated with each other as a whole are also referred to as labeled data. The whole of the plurality of labeled data is also referred to as a labeled data set.

[0035] Note that a plurality of captured image data may be added by so-called data augmentation. Furthermore, the generation of captured image data and the input of class information (and thus S110) may be performed in advance before the start of the training process.

[0036] In S120, the processor 210 divides each of the labeled dataset of the first class CL1 and the labeled dataset of the second class CL2 into a training dataset and a confirmation dataset. For example, the processor 210 adopts 70% of the randomly selected labeled dataset as the training dataset, and the remaining 30% of the labeled dataset as the confirmation dataset. The division of the labeled dataset of the first class CL1 and the division of the labeled dataset of the second class CL2 are performed independently. Hereinafter, it is assumed that the total number of labeled data in the training dataset is Nt, and the total number of labeled data in the confirmation dataset is Nv (Nt and Nv are both integers equal to or greater than 2). Hereinafter, the photographed image data included in the training dataset is also referred to as the training photographed image data, and the image represented by the training photographed image data is also referred to as the training photographed image. Moreover, the photographed image data included in the confirmation dataset is also referred to as the confirmation photographed image data, and the image represented by the confirmation photographed image data is also referred to as the confirmation photographed image.

[0037] In S130, the processor 210 calculates the average luminance value TL and the average white balance values ​​Ta and Tb of the learning data set. In this embodiment, the processor 210 first converts the color values ​​(R, G, B) of each of the pixels of the multiple captured image data of the learning data set into color values ​​(L*, a*, b*) of the CIELAB color space. The CIELAB color space is a color space recommended by the International Commission on Illumination (CIE) in 1976, and is also called the CIE 1976 L*a*b* color space. When the color space expressing the color values ​​of the captured image data is a device-independent color space (e.g., sRGB color space), a known conversion formula that defines the correspondence between two color spaces is used as the conversion formula for the color values. When the color space expressing the color values ​​of the captured image data is a device-dependent color space, the conversion formula for the color values ​​is experimentally determined in advance, taking into account the characteristics of the digital camera 100.

[0038] The processor 210 calculates the average luminance value TL and the average white balance values ​​Ta and Tb using the color values ​​(L*, a*, b*). FIG. 5(A) is an explanatory diagram showing an example of a calculation formula for the average luminance value TL. The processor 210 calculates the average value PSiL (also called the image luminance value PSiL) of the L* values ​​for each captured image. The processor 210 then calculates the average luminance value TL, which is the average value of the image luminance values ​​PSiL of the Nt captured images. PSi(x, y)L in FIG. 5(A) is the L* value of the pixel at coordinates (x, y) of the i-th captured image. The coordinate value x indicates the position of the pixel in the first direction Dx (FIG. 2(A)), and the coordinate value y indicates the position of the pixel in the second direction Dy. The coordinate values ​​x and y are assigned in order from 1 so as to be arranged in ascending order in the directions Dx and Dy, respectively. PSiw is the number of pixels in the first direction Dx of the i-th captured image, which is the same as the width W1 (Figure 2(A)). PSih is the number of pixels in the second direction Dy of the i-th captured image, which is the same as the height H1 (Figure 2(A)). The image brightness value PSiL is the average of the L* values ​​of all pixels (PSih x PSiw pixels) of the i-th captured image. The average brightness value TL is the average of the Nt image brightness values ​​PSiL.

[0039] The calculation method of the average white balance values ​​Ta and Tb is the same as the calculation method of the average lightness value TL, except that a* and b* are used instead of L*. FIG. 5B is an explanatory diagram showing a calculation formula of the first average white balance value Ta. PSi(x, y)a in FIG. 5B is the a* value of the pixel at the coordinates (x, y) of the i-th photographed image. The image first value PSia is the average of the a* values ​​of all the pixels of the i-th photographed image. The first average white balance value Ta is the average of the Nt image first values ​​PSia. FIG. 5C is an explanatory diagram showing a calculation formula of the second average white balance value Tb. PSi(x, y)b in FIG. 5C is the b* value of the pixel at the coordinates (x, y) of the i-th photographed image. The image second value PSib is the average of the b* values ​​of all the pixels of the i-th photographed image. The second average white balance value Tb is the average of the Nt image second values ​​PSib.

[0040] The average luminosity value TL indicates the average brightness of the multiple captured images in the learning data set. The a* and b* values ​​indicate the magnitude of deviation from achromatic color. Therefore, the average white balance values ​​Ta and Tb indicate the average bias of the white balance of the multiple captured images in the learning data set. These values ​​TL, Ta, and Tb may vary depending on the conditions of the light source LS at the time of shooting.

[0041] In S140 (FIG. 4), the processor 210 stores the correction parameter data 237, which is data indicating the values ​​TL, Ta, and Tb, in the storage device 215 (for example, the non-volatile storage device 230). The values ​​TL, Ta, and Tb are used in the color correction process described below. Hereinafter, the values ​​TL, Ta, and Tb are also referred to as color correction parameters TL, Ta, and Tb.

[0042] In S150, the processor 210 initializes a plurality of calculation parameters (including parameter sets P1s-P6s) of the machine learning model 238. For example, each calculation parameter is set to a random value.

[0043] In S158, the processor 210 executes a correction process to correct the color of each of the Nv pieces of confirmation captured image data. Fig. 6(A) is a flowchart showing an example of the correction process. The correction process in Fig. 6(A) shows the correction process for one piece of captured image data. The processor 210 executes the correction process in Fig. 6(A) for each of the Nv pieces of confirmation captured image data.

[0044] In S510, the processor 210 converts the color values ​​(R, G, B) of each of the pixels of the confirmation captured image data to be processed into color values ​​(L*, a*, b*) in the CIELAB color space. The color value conversion process is the same as the conversion process described in S130.

[0045] In S520, the processor 210 performs a brightness correction process to correct the brightness of the confirmation captured image. In this embodiment, the L* value is corrected. FIG. 6B is a graph showing an example of the L* value correction process. The horizontal axis indicates the input value Lin, which is the L* value before correction, and the vertical axis indicates the output value Lout, which is the L* value after correction. Lmax in the figure is the maximum value of L*. The curve TCi showing the correspondence between the input value Lin and the output value Lout is also called a tone curve.

[0046] In this embodiment, the processor 210 performs gamma correction processing of the L* value. The processor 210 calculates an image brightness value PViL, which is an average of the pre-correction L* values ​​of the confirmation captured image data to be processed. The processor 210 determines a tone curve TCi for the gamma correction processing so that the image brightness value PViL is converted to an average brightness value TL. Then, the processor 210 corrects the L* value of each of the multiple pixels of the confirmation captured image data based on the tone curve TCi. In this way, the color correction parameter TL indicates a reference for the L* value.

[0047] Fig. 5(E) is an explanatory diagram showing an example of a calculation formula for the image brightness value PViL. PVi(x, y)L in Fig. 5(E) is the L* value of the pixel at coordinates (x, y) of the i-th confirmation-captured image. PViw is the number of pixels in the first direction Dx of the i-th confirmation-captured image, which is the same as the width W1 (Fig. 2(A)). PVih is the number of pixels in the second direction Dy of the i-th confirmation-captured image, which is the same as the height H1 (Fig. 2(A)). The image brightness value PViL is the average of the L* values ​​of all pixels (PVih x PViw pixels) of the i-th confirmation-captured image.

[0048] In S530 (FIG. 6(A)), the processor 210 performs a white balance correction process to correct the white balance of the confirmation captured image to be processed. In this embodiment, S530 includes a process S533 to correct the a* value and a process S536 to correct the b* value. FIG. 6(C) is an explanatory diagram showing the a* correction formula Ea and the b* correction formula Eb. In this embodiment, the a* value and the b* value of each pixel are corrected so that the average value of the a* value after correction approaches the first average white balance value Ta, and the average value of the b* value after correction approaches the second average white balance value Tb. In this way, the first average white balance value Ta indicates the reference of the a* value, and the second average white balance value Tb indicates the reference of the b* value.

[0049] Specifically, it is as follows. PVi(x, y)a in the first correction formula Ea is the a* value of the pixel at coordinates (x, y) of the i-th confirmation captured image. PVi(x, y)a on the left side indicates the a* value after correction, and PVi(x, y)a on the right side indicates the a* value before correction. PVia is the average a* value of all pixels in the i-th confirmation captured image (see FIG. 5(F)). In this way, the processor 210 calculates the average a* value PVia of the captured image data to be processed, and corrects the a* value of each pixel by adding the difference obtained by subtracting the average value PVia from the first average white balance value Ta. As a result, the average a* value after correction becomes closer to the first average white balance value Ta.

[0050] The second correction formula Eb for the b* value is determined in the same manner. PVi(x, y)b is the b* value of the pixel at the coordinates (x, y) of the i-th confirmation captured image. PVi(x, y)b on the left side indicates the b* value after correction, and PVi(x, y)b on the right side indicates the b* value before correction. PVib is the average b* value of all pixels of the i-th confirmation captured image (see FIG. 5(G)). The processor 210 calculates the average b* value PVib of the captured image data to be processed, and corrects the b* value of each pixel by adding the difference obtained by subtracting the average PVib from the second average white balance value Tb. As a result, the average b* value after correction becomes closer to the second average white balance value Tb.

[0051] In S540 (FIG. 6(A)), the processor 210 converts the corrected color values ​​(L*, a*, b*) into color values ​​(R, G, B) in the original color space. This color conversion process is a conversion process in the opposite direction to the color conversion process of S510. In this way, the processor 210 generates corrected confirmation photographed image data. The processor 210 generates Nv corrected confirmation photographed image data by executing the correction process of FIG. 6(A) on each of the Nv confirmation photographed image data. Then, the process of FIG. 6(A), i.e., S158 of FIG. 4, ends. Thereafter, the processor 210 uses the entire corrected confirmation photographed image data and the corresponding label data as corrected labeled data. In addition, the set of Nv corrected labeled data is used as a corrected confirmation data set.

[0052] In S160, the processor 210 calculates the learning loss by using the learning data set. Specifically, the processor 210 inputs Nt pieces of learning photographed image data to the machine learning model 238 to generate Nt pieces of defect data Dout. Then, the processor 210 calculates the learning loss by using the Nt pieces of defect data Dout and the Nt pieces of label data associated with the Nt pieces of learning photographed image data. FIG. 5(D) is an explanatory diagram showing an example of a calculation formula (also called a loss function) for the learning loss. Oi is a value of the defect data Dout output by using the i-th learning photographed image data (also called a predicted value Oi). Ti is a value of the label data associated with the i-th learning photographed image data (also called a teacher value Ti). The learning loss Loss_s is the average value of the square root of the difference between these values ​​Oi and Ti. Such a learning loss Loss_s indicates the average value of the absolute value of the difference between the values ​​Oi and Ti. As described above, these values ​​Oi and Ti indicate 1 or zero. When the predicted value Oi is correct, the absolute value of the difference between the values ​​Oi and Ti is 0. When the predicted value Oi is incorrect, the absolute value of the difference between the values ​​Oi and Ti is 1. The learning loss Loss_s becomes smaller as the proportion of correct predicted values ​​Oi among the Nt predicted values ​​Oi increases.

[0053] In S170 (FIG. 4), the processor 210 uses the learning loss Loss_s to adjust a plurality of calculation parameters (including the parameter sets P1s-P6s) of the machine learning model 238. Specifically, the processor 210 adjusts the calculation parameters according to a predetermined algorithm so as to reduce the learning loss Loss_s. As the predetermined algorithm, for example, an algorithm using the backpropagation method and the gradient descent method is used.

[0054] In S190, the processor 210 calculates the confirmation loss using the corrected confirmation data set. The calculation method of the confirmation loss is the same as the calculation method of the learning loss Loss_s described in S160, except that the corrected confirmation photographed image data is used instead of the learning photographed image data. Specifically, the processor 210 inputs the Nv corrected confirmation photographed image data to the machine learning model 238 having the calculation parameters updated in S170 to generate Nv defect data Dout. Then, the processor 210 calculates the confirmation loss using the Nv defect data Dout and the Nv label data associated with the Nv corrected confirmation photographed image data. FIG. 5(H) is an explanatory diagram showing an example of a calculation formula (also called a loss function) of the confirmation loss. Oi is the value of the defect data Dout output using the i-th corrected confirmation photographed image data (i.e., the predicted value Oi). Ti is the value of the label data associated with the i-th corrected confirmation photographed image data (i.e., the teacher value Ti). The confirmation loss Loss_v is the average of the square root of the squares of the differences between these values ​​Oi and Ti.

[0055] In S200 (FIG. 4), the processor 210 determines whether the training is completed. The condition for completing the training may be various conditions. In this embodiment, the condition for completing the training is that both the learning loss Loss_s and the confirmation loss Loss_v are equal to or less than a predetermined reference value (e.g., 0.00001). Note that the condition for completing the training may be various conditions that indicate that both the learning loss Loss_s and the confirmation loss Loss_v are small. For example, the reference value of the learning loss Loss_s may be different from the reference value of the confirmation loss Loss_v.

[0056] If the training is not completed (S200: No), the processor 210 proceeds to S160 and continues the training. If the training is completed (S200: Yes), in S210, the processor 210 stores the machine learning model 238 including the adjusted calculation parameters as a trained model in the storage device 215 (here, the non-volatile storage device 230). Then, the processor 210 ends the processing of FIG. 4.

[0057] A5.Inspection process: 7 is a flowchart showing an example of an inspection process for an object. In this embodiment, it is inspected whether or not a label LB is attached to the printer DV (FIG. 2(A)). The processor 210 performs the inspection process according to the second program 232.

[0058] In S710, the processor 210 performs an acquisition process of the defect data Dout. FIG. 8 is a flowchart showing an example of the acquisition process. In S310, the processor 210 acquires captured image data of the target printer, which is the printer to be inspected. In this embodiment, it is assumed that the target printer is placed within the shooting range of the digital camera 100 (FIG. 1). The processor 210 supplies a shooting instruction to the digital camera 100. In response to the instruction, the digital camera 100 captures an image of the target printer and generates captured image data. The processor 210 acquires the captured image data from the digital camera 100. Hereinafter, the captured image data of the target printer is also referred to as the target captured image data.

[0059] In S320, the processor 210 acquires the correction parameter data 237 from the storage device 215. In S330, the processor 210 executes a correction process for correcting the color of the target photographed image data. This correction process is the same as the correction process in S158 of Fig. 4 and Fig. 6(A) except that the image data to be processed is the target photographed image data rather than the confirmation photographed image data. By this correction process, the average values ​​of L*, a*, and b* of the corrected target photographed image data become closer to the color correction parameters TL, Ta, and Tb, respectively.

[0060] In S340, the processor 210 inputs the corrected captured image data of interest into the trained machine learning model 238, thereby acquiring defect data Dout of the printer of interest from the machine learning model 238 (hereinafter, also referred to as the defect data of interest Dout). In S350, the processor 210 stores the defect data of interest Dout in the storage device 215 (e.g., the non-volatile storage device 230). With the above, the process of FIG. 8, that is, S710 of FIG. 7, ends. Note that the information processing device 200 (FIG. 1) that executes the process of FIG. 8 is an example of an acquisition system that acquires the defect data Dout.

[0061] In S720, the processor 210 outputs the inspection result. Then, the processor 210 ends the process of FIG. 7. The inspection result is an inspection result of the defect of the noted printer identified using the noted defect data Dout. In this embodiment, the inspection result is either "pass" or "fail", and is the same as the information indicated by the noted defect data Dout. The method of outputting the inspection result may be any method. For example, the processor 210 may output the inspection result data indicating the inspection result to a storage device (for example, the non-volatile storage device 230, or an external storage device not shown connected to the information processing device 200) (thereby, the inspection result data is stored in the storage device). In addition, the processor 210 may output image data indicating the inspection result to the display unit 240 (thereby, the inspection result is displayed on the display unit 240). In response to the completion of the output of the inspection result (S720), the processor 210 ends the process of FIG. 7.

[0062] As described above, in this embodiment, the processor 210 acquires defect data by executing the process of FIG. 8. Specifically, in S330, the processor 210 corrects the color of the target captured image data indicating the captured image of the target printer using the color correction parameters TL, Ta, and Tb indicating the color reference (see FIG. 6(A)). In S340, the processor 210 inputs the corrected target captured image data to the trained machine learning model 238, thereby acquiring the target defect data Dout from the trained machine learning model 238. As described in FIG. 2(A), FIG. 2(B), and FIG. 3, the target defect data Dout indicates information on a defect in the appearance of the target printer. Also, as described in FIG. 3 and FIG. 4, the machine learning model 238 is trained to indicate the relationship between the captured image data indicating the captured image of the printer and the defect data related to the defect in the appearance of the printer.

[0063] With this configuration, the processor 210 can obtain appropriate attention defect data Dout when the color of the attention photographed image data may deviate from the standard. For example, the attention printer may be photographed on a day different from the day of photographing for the training photographed image data. Therefore, the condition of the light source LS when the attention printer is photographed may be different from the condition of the light source LS when the attention printer is photographed for training. As a result, the color of the attention photographed image data may deviate from the color of the photographed image data used for training the machine learning model 238. When the attention photographed image data having the deviated color is input to the machine learning model 238, inappropriate attention defect data Dout may be generated. In this embodiment, the corrected attention photographed image data is input to the machine learning model 238, so that appropriate attention defect data Dout can be obtained.

[0064] As described in FIG. 3, the trained machine learning model 238 includes a plurality of parameters (specifically, parameter sets P1s-P6s). As described in S160 and S170 in FIG. 4, the plurality of parameters (including parameter sets P1s-P6s) of the trained machine learning model 238 are adjusted using Nt pieces of defect data Dout acquired by inputting Nt pieces of learning photographed image data to the machine learning model 238. Then, as described in S130, the color correction parameters TL, Ta, and Tb are calculated using Nt pieces of learning photographed image data. Therefore, the correction of the color of the target photographed image data (FIG. 8: S330) can reduce the deviation between the color of the corrected target photographed image data and the standard (in this embodiment, the color correction parameters TL, Ta, and Tb) calculated using the multiple learning photographed image data. As a result, in the acquisition process in FIG. 8, the processor 210 can acquire appropriate target defect data Dout when the color of the target photographed image data may deviate from the standard calculated using the multiple learning photographed image data.

[0065] As described in S158 of FIG. 4, the processor 210 generates Nv corrected confirmation captured image data by correcting the colors of the Nv confirmation captured image data using the color correction parameters TL, Ta, and Tb. In S190, the processor 210 inputs the Nv corrected confirmation captured image data to the machine learning model 238 having the calculation parameters adjusted in S170, and generates Nv defect data Dout. The processor 210 calculates a confirmation loss Loss_v indicating a difference between the Nv defect data Dout and the Nv teacher data using the Nv defect data Dout and the Nv label data (i.e., teacher data) respectively associated with the Nv confirmation captured image data. In S200, the processor 210 ends the training when a condition including that the confirmation loss Loss_v is equal to or less than a reference value is satisfied. That is, multiple parameters of the trained machine learning model 238 (including parameter sets P1s-P6s) are adjusted so that conditions including the validation loss Loss_v being below a reference are satisfied.

[0066] In this way, the machine learning model 238 is trained to output appropriate defect data Dout when the confirmation photographed image data corrected using the color correction parameters TL, Ta, and Tb is input to the machine learning model 238. That is, the multiple parameters (including the parameter sets P1s-P6s) of the machine learning model 238 are adjusted to be suitable for the multiple confirmation photographed image data corrected using the color correction parameters TL, Ta, and Tb. Therefore, the machine learning model 238 can output appropriate noted defect data Dout for various noted photographed image data corrected using the color correction parameters TL, Ta, and Tb. In the acquisition process (FIG. 8), the processor 210 can acquire appropriate noted defect data Dout by using such a machine learning model 238.

[0067] As described in S130 of FIG. 4 and in FIGS. 6(A)-6(C), the color correction parameters TL, Ta, and Tb indicate a value TL indicating a standard of brightness (also called a brightness standard TL) and values ​​Ta and Tb indicating a standard of white balance (also called white balance standards Ta and Tb). The correction process of the target photographed image data (S330 of FIG. 8 and FIG. 6(A)) includes a brightness correction process (S520 of FIG. 6(A)) and a white balance correction process (S530 of FIG. 6(A)). The brightness correction process is a process of bringing the brightness of the target photographed image data closer to the brightness standard TL, as described in FIG. 6(B). Specifically, the average value of L* of the corrected target photographed image data approaches the brightness standard TL. The white balance correction process is a process of bringing the white balance of the target photographed image data closer to the white balance standards Ta and Tb, as described in FIG. 6(C). Specifically, the average value of a* and the average value of b*, which indicate the white balance of the corrected target photographed image data, approach the white balance standards Ta and Tb. As a result, in the acquisition process of Fig. 8, the processor 210 can acquire appropriate target defect data Dout when the light source conditions at the time of photographing the target printer can change.

[0068] As described in S130 of FIG. 4 and FIG. 6(A), the lightness reference TL indicates the reference of the L* component of the CIELAB color space. The white balance references Ta and Tb indicate the reference of the a* component and the b* component of the CIELAB color space, respectively. The lightness correction process (FIG. 6(A): S520) includes a process of correcting the L* component of the target photographed image data using the reference TL of the L* component. The white balance correction process (FIG. 6(A): S530) includes a process of correcting the a* component of the target photographed image data using the reference Ta of the a* component (S533) and a process of correcting the b* component of the target photographed image data using the reference Tb of the b* component (S536). As described above, the processor 210 can appropriately correct the lightness and white balance of the target photographed image data, and therefore can obtain appropriate defect data.

[0069] B. Second Example: B1. Training process: FIG. 9 is a flowchart showing a second embodiment of the training process of the machine learning model 238 (FIG. 3). The difference from the embodiment of FIG. 4 is that predetermined color correction parameters TL2, Ta2, Tb2 are used instead of the color correction parameters TL, Ta, Tb, and corrected learning captured image data is used instead of the learning captured image data. Hereinafter, the color correction parameters TL, Ta, Tb of the first embodiment are also referred to as first-type correction parameters TL, Ta, Tb, and the color correction parameters TL2, Ta2, Tb2 of the second embodiment are also referred to as second-type correction parameters TL2, Ta2, Tb2. In this embodiment, in order to train the machine learning model 238 to be suitable for the second-type correction parameters TL2, Ta2, Tb2, in addition to the confirmation captured image data, the learning captured image data is also corrected using the second-type correction parameters TL2, Ta2, Tb2. The configuration of the information processing device that executes the training process is the same as the configuration of the information processing device 200 of FIG. 1. Correction parameter data 237 indicating the second type correction parameters TL2, Ta2, and Tb2 are stored in advance in the storage device 215 (for example, the non-volatile storage device 230). Various methods may be used to determine the second type correction parameters TL2, Ta2, and Tb2. For example, the printer DV may be photographed to determine the second type correction parameters TL2, Ta2, and Tb2. Then, representative values ​​of L*, a*, and b* (for example, values ​​having a correlation with color values ​​of multiple pixels, such as average values ​​and median values) of the photographed image data acquired by this photographing may be used as the second type correction parameters TL2, Ta2, and Tb2.

[0070] S110 and S120 in Fig. 9 are the same as S110 and S120 in Fig. 4. The following S150 is the same as S150 in Fig. 4.

[0071] In S154a, the processor 210 executes a correction process for correcting the color of each of the Nt pieces of learning captured image data. This correction process is the same as S158 in FIG. 4 and the correction process in FIG. 6(A), except that the second correction parameters TL2, Ta2, and Tb2 are used instead of the first correction parameters TL, Ta, and Tb, and the image data to be processed is learning captured image data instead of confirmation captured image data. The processor 210 acquires correction parameter data 237 indicating the second correction parameters TL2, Ta2, and Tb2 from the storage device 215. Then, the processor 210 executes a correction process for each of the Nt pieces of learning captured image data to generate Nt pieces of corrected learning captured image data. By this correction process, the average values ​​of L*, a*, and b* of the corrected learning captured image data become closer to the second correction parameters TL2, Ta2, and Tb2, respectively. Thereafter, the processor 210 uses the entire corrected learning captured image data and the corresponding label data as corrected learning labeled data. The processor 210 also uses a set of Nt corrected learning labeled data as a corrected learning data set.

[0072] In S158a, the processor 210 executes a correction process to correct the color of each of the Nv pieces of confirmation photographed image data. This correction process is the same as S158 in FIG. 4 and the correction process in FIG. 6(A) except that the second type correction parameters TL2, Ta2, Tb2 are used instead of the first type correction parameters TL, Ta, Tb. The processor 210 acquires the correction parameter data 237 indicating the second type correction parameters TL2, Ta2, Tb2 from the storage device 215. Then, the processor 210 executes a correction process on each of the Nv pieces of confirmation photographed image data to generate Nv pieces of corrected confirmation photographed image data. By this correction process, the average values ​​of L*, a*, b* of the corrected confirmation photographed image data become closer to the second type correction parameters TL2, Ta2, Tb2, respectively. Thereafter, the processor 210 uses the entirety of the corrected confirmation photographed image data and the corresponding label data as the corrected confirmation labeled data. The processor 210 also utilizes the set of Nv corrected verification labeled data as a corrected verification data set.

[0073] In S160a, the processor 210 calculates the learning loss Loss_s by using the corrected learning data set. The calculation process of the learning loss Loss_s is the same as the process of S160 in FIG. 4, except that the corrected learning data set is used instead of the uncorrected learning data set.

[0074] S170, S190, and S200 are the same as S170, S190, and S200 in Fig. 4, respectively. If the training is not completed (S200: No), the processor 210 proceeds to S160a and continues the training. If the training is completed (S200: Yes), in S210, the processor 210 stores the machine learning model 238 including the adjusted calculation parameters as a trained model in the storage device 215 (here, the non-volatile storage device 230). Then, the processor 210 ends the processing of Fig. 9.

[0075] B2. Specific Processing: FIG. 10 is a flowchart showing a second embodiment of the acquisition process. This acquisition process is the same as the specification process of FIG. 8, except that second-type correction parameters TL2, Ta2, and Tb2 are used instead of the first-type correction parameters TL, Ta, and Tb. Specifically, the specification process of FIG. 10 is obtained by replacing S320 and S330 of the specification process of FIG. 8 with S320a and S330a using the second-type correction parameters TL2, Ta2, and Tb2, respectively. In S320a, the processor 210 acquires the correction parameter data 237 indicating the second-type correction parameters TL2, Ta2, and Tb2 from the storage device 215. In S330a, the processor 210 corrects the color of the target captured image data (see FIG. 6(A)). Here, the second-type correction parameters TL2, Ta2, and Tb2 are used instead of the first-type correction parameters TL, Ta, and Tb.

[0076] As described above, in this embodiment, the color correction parameters TL2, Ta2, and Tb2 are determined in advance. Then, as described in S154a of FIG. 9, the processor 210 generates Nt corrected learning photographed image data by correcting the colors of the Nt learning photographed image data using the color correction parameters TL2, Ta2, and Tb2. In S160a, the processor 210 inputs the Nt corrected learning photographed image data to the machine learning model 238 to obtain Nt defect data Dout. In S160a and S170, the processor 210 adjusts a plurality of calculation parameters (including the parameter sets P1s-P6s) of the machine learning model 238 using the Nt defect data Dout. In this way, a plurality of calculation parameters of the trained machine learning model 238 are adjusted using a plurality of defect data Dout. A plurality of defect data Dout is obtained by inputting a plurality of corrected learning photographed image data to the machine learning model 238. The multiple corrected learning captured image data are corrected using the color correction parameters TL2, Ta2, and Tb2. In this manner, the multiple calculation parameters of the machine learning model 238 are adjusted to be suitable for the color correction parameters TL2, Ta2, and Tb2. Therefore, the machine learning model 238 can output appropriate noted defect data Dout for various noted captured image data corrected using the color correction parameters TL2, Ta2, and Tb2. In the acquisition process (FIG. 10), the processor 210 can acquire appropriate noted defect data Dout by using such a machine learning model 238.

[0077] The training process and the specific process of the second embodiment are the same as those of the first embodiment, except that the second correction parameters TL2, Ta2, and Tb2 are used instead of the first correction parameters TL, Ta, and Tb, and corrected learning captured image data is used instead of learning captured image data. Therefore, the second embodiment can achieve various advantages similar to those of the first embodiment.

[0078] C. Variations: (1) The object from which the defect data Dout is acquired is not limited to a printer, but may be any product such as a scanner, a multifunction device, a digital camera, a cutting machine, a mobile terminal, etc. Also, a case for housing the product may be the object.

[0079] (2) The captured image data used for training a machine learning model (e.g., machine learning model 238 in FIG. 3) and acquiring defect data may be various image data representing captured images of objects. For example, the color space of the captured image data may be another color space such as a CMYK color space instead of RGB. The captured image data may represent an image by a luminance value. The captured image data may be generated by performing various image processing such as a resolution conversion process and a cropping process on the image data generated by the digital camera 100.

[0080] (3) The process of correcting the color of the captured image data may be various other processes instead of the process described in FIG. 6(A) and the like. For example, the color components used for the correction may be other color components instead of L*, a*, and b*. For example, three color components Y, Cb, and Cr may be corrected. The Y value is a color value indicating brightness, similar to the L* value. Therefore, the correction of the Y value can correct the brightness. The Y value may be corrected by gamma correction, similar to the correction of L*. Moreover, the Cr value and the Cb value indicate the magnitude of deviation from achromatic color, similar to the a* value and the b* value. Therefore, the correction of the Cr value and the Cb value can correct the white balance. The Cr value and the Cb value may be corrected by gamma correction, similar to the correction of L*, for example.

[0081] Also, the three color components R, G, and B may be corrected. The three color components R, G, and B collectively indicate brightness and white balance. Therefore, the correction of the R value, the G value, and the B value can correct the brightness and white balance. The R value, the G value, and the B value may be corrected by gamma correction, for example, similar to the correction of L*.

[0082] Also, the white balance correction may be omitted and the brightness correction may be performed. Alternatively, the brightness correction may be omitted and the white balance correction may be performed.

[0083] In either case, the correction process may be various processes that correct the color using a color correction parameter that indicates a color reference, instead of the gamma correction process as shown in FIG. 6B and the process of adding the difference between the representative value and the reference as shown in FIG. 6C. For example, the correction process may be various processes that bring the representative value of the color component to be corrected in the captured image data closer to the reference. In the embodiment of FIG. 6B and FIG. 6C, an average value is used as the representative value (specifically, the image brightness value PViL and the average values ​​PVia and PVib). The representative value is not limited to the average value, and may be various values ​​calculated using the color values ​​of multiple pixels in the captured image data (for example, values ​​that have a correlation with the color values ​​of multiple pixels, such as an average value, a median value, or an integrated value). The representative value may be calculated according to a predetermined calculation formula. The reference may be a target value of the representative value. The correction process may be a process of multiplying the color value by a ratio (reference / representative value).

[0084] (4) The machine learning model 238 (FIG. 3) may be trained to predict the presence or absence of various other defects instead of the presence or absence of the label LB. For example, the machine learning model 238 may predict the presence or absence of a part attached to the object using photographed image data of the object. The object may be a case that houses a part. The machine learning model 238 may predict the presence or absence of a part housed in the case using photographed image data of the object. In either case, the configuration of the machine learning model may be any configuration that classifies the photographed image into two classes instead of the configuration of FIG. 3. For example, the total number of each of the convolutional layers, pooling layers, and fully connected layers may be appropriately changed. The convolutional layers and pooling layers may be omitted, and the machine learning model may be composed of multiple fully connected layers. The machine learning model may be a CNN called VGG16. A random forest or a support vector machine may be adopted.

[0085] Further, the machine learning model may be various models that output defect data related to defects in the appearance of an object instead of a model that outputs binary defect data Dout indicating the presence or absence of a defect. For example, the machine learning model may be an object detection model that detects scratches on an object in a captured image. As the object detection model, various models can be adopted, such as a model called YOLO (You only look once) and a model called SSD (Single Shot Detector). Such an object detection model outputs data indicating, for example, an area (also called a bounding box) indicating an object such as a scratch and a certainty factor for each type of object.

[0086] Any of the machine learning models can be trained by the process of FIG. 4 or FIG. 9. Here, data showing the correct answer (i.e., target value) of the data output from the machine learning model is used as the teacher data. For example, when training an object detection model for detecting scratches, learning captured image data of an object having a scratch is used. Then, data showing the correct area showing the scratch, a high confidence level associated with the scratch, and a low confidence level associated with other types of objects is used as the teacher data.

[0087] Further, the training method of the machine learning model may be various methods suitable for the machine learning model instead of the method described in FIG. 4 and FIG. 9. For example, in the embodiment of FIG. 4 and FIG. 9, the loss function used to calculate the loss may be a function that calculates other types of loss, such as a square sum error or a cross entropy error, instead of the function of FIG. 5(D) and FIG. 5(H). In general, the loss function may be various functions that calculate an evaluation value of the difference between the data output from the machine learning model and the teacher data. For example, when an object detection model is used, the loss function may be various functions that calculate an error that is correlated with the error of the area indicating the object and the error of the confidence for each type of object.

[0088] Moreover, the method of adjusting the calculation parameters included in the machine learning model may be various other methods such as a method of propagating a target value (also called Target Propagation) instead of the error backpropagation method. Moreover, the condition for completing the training may be various conditions that indicate that the difference between the data output from the machine learning model and the teacher data is small. For example, the confirmation loss Loss_v may be omitted from the condition for completing the training. In this case, in the processing of FIG. 4 and FIG. 9, all the labeled data sets may be used as the learning data sets. Moreover, the processor 210 may determine that the training is completed when an instruction to complete is input from the operator, and may determine that the training is not completed when an instruction to continue the training is input. For example, the operator may determine whether or not to end the training by referring to the defect data output using the confirmation photographed image data. Alternatively, the condition for completing the training may be that the calculation of the learning loss (for example, the learning loss Loss_s) and the update of the calculation parameters (for example, S160-S170, S160a-S170) are repeated a predetermined number of times.

[0089] (5) The inspection result output in S720 of the inspection process (FIG. 7) is not limited to the presence or absence of a defect, and may be various information specified using defect data output from the machine learning model. For example, when the machine learning model is an object detection model, the processor 210 may output image data showing the detected object to the display unit 240 (so that an image of the object is displayed on the display unit 240). The worker can easily confirm the detected object (e.g., a scratch) by observing the display unit 240. In addition, the processor 210 may output image data showing the detected object to a storage device (e.g., the non-volatile storage device 230 or an external storage device (not shown) connected to the information processing device 200) (the image data is stored in the storage device).

[0090] (6) The machine learning model training process (e.g., the process in FIG. 4 and FIG. 9), the defect data acquisition process (e.g., the process in FIG. 8 and FIG. 10), and the inspection result output process (FIG. 7: S720) may each be executed by a different information processing device. Any process selected from these processes may be shared by a plurality of devices (e.g., information processing devices such as computers) that can communicate with each other via a network. For example, the defect data acquisition process may be executed by an information processing system (an example of an acquisition system) that includes a plurality of information processing devices.

[0091] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the function of the machine learning model 238 in FIG. 1 may be realized by a dedicated hardware circuit.

[0092] Furthermore, when some or all of the functions of the present invention are realized by a computer program, the program can be provided in a form stored in a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used in a state stored in the same or a different recording medium (computer-readable recording medium) from when it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0093] The present invention has been described above based on examples and modifications, but the above-described embodiments of the invention are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the invention, and equivalents thereof are included in the present invention. [Explanation of symbols]

[0094] 100... digital camera, 200... information processing device, 210... processor, 215... storage device, 220... volatile storage device, 230... non-volatile storage device, 231... first program, 232... second program, 237... correction parameter data, 238... machine learning model, 240... display unit, 250... operation unit, 270... communication interface, 510... first convolution layer, 520... first pooling layer, 530... second convolution layer, 540...second pooling layer, 550...first fully connected layer, 560...second fully connected layer, 570...third fully connected layer, 580...judgment unit, P1s-P6s...parameter set, x...coordinate value, y...coordinate value, Loss_s...learning loss, Loss_v...confirmation loss, Dout...defect data, W1...width, H1...height, LB...label, LS...light source, DV...target (printer), Dx...first direction, Dy...second direction, TCi...tone curve

Claims

1. A method for acquiring data on defects in the appearance of an object, comprising the steps of: a correction step of correcting a color of the target captured image data indicating a captured image of the target object by using a color correction parameter indicating a color reference; An acquisition process of acquiring defect data of interest relating to defects in the appearance of the target object from a trained machine learning model by inputting the corrected captured image data of interest into the machine learning model trained to indicate a relationship between captured image data showing an image of the target object and defect data relating to defects in the appearance of the target object; Equipped with The trained machine learning model includes a plurality of parameters; The plurality of parameters of the trained machine learning model are adjusted using a plurality of defect data acquired by inputting a plurality of learning photographed image data into the machine learning model; The color correction parameters are calculated using the plurality of learning captured image data. How to get it.

2. The acquisition method according to claim 1 , The plurality of parameters of the trained machine learning model are adjusted so that a condition including a loss indicating a difference between a plurality of defect data acquired by inputting the plurality of confirmation photographed image data corrected using the color correction parameters into the machine learning model having a plurality of adjusted parameters and a plurality of teacher data respectively corresponding to the plurality of confirmation photographed image data is equal to or less than a reference value is satisfied. How to get it.

3. A method for acquiring data on defects in the appearance of an object, comprising the steps of: a correction step of correcting a color of the target captured image data indicating a captured image of the target object by using a color correction parameter indicating a color reference; An acquisition process of acquiring defect data of interest relating to defects in the appearance of the target object from a trained machine learning model by inputting the corrected captured image data of interest into the machine learning model trained to indicate a relationship between captured image data showing an image of the target object and defect data relating to defects in the appearance of the target object; Equipped with The color correction parameters are predetermined, The trained machine learning model includes a plurality of parameters; The plurality of parameters of the trained machine learning model are adjusted using a plurality of defect data obtained by inputting a plurality of learning photographed image data corrected using the color correction parameters into the machine learning model. How to get it.

4. The acquisition method according to any one of claims 1 to 3, the color correction parameters indicate a brightness criterion and a white balance criterion; The correction step includes: performing a brightness correction process for bringing the brightness of the target photographed image data closer to the brightness standard; a step of performing a white balance correction process for bringing the white balance of the target photographed image data closer to the white balance standard; Including, how to obtain.

5. The acquisition method according to claim 4, The lightness standard indicates a standard of the L* component of the CIELAB color space, the white balance standard indicates a standard of the a* component and a standard of the b* component of the CIELAB color space, the brightness correction process includes a process of correcting the L* component of the target photographed image data using the reference for the L* component, The white balance correction process includes: a process of correcting the a* component of the target photographed image data using the reference for the a* component; a process of correcting the b* component of the target photographed image data using the reference for the b* component; Including, how to obtain.

6. 1. An acquisition system for acquiring data relating to defects in the appearance of an object, comprising: a correction unit that corrects a color of target captured image data indicating a captured image of a target object by using a color correction parameter indicating a color reference; An acquisition unit that acquires, from a trained machine learning model that is trained to indicate a relationship between photographed image data showing a photographed image of an object and defect data related to defects in the appearance of the object, the target defect data related to defects in the appearance of the object by inputting the corrected photographed image data of interest into the trained machine learning model; Equipped with The trained machine learning model includes a plurality of parameters; The plurality of parameters of the trained machine learning model are adjusted using a plurality of defect data acquired by inputting a plurality of learning photographed image data into the machine learning model; The color correction parameters are calculated using the plurality of learning captured image data. Acquisition system.

7. 1. An acquisition system for acquiring data relating to defects in the appearance of an object, comprising: a correction unit that corrects a color of target captured image data indicating a captured image of a target object by using a color correction parameter indicating a color reference; An acquisition unit that acquires, from a trained machine learning model that is trained to indicate a relationship between photographed image data showing a photographed image of an object and defect data related to defects in the appearance of the object, the target defect data related to defects in the appearance of the object by inputting the corrected photographed image data of interest into the trained machine learning model; Equipped with The color correction parameters are predetermined, The trained machine learning model includes a plurality of parameters; The plurality of parameters of the trained machine learning model are adjusted using a plurality of defect data obtained by inputting a plurality of learning photographed image data corrected using the color correction parameters into the machine learning model. Acquisition system.

8. A computer program for acquiring data relating to defects in the appearance of an object, comprising: a correction function for correcting the color of the target captured image data representing a captured image of the target object by using a color correction parameter representing a color reference; An acquisition function for acquiring defect data of interest relating to defects in the appearance of the target object from a trained machine learning model by inputting the corrected captured image data of interest into the machine learning model trained to indicate a relationship between captured image data showing an image of the target object and defect data relating to defects in the appearance of the target object; This is realized on a computer. The trained machine learning model includes a plurality of parameters; The plurality of parameters of the trained machine learning model are adjusted using a plurality of defect data acquired by inputting a plurality of learning photographed image data into the machine learning model; The color correction parameters are calculated using the plurality of learning captured image data. Computer program.

9. A computer program for acquiring data relating to defects in the appearance of an object, comprising: a correction function for correcting the color of the target captured image data representing a captured image of the target object by using a color correction parameter representing a color reference; An acquisition function for acquiring defect data of interest relating to defects in the appearance of the target object from a trained machine learning model by inputting the corrected captured image data of interest into the machine learning model trained to indicate a relationship between captured image data showing an image of the target object and defect data relating to defects in the appearance of the target object; This is realized on a computer. The color correction parameters are predetermined, The trained machine learning model includes a plurality of parameters; The plurality of parameters of the trained machine learning model are adjusted using a plurality of defect data obtained by inputting a plurality of learning photographed image data corrected using the color correction parameters into the machine learning model. Computer program.

Images