Program and image processing device

The program addresses brightness variability in image processing by using a generative model and threshold-based execution to ensure accurate and consistent processing of object appearances, enhancing abnormality detection.

JP2024168483AInactive Publication Date: 2024-12-05BROTHER KOGYO KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023085200
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-12-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing image processing techniques fail to account for the variability in image brightness, leading to inconsistent and potentially inaccurate processing results.

Method used

A program that includes a first histogram generation function to represent the intensity distribution of target color components in an input image, a generative model for reconstructing images, and a threshold-based execution function to ensure processing is dependent on image brightness, using a variational autoencoder for abnormality detection in object appearances.

Benefits of technology

Ensures that image processing is conducted based on appropriate brightness levels, reducing the likelihood of inaccurate reconstructions and improving the detection of abnormalities in object appearances.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168483000001_ABST
    Figure 2024168483000001_ABST
Patent Text Reader

Abstract

To advance processing according to the brightness of an image.SOLUTION: An image processing device generates a first histogram representing the intensity distribution of target color components in an input image that represents an object of interest, inputs the data of the input image to a trained generation model so as to generate image data representing an object without an appearance abnormality from the image data representing an object with an appearance abnormality to generate the data of a reconstructed image, generates a second histogram representing the intensity distribution of the target color components in the reconstructed image, and executes specific processing if a difference between the first histogram and the second histogram is greater than or equal to a difference threshold.SELECTED DRAWING: Figure 8
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present specification relates to techniques for processing images representing objects. [Background technology]

[0002] Conventionally, image data representing an object has been used in processing related to the appearance of the object, such as in appearance inspection. Image data obtained by photography can be used as the image data. Here, the following technology has been proposed for designing an appropriate illumination pattern. That is, a light source is caused to emit light in a plurality of evaluation illumination patterns, a plurality of evaluation workpiece images relating to at least one workpiece illuminated with the plurality of evaluation illumination patterns are photographed by a camera, and a light emission pattern is simulated by changing a parameter of a continuous value based on the plurality of evaluation workpiece images, and a parameter representing a light emission pattern for identifying the state of the workpiece is calculated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-49874 Summary of the Invention [Problem to be solved by the invention]

[0004] There was room for improvement in terms of progressing the processing depending on the brightness of the image.

[0005] This specification discloses a new technique for progressing processing depending on the brightness of an image. [Means for solving the problem]

[0006] The technology disclosed in this specification can be realized in the following application examples.

[0007] [Application Example 1] A program that causes a computer to realize the following: a first generation function that generates a first histogram representing the intensity distribution of a target color component of an input image representing an object of interest; an image generation function that generates data of a reconstructed image by inputting data of the input image into a generative model that has been trained to generate image data representing an object without an appearance abnormality from image data representing an object with an appearance abnormality; a second generation function that generates a second histogram that represents the intensity distribution of the target color component of the reconstructed image; and an execution function that executes a specific process if the difference between the first histogram and the second histogram is equal to or greater than a difference threshold.

[0008] According to this configuration, when the difference between a first histogram representing the intensity distribution of a target color component of the input image and a second histogram representing the intensity distribution of a target color component of the reconstructed image is greater than or equal to a difference threshold, a specific process is executed, so that the process can proceed according to the brightness of the input image.

[0009] The technology disclosed in this specification can be realized in various forms, for example, in the form of an image processing method and an image processing device, a computer program for realizing the functions of those methods or devices, a recording medium on which that computer program is recorded (e.g., a non-temporary recording medium), etc. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating a data processing device according to an embodiment; [Diagram 2] 1A is a perspective view of a product 300. FIG. 1B is a diagram showing an example of a label L. FIG. 1C is a diagram showing an example of a photographed image. FIG. 1D is a diagram showing an example of an image of a label portion. [Diagram 3] 13A to 13C are diagrams showing examples of a captured image, a reconstructed image, and a difference image. [Figure 4] 13A-13C are diagrams showing examples of label brightness and difference images. [Diagram 5]FIG. 2 is a block diagram showing an example of the configuration of an image generation model M2. [Figure 6] 13 is a flowchart showing an example of a training process for an image generation model M2. [Figure 7] 13 is a flowchart showing an example of an inspection process for a label L. [Figure 8] 13 is a flowchart illustrating an example of an image determination process. [Figure 9] 13A to 13C are diagrams showing example histograms. [Figure 10] 1A is a diagram showing an example of a formula for calculating correlation c, and FIG. 1B is a diagram showing an example of a formula for calculating a total difference d. [Figure 11] 13 is a flowchart illustrating an example of an abnormal portion detection process. [Figure 12] 1A is a diagram showing an example of a heat map image, FIG. 1B is a diagram showing an example of extracted abnormal pixels, and FIG. 1C is a diagram showing an example of extracted abnormal parts. [Figure 13] 10 is a flowchart showing another embodiment of the image determination process. [Figure 14] 10 is a flowchart showing another embodiment of the image determination process. [Figure 15] 10 is a flowchart showing another embodiment of the image determination process. [Figure 16] 13 is a flowchart illustrating an example of a process for determining a difference threshold. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A. First Example: A1. Equipment configuration: FIG. 1 is an explanatory diagram showing a data processing device as an embodiment. The data processing device 100 is, for example, a personal computer. In this embodiment, a label L is affixed to a product 300 (for example, a multifunction peripheral). The data processing device 100 processes image data representing the appearance of the label L in order to inspect the appearance of the label L. The data processing device 100 is an example of an image processing device that processes image data representing the appearance of an object of interest (in this embodiment, the label L).

[0012] The data processing device 100 includes a processor 110, a storage device 115, a display unit 140, an operation unit 150, a communication interface 170, and a GPU 190. These elements are connected to each other via a bus. The storage device 115 includes a volatile storage device 120 and a non-volatile storage device 130.

[0013] The processor 110 is a device configured to process data. The processor 110 may be, for example, a central processing unit (CPU) or a system on a chip (SoC). The volatile storage device 120 is, for example, a DRAM, and the non-volatile storage device 130 is, for example, a flash memory. The non-volatile storage device 130 stores data for programs 131 and 132, an object detection model M1, an image generation model M2, a copy image D1, and a background image group D2. Details of these data will be described later. The third program 133 and the reference histogram D3 are used in other embodiments described later.

[0014] The display unit 140 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 150 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel arranged overlaid on the display unit 140. A user can input various instructions to the data processing device 100 by operating the operation unit 150. The communication interface 170 is an interface for communicating with other devices. The communication interface 170 includes, for example, one or more of a USB interface, a wired LAN interface, an IEEE802.11 wireless interface, and an industrial camera interface (for example, CameraLink, CoaXPress, etc.). The imaging device 400 is connected to the communication interface 170.

[0015] The GPU (Graphics Processing Unit) 190 is a computing device configured to execute various numerical calculations such as image processing and machine learning. In this embodiment, as described later, the processor 110 and the GPU 190 cooperate to execute various calculations. Note that a driver program (not shown) for controlling the GPU 190 may be provided by the manufacturer of the GPU 190.

[0016] The photographing device 400 photographs a subject using a two-dimensional image sensor, thereby generating image data (also called photographed image data) representing the subject. In this embodiment, the photographing device 400 is a digital camera. The photographed image data is bitmap data representing an image including a plurality of pixels. In this embodiment, the photographed image data is RGB image data representing the color of each pixel by the gradation values ​​of three color components, red R, green G, and blue B. The R value, G value, and B value are expressed in 256 gradations from zero to 255, for example. Note that the photographed image data may represent the color of each pixel by the gradation values ​​of other color components (for example, luminance).

[0017] The photographing device 400 photographs the product 300 to which the label L is affixed, and generates photographed image data representing a photographed image of a portion of the product 300 including the label L. The photographed image data is used to detect abnormalities (e.g., scratches, stains, etc.) in the appearance of the label L. When photographing the product 300, a light source 500 (e.g., a light-emitting diode, an incandescent lamp, a fluorescent lamp, etc.) illuminates the product 300 (and thus the label L). For appropriate processing, it is preferable that the photographed image shows the label L with appropriate brightness (i.e., it is preferable that the brightness of the label L at the time of photographing is appropriate). The data processing device 100 uses the photographed image data to determine whether the brightness of the label L in the photographed image is appropriate (details will be described later).

[0018] A2. Label and photo: 2A is a perspective view of the product 300. In this embodiment, the product 300 is a multifunction peripheral having a substantially rectangular parallelepiped shape. In the manufacturing process of the product 300, a label L is attached to a predetermined position on the outer surface of the product 300. Then, in order to inspect the appearance of the label L, the photographing device 400 photographs the label L on the product 300. When photographing the label L, the position and orientation of the photographing device 400 relative to the product 300 are adjusted so that the label L is located within the photographing range CV of the photographing device 400.

[0019] FIG. 2(B) is a diagram showing an example of the label L. In this embodiment, the label L is a rectangular sheet. The label L may represent various information such as the manufacturer's name, the manufacturer's logo, a brand logo, and a certification mark. Appropriate information to be represented by the label L is associated with the product 300 in advance. Note that the shape of the label L may be various other shapes instead of a rectangle (e.g., a circle, an ellipse, a polygon, etc.).

[0020] FIG. 2(C) is a diagram showing an example of a captured image. The captured image i1 represents a portion of the product 300 including the label L. The processor 110 acquires a portion representing the label L from such a captured image (the acquired partial image is also referred to as a "label partial image"). FIG. 2(D) is a diagram showing an example of an acquired label partial image. In the label partial image i2, the proportion of the portion other than the label L is smaller than in the captured image i1. Also, in the example of FIG. 2(D), the label partial image i2 represents the entire label L. The label partial image is used to detect an abnormality in the appearance of the label L.

[0021] In this embodiment, the photographed image i1 and the labeled partial image i2 are rectangular images having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy perpendicular to the first direction Dx. The images i1 and i2 are represented by the color values ​​of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy. In this embodiment, the color value of each pixel of the labeled partial image i2 is represented by three gradation values ​​of red R, green G, and blue B, similar to the color value of each pixel of the photographed image i1. Each gradation value is represented, for example, in 256 steps from 0 to 255. Note that the labeled partial image may represent the color of each pixel by the gradation value of another color component (for example, luminance).

[0022] A3. Overview of appearance anomaly detection: The processor 110 (FIG. 1) detects an abnormality in the appearance of the label L by using a label partial image, which is a photographed image of the label L, and a reconstructed image of the label L generated from the label partial image by the calculation of the image generation model M2. The image generation model M2 may be any of various machine learning models that generate an image of the same label without an abnormality in appearance from an image of a label having an abnormality in appearance. In this embodiment, the image generation model M2 is a so-called variational auto-encoder (VAE). Details of the image generation model M2 will be described later.

[0023] 3(A)-3(C) are diagrams showing examples of a captured image, a reconstructed image, and a difference image. The label partial image i2a in FIG. 3(A) represents a label La without an abnormality in appearance. The processor 110 generates a reconstructed image i2ag representing a label Lag without an abnormality in appearance by computing the image generation model M2 using the label partial image i2a. The difference image i2ad shows the difference between the label partial image i2a and the reconstructed image i2ag. The color value of each pixel in the difference image i2ad represents the difference in the gradation value of the corresponding pixel between the label partial image i2a and the reconstructed image i2ag (for example, the absolute value of the difference in brightness value). If the label La of the label partial image i2a does not have an abnormality in appearance, the difference between the reconstructed image i2ag and the label partial image i2a is small. The gradation value (i.e., the difference) of each pixel in the difference image i2ad is close to zero. However, the reconstructed image i2ag is not identical to the label partial image i2a. Therefore, the gradation value of each pixel of the difference image i2ad may differ from pixel to pixel.

[0024] The labeled partial image i2b in FIG. 3B represents the label Lb having the flaw Db. The reconstructed image i2bg generated from the labeled partial image i2b by the calculation of the image generation model M2 represents the label Lbg having no abnormal appearance. The difference image i2bd between these images i2b and i2bg represents the flaw Dbd which is similar to the flaw Db. In the difference image i2bd, the gradation value (i.e., the difference) of each pixel is close to zero in the portion other than the flaw Dbd.

[0025] The labeled partial image i2c in FIG. 3C represents the label Lc having the stain Dc. The reconstructed image i2cg generated from the labeled partial image i2c by the calculation of the image generation model M2 represents the label Lcg without any abnormal appearance. The difference image i2cd between these images i2c and i2cg represents the stain Dcd similar to the stain Dc. In the difference image i2cd other than the stain Dcd, the grayscale value (i.e., the difference) of each pixel is close to zero.

[0026] In this way, the difference image between the label partial image and the reconstructed image may show abnormalities in the appearance of the label (such as scratches or dirt). When the brightness of the label in the captured image (i.e., the brightness of the label at the time of capture) is dark or bright, the calculation of the image generation model M2 may not be able to generate an appropriate reconstructed image. Figures 4(A) to 4(C) are diagrams showing examples of the brightness of the label and the difference image. Figure 4(A) shows a case where the brightness of the label in the captured image (i.e., the brightness of the label at the time of capture) is appropriate. The label partial image i2a, the reconstructed image i2ag, and the difference image i2ad are the same as those in Figure 3(A), respectively. The reconstructed image i2ag properly shows the label Lag, which has no abnormalities in appearance.

[0027] FIG. 4B shows a case where the brightness of the label in the captured image (i.e., the brightness of the label at the time of capture) is dark. The processor 110 acquires a label partial image i2x representing the label La from the captured image i1x of the product 300. The label La appears dark in the label partial image i2x. The label La may appear dark due to various reasons such as insufficient light from the light source 500. A reconstructed image i2xg is generated from the label partial image i2x. If the label La in the image i2x input to the image generation model M2 is dark, the image generation model M2 may not be able to properly reconstruct the label La. For example, information such as logos and characters may be blurred on the label Lax represented by the reconstructed image i2xg. Even if there is no abnormality in the appearance of the label La, the difference image i2xd between the label partial image i2x and the reconstructed image i2xg may show a large difference over a wide range.

[0028] FIG. 4(C) shows a case where the brightness of the label in the captured image (i.e., the brightness of the label at the time of capture) is bright. A label partial image i2z representing the label La is acquired from the captured image i1z of the product 300. The label La appears bright in the label partial image i2z. The label La may appear bright due to various reasons, such as an excessive amount of light from the light source 500. A reconstructed image i2zg is generated from the label partial image i2z. If the label La in the label partial image i2z is bright, the image generation model M2 may not be able to properly reconstruct the label La. For example, information such as logos and characters may be blurred on the label Laz represented by the reconstructed image i2zg. Therefore, even if there is no abnormality in the appearance of the label La, the difference image i2zd between the label partial image i2z and the reconstructed image i2zg may show a large difference over a wide range.

[0029] In this way, the brightness of the label at the time of shooting, i.e., the brightness of the label in the captured image, can affect image processing. For example, the possibility that an inappropriate reconstructed image is generated by the image generation model M2 varies depending on the brightness of the label in the captured image. In this embodiment, the processor 110 executes processing to reduce the possibility of using an inappropriate reconstructed image (details will be described later). Below, first, the image generation model M2 will be described, and then the inspection processing will be described.

[0030] A4. Image generation model M2: 5 is a block diagram showing an example of the configuration of the image generation model M2. In this embodiment, the image generation model M2 is a so-called variational autoencoder. The variational autoencoder is disclosed in, for example, the following paper. Diederik P. Kingma, Max Welling, "Auto-Encoding Variational Bayes", arXiv:1312.6114, https: / / arxiv.org / abs / 1312.6114

[0031] The image generation model M2 has an encoder Ve and a decoder Vd. The encoder Ve performs a dimension reduction process on the data of the input image IIg to generate a latent variable vector z representing the features of the input image IIg. In the variational autoencoder, it is assumed that each element of the latent variable vector z follows a probability distribution (for example, a normal distribution). The decoder Vd performs a dimension restoration process on the latent variable vector z to generate data of a reconstructed image OIg having the features represented by the latent variable vector z. As will be described later, in the training of the image generation model M2, data of an input image IIg representing a label L (FIG. 2(A)) without an abnormal appearance is input to the image generation model M2. Then, the image generation model M2 is trained so that data of a reconstructed image OIg representing the same label L without an abnormal appearance is generated. The trained image generation model M2 can generate a reconstructed image of a label without an abnormal appearance from an image of a label having an abnormal appearance, as described in FIG. 3(A)-FIG. 3(C).

[0032] The encoder Ve may have various configurations for calculating the latent variable vector z from the data of the input image IIg. In this embodiment, the encoder Ve has convolution layers Ve21 and Ve22, fully connected layers Ve23 and Ve24, and an arithmetic layer Ve25.

[0033] Data of an input image IIg is input to a first convolutional layer Ve21. Data output from the first convolutional layer Ve21 is input to a second convolutional layer Ve22. The convolutional layers Ve21 and Ve22 perform processing including a convolution process and a bias addition process on the input data. The convolutional layers Ve21 and Ve22 each have a set of calculation parameters including a plurality of weights and a plurality of biases of a plurality of filters used in the convolution process. The activation function of each of the convolutional layers Ve21 and Ve22 is, for example, Relu (Rectified Linear Unit).

[0034] The first fully connected layer Ve23 uses the data output from the second convolutional layer Ve22 to generate a mean vector μ indicating the average of the probability distribution of each of the elements of the latent variable vector z. The second fully connected layer Ve24 uses the data output from the second convolutional layer Ve22 to generate a standard deviation vector σ indicating the standard deviation of the probability distribution of each of the elements of the latent variable vector z. Each of the fully connected layers Ve23 and Ve24 generates data for output by inputting the sum of the inner product of the vector represented by the input data and the weight vector and the bias into an activation function. Each of the fully connected layers Ve23 and Ve24 has a set of calculation parameters including a weight vector and a bias. The activation function of each of the fully connected layers Ve23 and Ve24 is, for example, Relu.

[0035] The computation layer Ve25 determines multiple elements of the latent variable vector z based on multiple probability distributions defined by the mean vector μ and the standard deviation vector σ. Here, the latent variable vector z is determined according to a technique called the "reparameterization trick" described in the above paper. In this embodiment, the computation layer Ve25 generates a noise vector consisting of multiple random values ​​based on a normal distribution (N(0,1)) defined by a mean of zero and a variance of one. The computation layer Ve25 calculates the latent variable vector z by adding the mean vector μ to the product (here, the product per element) of the noise vector and the standard deviation vector σ.

[0036] The decoder Vd may have various configurations for generating a reconstructed image OIg from the latent variable vector z. In this embodiment, the decoder Vd has a fully connected layer Vd21 and transposed convolution layers Vd22 and Vd23 connected in series downstream of the fully connected layer Vd21.

[0037] The latent variable vector z from the encoder Ve is input to the fully connected layer Vd21. The fully connected layer Vd21 generates output data by inputting the sum of the inner product of the latent variable vector z and a weight vector and a bias to an activation function. The fully connected layer Vd21 has a set of calculation parameters including the weight vector and the bias. The activation function of the fully connected layer Vd21 is, for example, Relu.

[0038] The transposed convolution layers Vd22 and Vd23 execute processing including a transposed convolution process and a bias addition process on the input data. The transposed convolution process is a process of increasing the number of elements by appropriately adding elements (e.g., elements indicating zero) to the input data according to the stride and padding, and then performing a convolution operation using a filter. The data output from the last transposed convolution layer Vd23 represents a reconstructed image OIg. The transposed convolution layers Vd22 and Vd23 each have a set of operation parameters including multiple weights and multiple biases of multiple filters used in the transposed convolution process. The activation function of the last transposed convolution layer Vd23 is a function (e.g., a Sigmoid function or a Tanh function) suitable for generating the reconstructed image OIg. The activation function of the other transposed convolution layer Vd22 is, for example, Relu.

[0039] The input data and output data of each layer Ve21-Ve25, Vd21-Vd23 may have various configurations. In this embodiment, the input image IIg and the reconstructed image OIg are color bitmap data that indicate the color of each pixel in three channels, red R, green G, and blue B. The number of elements of the data of the input image IIg (here, height*width*number of channels) may be 256*512*3. Each color value of RGB may be expressed in 256 gradations from zero to 255. The height indicates the number of pixels in the second direction Dy (FIG. 4(A)), and the width indicates the number of pixels in the first direction Dx. The encoder Ve may reduce the number of elements corresponding to the height and the number of elements corresponding to the width. For example, the convolutional layers Ve21 and Ve22 may change the number of data elements from "256*512*3" to "128*256*64" and "64*128*128" in that order. The fully connected layers Ve23 and Ve24 may each change the number of data elements from "64*128*128" to "32 (i.e., 1*1*32)". In this way, the mean vector μ, the standard deviation vector σ, and the latent variable vector z may each be composed of 32 elements. The decoder Vd may increase the number of elements corresponding to the height and the number of elements corresponding to the width. For example, the fully connected layer Vd21 of the decoder Vd may change the number of data elements from "32" to "64*128*128". The transposed convolution layers Vd22 and Vd23 may change the number of data elements in the order of "64*128*128", "128*256*64", and "256*512*3". In either case, the data configuration of the reconstructed image OIg (specifically, height*width*number of channels) is set to be the same as that of the input image IIg.

[0040] A number of calculation parameters of the image generation model M2 are adjusted by a training process described below.

[0041] A5. Training process of image generation model M2: 6 is a flowchart showing an example of a training process for the image generation model M2. The processor 110 of the data processing device 100 (FIG. 1) trains the image generation model M2 by executing the first program 131. Before training, a plurality of calculation parameters of the image generation model M2 are set to initial values ​​such as random numbers.

[0042] In S110, the processor 110 (FIG. 1) acquires data of the copy image D1 from the non-volatile storage device 130. The copy image D1 is an image representing a label L (FIG. 2(B)) that has no abnormal appearance, and is prepared in advance for the manufacture of the label L. In the manufacturing process of the product 300, the copy image D1 is printed on a sheet to form the label L.

[0043] In S120, the processor 110 executes data expansion processing of the copy image D1. The data expansion processing is processing to increase image data by image processing. The image processing may be various processing to generate another image representing the label L without abnormal appearance from the copy image D1. In this embodiment, the processor 110 selects one or more image processing from among resizing processing, rotation processing, brightness change processing, color change processing, blurring processing, and noise addition processing using a random number. The processor 110 executes the one or more selected image processing on the data of the copy image D1 to generate data of the processed copy image.

[0044] The resizing process may be various processes that change the size (i.e., the number of pixels) of the label L in the image. For example, the processor 110 may change the size of the label L in the image according to a ratio determined using a random number. The ratio may be determined, for example, within a range of 0.8 to 1.2.

[0045] The rotation process may be various processes that change the rotation position of the label L in the image. For example, the processor 110 may rotate the label L in the image around the center of gravity of the label L according to a rotation angle determined using a random number. The rotation angle may be determined, for example, within a range of -3 degrees or more and +3 degrees or less (a positive rotation angle indicates a clockwise rotation, and a negative rotation angle indicates a counterclockwise rotation).

[0046] Note that the color value of each pixel may be determined by various methods based on the resizing and rotation processes (for example, nearest neighbor, bilinear, bicubic, etc.).

[0047] The brightness change process may be various processes that change the overall brightness of the image. For example, the processor 110 may adjust the gradation values ​​of R, G, and B using a gamma curve represented by a gamma value determined using random numbers. The gamma value may be determined within a range of, for example, 0.7 to 1.3.

[0048] The color modification process may be any process that modifies the color of an image. For example, the processor 110 may add a modification amount determined by using a random number for each color component to the gradation value of each of R, G, and B. The modification amount may be determined within a range of, for example, -10 or more and +10 or less.

[0049] The blurring process may be any process that blurs the label L in the image. The blurring process is also called smoothing. For example, the processor 110 may perform a filter process using a smoothing filter (e.g., a median filter).

[0050] The noise addition process may be any process that adds noise to an image. For example, the processor 110 may add a modification amount determined using a random number for each color component to the R, G, and B gradation values ​​of a plurality of pixels selected using a random number. The modification amount may be determined for each pixel.

[0051] The data augmentation process may include other image processes (e.g., a process of shifting the position) in addition to these processes. Also, the data augmentation process may include other image processes instead of all or part of these processes.

[0052] In S130, the processor 110 (FIG. 1) acquires data of one background image to be processed from data of the background image group D2 in the non-volatile storage device 130. In this embodiment, each background image in the background image group D2 is a photographed image obtained by photographing various subjects (e.g., landscapes, devices (e.g., multifunction devices), etc.) using a digital camera. Without being limited to this, the background image may include a read image obtained by scanning an original such as a picture or a photograph with a scanner, and may include an image generated by a computer such as an illustration.

[0053] In S140, processor 110 generates data for a composite image using data of the background image acquired in S130 and data of the processed copy image generated in S120. The composite image represents an image obtained by superimposing the processed copy image on the background image. Processor 110 determines the superimposition position of the processed copy image on the background image using a random number. For example, the superimposition position is determined within a range in which the entirety of the processed copy image is located within the background image.

[0054] In S150, the processor 110 executes an extension process for the composite image data. The extension process is selected from one or more image processes, similar to the extension process in S120. The processor 110 uses a random number to select one or more image processes, and executes the selected one or more image processes on the composite image data to generate processed composite image data. The processor 110 generates processed composite image data having a data configuration (here, height*width*number of channels) that is acceptable to the image generation model M2. If gaps (i.e., parts where color values ​​are not defined) occur due to the extension processes in S120 and S150 (e.g., rotation processes, resizing processes, etc.), the processor 110 may set the color of the pixels in the gaps to a predetermined color (e.g., white).

[0055] In S170, the processor 110 stores the processed composite image data as training image data in the storage device 115 (e.g., the non-volatile storage device 130). The training image represented by the training image data represents the label L without abnormal appearance, similar to the block copy image D1. However, various parameters related to the expression of the label L, such as the overall brightness of the image, the inclination of the label L in the image, the size of the label L in the image, the position of the label L in the image, and the degree of blur of the label L, are different between the training image and the block copy image D1.

[0056] In S180, the processor 110 determines whether a generation end condition for ending the generation of the training image data is satisfied. The generation end condition may be various conditions indicating that a plurality of training image data for training the image generation model M2 has been generated. In this embodiment, the generation end condition is that the total number of generated training image data is equal to or greater than a predetermined threshold (e.g., several hundred or several thousand).

[0057] If the generation end condition is not satisfied (S180: No), the processor 110 proceeds to S120 to generate new training image data. If the generation end condition is satisfied (S180: Yes), the processor 110 proceeds to S210.

[0058] In S210, the processor 110 selects a subset consisting of a plurality of unprocessed training image data from the plurality of training image data generated in S120 to S170. The total number of image data in the subset is predetermined.

[0059] In S220, the processor 110 generates image data (called generated image data) by inputting the training image data included in the subset to the image generation model M2. The processor 110 uses the training image data to proceed with the operation of the image generation model M2. The processor 110 may cause the GPU 190 to execute a part or all of the operation of the image generation model M2.

[0060] In S230, the processor 110 calculates an evaluation value using the training image data and the generated image data. The calculation of the evaluation value is performed for each training image data. Then, the processor 210 calculates a loss using a plurality of evaluation values ​​calculated from a plurality of training image data included in the subset. The loss may be, for example, the sum of the plurality of evaluation values.

[0061] The method of calculating the evaluation value may be any method suitable for training the image generation model M2. The evaluation value preferably indicates the magnitude of the difference between the training image and the generated image. In this embodiment, the evaluation value is the sum of the reconstruction error and the regularization term described in the above paper on the variational autoencoder. The reconstruction error may be, for example, the cross-entropy loss between the training image and the generated image. The regularization term may be calculated using, for example, the Kullback-Leibler divergence, which indicates the degree of agreement between the probability distribution of the latent variable and a normal distribution (mean = zero, standard deviation = 1). In this embodiment, each element of the latent variable vector z (FIG. 5) indicates a latent variable. The probability distribution of each latent variable is determined by the mean and standard deviation represented by the corresponding elements of the mean vector μ and the standard deviation vector σ.

[0062] In S240, the processor 110 adjusts a plurality of calculation parameters of the image generation model M2 using the loss. In this embodiment, the processor 110 adjusts the plurality of calculation parameters so that the loss is small. The adjustment method may be any method. For example, a method using an error backpropagation method and a gradient descent method may be adopted. Here, the processor 110 may perform so-called Adam optimization. The processor 110 may cause the GPU 190 to execute a part or all of the calculation for adjusting the plurality of calculation parameters.

[0063] In S250, the processor 110 determines whether a training end condition for ending the training of the image generation model M2 is satisfied. The training end condition may be any condition indicating that the image generation model M2 has been properly trained. The training end condition may be, for example, that the loss calculated in S230 is equal to or less than a predetermined loss threshold. Alternatively, the training end condition may be that the change in the loss calculated in the repeated S230 is equal to or less than a predetermined change threshold, or that the number of repetitions of S240 is equal to or greater than a predetermined number threshold.

[0064] If the training end condition is not satisfied (S250: No), the processor 110 proceeds to S210 and executes processing of a new subset. If the training end condition is satisfied (S250: Yes), in S260, the processor 110 stores data representing the trained image generation model M2 in the storage device 115 (here, the non-volatile storage device 130). Then, the processor 110 ends the processing of FIG. 6.

[0065] As described above, the image generation model M2 is trained to generate image data of the same label without an abnormality in appearance from image data of the label L without an abnormality in appearance. That is, the image generation model M2 is trained to reconstruct the features of an image of the label L without an abnormality in appearance. Therefore, even if the image data input to the image generation model M2 represents the labels Lb, Lc having an abnormality in appearance (FIGS. 3(B) and 3(C)), the image data generated by the trained image generation model M2 can represent the labels Lbg, Lcg without an abnormality in appearance. In this way, the image generation model M2 is trained to generate image data representing the labels without an abnormality in appearance, regardless of whether the labels in the image input to the image generation model M2 have an abnormality in appearance.

[0066] In addition, the image generation model M2 is trained to generate a reconstructed image that is the same as the training image input to the image generation model M2. Therefore, when the brightness of the label of the image input to the image generation model M2 is comparable to the brightness of the label of the training image used to train the image generation model M2, the image generation model M2 can generate an appropriate reconstructed image. The distribution range of the brightness of the labels of the multiple training images used to train the image generation model M2 indicates the appropriate brightness range of the labels for generating an appropriate reconstructed image. When the brightness of the labels in the image is significantly different from the brightness of the labels in the training image, as described in FIG. 4(B) and FIG. 4(C), the image generation model M2 may generate an inappropriate reconstructed image.

[0067] A6.Inspection process: FIG. 7 is a flowchart showing an example of an inspection process of the label L of the product 300 (FIG. 2(A)). The processor 110 of the data processing device 100 (FIG. 1) executes the second program 132 to perform the inspection process. As described in FIG. 2(A), the relative arrangement between the product 300 and the photographing device 400 is adjusted for the inspection. In this embodiment, the position and orientation of the product 300 are adjusted by a machine (such as a belt conveyor or a turntable). After the product 300 is arranged, an instruction to start the inspection process is input to the data processing device 100. In this embodiment, the operator inputs the start instruction by operating the operation unit 150. The processor 110 starts the inspection process in response to the start instruction. Note that the arrangement of the product 300 may be adjusted by the operator. Instead of the position and orientation of the product 300, the position and orientation of the photographing device 400 may be adjusted. The start instruction may be input to the data processing device 100 via the communication interface 170 by another device different from the data processing device 100. The process in Fig. 7 illustrates an inspection process for one product 300. The processor 110 repeats the process in Fig. 7 to inspect each of the multiple products 300.

[0068] In S310, the processor 110 supplies a shooting instruction to the photographing device 400. In response to the shooting instruction, the photographing device 400 photographs a part of the product 300 including the label L and generates data of the photographed image. The processor 110 acquires the photographed image data from the photographing device 400. This photographed image represents a part of the product 300 including the label L, like the photographed image i1 in FIG. 2(C).

[0069] In S320, the processor 110 determines whether a judgment execution condition is satisfied. The judgment execution condition is a condition for executing an image judgment process described later. In this embodiment, the judgment execution condition is that the current inspection process is the 100th inspection process counted from the last inspection process involving an image judgment process. In other words, the judgment execution condition is satisfied once in 100 times.

[0070] If the judgment execution condition is satisfied (S320: Yes), in S330, the processor 110 executes image judgment processing. Fig. 8 is a flowchart showing an example of the image judgment processing. In S460, the processor 110 detects a partial image (i.e., a label partial image) representing the label L (Fig. 2(B)) from the captured image, and acquires data of the label partial image. The label partial image i2 in Fig. 2(D) is an example of a label partial image detected from the captured image i1 in Fig. 2(C).

[0071] The method of detecting the portion representing the label L may be any method. In this embodiment, the processor 110 detects the label L using an object detection model M1 trained to detect the label L. The object detection model M1 may be various object detection models. In this embodiment, the object detection model M1 is a machine learning model called YOLOv4. YOLOv4 is disclosed in, for example, the following paper. Alexey Bochkovskiy, Chien-Yao Wang, Hong-Yuan Mark Liao, "YOLOv4: Optimal Speed ​​and Accuracy of Object Detection", arXiv:2004.10934, https: / / arxiv.org / abs / 2004.10934

[0072] The YOLOv4 model predicts a region (also called a bounding box) that represents an object (in this embodiment, label L). The training method of the object detection model M1 may be any method that trains the object detection model M1 to detect a bounding box that represents the label L. For example, the training method described in the above paper may be adopted.

[0073] In S460 (FIG. 8), the processor 110 detects a bounding box representing the label L by performing calculations of the object detection model M1 using the captured image data. The bounding box BB in FIG. 2(C) is an example of a bounding box detected from the captured image i1. The bounding box BB surrounds the label L. The processor 110 may cause the GPU 190 to execute a part or all of the calculations of the object detection model M1.

[0074] The processor 110 extracts an area including a bounding box from the captured image by performing a trimming process on the captured image data. The area EA in FIG. 2C is an example of an area extracted from the captured image i1. The area EA includes the bounding box BB.

[0075] The aspect ratio of the bounding box (here, the ratio of height to width) may be different from the processing aspect ratio, which is the aspect ratio accepted by the image generation model M2. In this case, the processor 110 extracts a region that includes the bounding box and has the processing aspect ratio. Here, the processor 110 may extract a minimum region that includes the bounding box. Note that a part of the label L may extend outside the bounding box. The processor 110 may extract a minimum region that includes the bounding box and a peripheral region (e.g., a region of a predetermined width) that surrounds the bounding box. The region that includes the bounding box and the peripheral region may include the entire label L. The position of the extracted region relative to the bounding box may be various positions. For example, the center of gravity of the extracted region may be located at the center of gravity of the bounding box.

[0076] The processor 110 obtains data of the labeled portion image by executing a resolution conversion process on the image of the extracted region. The data of the labeled portion image has a data structure (specifically, height*width*number of channels) that is accepted by the image generation model M2 (FIG. 5). The labeled portion image i2 in FIG. 2(D) represents the image of the region EA extracted from the photographed image i1 (FIG. 2(C)). The method of determining the color value of each pixel by the resolution conversion process may be various methods (e.g., nearest neighbor, bilinear, bicubic, etc.).

[0077] In S505 (FIG. 8), the processor 110 uses the data of the label partial image to generate data of the reconstructed image by proceeding with the calculation of the image generation model M2. As shown in FIG. 3(A)-FIG. 3(C), the reconstructed image may represent a label without an abnormal appearance, regardless of whether the label has an abnormal appearance. However, as shown in FIG. 4(B) and FIG. 4(C), if the brightness of the label in the label partial image is inappropriate, the reconstructed image may represent an inappropriate label. The processor 110 may cause the GPU 190 to execute a part or all of the calculation of the image generation model M2.

[0078] In S510 (FIG. 8), the processor 110 acquires the luminance V of each pixel of the labeled partial image. In this embodiment, the processor 110 calculates the luminance V by using a predetermined correspondence relationship between the original color value (in this embodiment, the gradation value of the three color components of RGB) and the luminance V. The correspondence relationship may be, for example, the correspondence relationship between the RGB color space and the HSV color space.

[0079] In S515, the processor 110 acquires the luminance V of each pixel of the reconstructed image. The method of acquiring the luminance V is the same as the method of acquiring the luminance V in S510.

[0080] In S520, the processor 110 generates a histogram HI of the luminance V of the labeled partial image. In S525, the processor 110 generates a histogram HR of the luminance V of the reconstructed image. FIG. 9(A)-FIG. 9(C) are diagrams showing examples of histograms. In FIG. 9(A)-FIG. 9(C), labeled partial images i2a, i2x, and i2z, reconstructed images i2ag, i2xg, and i2zg, and a graph are shown. In the graph, the histogram HI of the labeled partial image (called an input histogram HI) and the histogram HR of the reconstructed image (called a reconstructed histogram HR) are shown superimposed. The horizontal axis indicates the luminance V, and the vertical axis indicates the frequency F.

[0081] Fig. 9(A) shows an example of a graph obtained from the labeled partial image i2a and the reconstructed image i2ag in Fig. 4(A). If the labeled partial image i2a represents the label La with appropriate brightness, the reconstructed image i2ag may represent the label Lag with brightness comparable to that of the label La in the labeled partial image i2a. The reconstructed histogram HR is similar to the input histogram HI.

[0082] FIG. 9(B) shows an example of a graph obtained from the label partial image i2x and the reconstructed image i2xg in FIG. 4(B). As described in FIG. 6, the image generation model M2 is trained to generate an image representing a label with the same brightness as the label of the training image. Therefore, when the label La of the label partial image i2x is excessively dark, the image generation model M2 generates an image representing a label with the same brightness as the label of the training image, that is, a reconstructed image i2xg representing a label Lax that is brighter than the label La of the label partial image i2x. As shown in the graph, the difference between the reconstructed histogram HR and the input histogram HI is large. Specifically, in the range of high luminance V, the frequency F of the reconstructed histogram HR is large, and the frequency F of the input histogram HI is small.

[0083] FIG. 9(C) shows an example of a graph obtained from the label partial image i2z and the reconstructed image i2zg in FIG. 4(C). When the label La of the label partial image i2z is excessively bright, the image generation model M2 generates an image representing a label with a brightness similar to that of the label of the training image, that is, a reconstructed image i2zg representing a label Laz that is darker than the brightness of the label La of the label partial image i2z. As shown in the graph, the difference between the reconstructed histogram HR and the input histogram HI is large. Specifically, in the range of high brightness V, the frequency F of the reconstructed histogram HR is small, and the frequency F of the input histogram HI is large.

[0084] In S530 (FIG. 8), the processor 110 calculates an index value of the difference between the input histogram HI of the label partial image and the reconstructed histogram HR of the reconstructed image (also called an input index value). The input index value may be various values ​​that indicate the magnitude of the difference between the input histogram HI and the reconstructed histogram HR. In this embodiment, the processor 110 calculates the correlation between the input histogram HI and the reconstructed histogram HR as the input index value. FIG. 10(A) is a diagram showing an example of a calculation formula for the correlation c. This correlation c is the Pearson product-moment correlation coefficient. The Pearson product-moment correlation coefficient is a measure that indicates the linear correlation between two data sets. The Pearson product-moment correlation coefficient is calculated by dividing the covariance of the two data sets by the product of their standard deviations. In this embodiment, the histograms HI and HR indicate the frequency F of each of n intervals (also called bins) from 1 to n. That is, the entire range of the brightness V is divided into n intervals. In Fig. 10(A), the frequency HI_k is the frequency of the kth section of the input histogram HI. The frequency HR_k is the frequency of the kth section of the reconstructed histogram HR. The average HI_m is the average of the n frequencies F of the input histogram HI. The average HR_m is the average of the n frequencies F of the reconstructed histogram HR.

[0085] When the reconstructed histogram HR is similar to the input histogram HI, as in Fig. 9(A), the correlation c is large. When the difference between the histograms HR and HI is large, as in Fig. 9(B) and Fig. 9(C), the correlation c is small.

[0086] In S540 (FIG. 8), the processor 110 uses the input index value to determine whether the difference between the histograms HI and HR is equal to or greater than the difference threshold. In this embodiment, the processor 110 determines that the difference is equal to or greater than the difference threshold when the correlation c is equal to or less than a predetermined first determination threshold. When the histograms HI and HR in FIG. 9(A) are processed, it is determined that the correlation c is greater than the first determination threshold (i.e., the difference between the histograms is less than the difference threshold). When the histograms HI and HR in FIG. 9(B) and FIG. 9(C) are processed, it is determined that the correlation c is equal to or less than the first determination threshold (i.e., the difference between the histograms is equal to or greater than the difference threshold). Although the illustration of the histograms is omitted, when the labeled partial images i2b and i2c and the reconstructed images i2bg and i2cg in FIG. 3(B) and FIG. 3(C) are processed, it is determined that the difference between the histograms is less than the difference threshold. The difference threshold (here, the first judgment threshold) is experimentally determined in advance so that when the brightness of the labels in the labeled partial image is appropriate, the difference between the histograms HI and HR is less than the difference threshold, and when the brightness of the labels in the labeled partial image is inappropriate, the difference between the histograms HI and HR is greater than or equal to the difference threshold.

[0087] If the judgment result of S540 is Yes (i.e., the difference between the histograms HI and HR is equal to or greater than the threshold), in S550, the processor 110 executes a first judgment result process. The first judgment result process may be any process associated with the judgment result of S540 (here, Yes). In this embodiment, the first judgment result process includes S552, S554, and S556. In S552, the processor 110 determines the result of the image judgment to be "brightness abnormality." In S554, the processor 110 notifies the operator of the result of the image judgment. The method of notification may be any method capable of transmitting the result of the image judgment to the operator. For example, the processor 110 may display information indicating the result of the image judgment on the display unit 140. In S556, the processor 110 stores result data indicating the result of the image judgment in the storage device 115 (for example, the non-volatile storage device 130). After S550, processor 110 ends the process of FIG. 8, that is, the process of S330 in FIG.

[0088] In this embodiment, if the result of the image judgment is "abnormal brightness", the operator who is notified of the result in S554 adjusts the light intensity of the light source 500 to an appropriate level before starting the next inspection process (for example, in S559 after S550). This enables the imaging device 400 to generate an image showing the label with appropriate brightness in the next inspection process.

[0089] If the determination result of S540 (FIG. 8) is No (i.e., the difference between the histograms HI and HR is less than the threshold), in S560, the processor 110 executes a second determination result process. The second determination result process may be any process associated with the determination result of S540 (here, No). In this embodiment, the second determination result process includes S562, S564, and S566. In S562, the processor 110 determines the result of the image determination to be "normal brightness". S564 and S566 are the same as S554 and S556 included in S550, respectively. After S560, the processor 110 ends the process of FIG. 8, that is, the process of S330 in FIG. 7.

[0090] In S340 (FIG. 7), the processor 110 determines whether the result of the image judgment is “normal brightness.” If the result is “normal brightness” (S340: Yes), in S360, the processor 110 executes an abnormal portion detection process using the labeled portion image and the reconstructed image.

[0091] FIG. 11 is a flowchart showing an example of the abnormal part detection process. In S362, the processor 110 generates data of a heat map image showing the possibility of abnormality in the appearance of the label represented by the label partial image. FIG. 12(A) is a diagram showing an example of a heat map image. In this embodiment, the heat map image is a difference image showing the difference between the label partial image and the reconstructed image. The difference image i2bd in the figure shows an example of a difference image generated from the label partial image i2b and the reconstructed image i2bg in FIG. 3(B). As described above, the reconstructed image is not the same as the label partial image. Therefore, the difference image may show various noises, not limited to abnormalities in the appearance of the label. For example, the difference image i2bd may show multiple noise parts D21, D22, and D23 in addition to the scratch Dbd (the noise parts D21, D22, and D23 are omitted in FIG. 3(B)).

[0092] The color value of each pixel of the difference image may be various values ​​indicating the magnitude of the difference between the color values ​​of the corresponding pixels of the labeled partial image and the reconstructed image. In this embodiment, the color value of a pixel of the difference image indicates the absolute value of the difference in luminance V. The color value of the difference image may indicate each gradation value (e.g., absolute value of the difference) of multiple color components (e.g., RGB).

[0093] In S364 (FIG. 11), processor 110 extracts abnormal pixels having gradation values ​​equal to or greater than a predetermined gradation threshold from the difference image. Image i2bp in FIG. 12(B) shows an example of abnormal pixels extracted from difference image i2bd (FIG. 12(A)). Pixels indicating the flaw Dbd and noise portions D21, D22 are extracted. Noise portion D23 (FIG. 12(A)) is not extracted. Noise indicating a small difference is removed. The gradation threshold is experimentally determined in advance so that the gradation value (i.e., the difference) of pixels indicating noise is smaller than the gradation threshold and the gradation value of pixels indicating an abnormality is larger than the gradation threshold.

[0094] In S366 (FIG. 11), the processor 110 extracts a continuous region having an area equal to or greater than a predetermined area threshold as an abnormal portion. A continuous region is a region in which multiple abnormal pixels are continuous. The area is, for example, measured using the number of pixels. Image i2bn in FIG. 12(C) shows an example of an abnormal portion extracted from image i2bp in FIG. 12(B). A continuous region showing a scratch Dbd is extracted as an abnormal portion, and small noise portions D21, D22 (FIG. 12(B)) are not extracted. Small noise areas are removed. The area threshold is experimentally determined in advance so that the area of ​​a continuous region showing noise is smaller than the area threshold, and the area of ​​a continuous region showing an abnormality is larger than the area threshold.

[0095] 3A, if the label La has no abnormality in appearance, a difference image i2ad without any abnormality can be generated. No abnormality is detected from such a difference image i2ad.

[0096] After S366 (FIG. 11), processor 110 ends the process of FIG. 11, that is, the process of S360 in FIG.

[0097] In S370, the processor 110 determines whether or not an abnormal portion is detected. If no abnormal portion is detected (S370: No), in S380, the processor 110 executes a first inspection result process. The first inspection result process may be any process associated with a case where no abnormal portion is detected. In this embodiment, the processor 110 stores inspection result data indicating that the inspection result is normal in the storage device 115 (e.g., the non-volatile storage device 130). After S380, the processor 110 ends the inspection process.

[0098] If an abnormal part is detected (S370: Yes), in S390, the processor 110 executes a second test result process. The second test result process may be any process associated with the case where an abnormal part is detected. In this embodiment, the processor 110 stores test result data indicating that the test result is abnormal in the storage device 115 (e.g., the non-volatile storage device 130). After S390, the processor 110 ends the test process.

[0099] In S340, if the result of the image judgment is not "brightness normal" (S340: No), the processor 110 does not inspect the appearance of the label and ends the process in Fig. 7. Therefore, the possibility that the inspection is performed using an inappropriate reconstructed image such as the reconstructed images i2xg and i2zg in Fig. 4(B) and Fig. 4(C) is reduced.

[0100] If the judgment execution condition is not satisfied (FIG. 7: S320: No), in S350, the processor 110 executes the label detection process. The process of S350 is the same as the process of S460 in FIG. 8. After S350, in S355, the processor 110 generates data of the reconstructed image. The process of S355 is the same as the process of S505 in FIG. 8. After S355, the processor 110 proceeds to S360. In this way, if the judgment execution condition is not satisfied, the processor 110 skips the image judgment process. As described in S550 and S559 (FIG. 8), in this embodiment, if the result of the image judgment is "abnormal brightness", the worker adjusts the light amount of the light source 500 to an appropriate light amount. Therefore, when the image judgment process is executed, it is estimated that the imaging device 400 can generate data of the captured image representing the label at an appropriate brightness for a while thereafter (called a grace period) (i.e., it is estimated that the image judgment process is unnecessary during the grace period). The judgment execution condition is determined in advance so that it is not satisfied during the grace period, but is satisfied after the grace period has elapsed. In this embodiment, the grace period is a period during which 99 inspection processes are performed. The grace period may be determined by time (e.g., days, hours, etc.) instead of the number of inspection processes. The brightness of the label in the label portion image may change due to environmental changes such as changes in the amount of light from the light source 500. The grace period may be determined in advance experimentally to be a period during which the brightness of the label in the label portion image is maintained within an appropriate range.

[0101] As described above, in this embodiment, the processor 110 executes the following process. As described in Figs. 3(A)-3(C) and 9(A)-9(C), in S520 (Fig. 8), the processor 110 generates a histogram HI of the luminance V of the labeled partial image (e.g., labeled partial images i2a-i2c, i2x, i2z) (hereinafter, the histogram HI is also referred to as the first histogram HI). The first histogram HI represents the intensity distribution of the luminance V. The luminance V is an example of a target color component that is a color component of the intensity distribution represented by the histogram HI. The labeled partial image is an example of an input image that is an image to be processed. In this embodiment, the labeled partial image represents a label (e.g., label La-Lc (Figs. 3(A)-3(B), 9(A)-9(C)). The label is an example of a target object that is an object of interest.

[0102] As shown in Figures 3(A)-3(C), in S505 (Figure 8), the processor 110 generates data of a reconstructed image (e.g., reconstructed images i2ag-i2cg, i2xg, i2zg) by inputting data of a labeled partial image (e.g., labeled partial images i2a-i2c, i2x, i2z) to the image generation model M2. As described in Figures 3(A)-3(C) and 6, the image generation model M2 has been trained to generate image data representing a label without an appearance abnormality (e.g., Lbg, Lcg) from image data representing a label with an appearance abnormality (e.g., labels Lb, Lc).

[0103] As described in Fig. 9(A)-Fig. 9(C), in S525 (Fig. 8), the processor 110 generates a histogram HR of the luminance V of the reconstructed image (e.g., reconstructed images i2ag, i2xg, i2zg) (hereinafter, the histogram HR is also referred to as a second histogram HR). The second histogram HR represents the intensity distribution of the luminance V.

[0104] In S540 and S550 (FIG. 8), the processor 110 executes a first judgment result process (S550) when the correlation c is equal to or less than the first judgment threshold (S540: Yes). As described in S540 and FIG. 10(A), the correlation c represents the correlation between the first histogram HI and the second histogram HR. The correlation c indicates the similarity between the first histogram HI and the second histogram HR. The fact that the correlation c is equal to or less than the first judgment threshold indicates that the difference between the first histogram HI and the second histogram HR is equal to or more than the difference threshold (the first judgment threshold is an example of a parameter indicating the difference threshold). The first judgment result process is an example of a specific process associated with the case where the difference between the first histogram HI and the second histogram HR is equal to or more than the difference threshold.

[0105] In this way, the processor 110 executes the first determination result process when the difference between the first histogram HI and the second histogram HR is equal to or greater than the difference threshold, and therefore can proceed with the process according to the brightness of the labeled partial image.

[0106] Here, the first determination result process preferably includes a process that is not executed when the difference between the histograms HI and HR is less than the difference threshold (S540: No) (for example, S552 in FIG. 8). This allows the processor 110 to execute a process when the difference between the histograms HI and HR is equal to or greater than the difference threshold, different from the process executed when the difference between the histograms HI and HR is less than the difference threshold.

[0107] Furthermore, when multiple labels are processed, the label expressions, such as brightness and the position of the label in the image, may differ between multiple labeled partial images. In this embodiment, the processor 110 uses the data of the labeled partial image and the image generation model M2 to generate data of a reconstructed image suitable for the labeled partial image. Therefore, the advance preparation of the second histogram HR suitable for comparison with the first histogram HI of the unknown labeled partial image (and thus the reference image to be used for generating the second histogram HR) is omitted. In this way, the burden of the judgment process using the histograms HI and HR is reduced.

[0108] 9(A) to 9(C), the histograms HI and HR represent the intensity distribution of the luminance V. Therefore, the processor 110 can proceed with the process according to the brightness of the labeled portion image, regardless of the color of the label.

[0109] Also, in S540 and S560 (FIG. 8), if the correlation c is greater than the first judgment threshold (S540: No), the processor 110 determines the result of the image judgment as "brightness normal" (S560). The fact that the correlation c is greater than the first judgment threshold indicates that the difference between the first histogram HI and the second histogram HR is less than the difference threshold. Then, as described in S340, S360-S390 (FIG. 7), if the result of the image judgment is "brightness normal" (S340: Yes), that is, if the difference between the histograms HI and HR is less than the difference threshold, the processor 110 executes the process of S360-S390. In S360-S390, the processor 110 uses the label portion image and the reconstructed image to judge whether the label has an abnormality in appearance. In this embodiment, in S360, the processor 110 uses the label portion image and the reconstructed image to detect an abnormal part of the label. If an abnormal portion is detected (S370: Yes), in S390, processor 110 determines that the inspection result is abnormal, i.e., that the label has an appearance abnormality. If no abnormal portion is detected (S370: No), in S380, processor 110 determines that the inspection result is normal, i.e., that the label does not have an appearance abnormality.

[0110] Here, the determination of whether the label has an abnormality in appearance is called an abnormality determination. The condition that the difference between the first histogram HI and the second histogram HR is less than the difference threshold is called a histogram condition. The condition under which the execution of the abnormality determination using the label partial image and the reconstructed image is permitted is called an allowable condition. In this embodiment, the allowable condition includes a histogram condition. Specifically, as shown in S340 (FIG. 7), the allowable condition is the same as the histogram condition. When the allowable condition including the histogram condition is satisfied (FIG. 7: S340: Yes), the processor 110 can execute an appropriate abnormality determination using a label partial image that represents the label with appropriate brightness and a reconstructed image that is generated using the label partial image. When the allowable condition including the histogram condition is not satisfied (FIG. 7: S340: No), the processor 110 does not execute the abnormality determination and ends the inspection process. Therefore, the processor 110 can avoid an inappropriate abnormality determination using a label partial image that represents the label with inappropriate brightness.

[0111] B. Second Example: Figures 13 and 14 are flowcharts showing another embodiment of the image determination process. Figure 14 shows a process subsequent to the process of Figure 13. The image determination process of this embodiment is executed in S330 (Figure 7) instead of the process of Figure 8.

[0112] In S410, the processor 110 acquires the luminance V of each pixel of the captured image acquired in S310 (FIG. 7). The method of acquiring the luminance V is the same as the method of acquiring the luminance V in S510 (FIG. 8).

[0113] In S420, the processor 110 generates a histogram of the luminance V of the captured image (called a captured histogram). Although not shown, the captured histogram represents the intensity distribution of the luminance V, similar to the histograms HI and HR in Figures 9(A) to 9(C).

[0114] In S430, the processor 110 calculates an index value of the difference between the photographed histogram and the reference histogram D3 (FIG. 1) (also called a photographed index value). The reference histogram D3 is a histogram of the brightness V of a reference image prepared in advance. The reference image is an image indicating a reference for the photographed image, and represents a portion of the product 300 including the label L, similar to the photographed image (e.g., the photographed image i1 in FIG. 2(C)). The reference image represents the label L without any abnormality in appearance with appropriate brightness. Data of the reference image is prepared in advance. Data of the reference histogram D3 is prepared in advance using data of the reference image. In this embodiment, data of the reference histogram D3 is stored in the non-volatile storage device 130 in advance.

[0115] The calculation formula for the shooting index value is the same as that for the input index value in S530 (FIG. 8), except that the shooting histogram and the reference histogram D3 are used instead of the histograms HI and HR. In this embodiment, the processor 110 calculates the correlation (FIG. 10(A)).

[0116] Note that, instead of the data of the reference histogram D3, data of the reference image may be stored in advance in the storage device 115 (for example, the non-volatile storage device 130). In S430, the processor 110 may generate the reference histogram using the data of the reference image.

[0117] In S440, the processor 110 determines whether the brightness of the captured image is within the allowable range. As described above, the reference image represents the label L with appropriate brightness. The reference histogram shows an example of a histogram of a captured image representing the label L with appropriate brightness. When the brightness of the captured image is appropriate, the difference between the captured histogram and the reference histogram is small. Therefore, in this embodiment, the processor 110 determines that the brightness of the captured image is within the allowable range when the difference between the captured histogram and the reference histogram is equal to or less than a threshold value.

[0118] In this embodiment, when the photographing index value (here, correlation) is equal to or greater than a predetermined first photographing threshold, the processor 110 determines that the histogram difference is equal to or less than the threshold, that is, determines that the brightness of the photographed image is within an allowable range. The first photographing threshold is experimentally determined in advance such that when the brightness of the photographed image is appropriate, the histogram correlation is equal to or greater than the first photographing threshold, and when the brightness of the photographed image is inappropriate, the histogram correlation is less than the first photographing threshold.

[0119] The label partial images i2a-i2c in Fig. 3(A)-Fig. 3(C) all show the labels La-Lc with appropriate brightness. When a captured image (not shown) including such label partial images i2a-i2c is processed, the judgment result in S440 may be Yes. The captured images i1x and i1z in Fig. 4(B) and Fig. 4(C) all show the label La with inappropriate brightness. When such captured images i1x and i1z are processed, the judgment result in S440 may be No.

[0120] If the determination result of S440 is No, in S450, the processor 110 executes the first pre-stage process. The first pre-stage process may be any process associated with the determination result of S440 (here, No). In this embodiment, the first pre-stage process includes S452, S454, and S456. S452, S454, and S456 are the same as S552, S554, and S556 included in S550 of FIG. 8, respectively. After S450, the processor 110 ends the process of FIG. 13, that is, the process of S330 of FIG. 7. The operator who has received the result in S454 adjusts the light amount of the light source 500 to an appropriate light amount before the start of the next inspection process (for example, in S459 after S450). As a result, in the next inspection process, the imaging device 400 can generate a captured image showing the label with appropriate brightness.

[0121] If the determination result in S440 is Yes, in S460, the processor 110 executes a label detection process. The process of S460 is the same as the process of S460 in Fig. 8. In S470, the processor 110 determines whether or not a label is detected.

[0122] If the label is not detected (S470: No), in S480, the processor 110 executes the second pre-stage process. The second pre-stage process may be any process associated with the judgment result of S470 (here, No). In this embodiment, the second pre-stage process includes S482, S484, and S486. In S482, the processor 110 determines the result of the image judgment to be "not detected". S484 and S486 are the same as S454 and S456 included in S450, respectively. In S484, the processor 110 notifies the operator of the result of the image judgment (here, "not detected"). After S480, the processor 110 ends the process of FIG. 13, that is, the process of S330 in FIG. 7. If the result of the image judgment is "not detected", the judgment result of S340 in FIG. 7 is "No". The processor 110 ends the process of FIG. 7 without inspecting the appearance of the label L. Therefore, the possibility of performing an inspection using a captured image in which the label L is not detected is reduced.

[0123] If a label is detected (S470: Yes), the processor 110 proceeds to S505 in FIG. 14. S505-S566 in FIG. 14 are the same as S505-S566 in FIG. 8, respectively. In S505-S540, the processor 110 determines whether the difference between the input histogram HI of the labeled partial image and the reconstructed histogram HR of the reconstructed image is equal to or greater than a difference threshold. Then, the processor 110 executes a process (S550 or S560) associated with the result of this determination. Then, the processor 110 ends the process of FIG. 13, that is, the process of S330 in FIG. 7.

[0124] As described above, the image determination process (FIGS. 13 and 14) of this embodiment includes the process of FIG. 8. Therefore, this embodiment has the same various advantages as the embodiment of FIG. 8. For example, when the difference between the first histogram HI and the second histogram HR is equal to or greater than the difference threshold (S540: Yes), the processor 110 executes the first determination result process (S550), so that the process can proceed according to the brightness of the labeled portion image. Also, since the histograms HI and HR represent the intensity distribution of the luminance V, the processor 110 can proceed with the process according to the brightness of the labeled portion image, regardless of the color of the label.

[0125] In addition, in S410-S440 (FIG. 13), the processor 110 uses the data of the captured image of the label to determine whether the brightness of the captured image is within the allowable range. If it is determined that the brightness of the captured image is within the allowable range (S440: Yes), in S460, the processor 110 detects a label partial image, which is a portion representing a label, from the captured image (for example, label partial images i2a-i2c in FIG. 3(A)-FIG. 3(C)). If it is determined that the brightness of the captured image is outside the allowable range (S440: No), the processor 110 cancels the detection of the label partial image (S460). In this embodiment, the processor 110 executes a predetermined process (S450) instead of the detection of the label partial image (S460). In this way, if it is determined that the brightness of the captured image is outside the allowable range (S440: No), the detection of the label partial image is canceled, thereby reducing the possibility of the process using an inappropriate label partial image. For example, when a detection process for a labeled partial image is performed on a dark captured image such as the captured image i1x (FIG. 4(B)), a part other than the part representing the label La may be detected. The possibility that a process (e.g., an inspection process) will proceed using such an inappropriate partial image is reduced.

[0126] Also, in this embodiment, in S410-S420 (FIG. 13), processor 110 generates a shooting histogram using data of the captured image. In S430-S440, processor 110 uses the shooting histogram and reference histogram D3 to determine whether or not the brightness of the captured image is within an acceptable range. In this way, processor 110 uses data of the captured image and data of reference histogram D3 to determine whether or not the brightness of the captured image is within an acceptable range. Reference histogram D3 is an example of reference data associated with a reference image prepared in advance. Processor 110 can appropriately determine whether or not the brightness of the captured image is within an acceptable range by using reference data associated with such a reference image.

[0127] C. Third Example: Fig. 15 is a flowchart showing another embodiment of the image determination process. In the figure, S532, S534, S536, and S540c, which are parts of the image determination process, are shown. The processor 110 executes these processes S532, S534, S536, and S540c instead of S540 (Figs. 8 and 14).

[0128] After S530 (FIG. 8, FIG. 14), in S532 (FIG. 15), the processor 110 generates a histogram (referred to as an input histogram) of each of a plurality of chromatic components (here, R, G, B) of a labeled partial image (e.g., labeled partial images i2a, i2x, i2z (FIG. 9(A)-FIG. 9(C))). In S534, the processor 110 generates a histogram (referred to as a reconstructed histogram) of each of a plurality of chromatic components (here, R, G, B) of a reconstructed image (e.g., reconstructed images i2ag, i2xg, i2zg (FIG. 9(A)-FIG. 9(C))). The configuration of the histogram (e.g., the total number of intervals) is the same between the input histogram and the reconstructed histogram corresponding to the same color component.

[0129] In S536, the processor 110 calculates an index value of the difference between the input histogram and the reconstructed histogram corresponding to the same color component (also called an input index value). The method of calculating the input index value of each of R, G, and B is the same as the method of calculating the input index value of the luminance V in S530 (FIGS. 8 and 14).

[0130] In S540c, the processor 110 judges whether the difference between the input histogram and the reconstructed histogram is equal to or greater than the difference threshold. This judgment is performed for each of the V, R, G, and B color components. The judgment method is the same as the judgment method of S540 (FIGS. 8 and 14). The difference threshold (here, the first judgment threshold) is predetermined for each of the V, R, G, and B color components. The difference threshold is experimentally determined in advance so that the histogram difference is less than the difference threshold when the label brightness of the labeled partial image is appropriate, and the histogram difference is equal to or greater than the difference threshold when the label brightness of the labeled partial image is inappropriate.

[0131] Processor 110 determines whether the difference between one or more color components among V, R, G, and B is equal to or greater than the corresponding difference threshold (in this embodiment, whether the correlation between one or more color components is equal to or less than the corresponding first determination threshold). If the difference between one or more color components is equal to or greater than the difference threshold (S540c: Yes), processor 110 proceeds to S550 (FIGS. 8 and 14). If the differences between all color components are less than the difference threshold (S540c: No), processor 110 proceeds to S560 (FIGS. 8 and 14).

[0132] As described above, in this embodiment, in S532 (FIG. 15), the processor 110 generates an input histogram (called the third histogram) of a specific chromatic color component (e.g., red R) of a labeled partial image (e.g., labeled partial images i2a-i2c, i2x, i2z (FIGS. 3(A)-3(C), 9(A)-9(C))). The third histogram represents the intensity distribution of the specific chromatic color component. In S534, the processor 110 generates a reconstructed histogram (called the fourth histogram) of a specific chromatic color component of a reconstructed image (e.g., reconstructed images i2ag-i2cg, i2xg, i2zg). The fourth histogram represents the intensity distribution of the specific chromatic color component.

[0133] In S540c (FIG. 15) and S550 (FIG. 8, FIG. 14), the processor 110 executes the first determination result process (S550) when the difference between one or more color components among V, R, G, and B is equal to or greater than the difference threshold corresponding to the color component. That is, the first determination result process (S550) is executed in the following specific cases. The specific cases include a case where the difference between the first and second histograms of the luminance V is equal to or greater than the difference threshold corresponding to the luminance V, and a case where the difference between the third and fourth histograms of a specific chromatic color component (e.g., red R) is equal to or greater than the difference threshold corresponding to the specific chromatic color component (referred to as the chromatic difference threshold). In this embodiment, the specific cases further include a case where the difference between the histograms of other chromatic color components (e.g., green G or blue B) is equal to or greater than the difference threshold corresponding to the color component, and a case where the difference between two or more color components among V, R, G, and B is equal to or greater than the difference threshold.

[0134] The color of the light illuminating the label during shooting may change. For example, the color of the light of the light source 500 may change over time. Also, when external light (e.g., sunlight) illuminates the label through a window in the room, the color of the light may change over time. If the color of the light illuminating the label during shooting deviates from the intended color (e.g., white), the color of the label represented by the label partial image may be inappropriate, and an inappropriate reconstructed image may be generated by the image generation model M2 (e.g., even if there is no abnormality in the appearance of the label, the reconstructed image may show an abnormality in the label). The intensity distribution of the chromatic color components may differ between such a label partial image and the reconstructed image. In this embodiment, if the difference between the third histogram and the fourth histogram of a specific chromatic color component is equal to or greater than the difference threshold (FIG. 15: S540c: Yes), the processor 110 executes the first judgment result process (FIG. 8, FIG. 14: S550). In this way, the processor 110 can proceed with the process according to the color of the label partial image.

[0135] The specific chromatic color component may be any color component suitable for the color of the light illuminating the label (e.g., the color of the light of the light source 500). For example, if the color of the light of the light source 500 is likely to unintentionally change to green, green G may be adopted as the specific chromatic color component. The total number of chromatic color components used to determine the difference in the histogram is not limited to 3, but may be 1, 2, or 4 or more. The chromatic color components are not limited to color components selected from R, G, and B, but may be various color components such as cyan C, magenta M, or yellow Y.

[0136] In addition, in S540c (FIG. 15) and S560 (FIG. 8, FIG. 14), if the correlation of each of the color components V, R, G, and B is greater than the first judgment threshold of the corresponding color component, that is, if the difference between the histograms of each of the color components V, R, G, and B is less than the difference threshold of the corresponding color component (S540c: No), the processor 110 determines the result of the image judgment to be "normal brightness" (S560). Then, as described in S340, S360-S390 (FIG. 7), if the result of the image judgment is "normal brightness" (S340: Yes), that is, if the difference between the histograms of each of the color components V, R, G, and B is less than the corresponding difference threshold, the processor 110 executes the process of S360-S390. In S360-S390, the processor 110 uses the label portion image and the reconstructed image to judge whether the label has an abnormality in appearance. In this way, the permissible conditions for allowing the execution of the abnormality determination include a condition that the difference in the histogram of the luminance V is less than the difference threshold corresponding to the luminance V, and a condition that the difference in each of the histograms of the other color components (here, R, G, B) is less than the corresponding difference threshold. If the permissible conditions are satisfied (S340: No), the processor 110 can perform an appropriate abnormality determination using a label partial image that represents a label with appropriate brightness and appropriate color, and a reconstructed image generated using the label partial image. If the permissible conditions are not satisfied (S340: No), the processor 110 ends the inspection process without performing the abnormality determination. Therefore, the processor 110 can avoid an inappropriate abnormality determination using a label partial image that represents a label with inappropriate brightness or inappropriate color.

[0137] D. Fourth Example: Fig. 16 is a flowchart showing an example of a process for determining a difference threshold. This process is a process for determining a difference threshold (here, the first determination threshold) used in S540 (Figs. 8 and 14). Processor 110 of data processing device 100 (Fig. 1) executes the process of Fig. 16 in accordance with third program 133.

[0138] S605, S610, S615, S620, S625, and S630 are the same as S505, S510, S515, S520, S525, and S530 in FIG. 8 and FIG. 14, respectively, except that a training image is used instead of the label partial image. In S605, the processor 110 generates data of a reconstructed image by inputting data of a training image representing a label L without an abnormality in appearance to a trained image generation model M2 (the reconstructed image generated is also called a generated image). The data of the training image is data used in the training process (FIG. 6) of the image generation model M2. The processor 110 executes S610-S630 to calculate an index value of the difference between the histogram of the luminance V of the training image and the histogram of the luminance V of the generated image (also called a reference index value). The calculation formula for the reference index value is the same as that for the input index value in S530 (FIGS. 8 and 14), except that the histograms of the training image and the histogram of the generated image are used instead of the histograms HI and HR. In this embodiment, the processor 110 calculates the correlation (FIG. 10(A)).

[0139] As described in FIG. 6, a plurality of training images are used for training the image generation model M2. The plurality of training images include a plurality of training images that represent labels at different brightnesses. In the process of FIG. 16, a portion of the plurality of training images used for training is used. The plurality of training images used in FIG. 16 include a plurality of training images that represent labels at different brightnesses. The processor 110 calculates a plurality of reference index values ​​by executing the process of S605-S630 for each of the plurality of training images. Each of the plurality of reference index values ​​obtained from the plurality of training images indicates an input index value (FIG. 8: S530) that can be calculated from a label partial image that represents a label at an appropriate brightness. Note that, in the process of FIG. 16, all of the plurality of training images used for training may be used.

[0140] In S640, the processor 110 determines the difference threshold value using a plurality of reference index values. In this embodiment, when it is assumed that the judgment in S540 (FIGS. 8 and 14) is performed using the reference index values, the processor 110 determines the difference threshold value so that the judgment result for each of the plurality of reference index values ​​is "the difference is less than the difference threshold value." This reduces the possibility that the judgment result for a label partial image showing a label with appropriate brightness is "the difference is equal to or greater than the difference threshold value."

[0141] When the index value indicates a correlation c(10(A)), the smaller the index value, the greater the difference between the histograms. The processor 110 determines the difference threshold (here, the first judgment threshold) to be a value smaller than the multiple reference index values. For example, the first judgment threshold is set to a value obtained by multiplying the smallest reference index value among the multiple reference index values ​​by a coefficient less than 1 (for example, 0.8).

[0142] In S650, processor 110 stores data indicating the determined difference threshold in storage device 115 (e.g., non-volatile storage device 130). Then, processor 110 ends the processing of Fig. 16. In S540 (Figs. 8 and 14), processor 110 uses the difference threshold determined in the processing of Fig. 16.

[0143] As described above, in this embodiment, the processor 110 determines the difference threshold using the training image data input to the image generation model M2 for training the image generation model M2 and the generated image data generated by inputting the training image data to the trained image generation model M2. Therefore, the processor 110 can determine a difference threshold suitable for the image generation model M2. For example, the processor 110 may determine the difference threshold so that the difference between the histogram of the training image and the histogram of the generated image is less than the difference threshold. In this case, when the label partial image represents the label with a brightness similar to the brightness of the label in the training image, the processor 110 can determine that the difference between the histograms is less than the difference threshold. Note that when an image of a label having an abnormality in appearance is used to determine the difference threshold, an inappropriate difference threshold may be determined due to the image. It is preferable to use a training image representing a label without an abnormality in appearance to determine the difference threshold.

[0144] The processor 110 may determine the difference threshold so that a determination result that "the histogram difference is equal to or greater than the difference threshold" is obtained from some of the reference index values ​​(e.g., the reference index values ​​showing the largest difference). For example, the processor 110 may set the first determination threshold to a value obtained by multiplying the smallest reference index value among the reference index values ​​by a coefficient greater than 1 (e.g., 1.2). In this case, when an inappropriate reconstructed image is generated, the possibility of obtaining a determination result that "the histogram difference is less than the difference threshold" is reduced. The first determination threshold may be set to the same value as the smallest reference index value.

[0145] The threshold determination process of this embodiment may be applied to the embodiment of Fig. 15. In the embodiment of Fig. 15, a difference threshold is determined for each color component. The processor 110 may determine the difference threshold for each color component by executing the process of Fig. 16 for each color component. Here, in S610 and S615, the gradation value of the color component to be processed is obtained. In S620 and S625, a histogram of the color component to be processed is generated.

[0146] E. Variations: (1) The image determination process may be various other processes instead of the processes of FIG. 8, FIG. 13-FIG. 14, and FIG. 15. For example, the target color component, which is the color component of the histograms HI and HR, may be various other color components instead of the luminance V. For example, any of red R, green G, and blue B may be adopted. The target color component is preferably a color component that has a correlation with the brightness of the label in the input image. The gradation value of such a target color component changes according to the brightness of the label in the input image. Therefore, the processor 110 can appropriately determine whether the brightness of the label in the input image is appropriate by using the difference between the histograms HI and HR. As the color component that has a correlation with the brightness, for example, a color component that changes between a gradation value corresponding to white (for example, a maximum gradation value such as 255) and a gradation value corresponding to black (for example, zero) can be adopted.

[0147] (2) The index value of the histogram difference may be various values ​​indicating the magnitude of the histogram difference, instead of the correlation c (FIG. 10(A)). For example, the processor 110 may calculate the total difference between two histograms as the index value. FIG. 10(B) is a diagram showing an example of a calculation formula for the total difference d. The total difference d is the total value of the squares of the differences of the frequencies HI_k and HR_k of the n sections. As in FIG. 9(A), when the reconstructed histogram HR is similar to the input histogram HI, the total difference d is small. As in FIG. 9(B) and FIG. 9(C), when the difference between the reconstructed histogram HR and the input histogram HI is large, the total difference d is large. In S540 (FIGS. 8 and 14), the processor 110 may determine that the histogram difference is equal to or greater than the difference threshold when the total difference d is equal to or greater than a predetermined second determination threshold.

[0148] The second judgment threshold may be determined by the process of FIG. 16. In S630, the processor 110 calculates the total difference as the reference index value. In S640, the processor 110 may determine the second judgment threshold to be a value greater than the multiple reference index values. For example, the processor 110 may set the second judgment threshold to a value obtained by multiplying the maximum reference index value of the multiple reference index values ​​by a coefficient greater than 1 (e.g., 1.2). Alternatively, the processor 110 may set the second judgment threshold to a value obtained by multiplying the maximum reference index value of the multiple reference index values ​​by a coefficient less than 1 (e.g., 0.8). The second judgment threshold may also be set to a value equal to the maximum reference index value.

[0149] In the embodiment of Fig. 15, the total difference may be used instead of the correlation. In this case, a difference threshold (here, the second determination threshold) is determined for each color component. In S430 and S440 of Fig. 13, the total difference may be used instead of the correlation. In S440, when the total difference is equal to or less than a predetermined second shooting threshold, the processor 110 determines that the histogram difference is equal to or less than the threshold, that is, determines that the brightness of the captured image is within the allowable range.

[0150] (3) The method of determining the difference threshold may be various other methods instead of the method of Fig. 16. For example, in the process of Fig. 16, multiple captured images showing the labels with appropriate brightness may be used instead of the training images.

[0151] (4) The specific process executed when the difference between the histogram of the label partial image (i.e., the input image) and the histogram of the reconstructed image is equal to or greater than the difference threshold may be various processes, not limited to the process of S550 (FIGS. 8 and 14). For example, the processor 110 may execute a process of adjusting the amount of light of the light source 500. When the label partial image is brighter than the reconstructed image, the processor 110 may reduce the amount of light of the light source 500. When the label partial image is darker than the reconstructed image, the processor 110 may increase the amount of light of the light source 500. The processor 110 may adjust the amount of light by repeating the change of the amount of light and the calculation of the difference of the histogram based on the changed amount of light (for example, the same process as the process of S460, S505-S530 in FIG. 8) until the difference of the histogram becomes smaller than the difference threshold. Here, the light source 500 may be connected to the data processing device 100 (for example, the communication interface 170).

[0152] (5) The permissible condition for allowing the execution of the anomaly determination using the labeled partial image and the reconstructed image may be various conditions including a histogram condition that the difference between the first histogram HI and the second histogram HR is less than a difference threshold. The permissible condition may include, for example, that the maximum brightness in the labeled partial image is equal to or less than a predetermined upper limit.

[0153] (6) The process of detecting anomalies in the appearance of the label may be any process capable of detecting anomalies, instead of the process described in FIG. 11. For example, the processor 110 may determine the presence or absence of anomalies in the appearance of the label using a machine learning model trained to classify label partial images into images with anomalies and images without anomalies. The processor 110 may also detect anomalies in the appearance of the label using a technology called CFLOW-AD. CFLOW-AD is disclosed in the following paper: Denis Gudovskiy, Shun Ishizaka and Kazuki Kozuka (2022), "CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows", 2022 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV), 1819-1828. https: / / doi.org / 10.1109 / WACV51458.2022.00188

[0154] Also, in S362 of FIG. 11, the processor 110 may generate data of a heatmap image indicating a possibility of anomaly using a machine learning model. Here, a technology called PaDiM (a Patch Distribution Modeling Framework for Anomaly Detection and Localization) may be adopted. PaDiM is a technology for detecting anomalies using a machine learning model, and is disclosed in the paper "T. Defard, A. Setkov, A. Loesch, and R. Audigier, "Padim: a patch distribution modeling framework for anomaly detection and localization", arXiv:2011.08785(2020), https: / / arxiv.org / abs / 2011.08785, posted on 17 Nov 2020".

[0155] In PaDiM, a Convolutional Neural Network (CNN) that has been trained to identify an image representing an object of interest (hereinafter, the object of interest is assumed to be a label) is used. As the CNN, for example, a well-known model called ResNet18 is used. This CNN includes multiple convolution processing units, each of which includes one or more convolution layers, and a classification processing unit that includes a fully connected layer.

[0156] Prior to anomaly detection, a Gaussian matrix is ​​generated. In the following, it is assumed that the processor 110 generates the Gaussian matrix. The processor 110 generates the Gaussian matrix using images of labels that have no abnormal appearance (called normal images). N normal images are used to generate the Gaussian matrix (N is an integer equal to or greater than 1. N is, for example, about several tens or several hundreds). The normal images may include photographed images of label L (FIG. 2(B)). The normal images may also include images synthesized by a computer, such as the training images in FIG. 6.

[0157] The processor 110 inputs normal image data to the CNN to calculate feature maps for each of the multiple convolution processors. The processor 110 calculates a feature vector for each patch by using L feature maps (L is an integer equal to or greater than 1) preselected from the multiple feature maps of the first three convolution processors. The patch corresponds to a pixel in the largest feature map. Hereinafter, the position of the patch is represented by coordinates (i, j) (i is the pixel position in the first direction Dx, and j is the pixel position in the second direction Dy). Since the resolution of the feature map is lower than the resolution of the image input to the CNN, multiple pixels of the image are associated with one patch (i, j) (in other words, the image is divided into multiple patches).

[0158] A feature vector (i,j) associated with a patch (i,j) is an L-dimensional vector having L elements. The processor 110 obtains the L elements by using the above-mentioned L feature maps. Here, the processor 110 enlarges each feature map by a known enlargement process so that the resolution becomes the same as the resolution of the largest feature map. The L elements of the feature vector (i,j) are L values ​​of L pixels corresponding to coordinates (i,j) in the L feature maps having the same resolution. Note that, from one normal image, the feature vectors of each of the multiple patches are calculated. From N normal images, N feature vectors are calculated for each patch.

[0159] The Gaussian matrix represents the Gaussian parameters of each patch. The Gaussian parameters (i,j) include a mean vector (i,j) and a covariance matrix (i,j). The mean vector (i,j) is the average of the N feature vectors (i,j). The covariance matrix (i,j) is the covariance matrix of the N feature vectors (i,j). The processor 110 calculates the Gaussian parameters of each patch using the N feature vectors of each patch. The processor 110 stores the CNN data and data representing the Gaussian matrix in the storage device 115 (e.g., the non-volatile storage device 130).

[0160] In S362 of FIG. 11, the processor 110 generates a heat map image using the labeled partial image and the Gaussian matrix. The processor 110 inputs the data of the labeled partial image to the CNN to calculate feature maps by each of the multiple convolution processing units of the CNN. The processor 110 calculates feature vectors of each of the multiple patches (called test feature vectors) using the above L feature maps among the multiple feature maps calculated. The processor 110 calculates an anomaly score (i, j) of the patch (i, j) using the test feature vector (i, j) and the Gaussian parameters (i, j). As the anomaly score, the so-called Mahalanobis distance is adopted. The Mahalanobis distance (i, j) of the patch (i, j) is an evaluation value indicating the degree of difference between the N normal images and the labeled partial image in the patch (i, j). Therefore, it can be said that the Mahalanobis distance (i, j) is a value indicating the degree of anomaly of the labeled partial image at the coordinates (i, j). The processor 110 calculates the Mahalanobis distance, i.e., the anomaly score, of each patch according to the formula described in the above-mentioned paper of PaDiM. The processor 110 generates data of an image representing the anomaly score for each patch (i.e., pixel). The generated image is an example of a heat map image.

[0161] The processor 110 may normalize the anomaly score so that it is expressed in a predetermined range (e.g., a range of zero or more and one or less). The normalization may be performed, for example, using a predetermined minimum value and a predetermined maximum value. The minimum value and the maximum value may be determined in advance using a plurality of unnormalized anomaly scores calculated by inputting a plurality of images of a label having an appearance abnormality and a plurality of normal images into a CNN. The maximum value may be set to the maximum value of the plurality of unnormalized anomaly scores. The minimum value may be set to the minimum value of the plurality of unnormalized anomaly scores. The processor 110 may calculate the normalized anomaly score As according to the following formula using the unnormalized anomaly score Ao, the maximum value Amax, and the minimum value Amin. As = (Ao-Amin) / (Amax-Amin)

[0162] In addition, in the generation of a heat map image by PaDiM, a difference image obtained from a label image may be used instead of a label image. Specifically, a plurality of normal difference images and a plurality of abnormal difference images are used for training the CNN. The normal difference image represents the difference between a reconstructed image obtained by inputting data of a normal image into the image generation model M2 and a normal image (e.g., difference image i2ad in FIG. 3A). The abnormal difference image represents the difference between a reconstructed image obtained by inputting an abnormal image, which is an image of a label having an abnormal appearance, into the image generation model M2 and an abnormal image (e.g., difference images i2bd and i2cd in FIG. 3B and FIG. 3C). The CNN is trained to identify the type of image (e.g., normal, scratch, dirt, etc.). The processor 110 determines the Gaussian matrix by inputting data of a plurality of normal difference images into the CNN. As described above, the grayscale value (i.e., difference) of each pixel of the normal difference image is close to zero. However, the reconstructed image is not identical to the normal image. Therefore, the gradation value of each pixel of the normal difference image may be different for each pixel. In S362 (FIG. 11), the processor 110 generates a heat map image using the difference image and the Gaussian matrix obtained from the label portion image. When the label has an abnormality in appearance, such as the labels Lb and Lc in FIG. 3(B) and FIG. 3(C), the difference image shows the abnormality in the appearance of the label. In this case, the abnormality score is large in the part corresponding to the abnormal part in the heat map image. When the label does not have an abnormality in appearance, such as the label La in FIG. 3(A), the difference image does not show the abnormality in the appearance of the label. In this case, the abnormality score is small throughout the heat map image. When the difference image is used to generate the heat map image by PaDiM, the difference between the label portion image and the normal image may be more emphasized than when the label image is used.

[0163] (7) The process of determining whether the brightness of the captured image is within the allowable range may be various other processes instead of the processes of S410-S440 in FIG. 13. Instead of the histogram of the brightness V, the processor 110 may use various statistics (average value, median value, etc.) that represent the characteristics of the distribution of the brightness V. For example, the processor 110 may determine that the brightness of the captured image is within the allowable range when the median value of the brightness V of the captured image is within a predetermined allowable range. The allowable range of the median value of the brightness V may be determined in advance using a reference image that represents a label without an abnormal appearance with an appropriate brightness. For example, the allowable range may be a range of a predetermined width that includes the median value of the brightness V of the reference image. Data indicating the allowable range is stored in advance in the storage device 115 (for example, the non-volatile storage device 130). The processor 110 may determine whether the brightness of the captured image is within the allowable range using the data of the captured image and the data of the allowable range. The data of the allowable range is an example of reference data that is associated with a reference image that is prepared in advance. The allowable range of the median value of the luminance V may be determined independently of the reference image. The reference data may be data of the reference image. The processor 110 may determine the allowable range of the brightness of the captured image by analyzing the data of the reference image. In addition, other color components (e.g., green G) may be used in this determination instead of the luminance V.

[0164] (9) The image generation model M2 is not limited to a variational autoencoder, and may be any machine learning model that generates image data representing a label without an appearance anomaly from image data representing a label with an appearance anomaly. The image generation model M2 may be, for example, a model called VQ-VAE (Vector Quantised-Variational AutoEncoder). VQ-VAE is disclosed in the following paper. “Aaron van den Oord, Oriol Vinyals and Koray Kavukcuoglu. “Neural Discrete Representation Learning.” arXiv, May 30, 2018, http: / / arxiv.org / abs / 1711.00937” In either case, the specific configuration of the model (for example, the specific configuration of each layer (dimension of data, activation function, etc.), the number of layers, etc.) may be changed as appropriate.

[0165] (10) The training process of the image generation model M2 may be any process for training the image generation model M2 to generate image data representing labels without an appearance abnormality from image data representing labels with an appearance abnormality, instead of the process described in FIG. 6. For example, the training images may include images representing labels with an appearance abnormality. The calculation parameters of the image generation model M2 may be adjusted so that the difference between a reconstructed image generated using the training images representing labels with an appearance abnormality and a normal image obtained by removing the abnormality from the training images is reduced. The training of the image generation model M2 may be performed by the data processing device 100, or may be performed by another device different from the data processing device 100.

[0166] (10) The object detection model M1 may be any machine learning model that detects an object of interest (e.g., a label) instead of YOLOv4. The object detection model M1 may be any of SSD, R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN. Training of the object detection model M1 may be performed by the data processing device 100 or may be performed by another device different from the data processing device 100.

[0167] (11) The process of detecting a portion representing an object of interest (e.g., a label) from a captured image (S350 (FIG. 7), S460 (FIG. 8, FIG. 13)) may be various other processes instead of the process using the object detection model M1. For example, the processor 110 may detect the portion representing the label by template matching using a template image of a label prepared in advance.

[0168] (12) The image determination process (FIGS. 8, 13-14, and 15) may be executed by various other processes instead of the inspection process (FIG. 7). For example, the processor 110 may execute the image determination process by adjusting the amount of light from the light source 500 to capture an image of an object of interest (e.g., a label). The processor 110 may adjust the amount of light so that the determination results of S540 and S540c are No.

[0169] (13) The inspection process and the image determination process may be various other processes instead of the above processes. For example, S320 and S350 in FIG. 7 may be omitted. That is, S330 and S340 may be executed every time. The photographing of the label (S310) may be omitted, and the processor 110 may obtain data of the photographed image from another data processing device. The processor 110 may obtain data of the label portion image (i.e., the input image) from another data processing device. In this case, the label detection process (S350 (FIG. 7), S460 (FIG. 8, FIG. 13)) may be omitted. In S380 and S390 in FIG. 7, the processor 110 may display a heat map image on the display unit 140.

[0170] (14) The object of interest, which is the object to be processed, is not limited to a label sheet (e.g., label L (FIG. 2(B)) and may be any object. The object of interest may be an object attached to a product (e.g., a multifunction printer). The object of interest may include one or more elements selected from a plurality of elements including a label sheet, a three-dimensional inscription, and a painted pattern. The three-dimensional inscription and the painted pattern may each indicate various information about the product, such as the manufacturer's name, manufacturer's logo, brand logo, or certification mark. The product is not limited to a multifunction printer and may be any product, such as a sewing machine, cutting machine, machine tool, or mobile terminal.

[0171] (15) The image processing device that performs image processing using a histogram may be various other devices (e.g., digital camera, smartphone) instead of the above-described data processing device 100 (FIG. 1). A plurality of devices (e.g., computers) that can communicate with each other via a network may share a part of the image processing function of the image processing device, and provide the image processing function as a whole (a system including these devices corresponds to an image processing device).

[0172] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the function of generating a reconstructed image using the image generation model M2 (S505 (FIGS. 8 and 14), S605 (FIG. 16)) may be realized by a dedicated hardware circuit.

[0173] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored in a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used in a state stored in the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0174] The above-mentioned examples and modifications can be combined as appropriate. The above-mentioned examples and modifications are provided to facilitate understanding of the present disclosure, and do not limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and the present invention includes equivalents thereof. [Explanation of symbols]

[0175] 100...data processing device, 110...processor, 115...storage device, 120...volatile storage device, 130...non-volatile storage device, 131...first program, 132...second program, 133...third program, 140...display unit, 150...operation unit, 170...communication interface, 210...processor, 300...product, 400...imaging device, 500...light source, L...label

Claims

1. A program, a first generation function for generating a first histogram representing an intensity distribution of a target color component of an input image representing an object of interest; an image generation function that generates data of a reconstructed image by inputting data of the input image into a generative model that has been trained to generate image data representing an object without an appearance abnormality from image data representing an object with an appearance abnormality; a second generation function for generating a second histogram representing an intensity distribution of the target color component of the reconstructed image; an execution function that executes a specific process when a difference between the first histogram and the second histogram is equal to or greater than a difference threshold; A program that enables a computer to achieve this.

2. The program according to claim 1, The target color component is luminance. program.

3. The program according to claim 2, further comprising: a third generation function for generating a third histogram representing an intensity distribution of a specific chromatic color component of the input image; a fourth generation function for generating a fourth histogram representing an intensity distribution of the specific chromatic color component of the reconstructed image; This is realized on a computer. the execution function executes the specific process in specific cases including a case where the difference between the first histogram and the second histogram is equal to or greater than the difference threshold, and a case where the difference between the third histogram and the fourth histogram is equal to or greater than a chromatic difference threshold. program.

4. The program according to any one of claims 1 to 3, further comprising: A program that causes a computer to realize an abnormality determination function that determines whether or not the object of interest has an appearance abnormality using the input image and the reconstructed image when an acceptable condition is satisfied, including the difference between the first histogram and the second histogram being less than the difference threshold.

5. The program according to any one of claims 1 to 3, further comprising: The difference threshold is determined using training image data input to the generative model for training the generative model, and generated image data generated by inputting the training image data to the trained generative model. program.

6. The program according to any one of claims 1 to 3, further comprising: a brightness determination function that uses data of a captured image of the target object to determine whether or not the brightness of the captured image is within an allowable range; a detection function for detecting, from the captured image, the input image that is a portion representing the target object when the brightness of the captured image is determined to be within the allowable range; a cancel function that cancels detection of the input image when the brightness of the captured image is determined to be outside the allowable range; A program that enables a computer to achieve this.

7. The program according to claim 6, the brightness determination function uses data of the captured image and reference data associated with a reference image that is prepared in advance to determine whether the brightness of the captured image is within the allowable range; program.

8. An image processing device, a first generation unit that generates a first histogram representing an intensity distribution of a target color component of an input image representing an object of interest; an image generation unit that generates data of a reconstructed image by inputting data of the input image into a generative model that has been trained to generate image data representing an object without an appearance abnormality from image data representing an object having an appearance abnormality; a second generation unit that generates a second histogram representing an intensity distribution of the target color component of the reconstructed image; an execution unit that executes a specific process when a difference between the first histogram and the second histogram is equal to or greater than a difference threshold; An image processing device comprising:

Citation Information

Patent Citations

  • Inspection device, inspection method, and inspection program

    JP2022049874A