Image processing apparatus and image processing method

Abstract

To provide an image processing device and the like capable of causing a learning model to easily perform machine learning.SOLUTION: The image processing device 100 includes the acquisition unit 110 that acquires an original image showing a target object, the generation unit 140 that generates a plurality of pseudo-images having mutually different feature quantities of a predetermined measurement item by using the original image, and the output unit 150 that outputs the plurality of pseudo-images as teacher data for causing a learning model to perform machine learning, the learning model receiving an inspection image showing the target object and outputting an inspection result indicating whether or not the target object shown in the input inspection image has an abnormality.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]

[0002] Conventionally, there are systems that use machine-learned learning models (trained models) to diagnose analysis targets shown in images.

[0003] Patent Document 1 discloses a system that includes a trained model acquisition unit that acquires multiple trained models that are generated by machine learning and that input information based on an image showing the diagnostic target and output information indicating a diagnostic result for the diagnostic target, an image acquisition unit that acquires an image to be analyzed, and performs calculations based on the multiple trained models on the acquired image to calculate one piece of information related to a diagnosis using the multiple trained models. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 235335 Summary of the Invention [Problem to be solved by the invention]

[0005] Machine learning requires a lot of effort to collect training data such as images to train a learning model. Therefore, it is desirable to reduce this effort and make it easier to train a learning model.

[0006] The present disclosure provides an image processing device and the like that can easily perform machine learning on a learning model. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present disclosure includes an acquisition unit that acquires an original image showing an object, a generation unit that uses the original image to generate a plurality of pseudo images having different feature values ​​for predetermined measurement items, and an output unit that outputs the plurality of pseudo images as training data for machine learning into a learning model that takes an inspection image showing the object as input and outputs an inspection result indicating whether the object shown in the input inspection image is abnormal or not.

[0008] An image processing method according to one aspect of the present disclosure acquires an original image showing an object, uses the original image to generate multiple pseudo images with different feature values ​​for specified measurement items, and outputs the multiple pseudo images as training data for machine learning into a learning model that takes an inspection image showing the object as input and outputs an inspection result indicating whether the object shown in the input inspection image is abnormal. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to provide an image processing device or the like that can easily perform machine learning on a learning model. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing device according to an embodiment. [Figure 2] FIG. 2 is a diagram showing a specific example of an original image according to the embodiment. [Figure 3] FIG. 3 is a diagram showing a performance table according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining the process of setting the feature amount of the base material image according to the embodiment. [Figure 5] FIG. 5 is a diagram for explaining the process of setting the feature amount of a defect image according to the embodiment. [Figure 6] FIG. 6 is a diagram for explaining the annotation information setting process according to the embodiment. [Figure 7] FIG. 7 is a diagram showing a specific example of a plurality of pseudo images according to the embodiment. [Figure 8] FIG. 8 is a flowchart showing a processing procedure of the image processing device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the embodiments described below illustrate specific examples of the present disclosure. Therefore, the numerical values, shapes, materials, components, component placement positions, and connection configurations shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Therefore, among the components in the following embodiments, components that are not recited in independent claims will be described as optional components.

[0012] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and redundant explanations are omitted or simplified.

[0013] (Embodiment) [composition] First, the configuration of an image processing device 100 according to the embodiment will be described.

[0014] Fig. 1 is a block diagram showing the configuration of an image processing device 100 according to an embodiment. Fig. 2(a) to (c) are diagrams showing specific examples of original images according to the embodiment.

[0015] The image processing device 100 is a device that generates multiple images (pseudo images) for machine learning by a learning model using an image (original image) generated by an imaging device 200 such as a camera capturing an image of an object (work).

[0016] The image processing device 100 is, for example, a computer such as a personal computer or a tablet terminal. Specifically, for example, the image processing device 100 is realized by a communication interface for communicating with the imaging device 200, the display device 210, and the input device 220, a non-volatile memory in which a program is stored, a volatile memory that is a temporary storage area for executing the program, an input / output port for transmitting and receiving signals, a processor that executes the program, etc. The communication interface may be realized by a connector to which a communication line is connected to enable wired communication, or by an antenna and a wireless communication circuit to enable wireless communication.

[0017] The image processing device 100 includes an acquisition unit 110, a division unit 120, an extraction unit 130, a generation unit 140, an output unit 150, a reception unit 160, a learning unit 170, and a storage unit 180.

[0018] The acquisition unit 110 is a processing unit that acquires an original image in which an object is captured.

[0019] The object (inspection object) is an object to be inspected for abnormalities using a machine-learned learning model. The acquisition unit 110 acquires, for example, an original image showing the object from the imaging device 200, as shown in (a) to (c) of FIG.

[0020] The object is, for example, an electronic component such as an integrated circuit (IC).

[0021] The object is not limited to an electronic component, but may be any object such as a circuit board.

[0022] The acquisition unit 110 may also acquire the original image from a server device or the like via a communication interface provided in the image processing device 100.

[0023] The dividing unit 120 is a processing unit that divides an original image based on the characteristics of an object shown in the original image to generate a plurality of divided images. By dividing the original image, the dividing unit 120 generates, for example, a divided image that includes only the surface portion of the object, a divided image that includes only the line (straight) portion of the object, and a divided image that includes only the point (e.g., corner) portion of the object. The surface portion is, for example, an image that includes only the object shown in the original image and a background such as a stage on which the object is placed, and does not include the boundary between the object and the background.

[0024] For example, in the inspection of a surface (e.g., an inspection of a flat portion of an object with a relatively uniform surface brightness value), the presence or absence of defects such as scratches and dirt on the surface, the quality of a pattern on the surface, the presence or absence of hairlines, the presence or absence of pattern lines, etc. For example, the learning model judges the presence or absence of abnormalities such as the presence or absence of these defects and the quality or poorness of the pattern for the object reflected in the inspection image, and outputs the judgment result.

[0025] Furthermore, for example, when inspecting the edges of lines and dots (for example, inspecting the boundary between the object and the background), the presence or absence of defects such as chips and distortions at the edge of the object, the presence or absence of burrs, whether it is straight or curved, the sharpness of corners (whether it is sharp or rounded, etc.), and the overlap of masking (whether it protrudes or is within a specified range, etc.) are inspected.

[0026] As described above, the contents of the examination (specifically, the measurement items for determining whether something is normal or abnormal) may differ depending on the position of the object, that is, the characteristics of the object (more specifically, the characteristics of the object at each position). The dividing unit 120 divides the original image for each content of the examination, that is, for each characteristic of the object, as described above, for example.

[0027] Furthermore, for example, when a hole or the like is provided, the dividing unit 120 generates a divided image including the hole. Furthermore, for example, the dividing unit 120 further divides the divided image including the hole to generate a divided image including only the surface portion of the object and a divided image including only the line (straight line) portion of the object.

[0028] In this way, the dividing section 120 may further divide the divided image.

[0029] The number of divided images generated by the dividing unit 120 is not particularly limited.

[0030] Furthermore, the features of the object may be set arbitrarily. For example, the features of the object may include a printed portion on which characters or the like are printed, such as a surface portion, a line portion, a dot portion, etc. For example, when inspecting a printed portion (e.g., inspecting a portion having a characteristic character shape), the presence or absence of defects such as distortion, blurring, and misalignment of the characters is inspected.

[0031] Furthermore, the divided images generated by the dividing section 120 may have any shape (more specifically, outer shape). The dividing section 120 divides the original image so that each of the divided images has a rectangular shape, for example.

[0032] Furthermore, the sizes of the divided images may be the same or different from one another.

[0033] Furthermore, information indicating the features of the object may be acquired from the user via, for example, the input device 220. For example, the user may use the input device 220 to specify feature regions, such as flat portions and edges, to be used in dividing the original image. Furthermore, for example, the dividing unit 120 may identify features in the original image using a machine-learned learning model (feature identification model) for identifying the features of the object. For example, the dividing unit 120 divides the original image into pieces of a predetermined size, measures features such as brightness of the divided images, and classifies the divided images by feature. The feature identification model is, for example, an inference model trained by machine learning using the original image and training data (so-called annotation information) indicating the features included in the original image as training data. The feature identification model is, for example, a machine learning model using a neural network such as deep learning (e.g., a convolutional neural network (CNN)), but may also be another machine learning model.

[0034] The feature identification model is stored in advance in the storage unit 180, for example.

[0035] It should be noted that the image processing device 100 does not have to divide the original image. In this case, the image processing device 100 does not have to include the dividing unit 120.

[0036] The extraction unit 130 is a processing unit that extracts feature amounts of an original image. Specifically, the extraction unit 130 extracts feature amounts of each of a plurality of divided images generated by dividing the original image. That is, the extraction unit 130 extracts feature amounts based on (included in) the original image. For example, the extraction unit 130 extracts feature amounts of the original image by performing image analysis to measure the brightness of the original image, etc. Also, for example, if the feature amounts of the original image are specified by a user, the extraction unit 130 extracts the feature amounts of the original image by acquiring the feature amounts of the original image accepted by the acceptance unit 160.

[0037] The feature quantity is a numerical value of a predetermined measurement item. Examples of the predetermined measurement items include defect area, defect average brightness, defect brightness standard deviation, non-defective average brightness, non-defective brightness standard deviation, non-defective pattern frequency, brightness variation range, brightness standard deviation variation range, defect brightness variation, defect size range, number of images to be trained by the learning model, number of images to be inspected, ratio of good / bad images (ratio of images containing defects or other abnormalities to images containing defects or other abnormalities), image size, etc. For example, the predetermined measurement items are visually observable parameters such as brightness and size.

[0038] The extraction unit 130 measures the feature amounts of these predetermined measurement items, for example, by performing image analysis on each of the multiple divided images.

[0039] The types and number of predetermined measurement items may be determined arbitrarily. An instruction to specify the predetermined measurement items may be received from the user, or the predetermined measurement items may be determined in advance.

[0040] 2(c) is a diagram showing an image (e.g., a segmented image) showing a part of the surface portion of the original image. In the example shown in FIG. 2(c), the segmented image includes a non-defective portion 400 and a defective portion 410.

[0041] The non-defective part 400 is a part other than the defective part 410, and is, for example, a part that is judged (should be judged) as non-defective (normal) by inspection that appears in an inspection image using a learning model.

[0042] Defective portion 410 is a portion that is not normally included in the object, such as a scratch or stain, and is a portion that is (should be) determined to be defective (abnormal) by inspection of the inspection image using the learning model.

[0043] For example, the extraction unit 130 extracts (measures) from the divided image the feature amount of a predetermined measurement item in the non-defective part 400 and the feature amount of a predetermined measurement item in the defective part 410. The positions of the non-defective part 400 and the defective part 410 are specified by the user, for example.

[0044] 2(a) and 2(b), in a case where the original image does not include a defective portion 410, the user may specify an estimated value of a measurement item such as the luminance of the defective portion 410. For example, the receiving unit 160 receives the estimated value of the measurement item from the user via the input device 220.

[0045] Furthermore, for example, if the condition of the original image is poor and the measurement items cannot be measured from the original image, the user may specify estimated values ​​for the measurement items of both the non-defective portion 400 and the defective portion 410 .

[0046] The generating unit 140 is a processing unit that generates multiple pseudo images using an original image, each having different feature values ​​for a predetermined measurement item. For example, the generating unit 140 generates multiple pseudo images each having different feature values ​​for a predetermined measurement item by changing the feature value for the predetermined measurement item in the original image. Alternatively, the generating unit 140 generates multiple pseudo images by combining a defect image showing a defect and a base material image, which is an image of a portion of the original image that does not contain a defect, with a base material image, each having different feature values ​​for a predetermined measurement item. Furthermore, for example, when the dividing unit 120 generates multiple segmented images, the generating unit 140 generates multiple pseudo images for each of the multiple segmented images. Specific examples of the pseudo image generation process will be described later.

[0047] The output unit 150 is a processing unit that outputs a plurality of pseudo images as training data for machine learning into a learning model that receives an inspection image showing an object as input and outputs an inspection result indicating whether the object shown in the input inspection image is abnormal or not. For example, the output unit 150 outputs the plurality of pseudo images to the learning model. The output unit 150 may output the plurality of pseudo images to the storage unit 180, thereby storing the plurality of pseudo images in the storage unit 180. In this embodiment, the output unit 150 outputs the plurality of pseudo images to the learning unit 170, thereby causing the learning model to perform machine learning using the plurality of pseudo images.

[0048] The learning model is, for example, stored in advance in the storage unit 180. The learning model is, for example, a machine learning model using a neural network such as deep learning, but may be another machine learning model.

[0049] Also, for example, the output unit 150 outputs a plurality of pseudo images to the learning model to allow the learning model to perform machine learning, then outputs a test image to the learning model, and outputs corresponding information indicating the performance of the learning model based on the test results obtained as the output of the learning model.

[0050] The performance here refers to, for example, the accuracy rate at which defects can be correctly extracted when an original image is input into a machine-learned learning model, or the accuracy rate at which the absence of defects can be correctly determined.

[0051] 3 is a diagram showing a performance table according to the embodiment. Note that the performance table (data table) is an example of correspondence information.

[0052] As shown in FIG. 3 , for example, the performance information includes information indicating the correspondence between feature quantities of predetermined measurement items for each of the multiple pseudo images input to the learning model and the performance of the learning model based on the inspection results obtained as the output of the learning model using the input pseudo images. In this embodiment, the performance table includes feature quantities such as defect luminance difference (e.g., the luminance difference between a non-defective portion 400 and a defective portion 410) and background luminance standard deviation, as well as the inspection performance (in this embodiment, accuracy rate) when the feature quantities are detected. Ave is the average accuracy rate. Specifically, Ave is the average accuracy rate of the inspection using the learning model for each of the multiple segmented images of an inspection image of each of the multiple objects. Max is the maximum accuracy rate. Specifically, Max is the average accuracy rate of the highest average accuracy rate for each object among the average accuracy rates for each of the multiple segmented images of an inspection image of each of the multiple objects.

[0053] For example, output unit 150 outputs multiple pseudo images to the learning model (learning unit 170 in this embodiment) to allow the learning model to perform machine learning. Next, output unit 150 outputs an inspection image acquired from imaging device 200 via acquisition unit 110 or an inspection image stored in storage unit 180 to the learning model. Also, for example, extraction unit 130 extracts feature quantities of predetermined measurement items from the inspection image. Next, output unit 150 outputs the abnormality determination result output from the learning model and the inspection image to display device 210, thereby displaying the determination result and the inspection image on display device 210. Furthermore, receiving unit 160 receives the presence or absence of an abnormality in the inspection image from the user via input device 220. This allows the accuracy of the inspection determination result based on the learning model to be determined. By repeating the above process using different inspection images, a correspondence relationship between feature quantities and performance (accuracy rate) is acquired. For example, the output unit 150 creates a performance table such as that shown in FIG. 3 as correspondence information based on these correspondence relationships and outputs it to the display device 210, thereby causing the display device 210 to display the performance table.

[0054] In addition, the inspection image input into the learning model when creating correspondence information may be an image that has not been used in machine learning, a divided image obtained by dividing the image, a pseudo-image generated using the image, an original image used in machine learning, or a pseudo-image generated using the original image.

[0055] Furthermore, the performance table only needs to include the correct answer (performance) for each feature, and the feature corresponding to the accuracy rate may be a combination of two features as shown in Figure 3, or may be a single feature, or may be a combination of three or more features.

[0056] Furthermore, the correspondence information may be any information that indicates the correspondence between the performance of the test and the feature amount, and may be any information such as a table, a graph, or a mathematical formula.

[0057] The receiving unit 160 is a processing unit that receives user operations. The receiving unit 160 receives user operations via, for example, the input device 220. The receiving unit 160 receives, for example, input of position information indicating the position of a defect 410 included in an original image. For example, the user looks at the original image or divided images displayed on the display device 210, and inputs the position of the defect 410 included in the original image or divided images using the input device 220. The receiving unit 160 receives, for example, the input as position information.

[0058] Furthermore, for example, the receiving unit 160 receives information (annotation information) indicating whether each of the pseudo image, defect image, and / or base material image is normal or abnormal.

[0059] The learning unit 170 is a processing unit that causes the learning model to perform machine learning. For example, the learning unit 170 causes the learning model to perform machine learning so as to output a determination result of the presence or absence of an abnormality contained in the image, using as input the multiple pseudo images generated by the generation unit 140 and the annotation information received by the reception unit 160.

[0060] The acquiring unit 110, the dividing unit 120, the extracting unit 130, the generating unit 140, the output unit 150, the receiving unit 160, and the learning unit 170 are realized by, for example, one or more processors.

[0061] The storage unit 180 is a storage device that stores programs executed by processing units such as the acquisition unit 110, division unit 120, extraction unit 130, generation unit 140, output unit 150, reception unit 160, and learning unit 170 to perform each process, information required for the process, and inspection images. The storage unit 180 is realized by, for example, an HDD (Hard Disk Drive) and / or a semiconductor memory.

[0062] The imaging device 200 is a camera that captures an image of an object to generate an original image. The imaging device 200 is realized by, for example, a complementary metal oxide semiconductor (CMOS) image sensor.

[0063] The display device 210 is a display that displays images under the control of the image processing device 100 (more specifically, the output unit 150). The display device 210 displays, for example, an original image, a divided image, a base material image, a defect image, a pseudo image, and corresponding information. The display device 210 is realized by, for example, a display device such as a liquid crystal panel or an organic EL (Electro Luminescence) panel.

[0064] The input device 220 is a user interface that accepts user operations and is realized by a mouse, a keyboard, a touch panel, and / or hardware buttons.

[0065] [Example of pseudo-image generation process] Next, a specific example of the pseudo image generation process will be described.

[0066] FIG. 4 is a diagram for explaining the process of setting the feature amount of the base material image according to the embodiment.

[0067] First, the user operates the input device 220 to, for example, cause the image processing device 100 to start a pseudo image generation process. When the receiving unit 160 receives an instruction to start a pseudo image generation process, the acquiring unit 110 acquires an original image from the imaging device 200, the storage unit 180, or the like. The extracting unit 130 extracts feature quantities of predetermined measurement items from the original image acquired by the acquiring unit 110. When the acquiring unit 110 acquires the original images shown in FIGS. 2A and 2B as images of a normal object, for example, the output unit 150 outputs an image including the predetermined measurement items, the feature quantities of the predetermined measurement items extracted by the extracting unit 130, and the original images shown in FIGS. 2A and 2B (in this example, base material images 300 and 301) as shown in FIG. 4 to the display device 210, thereby displaying the image on the display device 210.

[0068] The information indicating whether the object shown in the original image is normal or abnormal is received by the receiving unit 160 from the user via the input device 220. For example, when the acquiring unit 110 acquires (a), (b), and (c) in FIG. 2 as original images, the output unit 150 selects an original image showing a normal object based on the information indicating whether the object shown in the original image is normal or abnormal.

[0069] The image displayed on the display device 210 also includes a display image for receiving instructions on feature quantities of predetermined measurement items set by the user. In the example shown in Fig. 4, the predetermined measurement items include "roughness," "particles," "aspect ratio," and "variation."

[0070] For example, "roughness" indicates the roughness of the shape of the particles included in the base material images 300 and 301. For example, the larger the value (feature amount) of "roughness" that is set, the more the shape (outline) of the particles reflected in the base material images 300 and 301 changes from circular to rough, like jagged edges.

[0071] Furthermore, for example, "particles" indicates the size of particles contained in the base material images 300 and 301. For example, the larger the value (feature amount) of "particles" that is set, the larger the size of the particles reflected in the base material images 300 and 301 is changed to be.

[0072] Furthermore, for example, the "aspect ratio" indicates the aspect ratio of the particles included in the base material images 300 and 301. For example, the larger the value (feature amount) of the "aspect ratio" that is set, the larger the aspect ratio of the particles shown in the base material images 300 and 301 is changed to be.

[0073] Furthermore, for example, "variation" indicates the degree of variation in the size, shape, etc. of particles included in the base material images 300 and 301. For example, the larger the value (feature amount) of "variation" that is set, the greater the variation in the size, shape, etc. of particles reflected in the base material images 300 and 301 becomes.

[0074] The user operates the input device 220 to input the specified measurement items of the base material images 300 and 301 used in generating the pseudo image, and the feature values ​​(e.g., maximum and minimum values) of the specified measurement items of the base material images.

[0075] FIG. 5 is a diagram for explaining the process of setting the feature amount of a defect image according to the embodiment.

[0076] When the output unit 150 receives input of the predetermined measurement items of the base material images 300 and 301 used in generating the pseudo image and the feature amounts of the predetermined measurement items of the base material images 300 and 301, it causes the display device 210 to display an image for the user to set the feature amounts of the predetermined measurement items in the defect images 310 and 311, as shown in Fig. 5, for example. In the example shown in Fig. 5, the defect images 310 and 311 are indicated by hatched circles.

[0077] Defect images 310 and 311 may be created based on a defect portion 410 in an original image as shown in FIG. 2(c), or may be arbitrarily created schematically by a user or the like. The created defect images 310 and 311 may be stored in storage unit 180. For example, extraction unit 130 extracts feature quantities of predetermined measurement items from defect images 310 and 311. Furthermore, output unit 150 outputs an image including the predetermined measurement items, the feature quantities of the predetermined measurement items extracted by extraction unit 130, and defect images 310 and 311 to display device 210, for example, to display the image.

[0078] The image displayed on the display device 210 also includes a display image for receiving instructions on feature quantities of predetermined measurement items set by the user. In the example shown in Fig. 5, the predetermined measurement items include "density," "size," "aspect ratio," and "variation."

[0079] For example, "intensity" indicates the color intensity of the defect images 310 and 311. For example, the larger the value (feature amount) of "intensity" that is set, the darker the color of the defect images 310 and 311 becomes, for example, the closer to black than to white it is changed.

[0080] Furthermore, for example, "size" indicates the size of the defect images 310 and 311. For example, the larger the value (feature amount) of the set "size" is, the larger the sizes of the defect images 310 and 311 are changed to be.

[0081] Furthermore, for example, the "aspect ratio" indicates the aspect ratio of the defect images 310 and 311. For example, the larger the value (feature amount) of the "aspect ratio" that is set, the larger the aspect ratio of the defect images 310 and 311 is changed to be.

[0082] Furthermore, for example, "variation" indicates the degree of variation in the size, shape, etc. of the defect images 310 and 311. For example, the larger the value (feature amount) of "variation" that is set, the greater the variation in the size, shape, etc. of the defect images 310 and 311 is changed to.

[0083] The user operates the input device 220 to input the specified measurement items of the defect images 310 and 311 used in generating the pseudo image, and the feature quantities (e.g., maximum and minimum values) of the specified measurement items of the defect images 310 and 311.

[0084] The types and number of predetermined measurement items displayed in the images shown in FIGS. 4 and 5 may be arbitrarily determined in advance. For example, the predetermined measurement items may include image brightness (e.g., average brightness), brightness variation, and pattern frequency. The settable ranges of the feature quantities for the predetermined measurement items and the feature quantities set when the images shown in FIGS. 4 and 5 are displayed may be arbitrarily determined in advance, or may reflect the results of extraction of the feature quantities of the original image by the extraction unit 130. For example, if the extraction unit 130 extracts that the particle size in the image shown in FIG. 2(a) is 5 mm and the particle size in the image shown in FIG. 2(b) is 15 mm, the generation unit 140 sets 5 mm as the minimum particle size and determines (provisionally determines) 15 mm as the maximum particle size. The output unit 150 causes the display device 210 to display information indicating the maximum and minimum particle sizes determined by the generation unit 140.

[0085] 6 is a diagram for explaining the annotation information setting process according to the embodiment. Note that the image of layer 0 shown in FIG. 6 is a base material image. Furthermore, the images of layers 1 through 1 shown in FIG. 6 are defect images.

[0086] When the generation unit 140 receives input of specified measurement items of the base material images 300 and 301 and the defect images 310 and 311 used in generating the pseudo images, and feature quantities of the specified measurement items of the base material images 300 and 301 and the defect images 310 and 311, the generation unit 140 generates multiple base material images 302 (e.g., pseudo images of the base material images 300 and 301) and multiple defect images 312 (e.g., pseudo images of the defect images 310 and 311) having different feature quantities of the specified measurement items based on the received feature quantities.

[0087] When the generation unit 140 generates a plurality of base material images 302 and a plurality of defect images 312 having different feature values ​​for a specified measurement item, the output unit 150 causes the display device 210 to display an image that allows the user to set information indicating whether each of the plurality of base material images 302 and the plurality of defect images 312 is to be judged as normal or abnormal.

[0088] For example, even if the base material image 302 has a very dark or light color, it may be considered to be abnormal. Also, even if the defect image 312 has a very dark or light color, or is very small, it may be judged to be normal without any problem depending on its relationship with the base material image 302.

[0089] Therefore, for each of the plurality of base material images 302 and the plurality of defect images 312 generated by the generating unit 140, information (annotation information) as to whether the image is normal (OK) or abnormal (NG) is received.

[0090] FIG. 7 is a diagram showing a specific example of a plurality of pseudo images according to the embodiment.

[0091] When the generation unit 140 receives information (annotation information) indicating whether each of the multiple base material images 302 and multiple defect images 312 generated by the generation unit 140 is normal (OK) or abnormal (NG), the generation unit 140 combines (i.e., synthesizes) the base material images 302 and the defect images 312 to generate multiple pseudo images 320.

[0092] For example, the generation unit 140 generates a pseudo image 320 ("OK base material only" shown in Figure 7) that is a base material image 302 that is considered normal (i.e., set by the user as normal) (i.e., a base material image 302 not combined with a defect image 312), a pseudo image 320 ("OK base material + OK defect" shown in Figure 7) that is considered normal by combining the base material image 302 that is considered normal and the defect image 312 that is considered normal, a pseudo image 320 (NG base material only) that is considered abnormal (i.e., the base material image 302 not combined with the defect image 312), a pseudo image 320 ("OK base material + NG defect" shown in Figure 7) that is considered abnormal by combining the base material image 302 that is considered normal and the defect image 312 that is considered abnormal, and a pseudo image 320 (not shown) that is considered abnormal by combining the base material image 302 that is considered normal and the defect image 312 that is considered abnormal. The output unit 150 outputs, for example, some of the plurality of pseudo images 320 generated by the generation unit 140 to the display device 210, thereby causing the display device 210 to display the image shown in FIG.

[0093] This allows the user to check whether there is a problem with the generated pseudo images 320, and if there is no problem, input an instruction to have the learning model perform machine learning using the input device 220. When the receiving unit 160 receives the instruction, the output unit 150 outputs the pseudo images 320 to the learning model together with annotation information.

[0094] Note that the output section 150 may output all of the pseudo images 320 generated by the generation section 140 to the display device 210, for example, to cause all of the pseudo images 320 to be displayed on the display device 210.

[0095] In addition, the output unit 150 may output multiple pseudo images 320 to the memory unit 180, and store each pseudo image 320 of a type such as ``OK base material only'' and ``OK base material + OK defects'' shown in Figure 7 in separate folders according to type in the memory unit 180.

[0096] As described above, the generating unit 140 generates the pseudo image 320 by, for example, combining the base material image 302 (background) and the defect image 312 (defect). Specifically, the generating unit 140 generates a plurality of pseudo images 320 by combining the defect image 312 and the base material image 302, which have different feature amounts for a predetermined measurement item for at least one of the defect image 312 showing a defect and the base material image 302, which is an image of a portion of the original image that does not include a defect. In this case, the generating unit 140 generates a plurality of base material images 302 and a plurality of defect images 312 having different feature amounts, for example, using the background generating mode shown in FIG. 4 and the defect generating mode shown in FIG. 5.

[0097] The feature quantities of the predetermined measurement items of each of the plurality of pseudo images may be determined based on input from the user as described above, or may be determined by the generation unit 140. For example, the generation unit 140 determines the feature quantities of the predetermined measurement items of each of the plurality of pseudo images based on the feature quantities of the predetermined measurement items in the original image extracted by the extraction unit 130. For example, the generation unit 140 generates the plurality of pseudo images by changing the feature quantities extracted by the extraction unit 130 by a predetermined value. The predetermined value and the number of the predetermined values ​​(i.e., the number of pseudo images) may be determined arbitrarily.

[0098] [Processing Procedure] Next, a processing procedure of the image processing device 100 according to the embodiment will be described.

[0099] FIG. 8 is a flowchart showing the processing procedure of the image processing device 100 according to the embodiment.

[0100] First, the acquisition unit 110 acquires an original image showing an object (S110).

[0101] Next, the generating unit 140 generates a plurality of pseudo images having different feature amounts of predetermined measurement items using the original image acquired by the acquiring unit 110 (S120).

[0102] Next, output unit 150 outputs the multiple pseudo images generated by generation unit 140 as training data for machine learning into a learning model that inputs an inspection image showing an object and outputs an inspection result indicating whether the object shown in the input inspection image is abnormal or not (S130). The object shown in the inspection image is, for example, the same type of object (e.g., the same type of part) as the object shown in the original image, but is a different individual.

[0103] For example, the output unit 150 outputs a plurality of pseudo images to the learning unit 170, thereby causing the learning model to perform machine learning using the plurality of pseudo images. Furthermore, for example, the output unit 150 outputs at least some of the plurality of pseudo images to the display device 210, thereby causing the display device 210 to display at least some of the plurality of pseudo images. For example, the receiving unit 160 receives information (annotation information) indicating whether each pseudo image is normal or abnormal from the user via the input device 220. For example, the output unit 150 associates the annotation information received by the receiving unit 160 with the plurality of pseudo images and outputs the information to the learning unit 170, thereby causing the learning model to perform machine learning.

[0104] After step S110, the dividing unit 120 may divide the original image to generate a plurality of divided images.

[0105] Furthermore, after step S110, the extraction unit 130 may extract the feature amount of each of the original image or the plurality of divided images.

[0106] In step S120, the generating unit 140 may generate a plurality of pseudo images for each of the plurality of divided images. The generating unit 140 may also determine predetermined measurement items and feature amounts of the plurality of pseudo images based on the feature amounts extracted by the extracting unit 130.

[0107] [Effects, etc.] Below, examples of techniques that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from the exemplified techniques will be described.

[0108] Technique 1 is an image processing device 100 that includes an acquisition unit 110 that acquires an original image showing an object, a generation unit 140 that uses the original image to generate a plurality of pseudo images that differ from each other in feature amounts of predetermined measurement items, and an output unit 150 that outputs the plurality of pseudo images as training data for machine learning to a learning model that takes an inspection image showing the object as input and outputs an inspection result indicating whether the object shown in the input inspection image is abnormal or not.

[0109] A large number of images are required for a learning model that uses images as training data to perform machine learning. Therefore, the image processing device 100 uses original images to generate multiple pseudo images with different feature amounts for predetermined measurement items. This allows, for example, a user to generate multiple pseudo images as learning images for machine learning in a learning model without collecting many original images. Therefore, the image processing device 100 can reduce the effort required to collect many original images, making it easy to train the learning model in machine learning.

[0110] Technology 2 is the image processing device 100 described in Technology 1, in which the generation unit 140 generates multiple pseudo images by combining a defect image showing a defect and a base material image, which is an image of a part of the original image that does not contain a defect, with a defect image and a base material image that have different feature values ​​for specified measurement items.

[0111] An example of a case in which the learning model determines an object to be abnormal is when the object contains defects such as scratches or stains. In this case, the defective portion of the original image often has significantly different feature values ​​for a predetermined measurement item from the non-defective portion. Therefore, for example, the generation unit 140 generates defect pseudo images, which are multiple pseudo images of the defect image, and background pseudo images, which are multiple pseudo images of the base material image, by separately modifying the feature values ​​of the defect image and the base material image, and then generates multiple pseudo images by combining these. This allows the learning model to learn defects and non-defective portions with various feature values, thereby improving the inspection performance of the learning model.

[0112] Technology 3 is the image processing device 100 according to Technology 1 or 2, further comprising an extraction unit 130 that extracts feature quantities of predetermined measurement items in the original image, and the generation unit 140 determines feature quantities of predetermined measurement items for each of the plurality of pseudo-images based on the feature quantities of the predetermined measurement items in the original image extracted by the extraction unit 130.

[0113] This allows multiple pseudo images with feature amounts corresponding to the actual object to be generated, making it possible to generate multiple pseudo images suitable for machine learning.

[0114] Technique 4 is the image processing device 100 according to any one of Techniques 1 to 3, in which the output unit 150 outputs a plurality of pseudo images to a learning model (in this embodiment, a learning unit 170 having a learning model).

[0115] This allows for easy machine learning of the learning model.

[0116] Technology 5 is the image processing device 100 described in Technology 4, in which the output unit 150 outputs a plurality of pseudo images to the learning model to allow the learning model to perform machine learning, then outputs a test image to the learning model, and outputs corresponding information indicating the performance of the learning model based on the test results obtained as the output of the learning model.

[0117] This allows the user to easily understand the inspection performance of the learning model by checking the corresponding information, and therefore, for example, the user can further train the learning model using images of feature values ​​for a specified measurement item that are lower than the performance desired by the user, thereby improving the inspection performance of the learning model.

[0118] Technology 6 is the image processing device 100 described in Technology 5, in which the correspondence information includes information indicating the correspondence between the feature values ​​of each of the specified measurement items of the multiple pseudo images input to the learning model and the performance of the learning model based on the test results obtained as the output of the learning model using the input pseudo images.

[0119] This allows, for example, a user to further train the learning model using images of feature quantities of a predetermined measurement item with low performance, thereby improving the inspection performance of the learning model. Furthermore, the correspondence information is used, for example, to predict the correspondence information (e.g., the performance table shown in FIG. 3 ) indicating the relationship between the inspection performance of a second object shown in an inspection image by a learning model when machine learning is performed using learning images showing a second object of a different type from the first object and the feature quantities of the inspection image, and the inspection performance of a first object shown in an object image by a learning model when machine learning is performed using learning images showing the first object, based on the feature quantities of the object image showing the first object. The first object is an example of the above-mentioned object, and is an object that has not been machine-learned by the learning model. The second object is an example of the above-mentioned object, and is an object that has already been machine-learned by the learning model. In other words, for example, based on the correspondence information indicating the inspection performance of an object by a learning model that has already been machine-learned, the inspection performance of an object that has not been machine-learned by the learning model when machine learning is performed by the learning model. For example, in the example shown in Figure 3, if the feature extracted by extraction unit 130 of the target image containing the first object is a defect brightness difference of 95 and a background brightness standard deviation of 45, it is predicted that the performance of the learning model when machine learning is performed using the learning image containing the first object can be appropriately inspected with an average accuracy rate of 96.4%.

[0120] Technique 7 is the image processing device 100 according to any one of techniques 1 to 6, further comprising a division unit 120 that generates a plurality of divided images by dividing the original image according to the characteristics of the object shown in the original image, and the generation unit 140 generates a plurality of pseudo images for each of the plurality of divided images.

[0121] For example, an object shown in an original image may have portions represented by lines and portions represented by surfaces. By segmenting the image into these features, such as lines and surfaces, each segmented image contains one of the simple elements of these features. This improves the performance of inspection using a learning model. Therefore, for example, by generating multiple pseudo images for each feature, the performance of object inspection using a learning model can be further improved.

[0122] Technique 8 is an image processing method that acquires an original image showing an object (S110), uses the original image to generate multiple pseudo images with different feature values ​​for specified measurement items (S120), and outputs the multiple pseudo images as training data for machine learning into a learning model that uses an inspection image showing the object as input and outputs an inspection result indicating whether the object shown in the input inspection image is abnormal or not (S130).

[0123] This provides the same effects as the image processing device 100.

[0124] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0125] (Other embodiments, etc.) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0126] Therefore, the components shown in the accompanying drawings and detailed description may include not only essential components for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are shown in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0127] For example, the image processing device 100 does not have to include the dividing unit 120, the extracting unit 130, the receiving unit 160, and / or the learning unit 170.

[0128] Furthermore, for example, data such as a base material image and a defect image created in the past may be stored in the storage unit 180, and when the receiving unit 160 receives an instruction from a user, the output unit 150 may output the data to the display device 210 for display. Furthermore, these data may be processed (customized) based on an instruction from the user.

[0129] In addition, when the receiving unit 160 receives an instruction to specify a specific area of ​​an image such as an original image, the generating unit 140 may measure the feature amount of the specific area and generate a base material image or a defect image, etc., so that the feature amount matches the measured feature amount.

[0130] Furthermore, the predetermined measurement items and their feature quantities used to generate the base material image and defect image may be specified by the user.

[0131] Furthermore, preset defect images (such as scratches, cracks, stains, and abrasions) may be stored in the storage unit 180 in advance.

[0132] Furthermore, the user may be able to arbitrarily select a defect image from among a plurality of defect images stored in the storage unit 180, for example.

[0133] Furthermore, a defect image created arbitrarily by the user may be stored in the storage unit 180 via the input device 220 or the like.

[0134] Furthermore, the generating section 140 may generate a defect image based on parameters (such as brightness value, size, frequency designation, and aspect ratio) specified by the user.

[0135] The defect image may also be a simple symbol such as a circle or a square.

[0136] Furthermore, a random map generation method used in games and the like may be employed to generate defect images.

[0137] Furthermore, the generating unit 140 may use the original image as the base image as is, or may generate the base image by processing the original image.

[0138] Furthermore, the generating unit 140 may use the defect included in the original image as the defect image as is, or may generate the defect image by processing the defect.

[0139] Furthermore, generating unit 140 may generate a pseudo image that is large in size (for example, larger than the display screen of display device 210). When generating such a pseudo image, for example, receiving unit 160 may receive a designation from the user of the proportion of the pseudo image that will extend beyond the display screen, and generating unit 140 may generate the pseudo image based on the designation.

[0140] Furthermore, the generating unit 140 may arrange the defect image at a position in the base material image designated by the user, or may automatically arrange the defect image at an arbitrary position in the base material image. The position in the base material image where the defect image is arranged may be random, fixed, or determined by an arbitrary function, etc.

[0141] Furthermore, for example, the orientation of the defect image, the number of defect images arranged on the base material image, and the enlargement / reduction of the size of the defect image in any direction may be changed by user specification. In other words, the number and positions of the defect images to be combined with the base material image may be determined arbitrarily. For example, a pseudo image may be generated by combining multiple defect images with the base material image.

[0142] The method of combining the defect image and the background image (blending method) may be simple addition, semi-transparency, or filling. These blending methods may be selected arbitrarily by the user, may be fixed, may be random, or may be used in any combination. The ratio at which each of these blending methods is used may be specified by the user.

[0143] The output unit 150 may also cause the display device 210 to display a list of the defect images generated by the generation unit 140.

[0144] In addition, the output unit 150 may display on the display device 210 one representative defect at a time from the multiple pseudo images generated by the generation unit 140, categorized by feature (e.g., by type of surface portion, etc.) and by defect (e.g., by type of shape of the defect image).

[0145] Furthermore, the ratio of normal pseudo images to abnormal pseudo images may be arbitrary among the multiple pseudo images generated by the generating unit 140. For example, the ratio may be specified by the user.

[0146] Furthermore, the output unit 150 may use the plurality of pseudo images generated by the generation unit 140 as training data to train a learning model, and display the performance (accuracy) of the inspection of the learning model on the display device 210. In other words, the image processing device 100 may have a function to test whether the inspection (determination) of the learning model trained by machine learning is arbitrary.

[0147] Furthermore, for example, in the above embodiment, the image processing device 100 is realized as a single device, but it may be realized by multiple devices. When the image processing device 100 is realized by multiple devices, the components of the image processing device 100 described in the above embodiment may be distributed among the multiple devices in any manner. For example, the image processing device 100 may be realized as a client-server system. In this case, the client device is a mobile terminal that acquires images, accepts user operations, and displays images, and the server device is an information terminal that performs information processing such as generating multiple pseudo images based on an original image.

[0148] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0149] In the above-described embodiments, each component (each processing unit) may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU (Central Processing Unit) or a processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0150] Furthermore, each component may be realized by hardware. Each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0151] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0152] For example, the present disclosure may be realized as an image processing method executed by a computer such as an image processing device. Furthermore, the present disclosure may be realized as a program for causing a computer to execute the image processing method, or as a computer-readable non-transitory recording medium on which such a program is recorded.

[0153] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would conceive, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0154] The present disclosure is useful as an image processing device that can determine the performance of a learning model from an original image. [Explanation of symbols]

[0155] 100 Image processing device 110 Acquisition Department 120 Division 130 Extraction part 140 Generation part 150 Output section 160 Reception 170 Learning Department 180 Storage section 200 Imaging device 210 Display device 220 Input Device 300, 301, 302 Base material images 310, 311, 312 Defect images 320 Pseudo Image 400 Good product department 410 Defective part

Claims

1. an acquisition unit that acquires an original image showing an object; a generating unit that generates a plurality of pseudo images, each of which has different feature amounts of a predetermined measurement item, using the original image; an output unit that outputs the plurality of pseudo images as training data for machine learning into a learning model that receives an inspection image showing the object as input and outputs an inspection result indicating whether the object shown in the input inspection image is abnormal or not, Image processing device.

2. the generating unit generates the plurality of pseudo images by combining a defect image showing a defect and a base material image, which is an image of a portion of the original image that does not include a defect, the defect image and the base material image having different feature amounts of the predetermined measurement item. The image processing device according to claim 1 .

3. further comprising an extraction unit that extracts a feature amount of the predetermined measurement item from the original image; the generation unit determines the feature amount of the predetermined measurement item of each of the plurality of pseudo images based on the feature amount of the predetermined measurement item in the original image extracted by the extraction unit. The image processing device according to claim 1 .

4. The output unit outputs the plurality of pseudo images to the learning model. The image processing device according to claim 1 .

5. the output unit outputs the plurality of pseudo images to the learning model to allow the learning model to perform machine learning, and then outputs the inspection image to the learning model, and outputs correspondence information indicating the performance of the learning model based on the inspection result obtained as an output of the learning model. The image processing device according to claim 4 .

6. The correspondence information includes information indicating a correspondence relationship between the feature amount of the predetermined measurement item of each of the plurality of pseudo images input to the learning model and the performance of the learning model based on the test result obtained as an output of the learning model using the input pseudo images. The image processing device according to claim 5 .

7. a dividing unit that divides the original image into a plurality of divided images by dividing the original image into features of the object shown in the original image, the generation unit generates the plurality of pseudo images for each of the plurality of divided images. The image processing device according to any one of claims 1 to 6.

8. Acquire the original image of the object, generating a plurality of pseudo images each having a different feature value for a predetermined measurement item using the original image; outputting the plurality of pseudo images as training data for machine learning into a learning model that receives an inspection image showing the object as an input and outputs an inspection result indicating whether the object shown in the input inspection image is abnormal; Image processing methods.

Images