Image analysis device, method, and storage medium

Abstract

The present invention provides an image analysis device that improves the accuracy of determination when determining each particle contained in an image of an object. The image analysis device of one embodiment of the present invention comprises a shape determination unit that determines the shape of each particle of a particle image extracted from an image of an object, and obtains an OK particle image of an OK particle that meets a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not meet the standard; a simulation image generation unit that generates a simulation image by overlapping a plurality of particle images including the OK particle image; and a similarity determination unit that determines whether the provisional NG particle image is similar to the simulation image, and when it is determined to be similar to the simulation image, determines that the provisional NG particle includes the OK particle.

Description

Technical Field

[0001] This application claims the benefit of priority from basic application No. 2019-098384 filed in the Japan Patent Office on May 27, 2019, and the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to an image analysis device, method, and program. Background Art

[0003] Conventionally, in the quality inspection of powders used as abrasive materials such as alumina powder and metal powder, images of alumina particles etc. obtained by optical microscope etc. are analyzed to determine whether the particles of alumina particles etc. meet the prescribed standards. Specifically, the image of each particle is extracted from the image of the alumina particles etc. photographed to determine whether the shape of each particle meets the prescribed standards.

[0004] (Prior art literature)

[0005] (Patent Document)

[0006] Patent Document 1: Japanese Patent Application Publication No. 2018-116391 Summary of the invention

[0007] (Problems to be solved by the present invention)

[0008] However, even if the particles actually meet the prescribed criteria, if the particles overlap or aggregate in the image, it will be difficult to distinguish the boundaries between the particles, and the particle group will be judged as not meeting the prescribed criteria.

[0009] Therefore, an object of the present invention is to improve the determination accuracy when determining each particle contained in an image of an object.

[0010] (Methods used to solve problems)

[0011] The present invention includes the following configurations.

[0012] [1] An image analysis device comprising:

[0013] A shape determination unit determines the shape of each particle in the particle image extracted from the image of the object, and obtains an OK particle image of an OK particle that satisfies a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not satisfy the standard;

[0014] a simulation image generating unit that generates a simulation image by overlapping a plurality of particle images including the OK particle image; and

[0015] The similarity determination unit determines whether the tentative NG particle image is similar to the simulation image, and when it is determined to be similar to the simulation image, determines that the tentative NG particles include the OK particles.

[0016] [2] In the image analysis device described in [1], the simulated image is a particle image of a particle group in which multiple particle images including at least one of the OK particle images overlap.

[0017] [3] An image analysis device comprising:

[0018] A shape determination unit determines the shape of each particle in the particle image extracted from the image of the object, and obtains an OK particle image of an OK particle that satisfies a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not satisfy the standard;

[0019] a simulated image generating unit that generates a simulated image using a generation model; and

[0020] The similarity determination unit determines whether the tentative NG particle image is similar to the simulation image, and when it is determined to be similar to the simulation image, determines that the tentative NG particles include the OK particles.

[0021] [4] In the image analysis device described in [3], the particle image used to generate the above-mentioned simulation image includes at least one of the above-mentioned OK particle images.

[0022] [5] In the image analysis device described in [3], the above-mentioned generative model is a generative adversarial network (GAN).

[0023] [6] In the image analysis device described in [5], a plurality of particle images including the OK particle image are used as input data, the provisional NG particle image is used as authentic data, and a simulated image (counterfeit) is generated as the simulated image.

[0024] [7] In the image analysis device described in [5] or [6], the above-mentioned OK particle image is used as input data, and the unclear image is used as authentic data to generate the above-mentioned simulated image.

[0025] [8] In the image analysis device described in [3], the above-mentioned generative model is a variational autoencoder (VAE).

[0026] [9] In the image analysis device described in [8], a plurality of particle images including the OK particle image are used as input data, and the provisional NG particle image is used as data similar to the input data to generate the simulated image.

[0027]

[10] In the image analysis device described in [8], the OK particle image is used as input data, and the unclear image is used as data similar to the input data to generate the simulated image.

[0028]

[11] In the image analysis device described in any one of [1] to

[10] , the similarity judgment unit judges whether the appearance of the provisional NG particles is similar to the appearance of the particles contained in the simulation image, thereby judging whether the provisional NG particle image is similar to the simulation image.

[0029]

[12] In the image analysis device described in any one of [1] to

[11] , the similarity judgment unit judges whether the concentration value of each pixel of the provisional NG particle image is similar to the concentration value of each pixel of the simulation image, thereby judging whether the provisional NG particle image is similar to the simulation image.

[0030]

[13] A method, executed by a computer, comprising the following steps:

[0031] An acquisition step of determining the shape of each particle of the particle image extracted from the image of the object, and acquiring an OK particle image of an OK particle that satisfies a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not satisfy the standard;

[0032] a generating step of generating a simulated image by overlapping a plurality of particle images including the OK particle image; and

[0033] The step of judging whether the tentative NG particle image is similar to the simulation image, and judging whether the tentative NG particle image is similar to the simulation image, and judging that the tentative NG particle includes the OK particle.

[0034]

[14] A program for causing a computer to function as:

[0035] A shape determination unit determines the shape of each particle in the particle image extracted from the image of the object, and obtains an OK particle image of an OK particle that satisfies a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not satisfy the standard;

[0036] a simulation image generating unit that generates a simulation image by overlapping a plurality of particle images including the OK particle image; and

[0037] The similarity determination unit determines whether the tentative NG particle image is similar to the simulation image, and when it is determined to be similar to the simulation image, determines that the tentative NG particles include the OK particles.

[0038] (Effects of the Invention)

[0039] In the present invention, it is possible to improve the accuracy of determination when determining each particle contained in an image of an object. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a diagram showing the configuration of an overall quality inspection system including an image analysis device according to one embodiment of the present invention.

[0041] Figure 2 This is a diagram showing the hardware configuration of an image analysis device according to one embodiment of the present invention.

[0042] Figure 3 This is a diagram showing functional blocks of an image analysis device according to one embodiment of the present invention.

[0043] Figure 4 This is a diagram for explaining an overview of image analysis processing according to one embodiment of the present invention.

[0044] Figure 5 It is a diagram for explaining particle image extraction and shape determination according to one embodiment of the present invention.

[0045] Figure 6 This is a diagram for explaining simulation image generation according to one embodiment of the present invention.

[0046] Figure 7 This is a flowchart showing the flow of image analysis processing according to one embodiment of the present invention.

[0047] Figure 8 This is a diagram for explaining machine learning according to one embodiment of the present invention.

[0048] Fig. 9 This is a diagram for explaining machine learning according to one embodiment of the present invention.

[0049] Fig.10 This is a diagram for explaining machine learning according to one embodiment of the present invention.

[0050] Fig.11 This is a diagram for explaining machine learning according to one embodiment of the present invention. DETAILED DESCRIPTION

[0051] Hereinafter, each embodiment will be described with reference to the accompanying drawings. It should be noted that in this specification and the accompanying drawings, for components having substantially the same functional configuration, the same reference numerals are given to omit repeated descriptions.

[0052] It should be noted that, although the present specification describes the case of determining whether alumina powder or the like meets a predetermined standard, the present invention can be applied to the case of determining whether a product such as a powder containing particles of any substance meets a predetermined standard. In addition, although the present specification describes the case of an image obtained by an optical microscope, the present invention can be applied to particle images obtained by any device including a scanning electron microscope (SEM).

[0053] The following is a description of an implementation method for generating a simulated image based on a predetermined generation rule (described in detail later) (hereinafter referred to as <Implementation Method 1>), and an implementation method for generating a simulated image through machine learning (hereinafter referred to as <Implementation Method 2>).

[0054] <Implementation Method 1>

[0055] <System Configuration>

[0056] Figure 1 1 is a diagram showing the overall system configuration including an image analysis device 102 according to an embodiment of the present invention. The image analysis device 102 can be used in a system for quality inspection (quality inspection system 100). The quality inspection system 100 can include an optical microscope 101, an image analysis device 102, and a user terminal 103. The image analysis device 102 obtains an image photographed by the optical microscope from the optical microscope 101 connected to the image analysis device 102. In addition, the image analysis device 102 sends and receives data with the user terminal 103 via an arbitrary network 104. The sending and receiving of data can be performed by a storage medium such as a semiconductor memory described later. The following will be described separately.

[0057] The optical microscope 101 is used to photograph an object (for example, aluminum oxide particles contained in aluminum oxide powder, etc.). The optical microscope 101 may include a photographing device such as a digital camera and a storage device for storing an image of the object photographed. In addition, the optical microscope 101 sends the image of the object obtained by photography to an image analysis device 102 connected to the optical microscope 101. The microscope of the optical microscope 101 may be a reflection microscope or a transmission microscope. In addition, the optical microscope 101 may include an ultra-high pressure mercury lamp, a xenon lamp, LEDs of various colors including three primary colors, ultraviolet LEDs, lasers, etc., and as an observation method of the image, a bright field observation method, a dark field observation method, a phase difference observation method, a differential interference observation method, a polarized light observation method, a fluorescence observation method, etc. can be used.

[0058] The image analysis device 102 is a device for determining whether aluminum oxide powder, etc., meets a predetermined standard. The image analysis device 102 is composed of, for example, one or more computers. Specifically, the image analysis device 102 analyzes the image of the object (for example, a plurality of particles of aluminum oxide powder, etc.) sent from the optical microscope 101 to determine whether the aluminum oxide powder, etc., meets a predetermined standard. In addition, the image analysis device 102 sends the data of the result of the quality inspection to the user terminal 103. In the latter part, refer to Figure 2 , Figure 3 , the image analysis device 102 is described in detail.

[0059] The user terminal 103 is a terminal used by a person who performs quality inspection. Specifically, the user terminal 103 sends data of a predetermined standard that should be satisfied by, for example, alumina powder, to the image analysis device 102. In addition, the user terminal 103 receives data of the result of the quality inspection from the image analysis device 102 and displays it on the user terminal 103 or on a display device (not shown) connected to the user terminal 103. The user terminal 103 is, for example, a computer such as a personal computer.

[0060] It should be noted that, although the image analysis device 102 and the user terminal 103 are described as separate computers in this specification, the image analysis device 102 and the user terminal 103 may be installed in one computer. In addition, the image analysis device 102 may have a part of the functions of the user terminal 103.

[0061] <Hardware Configuration of Image Analysis Device 102>

[0062] Figure 21 is a diagram showing an example of the hardware configuration of an image analysis device 102 according to an embodiment of the present invention. The image analysis device 102 includes a CPU (Central Processing Unit) 1, a ROM (Read Only Memory) 2, and a RAM (Random Access Memory) 3. The CPU 1, ROM 2, and RAM 3 form a so-called computer.

[0063] In addition, the image analysis device 102 may also include a GPU (Graphics Processing Unit) 4, an auxiliary storage device 5, an I / F (Interface) device 6, and a drive device 7. It should be noted that the hardware of the image analysis device 102 is connected to each other via a bus 8.

[0064] The CPU 1 is a computing device for executing various programs installed in the auxiliary storage device 5 .

[0065] ROM2 is a nonvolatile memory. ROM2 functions as a main storage device for storing various programs and data necessary for CPU1 to execute various programs installed in auxiliary storage device 5. Specifically, ROM2 functions as a main storage device for storing startup programs such as BIOS (Basic Input / Output System) and EFI (Extensible Firmware Interface).

[0066] The RAM 3 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or a SRAM (Static Random Access Memory), and functions as a main storage device that provides a work area for various programs installed in the auxiliary storage device 5 to be developed when the CPU 1 executes them.

[0067] GPU4 is a computing device dedicated to image processing.

[0068] The auxiliary storage device 5 is an auxiliary storage device for storing various programs and information used when the various programs are executed.

[0069] The I / F device 6 is a communication device for communicating with the optical microscope 101 and the user terminal 103 .

[0070] The drive device 7 is a device for placing the storage medium 9. The storage medium 9 mentioned here includes a medium that records information optically, electrically or magnetically, such as a CD-ROM, a floppy disk, a magneto-optical disk, etc. In addition, the storage medium 9 may include a semiconductor memory that records information electrically, such as a ROM, a flash memory, etc.

[0071] It should be noted that the various programs installed in the auxiliary storage device 5 are installed in the drive device 7 through, for example, a distributed storage medium 9, and the various programs recorded in the storage medium 9 are read and installed by the drive device 7. Alternatively, the various programs installed in the auxiliary storage device 5 can be downloaded from another network different from the network 104 through the I / F device 6 and installed.

[0072] <Functional Blocks of Image Analysis Device 102>

[0073] Figure 3 1 is a diagram showing functional blocks of an image analysis device 102 according to an embodiment of the present invention. The image analysis device 102 may include an object image acquisition unit 301, a particle image extraction unit 302, a shape determination unit 303, a simulated image generation unit 304, a similarity determination unit 305, and a pass / fail determination unit 306. In addition, the image analysis device 102 may function as the object image acquisition unit 301, the particle image extraction unit 302, the shape determination unit 303, the simulated image generation unit 304, the similarity determination unit 305, and the pass / fail determination unit 306 by executing a program. Each of these units will be described below.

[0074] The object image acquisition unit 301 acquires, from the optical microscope 101, an image of the object photographed by the optical microscope 101. In addition, the object image acquisition unit 301 stores the acquired image of the object in a storage device in a manner that the particle image extraction unit 302 can refer to it.

[0075] The particle image extraction unit 302, the shape determination unit 303, the simulation image generation unit 304, and the similarity determination unit 305 perform processing 3 for determining whether, for example, aluminum oxide powder satisfies a predetermined standard. Figure 4 , an overview of image analysis processing according to one embodiment of the present invention is described.

[0076] Figure 4 FIG. 1 is a diagram for explaining an overview of image analysis processing according to an embodiment of the present invention. Figure 4 The image shown in the upper part is an image of the object acquired by the object image acquisition unit 301 from the optical microscope 101. The image of the object includes images of a plurality of particles.

[0077] In step 401 (S401), the particle image extraction unit 302 extracts one or more overlapping particle images from the image of the object. And the shape determination unit 303 determines whether the shape of the particles contained in the extracted particle image meets the prescribed criterion. Hereinafter, the particle image of the particles that meet the prescribed criterion (hereinafter also referred to as OK particles) is referred to as an OK image (hereinafter also referred to as an OK particle image), and the particle image of the particles that do not meet the prescribed criterion (hereinafter also referred to as provisional NG particles) is referred to as a provisional NG image (hereinafter also referred to as a provisional NG image) (hereinafter also referred to as a provisional NG particle image). That is, the shape of each particle of the particle image extracted from the image of the object is determined to obtain an OK image and a provisional NG image.

[0078] Here, in the image of the object, there is sometimes an image with unclear outlines of particles (also called an edge-missing image). For this unclear outline image, when the same image can be reproduced by, for example, removing a part of the outline of the clear outline image, it can be processed in the same way as the clear outline image. That is, even if it is an image with unclear outlines of particles, for an image that can reproduce the same image as the clear outline image, it can be determined whether it meets the above-mentioned prescribed criteria and processed as an OK image or a provisional NG image.

[0079] In step 402 ( S402 ), the simulation image generation unit 304 generates a superimposed image (hereinafter referred to as a “simulation image”) by superimposing a plurality of particle images including the OK image obtained in S401 .

[0080] The simulation image generation unit 304 may generate a simulation image by overlapping a plurality of particle images including at least one OK image obtained in S401. In addition, the particle images used for generating the simulation image may include a plurality of OK images or a provisional NG image. As the provisional NG image used for generating the simulation image, a provisional NG image extracted from the object image may be used, or a provisional NG image extracted from other object images related to the same powder (for example, an image of an aluminum oxide powder different from the aluminum oxide powder being quality inspected, or other images of the aluminum oxide powder being quality inspected) may be used. Furthermore, a provisional NG image that is formally determined to be an NG image as a result of performing the analog judgment described later on other object images related to the same powder may be used.

[0081] In step 403 (S403), the similarity determination unit 305 determines whether the provisional NG image extracted in S401 is similar to the simulated image generated in S402. Furthermore, the similarity determination unit 305 determines that the provisional NG particles include OK particles when the two are similar (that is, it determines that particles that meet the prescribed criteria overlap, agglomerate and are included therein), and when the two are not similar, the provisional NG image is formally regarded as an NG image. That is, it is determined that the provisional NG particles of the provisional NG image determined to be similar to the simulated image include the same number of OK particles as the number of OK images used to generate the similar simulated image.

[0082] return Figure 3 As described above, the particle image extraction unit 302 extracts the image of the particle from the image of the object acquired by the object image acquisition unit 301. In addition, the particle image extraction unit 302 stores the extracted particle image in the storage device in a manner that the shape determination unit 303 can refer to it.

[0083] The shape determination unit 303 determines whether the shape of the particle contained in the image of the particle extracted by the particle image extraction unit 302 satisfies a predetermined standard. For example, the shape determination unit 303 may determine that the image including the particle is an OK image when the roundness of the particle is above a threshold value. Here, roundness refers to "the geometric deviation of a circular body from a correct circle" as defined in JIS B0621-1984 "Definition and Representation of Geometric Deviations". In addition, for example, when the roundness of the particle is lower than the threshold value, the shape determination unit 303 determines that the image including the particle is a provisional NG image. It should be noted that the standard that the shape of the particle should meet is not limited to roundness, and may include the case where the shape of the particle is an ellipse, etc., and a standard composed of feature quantities such as the area of ​​the particle, the length of the major diameter, minor diameter, equivalent circle diameter, directional wire diameter, or the circumference of the particle may be used.

[0084] Figure 5 It is a diagram for explaining particle image extraction and shape determination according to one embodiment of the present invention.

[0085] In step 501 (S501), the object image acquisition unit 301 acquires the object image from the optical microscope 101. Figure 5 An image of the object as shown (an image serving as a basis for image analysis).

[0086] In step 502 (S502), the particle image extraction unit 302 extracts the particle image. For example, the particle image extraction unit 302 binarizes the original image of S501 to generate Figure 5 The mask image shown in Figure 5In the figure, the region of the particle is shown in white, and the region other than the particle is shown in black). Furthermore, the particle image extraction unit 302 extracts the image of the particle without other particles around it from the original image of S501 based on the mask image (also referred to as cutting or clipping).

[0087] In step 503 ( S503 ), the shape determination unit 303 determines whether the particles included in the image of the particles extracted in S502 satisfy a predetermined criterion.

[0088] return Figure 3 The simulation image generation unit 304 generates a superimposed image (ie, a simulation image) by superimposing a plurality of particle images.

[0089] Here, the simulated image is explained. The simulated image is generated by overlapping two or more images including particles. That is, the simulated image can be generated using images of two particles, or it can be generated using images of three or more particles. In addition, the simulated image can be generated using at least one OK image. That is, the simulated image can be generated using only the OK image, or it can be generated using the OK image and an image other than the OK image (for example, a tentative NG image).

[0090] As the provisional NG image used for generating the simulation image, a provisional NG image extracted from the object image or a provisional NG image extracted from another object image of the same powder may be used. Furthermore, a provisional NG image formally regarded as an NG image as a result of analog judgment described later on another object image of the same powder may be used.

[0091] Below, refer to Figure 6 , the simulation image generation according to one embodiment of the present invention is described.

[0092] Figure 6 This is a diagram for explaining simulation image generation according to one embodiment of the present invention.

[0093] First, the simulated image generation unit 304 randomly selects images of particles for generating simulated images. Specifically, the simulated image generation unit 304 may select images of two or more particles including at least one OK image from among OK images and images other than OK images (eg, provisional NG images).

[0094] In step 601 (S601), the simulated image generation unit 304 performs position correction on the images of the two or more selected particles. Specifically, the simulated image generation unit 304 determines the positional relationship of the particle images in such a way that the particles in the particle images are in contact with each other. The simulated image generation unit 304 can generate various simulated images by adjusting the locations where the particles in the particle images are in contact with each other.

[0095] In step 602 (S602), the simulated image generation unit 304 performs a process of superimposing the images of the particles whose positions have been corrected in S601. The simulated image generation unit 304 can generate various simulated images by adjusting the degree of superposition.

[0096] In step 603 (S603), the simulated image generation unit 304 rotates the group of particle images superimposed in S602. The simulated image generation unit 304 can generate various simulated images by adjusting the degree of rotation of the group of particle images superimposed.

[0097] In addition to the adjustment of the above parameters, the simulated image generation unit 304 can generate various simulated images by adjusting various parameters. The parameters can be, for example, the number of particles, or the contact points of particles in the particle image, the degree of overlap, the degree of rotation of the overlapping particle image group, or any combination of the roundness, color, transparency, blur degree, etc. of the particles in the image. In addition, the simulated image generation unit 304 can generate a simulated image by using a method (also known as the Tetris (registered trademark) method) in which particles are dropped and accumulated from one direction within a frame.

[0098] return Figure 3 The similarity determination unit 305 determines whether the provisional NG image determined by the shape determination unit 303 is similar to the simulated image generated by the simulated image generation unit 304. In addition, the similarity determination unit 305 determines that the provisional NG particles contained in the provisional NG image include OK particles (that is, determines that multiple particles including OK particles overlap or agglomerate) when the two are similar, and formally determines the provisional NG image as an image of NG particles when the two are not similar. The following describes an example of the similarity determination between the two.

[0099] <Judgment based on similarity in appearance>

[0100] The similarity determination unit 305 compares the outer shape (contour) of the particles contained in the overlapping particle images contained in the simulation image with the outer shape (contour) of the particles contained in the provisional NG image. For example, if the difference between the outer shape of the particles contained in the overlapping particle images contained in the simulation image and the outer shape of the particles contained in the provisional NG image is less than a threshold value, the similarity determination unit 305 determines that the two are similar. In addition, for example, if the difference between the outer shape of the particles contained in the overlapping particle images contained in the simulation image and the outer shape of the particles contained in the provisional NG image is greater than a threshold value, the similarity determination unit 305 determines that the two are not similar.

[0101] <Judgment based on similarity>

[0102] In addition to the above-mentioned <judgment of similarity based on the outer shape>, or in place of the above-mentioned <judgment of similarity based on the outer shape>, the similarity judgment unit 305 may perform similarity judgment based on the shade (density) of each pixel of the image. Specifically, the similarity judgment unit 305 compares the density value of each pixel of the simulated image with the density value of each pixel of the provisional NG image. For example, the similarity judgment unit 305 judges that the two are similar when the difference between the density value of each pixel of the simulated image and the density value of each pixel of the provisional NG image is less than a threshold value. In addition, for example, the similarity judgment unit 305 judges that the two are not similar when the difference between the density value of each pixel of the simulated image and the density value of each pixel of the provisional NG image is greater than a threshold value. In the case of performing analogical judgment based on pixels, it is preferred that either particle image of the simulated image or the provisional NG image to be compared has 32×32 pixels or more, and more preferably has 64×64 pixels or more.

[0103] The pass / fail judgment unit 306 judges whether the object, i.e., the powder including particles, is pass / fail (i.e., the result of the quality inspection). Specifically, the pass / fail judgment unit 306 measures the number of OK particles judged as OK images by the shape judgment unit 303 and the number of OK particles contained in the provisional NG images judged to be similar to the simulated images by the similarity judgment unit 305, and makes judgment using the obtained measured values.

[0104] When judging whether a particle is qualified, the number of images of particles extracted from the image of the object is preferably 100 or more, more preferably 500 or more, and further preferably 1,000 or more. After repeatedly accumulating a series of steps from photographing the image of the object to analog judgment, a judgment on whether the particle is qualified can be made.

[0105] Two notification examples are described below.

[0106] <Notification of numerical values>

[0107] The pass / fail judgment unit 306 notifies the user terminal 103 of the percentage of the total value 1 and the total value 2 relative to the total value of the number of OK particles determined as OK images by the shape judgment unit 303 (number 1), the total value of the number of OK particles contained in the provisional NG image determined as similar to the simulated image by the similarity judgment unit 305 (number 2), and the total value of the number of NG particles officially determined as NG images by the similarity judgment unit 305 (number 3). In addition, if any value is selected as the radius of the particle from the example of the prescribed criterion for the above-mentioned shape judgment, the volume of the particle can be obtained as the cumulative distribution of the volume of the OK particles determined as OK images by the shape judgment unit 303 and the OK particles contained in the provisional NG image determined as similar to the simulated image by the similarity judgment unit 305, and the 50% cumulative volume particle diameter (D 50 ).

[0108] <Notification of whether or not to pass the test>

[0109] The conformity determination unit 306 determines that the powder etc. including the object is conformed when the percentage of the total value of the above-mentioned number of number 1 and the above-mentioned number of number 2 relative to the total value of the number of OK particles determined as OK images by the shape determination unit 303 (number of number 1), the total value of the number of OK particles contained in the provisional NG image determined as similar to the simulated image by the similarity determination unit 305 (number of number 2), and the total value of the number of NG particles formally determined as NG images by the similarity determination unit 305 (number of number 3) is greater than a prescribed value. At this time, the conformity determination unit 306 notifies the user terminal 103 that the powder etc. including the object is conformed. The prescribed value recognized as conformed is preferably 95%, more preferably 97%, and further preferably 99%.

[0110] Figure 7 This is a flowchart showing the flow of image analysis processing according to one embodiment of the present invention.

[0111] In step 700 (S700), the particle image extraction unit 302 extracts the image of the particle and creates the image of the extracted particle. In the extraction of the particle image, the above-mentioned processing such as cutting can be performed.

[0112] In step 701 ( S701 ), the shape determination unit 303 sets a criterion (eg, a threshold value) that the particles included in the particle image should satisfy. For example, the shape determination unit 303 may set a criterion specified by the user terminal 103 .

[0113] In step 702 (S702), the shape determination unit 303 determines whether the image of the particle extracted in S700 satisfies the criterion set in S701. If it is determined that the image does not satisfy the prescribed criterion (that is, it is a provisional NG image), the process proceeds to step 703, and if it is determined that the image satisfies the prescribed criterion (that is, it is an OK image), the process proceeds to step 707. In the shape determination unit 303, if there is an image with unclear outline as a particle, the image with unclear outline can be further determined.

[0114] In step 707 (S707), the shape determination unit 303 notifies the pass / fail determination unit 306 of the number of OK images.

[0115] In step 703 (S703), the simulation image generation unit 304 generates a simulation image.

[0116] It should be noted that the simulation image generating unit 304 may generate a simulation image each time a quality inspection is performed, or may use an already generated simulation image (that is, a simulation image generated using an OK image of the same material as the object to be quality inspected).

[0117] In step 704 (S704), the similarity determination unit 305 determines whether the provisional NG image determined in S702 is similar to the simulated image in S703. If not similar, the process proceeds to step 705, and if similar, the process proceeds to step 708.

[0118] In step 708 (S708), the similarity determination unit 305 notifies the pass / fail determination unit 306 of the number of OK particles included in the provisional NG image determined to be similar to the simulation image.

[0119] In step 705 (S705), the similarity determination unit 305 determines the provisional NG image as a formal NG image, and notifies the pass / fail determination unit 306 of the number of NG particles formally determined as NG images.

[0120] In step 706 (S706), the qualification determination unit 306 measures the number of OK particles in the OK image determined in S702, the number of OK particles contained in the provisional NG image determined in S708 to be similar to the simulated image (that is, the number of OK particles contained in the simulated image determined to be similar in S704), and the number of NG particles formally determined to be the NG image in S705.

[0121] Thus, in the present invention, the provisional NG particles in the provisional NG image that is temporarily determined as the provisional NG image and that are determined to be similar to the simulated image generated using the OK image are determined to include one or more OK particles. Therefore, the image of the particle group in which only the particles that meet the prescribed criteria overlap or agglomerate in the image of the particles that are determined to not meet the prescribed criteria in the previous image analysis can be processed as an OK particle image.

[0122] <Implementation Method 2>

[0123] Below, refer to Figures 8 to 11 , an implementation method for generating a simulated image close to a real object by machine learning is described. It should be noted that the description is mainly based on the points different from implementation method 1.

[0124] In <Implementation 2>, a generative model is used to generate a simulated image. A generative model is a method of learning learning data to generate new data similar to the data, and is a model that learns in a manner that the distribution of the learning data used for learning is consistent with the distribution of the generated data. As examples of generative models, for example, two types of generative adversarial networks (GAN) and variational autoencoders (VAE) can be cited. If a generative model is used to generate a simulated image, a simulated image close to the real object can be generated.

[0125] As a generation model, the case of using a generative adversarial network (GAN) is described with reference to the accompanying drawings.

[0126] first, Figure 8 as well as Fig. 9 This is a diagram of a GAN for illustrating one embodiment of the present invention.

[0127] exist Figure 8 In the generator, the input image category information (C(class)) and the input noise (Z(noise), such as a random number) are input into the generator network to generate a simulated image (fake) (X fake). In the discriminator network (Discriminator), the simulated image (fake) is compared with the real data (X real(data)) to identify the authenticity (real or fake) and category of the simulated image (fake). Fig. 9In the generator network, the process of deconvolution ("deconv") is shown to generate a simulated image (forgery) by upsampling the feature quantity. 100 in the figure is an example of the feature quantity, which is equivalent to Figure 8 The sum of C(class) and Z(noise) in the GAN structure.

[0128] In addition, the feature quantity of the object image can be obtained by machine learning and combined with GAN. Fig.10 As shown, machine learning is performed using a network composed of a front-stage network whose input layer is an image of an object (that is, input data including OK images and provisional NG images) and whose output layer is a feature quantity, and a rear-stage network whose input layer is a feature quantity output by the front-stage network and whose output layer is a simulated image generated by implementation mode 1, to extract feature quantities. Next, as Fig.11 As shown in the figure, GAN is used to generate simulated images (forgeries) by taking the tentative NG images of the judgment object as the authentic data. Specifically, the generator network (Generator) adjusts the feature quantity (in Fig.10 Here, the generation network (Generator) uses the same network as the later-stage network that inputs feature quantities and outputs simulated images. In addition, the recognition network (Disciminator) compares the generated simulated images (counterfeit) with the authentic data (authentic) (that is, the provisional NG images used as authentic data) to identify whether it is authentic or counterfeit. The generation network learns in a way that wants to deceive the recognition network, and the recognition network learns in a way that wants to identify more accurately, so that as learning proceeds, the generation network generates simulated images (counterfeit) that are closer to the authentic data.

[0129] When GAN is used as a generative model, multiple extracted particle images including OK images are used as input data to extract the above-mentioned feature quantity. More specifically, multiple extracted particle images including OK images and excluding provisional NG images of analog judgment objects are used as input data to extract the above-mentioned feature quantity, and the number of OK images used for generating simulated images (forgeries) is obtained as additional information. In addition, the generative network uses the provisional NG images of analog judgment objects as authentic data to generate simulated images (forgeries). As a result of GAN learning, simulated images (forgeries) that are judged to be similar to provisional NG images are generated. For provisional NG images of analog judgment objects, the number of OK particles contained in the provisional NG images can be obtained by referring to the additional information of the generated simulated images (forgeries).

[0130] In the image of the object, there are sometimes images with unclear outlines due to particles (also called edge loss images) and unclear images due to noise in pixels (also called noisy images). For such unclear images, when GAN is used as a generative model, more specifically, a simulated image can be generated by extracting feature quantities using an OK image as input data and using the unclear image of the object as authentic data. The obtained unclear simulated image can be treated as an OK image.

[0131] The case of using a variational autoencoder (VAE) is described. It is the same as GAN. VAE learns the features of the representation data and generates output data similar to the input data. In VAE, the mean vector and the dispersion vector are obtained by the encoder on the input layer side, and the latent variables are randomly extracted based on these. The latent variables extracted by the decoder on the output layer side are used to generate output data similar to the input data, which reproduces the original data. In VAE, the latent variables are adjusted in a way that the feature amount of the input data is maintained as much as possible. As a result of learning in this way, a particle image similar to the provisional NG image of the analog judgment object is generated.

[0132] When VAE is used as a generative model, multiple particle images including OK images are input, and overlapping images are used as input data on the input layer side, and a simulated image is generated using a provisional NG image as data similar to the input data. More specifically, extracted particle images including OK images and excluding a provisional NG image of an analog judgment object are input, and overlapping images are used as input data on the input layer side, and an average vector and a dispersion vector are obtained. Latent variables are randomly extracted based on these, and multiple simulated images similar to the provisional NG image of the analog judgment object are generated using the extracted latent variables, and the number of OK images used for the generation of the simulated images is obtained as additional information. For a simulated image that is judged to be similar to the provisional NG image of the analog judgment object, the number of OK particles contained in the provisional NG image can be obtained by referring to the additional information of the generated simulated image.

[0133] When VAE is used as a generative model, for the above-mentioned unsharp image, a simulated image can be generated by using an OK image as input data and an unsharp image as output data similar to the input data. A simulated image that is judged to be similar to the unsharp image of the object can be processed as an OK image.

[0134] Thus, in the <Implementation 2>, the image of the particle group expressed in the simulated image has the particle overlap and agglomeration methods close to the real thing, and the unnaturalness peculiar to the synthesis as performed in the above <Implementation 1> disappears. Therefore, it is possible to generate a simulated image like an image of an actual particle group (that is, an actual particle group that overlaps and agglomerates particles that meet the prescribed criteria).

[0135] It should be noted that the present invention is not limited to the configurations listed in the above embodiments, etc., in combination with other elements, etc. The present invention can be modified within the scope of the present invention and can be appropriately determined according to its application.

[0136] Description of Reference Numerals

[0137] 100% quality inspection system

[0138] 101 Optical Microscope

[0139] 102 Image analysis device

[0140] 103 User Terminal

[0141] 104 Network

[0142] 301 object image acquisition unit

[0143] 302 Particle image extraction unit

[0144] 303 shape determination unit

[0145] 304 simulation image generation unit

[0146] 305 Similarity Judgment Section

[0147] 306 Qualification Assessment Unit

Claims

1. An image analysis device, comprising: A shape determination unit determines the shape of each particle in the particle image extracted from the image of the object, and obtains an OK particle image of an OK particle that satisfies a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not satisfy the standard; a simulation image generating unit that generates a simulation image by overlapping a plurality of particle images including the OK particle image; and The similarity determination unit determines whether the tentative NG particle image is similar to the simulation image, and when it is determined to be similar to the simulation image, determines that the tentative NG particles include the OK particles.

2. The image analysis device according to claim 1, wherein: The simulation image is a particle image of a particle group in which a plurality of particle images including at least one of the OK particle images overlap.

3. An image analysis device, comprising: A shape determination unit determines the shape of each particle in the particle image extracted from the image of the object, and obtains an OK particle image of an OK particle that satisfies a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not satisfy the standard; a simulated image generating unit that generates a simulated image using the generation model, the simulated image being a simulated image of an image of an actual existing particle group of particles overlapping and agglomerated to meet a predetermined standard; and The similarity determination unit determines whether the tentative NG particle image is similar to the simulation image, and when it is determined to be similar to the simulation image, determines that the tentative NG particles include the OK particles.

4. The image analysis device according to claim 3, wherein: The particle images used to generate the above-mentioned simulation image include at least one of the above-mentioned OK particle images.

5. The image analysis device according to claim 3, wherein: The above-mentioned generative model is Generative Adversarial Networks (GAN).

6. The image analysis device according to claim 5, wherein: A plurality of particle images including the OK particle image are used as input data, the provisional NG particle image is used as authentic data, and a simulated image is generated as the simulated image.

7. The image analysis device according to claim 5, wherein: The above-mentioned OK particle image is used as input data and the unclear image is used as authentic data to generate the above-mentioned simulated image.

8. The image analysis device according to claim 3, wherein: The above generative model is a Variational AutoEncoder (VAE).

9. The image analysis device according to claim 8, wherein: The simulation image is generated by using a plurality of particle images including the OK particle image as input data and the provisional NG particle image as data similar to the input data.

10. The image analysis device according to claim 8, wherein: The above-mentioned OK particle image is used as input data, and the unclear image is used as data similar to the above-mentioned input data to generate the above-mentioned simulation image.

11. The image analysis device according to any one of claims 1 to 10, wherein: The similarity determination unit determines whether the outer shape of the provisional NG particle is similar to the outer shape of the particle included in the simulation image, thereby determining whether the provisional NG particle image is similar to the simulation image.

12. The image analysis device according to any one of claims 1 to 10, wherein: The similarity determination unit determines whether the density value of each pixel of the provisional NG particle image is similar to the density value of each pixel of the simulation image, thereby determining whether the provisional NG particle image is similar to the simulation image.

13. A method, which is executed by a computer, comprising the following steps: An acquisition step of determining the shape of each particle of the particle image extracted from the image of the object, and acquiring an OK particle image of an OK particle that satisfies a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not satisfy the standard; A generating step, which generates a simulated image by overlapping a plurality of particle images including the OK particle image; as well as The step of judging whether the tentative NG particle image is similar to the simulation image, and judging whether the tentative NG particle image is similar to the simulation image, and judging that the tentative NG particle includes the OK particle.

14. A storage medium storing a program for causing a computer to function as: A shape determination unit determines the shape of each particle in the particle image extracted from the image of the object, and obtains an OK particle image of an OK particle that satisfies a standard regarding a predetermined shape, and a provisional NG particle image of a provisional NG particle that does not satisfy the standard; a simulation image generating unit that generates a simulation image by overlapping a plurality of particle images including the OK particle image; and The similarity determination unit determines whether the tentative NG particle image is similar to the simulation image, and when it is determined to be similar to the simulation image, determines that the tentative NG particles include the OK particles.

Images