COMPUTER PROGRAM, LEARNING MODEL GENERATION DEVICE, DISPLAY DEVICE, PARTICLE IDENTIFICATION DEVICE, LEARNING MODEL GENERATION METHOD, DISPLAY METHOD, AND PARTICLE IDENTIFICATION METHOD

Abstract

To provide a computer program, a learning model generator, a display device, a particle identification device, a learning model generation method, a display method and a particle identification method which enable detection of an object, regardless of intensity change, complexity, and irregularity of the object.SOLUTION: An image identification device having a function as a learning model generator, a display device and a particle identification device has a control unit configured by a CPU, a ROM, a RAM, etc. The control unit acquires a first image in which an identification object is captured, acquires a second image which identifies the identification object by at least a binary label corresponding to a pixel of the first image, and generates a learning model for identifying the identification object on the basis of at least the acquired first image and the second image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a computer program, a learning model generation device, a display device, a particle identification device, a learning model generation method, a display method, and a particle identification method. [Background technology]

[0002] Powders (also called granular materials) are used in various industrial fields such as chemistry. Powder refers to an aggregate of fine particles, such as powder or grains, mainly in the micro to nano order. Powders are used in a wide range of industrial materials, and the main substances handled as powders are diverse, such as food, abrasives, cement, ceramics, magnetic materials, pigments, and cosmetics. Since the properties of the powder change depending on the shape characteristics of the fine particles (e.g. particle size, particle size distribution, shape, etc.), it is important to control the properties of each individual particle when using them as industrial materials.

[0003] As another example of industrial materials, polymers are used in various fields such as electronics, medicine, and biology. When using polymer materials, it is important to control not only the internal structure and physical properties but also the surface and interface. Various microscopes, including scanning electron microscopes (SEMs), are used to measure the surfaces of such powders and materials.

[0004] Patent Document 1 discloses an image processing system that performs required image processing on an image captured by an electron microscope. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2007-200769 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in images captured by an electron microscope, the brightness value changes depending on the shape of the captured object (e.g., powder, the surface of a substance, etc.), and for example, for a spherical object, the brightness histogram is distributed widely from low brightness to high brightness. Also, if the entire surface of the object is inclined, it becomes difficult to set a histogram range that can define the object in the image. Furthermore, if the object has a complex or irregular shape, it becomes difficult to detect the object in the image.

[0007] The present disclosure has been made in consideration of such circumstances, and discloses a computer program, a learning model generation device, a display device, a particle identification device, a learning model generation method, a display method, and a particle identification method that can easily detect an object to be identified. [Means for solving the problem]

[0008] The present application includes multiple means for solving the above-mentioned problems. As one example, a learning model generation device includes a processor, which acquires a first image in which an object to be identified is captured, acquires a second image in which the object to be identified is identified by at least a binary label corresponding to pixels of the first image, and generates a learning model for identifying the object to be identified based on at least the acquired first and second images. Effect of the Invention

[0009] According to the present disclosure, the object to be identified can be easily detected. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of a configuration of an image identification device according to an embodiment of the present invention; [Diagram 2] FIG. 4 is a schematic diagram showing a first example of a process for generating a detection model. [Diagram 3] FIG. 2 is an explanatory diagram showing a first example of a learning input image, which is a first image, and a corresponding label image, which is a second image. [Figure 4]FIG. 11 is a schematic diagram showing an example of a display screen when a learning model is generated by the image recognition device. [Diagram 5] FIG. 2 is an explanatory diagram showing an example of a learning input image which is a first image. [Figure 6] FIG. 2 is an explanatory diagram showing an example of an evaluation image which is a first image. [Figure 7] FIG. 13 is a schematic diagram showing an example of a learning progress graph. [Figure 8] FIG. 13 is a schematic diagram showing an example of an editing screen. [Figure 9] FIG. 13 is a schematic diagram showing an example of a label image in which a part of a first image corrected by an editing process is superimposed (replaced) on a second image. [Figure 10] FIG. 13 is an explanatory diagram showing a second example of a learning input image which is a first image and a corresponding label image which is a second image. [Figure 11] FIG. 13 is an explanatory diagram showing a third example of a learning input image, which is a first image, and a corresponding label image, which is a second image. [Figure 12] FIG. 13 is a schematic diagram illustrating an example of a generation process of a classification model. [Figure 13] FIG. 13 is an explanatory diagram showing a fourth example of learning input images, which are first images, and corresponding label images, which are second images. [Figure 14] FIG. 13 is an explanatory diagram showing a fifth example of a learning input image, which is a third image, and a corresponding labeled image, which is a fourth image. [Figure 15] FIG. 2 is a schematic diagram showing a first example of a particle identification process. [Figure 16] FIG. 13 is a schematic diagram showing an example of a display screen when particles are detected by the image identification device. [Figure 17] FIG. 13 is a schematic diagram showing a first example of a display screen when particle detection is completed. [Figure 18] FIG. 13 is a schematic diagram showing a second example of a display screen when particle detection is completed. [Figure 19] FIG. 2 is a schematic diagram showing an example of a particle size graph. [Figure 20] FIG. 13 is a schematic diagram showing a third example of a display screen when particle detection is completed. [Figure 21]FIG. 13 is a schematic diagram showing a fourth example of a display screen when particle detection is completed. [Figure 22] FIG. 13 is a schematic diagram showing a second example of an output image. [Figure 23] FIG. 13 is a schematic diagram showing a third example of an output image. [Figure 24] FIG. 13 is a schematic diagram showing a fourth example of an output image. [Diagram 25] FIG. 13 is a schematic diagram showing a fifth example of an output image. [Figure 26] FIG. 13 is a schematic diagram showing a sixth example of an output image. [Figure 27] 11 is a flowchart showing an example of a learning process procedure performed by the image recognition device. [Figure 28] 11 is a flowchart showing an example of a particle detection processing procedure performed by the image identification device. [Figure 29] 1 is a flowchart showing an example of a procedure for classifying particles by the image identification device. [Diagram 30] FIG. 11 is a schematic diagram showing a second example of the detection model generation process. [Diagram 31] FIG. 13 is an explanatory diagram showing a sixth example of a learning input image, which is a first image, and a corresponding label image, which is a second image. [Diagram 32] FIG. 11 is a schematic diagram showing a second example of a particle identification process. [Diagram 33] FIG. 13 is a schematic diagram showing another example of the editing screen. [Figure 34A] FIG. 13 is a schematic diagram showing an overview of a display method in mask display / semi-transparent mode. [Figure 34B] FIG. 13 is a schematic diagram showing an overview of a display method in mask display / semi-transparent mode. [Figure 34C] FIG. 13 is a schematic diagram showing an overview of a display method in mask display / semi-transparent mode. [Fig. 34D] FIG. 13 is a schematic diagram showing an overview of a display method in mask display / semi-transparent mode. [Figure 34E] FIG. 13 is a schematic diagram showing an overview of a display method in mask display / semi-transparent mode. [Diagram 35] 1A and 1B are schematic diagrams illustrating an example of the distribution of luminance variance and frequency variance of a cut-out image. [Figure 36A] FIG. 2 is a schematic diagram showing an example of a cut-out image. [Figure 36B] FIG. 2 is a schematic diagram showing an example of a cut-out image. [Figure 36C] FIG. 2 is a schematic diagram showing an example of a cut-out image. [Figure 37] FIG. 13 is a schematic diagram showing an example of a selection screen for a first image. [Figure 38] 11 is a flowchart showing an example of a learning process procedure using a three-value label image by the image recognition device. [Figure 39] 13 is a flowchart showing another example of the procedure of the particle detection process performed by the image identification device. [Diagram 40] FIG. 13 is a block diagram showing another example of the configuration of the image identification device according to the present embodiment. [Figure 41A] 11A and 11B are schematic diagrams showing an example of an evaluation result when particles and a background are binarized. [Figure 41B] 11A and 11B are schematic diagrams showing an example of an evaluation result when particles and a background are binarized. [Figure 41C] 11A and 11B are schematic diagrams showing an example of an evaluation result when particles and a background are binarized. [Figure 41D] 11A and 11B are schematic diagrams showing an example of an evaluation result when particles and a background are binarized. [Figure 41E] 11A and 11B are schematic diagrams showing an example of an evaluation result when particles and a background are binarized. [Figure 42A] 10A and 10B are schematic diagrams showing an example of an evaluation result when particles, boundaries, and a background are binarized; [Figure 42B] 10A and 10B are schematic diagrams showing an example of an evaluation result when particles, boundaries, and a background are binarized; [Figure 42C] 10A and 10B are schematic diagrams showing an example of an evaluation result when particles, boundaries, and a background are binarized; [Fig.42D] 10A and 10B are schematic diagrams showing an example of an evaluation result when particles, boundaries, and a background are binarized; [Figure 42E] 10A and 10B are schematic diagrams showing an example of an evaluation result when particles, boundaries, and a background are binarized; [Figure 43A]11A and 11B are schematic diagrams showing an example of an evaluation result when binarization is performed into a boundary and a non-boundary. [Figure 43B] 11A and 11B are schematic diagrams showing an example of an evaluation result when binarization is performed into a boundary and a non-boundary. [Figure 43C] 11A and 11B are schematic diagrams showing an example of an evaluation result when binarization is performed into a boundary and a non-boundary. [Fig. 43D] 11A and 11B are schematic diagrams showing an example of an evaluation result when binarization is performed into a boundary and a non-boundary. [Figure 43E] 11A and 11B are schematic diagrams showing an example of an evaluation result when binarization is performed into a boundary and a non-boundary. [Figure 44A] FIG. 13 is a schematic diagram showing an example of an evaluation result in the case where the particle, boundary, and background are ternary-valued; [Figure 44B] FIG. 13 is a schematic diagram showing an example of an evaluation result in the case where the particle, boundary, and background are ternary-valued; [Figure 44C] FIG. 13 is a schematic diagram showing an example of an evaluation result in the case where the particle, boundary, and background are ternary-valued; [Fig.44D] FIG. 13 is a schematic diagram showing an example of an evaluation result in the case where the particle, boundary, and background are ternary-valued; [Figure 44E] FIG. 13 is a schematic diagram showing an example of an evaluation result in the case where the particle, boundary, and background are ternary-valued; [Figure 45A] FIG. 13 is a schematic diagram showing an example of an okonomiyaki flour detection result. [Figure 45B] FIG. 13 is a schematic diagram showing an example of an okonomiyaki flour detection result. [Figure 46A] FIG. 13 is a schematic diagram showing an example of a detection result of sodium bicarbonate. [Figure 46B] FIG. 13 is a schematic diagram showing an example of a detection result of sodium bicarbonate. [Figure 47A] FIG. 13 is a schematic diagram showing an example of a detection result of potato starch. [Figure 47B] FIG. 13 is a schematic diagram showing an example of a detection result of potato starch. [Figure 48A] FIG. 13 is a schematic diagram showing an example of a detection result of table salt. [Figure 48B] FIG. 13 is a schematic diagram showing an example of a detection result of table salt. [Figure 49A]FIG. 13 is a schematic diagram showing an example of a detection result of a moth-eye film. [Figure 49B] FIG. 13 is a schematic diagram showing an example of a detection result of a moth-eye film. [Figure 50A] FIG. 2 is a schematic diagram showing an example of the detection results of a lid material for food containers containing silica particles. [Figure 50B] FIG. 2 is a schematic diagram showing an example of the detection results of a lid material for food containers containing silica particles. [Figure 51] FIG. 1 is a schematic diagram showing a first usage mode of an image identification device. [Figure 52] FIG. 13 is a schematic diagram showing a second usage form of the image identification device. [Figure 53] FIG. 13 is a schematic diagram showing a third usage form of the image identification device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of a configuration of an image recognition device 100 according to the present embodiment. The image recognition device 100 has functions as a learning model generating device, a display device, and a particle recognition device. The image recognition device 100 can detect a recognition target and classify the recognition target.

[0012] The identification target may be set in various ways depending on the purpose of observation. For example, it includes powder (also called powdered granules) used as industrial materials in various fields. Major substances treated as powders are diverse, such as food (flour, seasonings, etc.), abrasives, cement, ceramics, magnetic bodies, magnetic materials, pigments, paints, and cosmetics. In addition, the identification target may include specific shapes and patterns of polymeric materials and metal materials used in various fields such as electronics, medicine, and biotechnology. When measuring such an identification target, for example, various microscopes such as a scanning electron microscope are used to image the data. The imaged identification target has a complex shape or pattern, or an irregular shape or pattern, but the image identification device 100 of the present embodiment can easily detect and classify the identification target. In particular, application to an image in which multiple identification targets exist is preferable. In the following description, particles (powders) will be described as an example of an identification target.

[0013] The image recognition device 100 includes a control unit 10 that controls the entire device, an input unit 11, an operation unit 12, a detection processing unit 13, a classification processing unit 14, an edit processing unit 15, an output processing unit 16, a display panel 17, a learning processing unit 20, and a storage unit 30. The learning processing unit 20 includes a learning data input unit 21, a detection parameter learning unit 22, a classification parameter learning unit 23, and a model file output unit 24. The storage unit 30 stores a detection model file 31 and a classification model file 32 as learning models, as well as learning data 33, etc.

[0014] The control unit 10 can be composed of a CPU, a ROM, a RAM, and the like.

[0015] The input unit 11 has a function as an acquisition unit and acquires an input image. The input image is an image in which the state of particles is unknown and which requires detection or classification of particles.

[0016] The operation unit 12 is configured with, for example, a keyboard, a mouse, a touch pad, or a touch panel, and can accept operations for editing the image displayed on the display panel 17.

[0017] The detection processing unit 13 has a function as an input processing unit, and detects particles by inputting the input image acquired by the input unit 11 into the detection model file 31.

[0018] The classification processing unit 14 has a function as an input processing unit, and inputs the input image acquired by the input unit 11 into the classification model file 32, thereby classifying the types of particles.

[0019] The output processing unit 16 has a function as a display processing unit, and outputs an output image including a plurality of particles by superimposing the particle identification result output by the detection model file 31 on the input image. The output processing unit 16 also outputs an output image including a plurality of particles by superimposing the particle identification result output by the classification model file 32 on the input image.

[0020] The edit processing unit 15 can perform edit processing on the image displayed on the display panel 17 in accordance with an edit operation received by the operation unit 12 .

[0021] The display panel 17 can be configured with a liquid crystal display, an organic EL display, etc. Instead of the display panel 17, an external display device may be connected to the image identification device 100.

[0022] Next, the learning process of the image recognition device 100 will be described.

[0023] Fig. 2 is a schematic diagram showing a first example of a detection model generation process. As shown in Fig. 2, a detection model file created in advance by a multi-layered neural network (deep learning) is prepared. The detection model file stores, for example, a neural network configuration consisting of multiple layers for discriminating between the inside and outside of a particle and the weighting coefficients of the neurons in each layer.

[0024] The learning data input unit 21 has functions as a first acquisition unit and a second acquisition unit, and acquires learning input images and label images.

[0025] The detection parameter learning unit 22 has a function as a generation unit, and by inputting a learning input image and a label image to the detection model, it can recalculate the weight coefficients of the neurons in each layer to generate a detection model, output it as a detection model file, and store it in the storage unit 30. With the above-mentioned configuration, when an input image with unknown particles is input to the detection model, the detection model identifies whether or not each pixel on the input image is a particle, so that particles on the input image can be easily detected.

[0026] FIG. 3 is an explanatory diagram showing a first example of a learning input image, which is a first image, and a corresponding label image, which is a second image. The learning input image is called the first image, and is an image in which particles to be identified are captured, and is an image used when generating or updating a learning model. In the example of FIG. 3, a large number of particles of different sizes are displayed in a dense state in the learning input image. The label image is called the second image, and is an image in which a label is associated with a pixel corresponding to a pixel constituting the learning input image, and the label is used to identify the identification target (for example, to identify whether or not it is an identification target). The label is at least binary, and for example, as shown in FIG. 2, label=0 can be the background (a region that is not an identification target) and label=1 can be the particle (a region that is an identification target).

[0027] 4 is a schematic diagram showing an example of a display screen when a learning model is generated by the image identification device 100. On the display screen, an image selection icon 221, a particle detection icon 222, a report output icon 223, a model creation icon 224, and an end icon 225 are displayed as a menu. The following describes a state in which the model creation icon 224 is selected.

[0028] By opening the combo box 201, an original model file created in advance can be selected when generating a learning model. By dragging and dropping a data folder including a pair of learning input image files and label image files onto the learning file list 202, learning data can be selected. By dragging and dropping a data folder including evaluation data files onto the evaluation file list 203, evaluation data can be selected.

[0029] Parameters for learning can be set by operating the parameter icon 226. The parameters can be, for example, the number of learning trials, the number of tests to be performed, the number of mini-batches which is the number of samples used to update the weighting coefficients, and the learning rate which indicates the rate at which the weighting coefficients are updated.

[0030] Generation of a learning model is started by operating a learning start icon 227. A remaining time 228 is displayed to the right of the learning start icon 227. The learning process can be controlled by the pause, resume, and stop icons 229.

[0031] FIG. 5 is an explanatory diagram showing an example of a learning input image which is the first image, and FIG. 6 is an explanatory diagram showing an example of an evaluation image which is the first image. The learning input image is displayed in the learning image area 205. By specifying a required area of ​​the learning input image with, for example, a rectangular frame, the image within the frame of the learning input image is enlarged and displayed in the learning image (enlarged) area 206. An image of the test result can be displayed in the test result (enlarged) area 207. The evaluation input image is displayed in the evaluation image area 208. By specifying a required area of ​​the evaluation input image with, for example, a rectangular frame, the image within the frame of the evaluation input image is enlarged and displayed in the evaluation image (enlarged) area 209. An image of the evaluation result can be displayed in the evaluation result (enlarged) area 210.

[0032] Below the evaluation image (enlarged) area 209, the selected learning input image and the number of correct evaluation input images are displayed.

[0033] Fig. 7 is a schematic diagram showing an example of a learning progress graph. A graph such as that shown in Fig. 7 can be displayed in the learning progress graph area 204. In Fig. 7, the horizontal axis indicates the number of trials, and the vertical axis indicates the adaptation rate (correct answer rate). This makes it possible to grasp the learning progress.

[0034] If it is necessary to check the evaluation image and correct the label image, it can be done as follows. That is, drag and drop the data folder containing the label image to be corrected onto the learning file list 202, select the label image in the learning file list 202, and operate the learning image editing icon 230. When correcting the label image, the learning input image (corresponding to the predetermined first image) may be the learning input image input when training the learning model, or may be a completely different learning input image (for example, an image not used in training the learning model).

[0035] FIG. 8 is a schematic diagram showing an example of an editing screen. In the example of FIG. 8, a portion of the learning input image 242 surrounded by a frame is enlarged and displayed in the correction image area 241. In the correction image area 241, a label image displayed at least in binary may be displayed as a label image output by the learning model, but a pixel of a specific label (a pixel with label=1 in the example of FIG. 8) among the pixels of the label image displayed in binary may be replaced with a specific part of pixels of the learning input image corresponding to the pixel (a pixel of the learning input image corresponding to label=1, i.e., a particle in the example of FIG. 8). By displaying the pixel of the first image corresponding to the pixel of the second image to which a specific label is assigned as the second image, the learning result can be corrected while referring to information of the pixel of the first image, which is the original image (for example, a distribution of luminance values, which are pixel values). The label image shown here may be created so as to be displayed on the learning input image by replacing the pixel of the label image with label=0, i.e., the pixel corresponding to the background. By using the edit operation icon 243, for example, it is possible to switch between displaying and hiding the label image, change the edit mode (eraser / pen), change the size of the eraser / pen, change the display color of the label image, and the like.

[0036] Fig. 9 is a schematic diagram showing an example of a label image in which a part of the first image corrected by the editing process is superimposed (replaced) on the second image. Comparing the label image shown in Fig. 8 with the label image shown in Fig. 9, the pixels in the area indicated by the symbol S in Fig. 8 (dark part in the figure: particles) have been erased in Fig. 9 and corrected to a label indicating the bright part, i.e., the background. Also, the bright part in Fig. 8, i.e., the label indicating the background, has been corrected to a label in the area indicated by the symbol S in Fig. 9 (dark part in the figure: particles).

[0037] After correcting the label, the label image is overwritten and saved by operating the save icon 244. The display screen shown in Fig. 4 is returned to by operating the model creation icon 224. Here, the operations on the label image displayed by replacing pixels of a specific label among the pixels of the label image displayed in binary as the label image with a specific part of pixels of the learning input image corresponding to the pixels (label = 1, i.e., pixels of the learning input image corresponding to the particle) have been described, but as described above, the internal processing of the image recognition device is performed so as to instruct correction to the binary label image.

[0038] If the evaluation image is checked and the result is good, a learning model is generated by operating the learning model save icon 231, and the model file is saved in the storage unit 30. When learning is complete, a log including pairs of learning input images and label images, a learning progress graph, the original model file, etc. can be saved by operating the log read icon 232. The saved log can be displayed by operating the log read icon 232.

[0039] As described above, the editing operation can be an operation to correct the brightness value when the brightness value of the pixel of the labeled image is different from the value of the label. That is, the label of the labeled image can be corrected by the editing operation, and when some pixels of the labeled image are not labeled correctly, the label value can be corrected to make the labeled image correct.

[0040] By inputting the learning input image and the label image that has received the editing operation (i.e., the label image that has become correct) to the learning model, the weight coefficients of the neurons can be recalculated and the learning model can be updated.

[0041] With the above-described configuration, the learning model can be retrained, and the accuracy of particle detection by the learning model can be improved.

[0042] In the above example, the label image indicates that label=0 is outside the particle and label=1 is inside the particle, but the labels are not limited to this.

[0043] FIG. 10 is an explanatory diagram showing a second example of a learning input image, which is a first image, and a corresponding label image, which is a second image. In FIG. 10, the particles are enlarged for convenience. In reality, a large number of particles are displayed in a dense state. The label image can identify particles with at least three-value labels corresponding to the pixels constituting the learning input image. By increasing the number of labels from two values, the amount of information for identifying particles can be increased, making it easier for the learning model to identify particles, and improving the detection accuracy of particles. In particular, while adjacent pixel values ​​often change relatively gradually inside a particle, the boundary is the boundary between a particle and a background, and is a point where the pixel value representing a particle and the pixel value representing the background change, and adjacent pixel values ​​often change sharply, so that the detection accuracy of particles can be improved by learning them as separate labels.

[0044] More specifically, as shown in Fig. 10, the label image includes labels corresponding to pixels of the learning input image, where label = 0 indicates background (area not to be classified), label = 1 indicates boundary (boundary between area to be classified and area not to be classified), and label = 2 indicates particle (area to be classified). Note that the position of each label in Fig. 10 is merely an example and is not limited to the example in Fig. 10.

[0045] Here, by making it possible to specify the boundary area between the boundary to be identified and the area not to be identified as the boundary (label = 1) with a specified width, it is possible to have the learning model identify particles without having to specify in detail the boundary area that distinguishes which specific pixels are particles or not, thereby improving the accuracy of particle detection.

[0046] FIG. 11 is an explanatory diagram showing a third example of a learning input image, which is a first image, and a corresponding label image, which is a second image. Note that in FIG. 11 as well, the particles are enlarged and displayed for convenience. In reality, a large number of particles are displayed in a dense state. The label image includes labels corresponding to the pixels of the learning input image, and indicates that it is a particle region by at least two types of labels (in the example of FIG. 11, labels = 2, 3). Note that the position of each label in FIG. 11 is an example, and is not limited to the example of FIG. 11.

[0047] That is, the label image includes labels corresponding to pixels of the learning input image, where label=0 indicates background (region not to be identified), label=1 indicates boundary (boundary between region to be identified and region not to be identified), label=2 indicates first interior having a predetermined characteristic inside the particle, and label=3 indicates second interior having a predetermined characteristic different from the first interior inside the particle. Even in a particle region, the luminance value and distribution of luminance values ​​may differ depending on the type of particle. By including labels indicating at least two types of particle interiors in the label image, the learning model can learn the change in luminance value and the difference in distribution of luminance value inside the particle, and the detection accuracy of the particle can be improved regardless of the type of particle.

[0048] In the example of FIG. 11, the label image may include label=4 (outside the learning range). When the learning input image includes pixels that do not contribute to the generation of the learning model, the amount of learning of the learning model can be reduced by using a label image in which a label outside the learning range is assigned to the pixel, thereby improving the learning efficiency. In addition to the inside and boundary of a particle, it is possible to set areas that are not subject to learning, which improves the accuracy of the information provided to the learning model and the detection accuracy of particles.

[0049] Fig. 12 is a schematic diagram showing an example of a classification model generation process. As shown in Fig. 12, a classification model file created in advance by a multi-layered neural network (deep learning) is prepared. The classification model file stores, for example, a neural network configuration consisting of multiple layers for classifying types of particles based on the particle shapes and the weight coefficients of neurons in each layer.

[0050] The learning data input unit 21 acquires a learning input image, which is a first image, and a corresponding labeled image, which is a second image.

[0051] The classification parameter learning unit 23 has a function as a generation unit, and by inputting a learning input image, which is a first image, and a corresponding label image, which is a second image, into the classification model, it can recalculate the weight coefficients of the neurons in each layer to generate a classification model, output it as a classification model file, and store it in the storage unit 30. With the above-mentioned configuration, when an input image with unknown particle shapes is input into the classification model, the classification model identifies the type of particle based on the shape of the particle in the input image, so that the particles in the input image can be easily classified.

[0052] Fig. 13 is an explanatory diagram showing a fourth example of a learning input image, which is a first image, and a corresponding label image, which is a second image. Note that, for convenience, the particles are also enlarged and displayed in Fig. 13. In reality, a large number of particles are displayed in a dense state. In addition, although the shapes of the particles are actually complex and irregular, for convenience, the shapes of the particles are displayed as circles, ellipses, triangles, and squares.

[0053] In the label image, label=0 indicates background (area not to be identified), label=1 indicates a particle and the particle shape is a first shape (round in the figure), label=2 indicates a particle and the particle shape is a second shape (triangular in the figure), label=3 indicates a particle and the particle shape is a third shape (elliptical in the figure), and label=4 indicates a particle and the particle shape is a fourth shape (rectangular in the figure). Note that the position of each label in FIG. 13 is merely an example and is not limited to the example in FIG. 13.

[0054] That is, the label image indicates that the particle is a particle and the type of the particle based on the shape of the particle by at least a three-value label. By inputting the learning input image, which is the first image, and the corresponding label image, which is the second image, into the classification model, the weight coefficients of the neurons are recalculated to generate a classification model. In this way, when an input image of unknown particles is input into the classification model, the classification model identifies the shape of the particle in the input image, so that the particle in the input image can be easily identified and classified. Even if the particle has a complex and irregular shape, the shape can be identified and the particle can be classified based on the particle shape. The classification model is trained so that it can simultaneously identify the object to be identified (be a particle) and classify the type of the object to be identified (classify the particle).

[0055] 13, the label image includes labels 0 to 4, and a plurality of different shapes are simultaneously learned, but the present invention is not limited to this. The classification model can also be trained to classify the type of the classification target based on an image that has been classified as being a classification target or not.

[0056] The learning data input unit 21 acquires a learning input image which is a third image in which the classification target has been classified, and a label image which is a fourth image in which the type of classification target has been specified corresponding to the learning input image.

[0057] The classification parameter learning unit 23 has a function as a generation unit, and by inputting a learning input image, which is a third image, and a corresponding label image, which is a fourth image, into the classification model, it can recalculate the weight coefficients of the neurons in each layer to generate a classification model, output it as a classification model file, and store it in the storage unit 30. With the above-mentioned configuration, when an input image with unknown particle shapes is input into the classification model, the classification model identifies the type of particle based on the shape of the particle in the input image, so that the particles in the input image can be easily classified.

[0058] Fig. 14 is an explanatory diagram showing a fifth example of a learning input image, which is a third image, and a corresponding label image, which is a fourth image. Note that, for convenience, the particles are also enlarged and displayed in Fig. 14. In reality, a large number of particles are displayed in a dense state. In addition, although the shapes of the particles are actually complex and irregular, for convenience, the shapes of the particles are displayed as circles, ellipses, triangles, and squares.

[0059] The learning input image is called the third image, and is an image in which the particles to be identified are identified, and is used when generating or updating the learning model. The third image may be an image in which the particles are identified, and may be a label image displayed in at least two values ​​output by the learning model described above. Furthermore, the learning input image may be set together with the third image in which the identification target is identified, and the first image in which the identification target corresponding to the third image is captured. By learning the learning model using not only the third image but also the first image as the learning input image, it becomes possible to identify the particles by taking into account information on the distribution of pixel values ​​of the particles and the background in addition to information on the identification of the particles.

[0060] The label image is called the fourth image, and is an image in which a label is associated with a pixel corresponding to a pixel constituting the learning input image, and the type of particle is specified by the label. The label may be anything that can indicate the type of particle based on the shape of the particle. If the type of particle is known, the particle can be classified.

[0061] In the label image, label=1 indicates that the particle shape is a first shape (round in the figure), label=2 indicates that the particle shape is a second shape (triangular in the figure), label=3 indicates that the particle shape is a third shape (elliptical in the figure), and label=4 indicates that the particle shape is a fourth shape (rectangular in the figure). Note that label=0 may also be used to indicate that the particle is the background (a region that is not to be identified). The positions of the labels in FIG. 14 are merely examples and are not limited to the example in FIG. 14.

[0062] A learning model may be generated so that round particles can be detected using a learning input image and a label image including only labels whose particle shapes are round, a learning model may be generated so that quadrangular particles can be detected using a learning input image and a label image including only labels whose particle shapes are square, a learning model may be generated so that triangular particles can be detected using a learning input image and a label image including only labels whose particle shapes are triangular, and a learning model may be generated so that elliptical particles can be detected using a learning input image and a label image including only labels whose particle shapes are elliptical. Also, based on the information of the result of particle detection, small images cut out one by one of the particle parts may be used as learning input images, and text data or the like containing information corresponding to one of the labels 0 to 4 for each of the small images may be used as label information to generate a learning model.

[0063] Next, the detection process of the image identification device 100 will be described.

[0064] 15 is a schematic diagram showing a first example of particle identification processing. When an input image is input to a learning model (detection model file), the learning model, for example, distinguishes between particles and the background, and the output processing unit 16 outputs an output image. Also, when an input image is input to a learning model (classification model file), the learning model distinguishes between the shapes of particles and classifies the types of particles, and the output processing unit 16 outputs an output image.

[0065] FIG. 16 is a schematic diagram showing an example of a display screen when particles are detected by the image identification device 100. In the example of FIG. 16, the particle detection icon 222 is operated. In the file list 251, the file of the input image for particle detection is displayed. The input image to be processed can be selected in advance by dragging and dropping the image file into the file list 251. In the enlargement area selection box 252, the area to be enlarged in the input image can be specified. In the example shown in the figure, the enlargement area can be selected as a rectangular area. In the enlargement area selection box 252, the enlargement ratio, the enlargement position, etc. can be changed. In the enlargement area 253, the enlarged area of ​​the input image is displayed.

[0066] A model file can be selected by opening the combo box 254. The model file is the detection model file 31 or the classification model file 32 generated by the above-mentioned learning process. The detection process is started by operating the start icon 255.

[0067] 17 is a schematic diagram showing a first example of a display screen when particle detection is completed. A detection result area 256 displays an output image as a first example including a plurality of particles, and an enlarged detection result area 257 displays an output image of an enlarged area selected in the enlarged area selection box 252 from the output image.

[0068] As described above, when an input image is input to the learning model, the learning model outputs a classification result. The classification result can be, for example, information indicating whether each pixel of the input image is a particle or not. By superimposing the input image and the classification result, an output image including multiple particles can be displayed.

[0069] As a result, when an input image with unknown particles is input into the learning model, the learning model identifies whether each pixel in the input image is a particle or not, making it easy to detect particles in the input image even if the particles have a complex or irregular shape.

[0070] The output image displays the center of gravity of each particle (shown as a small circle in the example image), making it easy to grasp the position of each particle even when multiple particles are displayed close together.

[0071] In addition, the output image shows the particle area, so even if particles and non-particles are mixed, the range of particles can be easily grasped, which is useful for identifying the size and shape of the particles.

[0072] Fig. 18 is a schematic diagram showing a second example of the display screen when particle detection is completed. In Fig. 17, by operating the graph confirmation icon 258 or the report output icon 223, the display screen shown in Fig. 18 is displayed. In the graph display area 259, a graph of the particle diameter of the particles included in the output image can be displayed. The particle diameter can be switched to the maximum Feret's diameter, the minimum Feret's diameter, or the area diameter by operating the major diameter icon 260, the minor diameter icon 261, or the area diameter icon 262, respectively.

[0073] The required attribute of the object to be identified, for example, the attribute of a particle, can be a physical quantity that characterizes the required properties of the particle depending on the field in which the particle is used. The attribute of a particle can be, for example, a particle size, and the particle size includes a Feret diameter, an axial length of an equivalent ellipse, an area diameter, and the like. The Feret diameter includes a maximum Feret diameter as a major axis and a minimum Feret diameter as a minor axis, and the axial length of an equivalent ellipse includes a major axis length of the equivalent ellipse as a major axis and a minor axis length of the equivalent ellipse as a minor axis. The Feret diameter is the horizontal length and the vertical length when a rectangle is provided that circumscribes the outline when a particle is projected onto an xy plane. The maximum Feret diameter is the maximum width of the parallel lines when a particle is sandwiched between parallel lines from all directions. The minimum Feret diameter is the minimum width of the parallel lines when a particle is sandwiched between parallel lines from all directions. The major axis length of an equivalent ellipse is the major axis length of an ellipse (equivalent ellipse) that has the same area as the particle area and has the same first and second moments. The minor axis length of the equivalent ellipse is the minor axis length of an ellipse (equivalent ellipse) that has the same area as the particle area and has the same first and second moments. The area diameter is the diameter of a circle that has the same area as the particle area.

[0074] Fig. 19 is a schematic diagram showing an example of a particle size graph. In Fig. 19, the horizontal axis indicates the maximum Feret diameter as the major axis, and the vertical axis indicates the frequency (%) and cumulative total (%) of particles. When the minimum Feret diameter as the minor axis is selected instead of the maximum Feret diameter as the particle size, the horizontal axis indicates the minimum Feret diameter, and when the area diameter is selected, the horizontal axis indicates the area diameter. As shown in Fig. 19, the particle size graph displays the number of particles N, the minimum value d1 and the maximum value d2 of the maximum Feret diameter, the value dm of the maximum Feret diameter when the cumulative total is 50%, and the like.

[0075] In FIG. 18, by selecting the all images / display image icon 263, it is possible to switch between displaying the particle size graph of all images or displaying the particle size graph of the image currently being displayed.

[0076] As described above, when a desired particle size (e.g., maximum Feret diameter, minimum Feret diameter, area diameter) is selected, the distribution information of the selected particle size is displayed. In other words, it is possible to grasp the range in which the particle sizes of multiple particles are distributed, and therefore it is possible to easily grasp the shape characteristics of the particles.

[0077] Fig. 20 is a schematic diagram showing a third example of the display screen when particle detection is completed. By inputting required values ​​in the minimum diameter / maximum diameter field 264 and operating the display switching icon 265, it is possible to select and highlight only particles having the input minimum diameter to maximum diameter. In the example of Fig. 20, some of the particles in the output image shown in Fig. 18 are highlighted (for example, by changing the color) in the enlargement area 253a. In addition, it is possible to display the number of particles and the minimum and maximum particle diameters for the entire image or the image being displayed.

[0078] In this way, some particles among a plurality of particles in the output image are displayed so as to be distinguishable from the other particles, so that a desired particle can be easily recognized.

[0079] By operating the graph save icon 266, the particle size graph image can be saved.

[0080] Fig. 21 is a schematic diagram showing a fourth example of the display screen when particle detection is completed. Fig. 21 shows a particle detection screen similar to that of Fig. 17. The detection results can be corrected by operating a result correction icon 267. Specifically, a screen similar to that shown in Fig. 8 is displayed, and editing operations can be performed.

[0081] Fig. 22 is a schematic diagram showing a second example of an output image. In Fig. 22, the particles are enlarged and displayed for convenience. In reality, a large number of particles are displayed in a dense state. In addition, the shape of the particles is elliptical instead of circular as in the first example. The output processing unit 16 displays at least one of the major axis and minor axis of the particles, so that the size and shape of the particles can be easily grasped.

[0082] Specifically, as shown in FIG. 22, the output processing unit 16 displays a line segment indicating the long axis of the particle (a solid line in the figure) and a line segment indicating the short axis (a dashed line in the figure), so that the long axis direction and the short axis direction of the particle can be easily visually confirmed.

[0083] Fig. 23 is a schematic diagram showing a third example of an output image. In Fig. 23, the particle is displayed in an enlarged scale for convenience. The output processing unit 16 displays the ratio of the long axis to the short axis of the particle, so that the ratio of the long axis to the short axis of the particle can be easily visually recognized. Specifically, the intersection of a line segment in the long axis direction and a line segment in the short axis direction is displayed, and a rectangle having a diagonal line proportional to the ratio of the long axis to the short axis centered at the intersection is displayed, so that the ratio of the long axis to the short axis of the particle can be easily visually recognized.

[0084] Fig. 24 is a schematic diagram showing a fourth example of an output image. For convenience, the particles are enlarged and displayed in Fig. 24. The output processing unit 16 displays adjacent particles in different display modes (for example, different colors), so that even if the particles are displayed together, they can be easily distinguished from each other.

[0085] Fig. 25 is a schematic diagram showing a fifth example of an output image. For convenience, the particles are displayed in an enlarged scale in Fig. 25. The output processing unit 16 displays an animation of a wave motion from the center of gravity of the particle toward the boundary, so that the position of the particle and the range of the area can be easily grasped.

[0086] Fig. 26 is a schematic diagram showing a sixth example of an output image. In reality, the shapes of particles are complex and irregular, but for the sake of convenience, Fig. 26 shows the particle shapes divided into circles, ellipses, triangles, and squares. The learning model (classification model file 32) identifies which type of particle each pixel on the input image is, so that even if the particle shape is complex or irregular, the particles on the input image can be easily classified.

[0087] Next, the processing procedure of the image identification device 100 will be described.

[0088] 27 is a flowchart showing an example of a learning process procedure by the image recognition device 100. In the following, for convenience, the main subject of the process will be described as the control unit 10. The control unit 10 acquires an input image for learning (S11), and acquires a label image (S12). The control unit 10 acquires an image for evaluation (S13), and acquires a model file (S14).

[0089] The control unit 10 sets parameters (number of trials, number of mini-batches, number of trials to perform testing, learning rate, etc.) (S15) and performs a learning process (S16). The control unit 10 outputs an evaluation result in which an evaluation image is input into the generated model file (S17). The control unit 10 determines whether or not to correct the label image (S18). The determination of whether or not to correct the label image can be made based on, for example, whether or not the user has performed an editing operation to correct the label image.

[0090] If the label image is to be corrected (YES in S18), the control unit 10 displays the label image to be corrected (S19), corrects the label image based on the editing operation (S20), and performs the process of step S21 described below. If the label image is not to be corrected (NO in S18), the control unit 10 determines whether or not to re-learn (S21). Whether or not to re-learn is determined by the user based on the evaluation result using the evaluation image, for example, and can be performed depending on whether or not the user performs a re-learning operation.

[0091] If relearning is to be performed (YES in S21), the control unit 10 continues the processing from step S16 onwards. If relearning is not to be performed (NO in S21), the control unit 10 judges whether or not to generate another model file (S22). If another model file is to be generated (YES in S22), the control unit 10 continues the processing from step S11 onwards, and if another model file is not to be generated (NO in S22), the control unit 10 saves the generated model file (S23) and ends the processing.

[0092] 27, when generating a particle detection model file, it is sufficient to obtain a learning input image that is a first image and a label image that is a corresponding second image. When generating a particle classification model file, it is sufficient to obtain a learning input image that is a third image and a label image that is a corresponding fourth image.

[0093] 28 is a flowchart showing an example of a particle detection process procedure by the image identification device 100. The control unit 10 acquires an input image (S31), and performs particle detection processing using the generated learning model (detection model file) (S32). The control unit 10 generates and displays an output image based on the detection result of the learning model and the input image (S33).

[0094] The control unit 10 judges whether or not a display operation of the particle distribution information has been performed (S34). The distribution information is distribution information of particle attributes such as particle size, for example, distribution information of maximum Feret diameter, minimum Feret diameter, and area diameter. If a display operation has been performed (YES in S34), the control unit 10 displays the particle distribution information (S35) and performs the process of step S36 described below.

[0095] If there is no display operation (NO in S34), the control unit 10 judges whether or not to correct the detection result (S36). The judgment of whether or not to correct the detection result can be made, for example, based on whether or not the user has performed an editing operation to correct the detection result. If the detection result is to be corrected (YES in S36), the control unit 10 displays the input image and the label image in an overlapping manner (S37), and corrects the label image based on the editing operation (S38).

[0096] The control unit 10 stores the corrected label image (S39) and ends the process. If the detection result is not to be corrected (NO in S36), the control unit 10 ends the process.

[0097] 29 is a flowchart showing an example of a particle classification processing procedure by the image identification device 100. The control unit 10 acquires an input image (S51), and performs particle classification processing using the generated learning model (classification model file) (S52). The control unit 10 generates and displays an output image based on the detection result of the learning model and the input image (S53).

[0098] The control unit 10 determines whether or not to correct the detection result (S54). The determination of whether or not to correct the detection result can be made, for example, based on whether or not the user has performed an editing operation to correct the detection result. If the detection result is to be corrected (YES in S54), the control unit 10 displays the input image and the label image in an overlapping manner (S55), and corrects the label image based on the editing operation (S56).

[0099] The control unit 10 stores the corrected label image (S57) and ends the process. If the detection result is not to be corrected (NO in S54), the control unit 10 ends the process.

[0100] In the above example, the particle detection process and the classification process are described separately, but it is also possible to perform both the particle detection process and the classification process on the input image. Also, the particle classification process may be performed on the output image obtained as a result of particle detection.

[0101] Next, the learning process and classification process based on the three-value labels illustrated in FIG. 10 will be described in more detail.

[0102] Fig. 30 is a schematic diagram showing a second example of the generation process of the detection model. As shown in Fig. 30, a detection model file created in advance by a multi-layered neural network (deep learning) is prepared. The detection model file stores, for example, a neural network configuration consisting of multiple layers for identifying particles, boundaries, and background (neither particles nor boundaries) and the weighting coefficients of neurons in each layer.

[0103] The learning data input unit 21 acquires learning input images and label images. The difference from the case of Fig. 2 is that the label images in the example of Fig. 30 are ternary images.

[0104] The detection parameter learning unit 22 can recalculate the weight coefficients of neurons in each layer to generate a detection model by inputting a learning input image and a label image to the detection model, outputting it as a detection model file, and storing it in the storage unit 30. With the above-mentioned configuration, when an input image with unknown particles is input to the detection model, the detection model identifies whether each pixel on the input image is a particle, a boundary, or the background, so that the particles and the background are distinguished by the boundaries on the input image, and the particles can be detected with high accuracy.

[0105] FIG. 31 is an explanatory diagram showing a sixth example of a learning input image, which is a first image, and a label image, which is a corresponding second image. The learning input image is an image in which a particle to be identified is captured, and is an image used when generating or updating a learning model. In the example of FIG. 31, a large number of particles with different sizes and shapes are displayed in the learning input image. The label image is an image in which a label is associated with a pixel corresponding to a pixel constituting the learning input image. In the example of FIG. 31, the label can be expressed by three values, and for example, as shown in FIG. 31, label=0 can be the background, label=1 can be the boundary, and label=2 can be the particle (area to be identified). Note that the learning input image, which is the first image, may be selected by cutting out from an original image (an image larger in size than the first image) in which a particle is captured by a microscope, for example, using a method such as that shown in FIG. 35 (however, not limited to the method of FIG. 35) described later.

[0106] FIG. 32 is a schematic diagram showing a second example of particle identification processing. When an input image (for example, an image of particles captured by a microscope) is input to a learning model (detection model file), the learning model identifies, for example, particles, background, and boundaries, and the output processing unit 16 outputs an output image. That is, when an input image is input to the learning model, the learning model outputs an output image including particles, background, and boundaries. The identification result can be, for example, information indicating whether each pixel of the input image is a particle, a boundary, or a background. By superimposing the input image and the identification result, an output image including multiple particles can be output.

[0107] As a result, when an input image of a particle captured by a microscope is input into the learning model, the learning model identifies whether each pixel in the input image is a particle, a boundary, or the background, and outputs an output image that includes particles, background, and boundaries.Therefore, even if the particle has a complex or irregular shape, the particles can be distinguished by their boundaries, making it possible to accurately detect particles in the input image.

[0108] In the above, an example of an editing screen for binary labels is shown in Fig. 8, and an example of an edited label image is shown in Fig. 9. Below, a specific example of the editing screen and the display method during editing will be described.

[0109] Fig. 33 is a schematic diagram showing another example of the editing screen. As shown in Fig. 33, in addition to a correction image area 271, an editing screen 270 displays various icons including a display / non-display icon 272, a semi-transparent display icon 273, a fill icon 274, a pen size icon 274, a mask color display change icon 276 for changing the display color of the mask, and a save icon 278 for saving the editing results.

[0110] In the example of FIG. 33, the corrected image area 271 can display an image in three display modes: mask non-display, mask display, and mask display / semi-transparent mode. In the mask non-display display mode, only the learning input image (an input image in which particles are captured by a microscope) is displayed in the corrected image area 271, and the mask image is not displayed. The mask image is a binary label image in which the background portion is masked. That is, the binary label image is an image in which the area corresponding to the particles is transparent, and the area corresponding to the background and the boundary is opaque. In the mask display display mode, only the mask image is displayed, and the learning input image is not displayed. The mask display / semi-transparent mode display mode is a display mode in which a mask image is placed on the learning input image, and the area corresponding to the background and the boundary of the mask image is made semi-transparent, and the area corresponding to the particles of the learning input image is displayed as it is through the transparent part of the mask image, and the boundary and the background of the learning input image are displayed semi-transparently. In FIG. 33, the corrected image area 271 shows the display mode of the mask display / semi-transparent mode. Note that the mask image may be a three-value label image. In this case as well, the regions corresponding to the particles are transparent, and the regions corresponding to the background and the boundary are opaque.

[0111] 34A to 34E are schematic diagrams showing an overview of a display method in the mask display semi-transparent mode. As shown in FIG. 34A, in the mask display semi-transparent mode, a mask image is placed on an input image (for example, an SEM image), and the mask image has a transparent area corresponding to particles and a semi-transparent area corresponding to the boundary and background. FIG. 34B is the same as FIG. 33. FIG. 34C to FIG. 34E show an enlarged view of one particle portion (area surrounded by a rectangle) of an original image (input image). FIG. 34C shows the original image, in which the broken lines indicate the boundaries of the particles. FIG. 34D shows a mask image. In the figure, the black parts are transparent, and the white parts are semi-transparent. The area between the boundary indicated by the broken lines and the part of the particle indicated by the black lines can be the boundary area S. In FIG. 34E, the parts of the mask image other than the particles are displayed semi-transparently, so that the boundary area S can be easily recognized, and the work efficiency of filling in the boundary is improved.

[0112] As described above, a mask image is superimposed on the first image, in which the areas corresponding to the particles are transparent and the areas corresponding to the boundaries and background are semi-transparent. Therefore, the particle parts of the first image are displayed as is, and the non-particle parts (boundaries and background) are displayed semi-transparently, allowing the user to easily see the boundary parts.

[0113] That is, the editing processing unit 15 can easily generate the second image, which is a three-value label image, in response to the editing operation received by the operation unit 12. More specifically, in the mask image before the boundary portion is filled in, for example, the portion corresponding to the particle is given a label of 2 and is transparent, and the background (in this case, the background also includes the boundary) is given a label of 0 and is semi-transparent. By filling in the boundary portion, the label of the filled portion is changed from 0 to 1, and the label is set to 1 (boundary), and the second image, which is a three-value label image, can be easily generated.

[0114] In the above example, a ternary label image is generated by filling in the boundary portion from a binary label image, but the present invention is not limited to this. For example, a ternary label image can be obtained and edited by painting the particles, background, and boundary with three colors.

[0115] The image shown in Fig. 34 is a learning input image (first image) at the stage of generating the learning model, and may be a learning input image prepared before generating the learning model at the stage of relearning (updating) the learning model, or may be an input image when actually input to a trained learning model and detecting particles. The mask image shown in Fig. 34 is a ternary label image (second image) with a boundary added to a binary image of a particle and background at the stage of generating the learning model, and may be a second image prepared before generating the learning model at the stage of relearning (updating) the learning model, or may be a second image corrected (edited) from binary to ternary using an input image when actually input to a trained learning model and detecting particles, or a second image corrected (edited) a ternary label image.

[0116] Next, a method for cutting out the first image will be described. Note that the following method for cutting out the first image is not limited to a ternary label image, but can also be applied to a binary label image.

[0117] Fig. 35 is a schematic diagram showing an example of the distribution of brightness variance and frequency variance of a cut-out image. The cut-out image has the same size (resolution) as the first image, and has, for example, about 100 to 300 pixels, but is not limited to this. The numbers in the figure are sample numbers of the cut-out image. The cut-out image is an image cut out of an image of the same size as a rectangular area each time a rectangular area is scanned on an original image (larger in size than the cut-out image) in which a particle is photographed by a microscope.

[0118] The luminance variance is the variance of the luminance values ​​of each pixel in a rectangular region, and for example, the variance value can be calculated by summing the squares of the differences between the luminance values ​​of each pixel in the rectangular region and the average luminance value of the rectangular region for each pixel, and dividing the sum by the number of pixels. The frequency variance is the variance of each frequency component of an image that has been frequency-converted by a Fourier transform or the like, and can be calculated in the same manner as the luminance variance.

[0119] Figures 36A to 36C are schematic diagrams showing examples of cut-out images. Figure 36A shows examples of cut-out images (sample numbers 33 and 36) in which the luminance variance and frequency variance are in a relatively large range, Figure 36C shows examples of cut-out images (sample numbers 5 and 67) in which the luminance variance and frequency variance are in a relatively small range, and Figure 36B shows examples of cut-out images (sample numbers 44 and 50) in which the luminance variance and frequency variance are in a relatively medium range.

[0120] The selection of the first image is not limited to the method of cutting out a rectangular area from the original image, and a desired candidate for the first image may be set from a plurality of candidate first images prepared in advance.

[0121] The learning data input unit 21 can acquire the selected first image. The selection of the first image is performed by calculating at least one of the luminance variance and the frequency variance of a rectangular region, and selecting the first image based on the calculated luminance variance and at least one of the frequency variance. For example, an image with a luminance variance equal to or greater than a predetermined luminance variance threshold, or a frequency variance equal to or greater than a predetermined frequency variance threshold, or an image with both a luminance variance and a frequency variance equal to or greater than each of the predetermined luminance variance and the frequency variance can be selected as the first image.

[0122] This allows the selection of a first image in which the number of particles is appropriate, the number of particles is appropriately dense, the number of patterns is relatively large, and the first image has various characteristics, making it possible to collect optimal learning data. Also, the user may manually select a rectangular area on the image that he or she particularly wants to learn, and cut out the first image.

[0123] Fig. 37 is a schematic diagram showing an example of a selection screen for a first image. As shown in Fig. 37, the selection screen for the first image displays a display area for a cut-out image (first image), a "next cut-out image" button for selecting and displaying the next cut-out image, and a "previous cut-out image" button for selecting and displaying the previous cut-out image. The cut-out image can be selected, for example, as follows.

[0124] The distances from the origin of the two index values ​​shown in FIG. 35, which are the brightness variance and the frequency variance, can be calculated, and the images can be selected and displayed in order of the longest distance. That is, each time the "next cutout image" button is operated, the cutout images are displayed in order of decreasing distance. In the example of FIG. 35, the cutout images are selected in the direction in which the brightness variance and the frequency variance become smaller. Also, each time the "previous cutout image" button is operated, the cutout images are displayed in order of increasing distance. In the example of FIG. 35, the cutout images are selected in the direction in which the brightness variance and the frequency variance become larger. This allows the user to easily select a cutout image suitable as the learning data desired by the user from among many cutout image candidates, improving user convenience.

[0125] In the above example, the first image suitable as learning data is selected using the luminance variance and frequency variance, but the present invention is not limited to this. For example, the quality of the learning result using the first image selected by the user may be accumulated as data, and the first image suitable as learning data may be selected using the data with good learning results as teacher data. If the data indicating the quality of the learning result can be collected, for example, by a server, it is possible to construct the learning data on the server side. More specifically, the processor collects learning result data learned by the learning model based on a plurality of first images selected by the user (for example, the first image and the accuracy rate may be recorded and collected in association with each other). The processor may accumulate the collected learning result data, and select data with good learning results (for example, data with an accuracy rate equal to or higher than a predetermined threshold) from the accumulated learning result data as the first image.

[0126] FIG. 38 is a flowchart showing an example of a learning process procedure using a three-value label image by the image identification device 100. In the following, for convenience, the main subject of the process will be described as the control unit 10. The control unit 10 acquires an original image in which particles are photographed by a microscope (S61), and calculates the luminance distribution and frequency distribution of the rectangular area while scanning the acquired original image in a rectangular area (S62). The control unit 10 selects an input image for learning based on the calculated luminance distribution and frequency distribution (at least one of them may be used) (S63). The control unit 10 acquires a binarized label image (S64). Note that a large amount of data of the input image for learning and the label image can be prepared. The control unit 10 generates a three-value label image to which the boundary part of the particle is added by accepting an operation to add a boundary to the binarized label image (S65). The control unit 10 acquires an image for evaluation (S66) and acquires a model file (S67).

[0127] The control unit 10 sets parameters (number of trials, number of mini-batches, number of trials to perform testing, learning rate, etc.) (S68) and performs a learning process (S69). The control unit 10 outputs an evaluation result in which an evaluation image is input into the generated model file (S70). The control unit 10 determines whether or not to correct the label image (S71). The determination of whether or not to correct the label image can be made, for example, based on whether or not the user has performed an editing operation to correct the label image.

[0128] If the label image is to be corrected (YES in S71), the control unit 10 displays the label image to be corrected as a mask image, superimposed on the learning input image (S72), corrects the label image based on the editing operation (S73), and performs the process of step S74 described below. If the label image is not to be corrected (NO in S71), the control unit 10 determines whether or not to re-learn (S74). Whether or not to re-learn is determined by the user based on the evaluation result using the evaluation image, for example, and can be performed depending on whether or not the user performs a re-learning operation.

[0129] If re-learning is to be performed (YES in S74), the control unit 10 continues the processing from step S69 onwards. If re-learning is not to be performed (NO in S74), the control unit 10 judges whether or not to generate another model file (S75). If another model file is to be generated (YES in S75), the control unit 10 continues the processing from step S61 onwards, and if another model file is not to be generated (NO in S75), the control unit 10 saves the generated model file (S76) and ends the processing.

[0130] 39 is a flowchart showing another example of the particle detection processing procedure by the image recognition device 100. The control unit 10 acquires an input image (S81), and performs particle detection processing using the generated learning model (detection model file) (S82). The control unit 10 generates and displays an output image based on the detection result of the learning model and the input image (S83). Here, the output image allows particles, boundaries, and background to be individually identified.

[0131] The control unit 10 judges whether or not a display operation of the particle distribution information has been performed (S84). The distribution information is distribution information of particle attributes such as particle size, for example, distribution information of maximum Feret diameter, minimum Feret diameter, and area diameter. If a display operation has been performed (YES in S84), the control unit 10 displays the particle distribution information (S85) and performs the process of step S86 described later.

[0132] If there is no display operation (NO in S84), the control unit 10 judges whether or not to correct the detection result (S86). The judgment of whether or not to correct the detection result can be made, for example, based on whether or not the user has performed an editing operation to correct the detection result. If the detection result is to be corrected (YES in S86), the control unit 10 displays the label image as a mask image superimposed on the input image (S87). Here, the mask image can have parts corresponding to particles transparent and parts corresponding to the background and boundaries semi-transparent. The control unit 10 corrects the label image based on the editing operation (S88).

[0133] The control unit 10 stores the corrected label image (S89) and ends the process. If the detection result is not to be corrected (NO in S86), the control unit 10 ends the process.

[0134] Fig. 40 is a block diagram showing another example of the configuration of the image identification device 100 of this embodiment. In Fig. 40, reference numeral 120 denotes a computer. The computer 120 includes a control unit 121, an input unit 126, an operation unit 127, a display panel 128, an output unit 129, a recording medium reading unit 130, etc. The control unit 121 includes a CPU (processor) 122, a ROM 123, a RAM 124, an I / F (interface) 125, etc.

[0135] The input unit 126, the operation unit 127, the display panel 128, and the output unit 129 correspond to the input unit 11, the operation unit 12, the display panel 17, and the output processing unit 16 shown in Fig. 1, respectively. The control unit 121 corresponds to the detection processing unit 13, the classification processing unit 14, the editing processing unit 15, the learning processing unit 20, and the storage unit 30 shown in Fig. 1.

[0136] The recording medium reading unit 130 is capable of reading a computer program (e.g., the processing procedures shown in Figures 27, 28, 29, 38, and 39) from a recording medium M (e.g., a medium such as a DVD) on which a computer program is recorded.

[0137] Although not shown, the computer program recorded on the recording medium M is not limited to one recorded on a medium that is easily portable, but may also include a computer program transmitted via the Internet or other communication lines.

[0138] Next, the evaluation results when the labeling of the label images is changed will be described.

[0139] Figures 41A to 41E are schematic diagrams showing an example of an evaluation result when particles and background are binarized. Figure 41A shows an input image, Figure 41B shows a labeled image, and Figures 41C to 41E show output images in different trials. In Figure 41B, the part indicated by a circle is labeled as the background at the boundary where two particles overlap in order to teach the system that the particles are separated, but in the output image, as in the parts indicated by a circle in Figures 41C to 41E, the particles are not clearly separated and the boundary may be unclear.

[0140] Fig. 42A to 42E are schematic diagrams showing an example of an evaluation result when binarizing particles, boundaries, and backgrounds. Fig. 42A shows an input image, Fig. 42B shows a labeled image, and Fig. 42C to 42E show output images in different trials. In Fig. 42B, the areas shown by circles were labeled and learned as the boundaries being the background in order to teach the particles that they are separated, but as in the areas shown by circles in Fig. 42C to 42E, the particle identification results may not be stable.

[0141] 43A to 43E are schematic diagrams showing an example of an evaluation result when binarizing boundaries and non-boundaries. FIG. 43A shows an input image, FIG. 43B shows a label image, and FIG. 43C to 43E show output images in different trials. In FIG. 43B, the parts shown by circles were labeled and learned to learn that they are boundaries, but as shown by circles in FIG. 43C to 43E, there are some breaks in the traces, which may make it difficult to distinguish between particles and the background. However, it may be useful when it is desired to distinguish between boundaries between different materials, rather than between particles and non-particles, as in the case of an image of a sliced ​​surface of a mixed material.

[0142] 44A to 44E are schematic diagrams showing an example of an evaluation result when particles, boundaries, and background are ternarized. FIG. 44A shows an input image, FIG. 44B shows a label image, and FIG. 44C to 44E show output images in different trials. In FIG. 44B, the parts indicated by circles are labeled as boundaries where the boundaries are unclear due to overlapping of particles, etc., and the learning model is trained by learning the parts, so that the learning model can learn the parts where the boundaries are unclear as boundaries. This makes it possible to separate particles at the boundaries, as shown by circles in FIG. 44C to 44E, and it can be seen that the accuracy of particle detection is improved.

[0143] Next, the detection result of the object to be identified will be described.

[0144] 45A-45B are schematic diagrams showing an example of the detection result of okonomiyaki flour. Fig. 45A shows the input image, and Fig. 45B shows the output image. In this case, the label image is a binary image. It can be seen that even if the identification target is an ingredient with a variety of sizes and shapes, such as okonomiyaki flour, the identification target can be detected with high accuracy.

[0145] 46A-46B are schematic diagrams showing an example of the detection result of baking soda. Fig. 46A shows an input image, and Fig. 46B shows an output image. In this case, the label image is a binary image. It can be seen that even if the material to be identified is a mixture of various sizes and shapes, such as baking soda, the identification target can be detected with high accuracy.

[0146] 47A-47B are schematic diagrams showing an example of a detection result of potato starch. Fig. 47A shows an input image, and Fig. 47B shows an output image. In this case, the label image is a binary image. It can be seen that even if the identification target is a material with densely packed fine particles such as potato starch, the identification target can be detected with high accuracy.

[0147] Figures 48A and 48B are schematic diagrams showing an example of the detection result of table salt. Figure 48A shows the input image, and Figure 48B shows the output image. In this case, the label image is a binary image. It can be seen that even if the identification target is a material with large particles sparsely distributed such as table salt, the identification target can be detected with high accuracy.

[0148] 49A-49B are schematic diagrams showing an example of the detection result of a moth-eye film. Fig. 49A shows an input image, and Fig. 49B shows an output image. In this case, the label image is a binary image. It can be seen that even if the identification target is a material such as a moth-eye film, on whose surface fine nano-projections smaller than the wavelength of light are formed, the identification target can be detected with high accuracy.

[0149] Figures 50A and 50B are schematic diagrams showing an example of the detection result of a lid material containing silica particles for food containers. Figure 50A shows an input image, and Figure 50B shows an output image. In this case, the label image is a binary image. It can be seen that even if the material to be identified is a lid material containing silica particles, in which minute shapes are dispersed, the identification object can be detected with high accuracy.

[0150] Next, a usage pattern of the image identification device 100 of this embodiment will be described.

[0151] FIG. 51 is a schematic diagram showing a first usage form of the image identification device 100. As shown in FIG. 51, a server and a client terminal are connected via a communication network. In FIG. 51, the server corresponds to the image identification device 100. As shown in FIG. 51, the client terminal transmits an image (input image) in which the identification target is unknown or the identification target is to be identified, to the server. The server identifies (detects) the identification target using a model file prepared in advance, and transmits an output image to the client terminal. The client terminal can display the output image. As a result, the client side can display the detection result simply by uploading the input image to the server. There is no need to have a learning model on the client terminal side.

[0152] FIG. 52 is a schematic diagram showing a second usage form of the image identification device 100. As shown in FIG. 52, a server and a client terminal are connected via a communication network. In FIG. 52, the server corresponds to the image identification device 100 having a learning processing function, and the client terminal corresponds to the image identification device 100 having a detection processing function and a classification processing function. As shown in FIG. 52, the client terminal transmits training data (learning input image and label image) to the server. The server receives the training data, generates a model file (learning model) using the received training data, and transmits the generated model file to the client terminal. The client terminal receives and stores the model file. After this, the client terminal inputs an image (input image) in which the identification target is unknown or the identification target is to be identified into the learning model, and identifies (detects) the identification target.

[0153] In addition, in the case of Figure 52, by editing the teacher data on the client terminal side (for example, Figures 8, 9, 33, and 34) and sending the edited teacher data to the server, the learning model can be re-learned (updated) on the server side, and the re-learned model file can be sent to the client terminal.

[0154] FIG. 53 is a schematic diagram showing a third usage form of the image identification device 100. As shown in FIG. 53, a server and a client terminal are connected via a communication network. In FIG. 53, the server corresponds to the image identification device 100 having a learning processing function, a detection processing function, and a classification processing function. The learning processing function, the detection processing function, and the classification processing function may be shared by two or more servers, or one server may have all the functions of the learning processing function, the detection processing function, and the classification processing function. As shown in FIG. 53, the client terminal transmits training data (learning input image and label image) to the server. The server receives the training data and generates a model file (learning model) using the received training data. Next, the client terminal transmits an image (input image) in which the identification target is unknown or the identification target is to be identified to the server. The server uses the generated model file to identify (detect) the identification target, and transmits an output image to the client terminal. The client terminal can display the output image.

[0155] In addition, in the case of Figure 53, the teacher data can be edited on the client terminal side (for example, Figures 8, 9, 33, and 34) and the edited teacher data can be sent to the server, allowing the learning model to be re-learned (updated) on the server side.

[0156] The image identification device 100 of this embodiment can be configured with a CPU, a GPU, a ROM, a RAM, a recording medium reading unit, and the like. A computer program recorded on a recording medium can be read by the recording medium reading unit and stored in the RAM. The computer program stored in the RAM can be executed by the CPU and the GPU, thereby executing the processing performed by the image identification device 100. The CPU is a general term for a processor, a processor device, a processing unit, and the like, and can be configured with one or more. In addition to the CPU, one or more GPUs may be provided. Instead of being configured to be read by the recording medium reading unit, the computer program can also be downloaded via a network such as the Internet.

[0157] According to the present embodiment, it is possible to easily detect or classify particles, even if they have complex and irregular shapes, by preventing the detection or classification of particles from being arbitrarily determined based on the observer's observation purpose.

[0158] According to this embodiment, as long as the particles can be imaged by a microscope such as a scanning electron microscope, the particles can be detected from the image and powder properties such as particle size can be measured without using expensive equipment. Moreover, since the error from the actual particle size is determined by the resolution of the microscope, the particle size can be measured with high accuracy within the allowable error range regardless of the microscope used.

[0159] In the image identification device 100 of this embodiment, the classification parameter learning unit 23, the classification processing unit 14, and the classification model file 32 are not essential components. That is, the image identification device 100 may be configured not to classify particles. Also, the image identification device 100 of this embodiment may not be configured to include the learning processing unit 20. In this case, the detection model file 31 and the classification model file 32 generated by another device may be stored in the storage unit 30, so that only the detection or classification processing of particles is performed.

[0160] In this embodiment, it is possible to specify an area not to be used for learning (outside the learning range), and it is possible to specify an area of ​​the image for learning to be studied (sampled) with emphasis. In this case, the part studied with emphasis may be displayed using a heat map or the like. This can improve the efficiency of learning.

[0161] In this embodiment, when the evaluation image has correct answer data, the parts where the classification result of the learning model is correct and the parts where the classification result is incorrect can be displayed in different display modes. Also, a threshold value for the accuracy rate may be set, and the areas where the accuracy rate is equal to or higher than the threshold value and the areas where the accuracy rate is lower than the threshold value may be displayed in different display modes.

[0162] The learning model of this embodiment can use a neural network configuration for detecting objects from an image. For example, a convolutional neural network (CNN) or a U-Net, which is a type of encoder-decoder type fully convolutional network (FCN), can be used, but in this embodiment, a simpler neural network configuration can be used.

[0163] The computer program of this embodiment causes a computer to execute a process of acquiring a first image in which an object to be identified is captured, a process of acquiring a second image in which the object to be identified is identified by at least binary labels corresponding to pixels of the first image, and a process of generating a learning model for identifying the object to be identified based on at least the acquired first and second images.

[0164] The learning model generation device of this embodiment includes a processor that acquires a first image in which an object to be identified is captured, acquires a second image in which the object to be identified is identified by at least binary labels corresponding to pixels of the first image, and generates a learning model for identifying the object to be identified based on at least the acquired first and second images.

[0165] The learning model generation method of this embodiment acquires a first image in which an object to be identified is captured, acquires a second image in which the object to be identified is identified by at least binary labels corresponding to pixels in the first image, and generates a learning model for identifying the object to be identified based on at least the acquired first and second images.

[0166] The first image is an input image for learning in which an object to be identified is captured. The object to be identified may be, for example, a powder or a surface pattern of a substance such as a polymer. The second image is a label image (also called a ground truth image) and corresponds to a pixel constituting the first image to specify whether the pixel is an object to be identified or not. The label is at least binary, and for example, label=0 can be used to indicate that the pixel is not an object to be identified (outside the object to be identified) and label=1 can be used to indicate that the pixel is an object to be identified (inside the object to be identified).

[0167] The learning model is composed of a neural network structure including an input layer, an output layer, and multiple intermediate layers, and weight coefficients of each neuron, etc. By inputting at least the first image and the second image into the learning model, the weight coefficients of the neurons are recalculated to generate the learning model.

[0168] With the above-described configuration, when an input image with an unknown classification target is input to the learning model, the learning model identifies whether each pixel in the input image is a classification target or not, making it easy to detect the classification target in the input image.

[0169] In the learning model generation device of this embodiment, the processor displays a second image output by the learning model based on a specified first image, accepts editing operations on the displayed second image, and updates the learning model based on at least the specified first image and the second image for which the editing operations have been accepted.

[0170] The predetermined first image may be the first image input when training the learning model, or may be a completely different image (e.g., an image not used in training the learning model). The editing operation may be, for example, an operation to correct the luminance value when the luminance value of the pixel of the second image output by the learning model differs according to the label value. That is, the label of the second image can be corrected by the editing operation, and when some pixels of the second image are not correctly labeled, the label value can be corrected to make it a correctly labeled image.

[0171] By inputting at least the predetermined first image and the second image that has received the editing operation (ie, the second image that has become correct) to the learning model, the weight coefficients of the neurons are recalculated and the learning model is updated.

[0172] With the above-described configuration, the learning model can be retrained, and the accuracy of detection of the object to be identified by the learning model can be improved.

[0173] In the learning model generation device of this embodiment, the processor replaces pixels corresponding to a specific label in the second image with pixels of the specified first image corresponding to the pixels, and displays the second image in which the pixels have been replaced.

[0174] Each pixel of the second image corresponds to at least a binary label that corresponds to a specific pixel of the first image. The specific label can be one of the binary labels. A pixel corresponding to a specific label in the second image is replaced with a pixel of the first image that corresponds to the pixel. By displaying the second image in which the pixels have been replaced, information on the pixels of the original first image (e.g., the distribution of luminance values, which are pixel values, on the second image) can be referenced, making it easier to correct the learning results.

[0175] In the learning model generating device of this embodiment, the processor acquires a second image in which an object to be classified is identified by at least three-value labels corresponding to pixels of the first image.

[0176] The second image can identify the object to be identified by at least three-value labels corresponding to the pixels constituting the first image. By increasing the number of labels from two values, the amount of information for identifying the object to be identified can be increased, making it easier for the learning model to identify the object to be identified, and improving the detection accuracy of the object to be identified. In particular, while adjacent pixel values ​​often change relatively gently inside a particle, the boundary is the boundary between the particle and the background, and is the point where the pixel value representing the particle and the pixel value representing the background change, and adjacent pixel values ​​often change sharply. By learning each of these as a separate label, the detection accuracy of the particle can be improved.

[0177] In the learning model generation device of this embodiment, the processor acquires a second image that identifies the object to be identified using labels corresponding to pixels of the first image, the labels including a first label indicating that the area is not an area to be identified, a second label indicating that the area includes the boundary of the area to be identified, and a third label indicating that the area is an area to be identified.

[0178] The second image includes labels corresponding to the pixels of the first image, the first label indicating that the area is not a target area, the second label indicating that the area includes a boundary of the target area, and the third label indicating that the area is a target area. For example, even boundary parts where it is unclear whether the area is a target area or not are also labeled, so that the target area and the outside of the area can be clearly distinguished by the boundary parts, making it easier for the learning model to identify the target area, and improving the detection accuracy of the target area.

[0179] In the learning model generation device of this embodiment, the processor acquires a second image that identifies the object to be identified using labels that correspond to pixels of the first image and include at least two types of third labels that indicate an area to be identified.

[0180] The second image includes labels corresponding to the pixels of the first image, and indicates that the area is an object of identification by at least two types of third labels. For example, even in an area to be identified, the luminance value and distribution of luminance values ​​within the area may differ depending on the type of object to be identified. By including at least two types of third labels in the second image, the learning model can learn the changes in luminance values ​​within the area and the differences in the distribution of luminance values, and the detection accuracy of the object to be identified can be improved regardless of the type of object to be identified.

[0181] In the learning model generation device of this embodiment, the processor acquires a second image that identifies the type of the object to be classified based on the shape of the object to be classified, which is an area of ​​the object to be classified, using at least three-value labels corresponding to the pixels of the first image.

[0182] The second image includes labels corresponding to the pixels of the first image, and indicates the type of the object based on the shape of the object by at least a ternary label, which is the area of ​​the object to be classified. This makes it possible to identify the shape of the object even if the object has a complex and irregular shape, and to classify the object based on its shape.

[0183] In the learning model generating device of this embodiment, the processor acquires a second image that is identified as being outside the learning range by a label associated with a pixel of the first image.

[0184] The second image includes a label corresponding to the pixel of the first image, and the label indicates that the pixel is outside the learning range. This reduces the amount of learning for the learning model, and improves the learning efficiency.

[0185] In the learning model generation device of this embodiment, the processor performs a process of acquiring a third image in which the object to be classified is identified, a process of acquiring a fourth image in which the type of the object to be classified is identified based on the shape of the object to be classified using labels corresponding to the pixels of the third image, and generates a learning model for identifying the type of the object to be classified based on at least the acquired third and fourth images.

[0186] The third image is an image in which the object to be identified has been identified. If a learning model for identifying the object to be identified is called a detection model, the third image is an image based on the identification result output by the detection model when an input image in which the object to be identified is unknown is input to the detection model. The third image may also be a learning input image in which a plurality of objects to be identified with different shapes are captured.

[0187] The fourth image is a label image (also called a ground truth image), which corresponds to the pixels constituting the third image and identifies what type of classification object the pixel is. The label can be anything that can indicate the type of classification object based on the shape of the classification object. If the type of classification object is known, the classification object can be classified.

[0188] By inputting at least the third and fourth images to the learning model, the weight coefficients of the neurons are recalculated to generate a learning model. This learning model is also called a classification model. As a result, when an input image with an unknown object to be identified is input to the learning model, the learning model identifies the shape of the object to be identified in the input image, so that the object to be identified in the input image can be easily classified.

[0189] The computer program of this embodiment causes a computer to execute the following processes: acquiring an input image; inputting the acquired input image into a learning model generated based on a first image in which at least an object to be identified is captured and a second image in which the object to be identified is identified by at least binary labels corresponding to pixels of the first image; and displaying an output image including multiple objects to be identified based on the recognition results output by the learning model.

[0190] The display device of this embodiment includes a processor that acquires an input image, inputs the acquired input image into a learning model generated based on a first image in which at least an object to be identified is captured, and a second image in which the object to be identified is identified by at least a binary label corresponding to the pixels of the first image, and displays an output image including a plurality of objects to be identified based on the recognition result output by the learning model.

[0191] The display method of this embodiment acquires an input image, inputs the acquired input image into a learning model generated based on a first image in which at least an object to be identified is captured, and a second image in which the object to be identified is identified by at least a binary label corresponding to the pixels of the first image, and displays an output image including multiple objects to be identified based on the recognition results output by the learning model.

[0192] The input image is an image in which the object to be identified is unknown. The learning model is generated based on at least a first image (learning input image) in which the object to be identified is captured, and a second image (label image) in which the object to be identified is identified by at least a binary label corresponding to the pixels of the first image. The learning model is also called a detection model.

[0193] When an input image is input to the learning model, the learning model outputs a classification result. The classification result can be, for example, information indicating whether or not each pixel of the input image is a classification target. By superimposing the input image and the classification result, an output image including multiple classification targets can be displayed.

[0194] As a result, when an input image with an unknown object to be classified is input into the learning model, the learning model identifies whether each pixel in the input image is an object to be classified or not, making it easy to detect the object to be classified in the input image even if the object to be classified has a complex or irregular shape.

[0195] In the display device of this embodiment, the processor inputs the acquired input image to a learning model generated based on at least a third image in which the object to be classified is identified, and a fourth image that identifies the type of the object to be classified based on the shape of the object to be classified with labels corresponding to the pixels of the third image, and displays the type of the object to be classified in an identifiable manner based on the shape of the object to be classified in the output image based on the classification result output by the learning model.

[0196] The learning model is generated based on at least a third image (learning input image) in which the classification object is classified, and a fourth image (label image) in which the type of the classification object is specified based on the shape of the classification object by labels corresponding to the pixels of the third image. The learning model is also called a classification model.

[0197] When an input image is input to a learning model, the learning model outputs a classification result. The classification result may be, for example, information indicating the type of classification object based on the shape of the classification object for each pixel of the input image.

[0198] As a result, when an input image with an unknown classification target is input into the learning model, the learning model identifies what type of classification target each pixel in the input image represents, making it easy to classify the classification target in the input image even if the shape of the classification target is complex or irregular.

[0199] In the display device of the present embodiment, the processor displays at least one of the major axis and the minor axis of the identification object of the output image.

[0200] At least one of the major axis and minor axis of the object to be identified is displayed, so that the size and shape of the object to be identified can be easily grasped.

[0201] In the display device of the present embodiment, the processor displays a line segment indicating a major axis and a line segment indicating a minor axis of the identification object of the output image.

[0202] Since a line segment indicating the major axis and a line segment indicating the minor axis of the identification object are displayed, the major axis direction and the minor axis direction of the identification object can be easily visually recognized.

[0203] In the display device of the present embodiment, the processor displays the ratio of the major axis to the minor axis of the identification object of the output image.

[0204] Since the ratio of the major axis to the minor axis of the identification object is displayed, the ratio of the major axis to the minor axis of the identification object can be easily visually recognized.

[0205] In the display device of the present embodiment, the processor accepts an operation for selecting a desired attribute for a plurality of classification targets in the output image, and displays distribution information of the selected attribute.

[0206] The attributes of the objects to be identified can be physical quantities that characterize the required properties of the objects to be identified depending on the field in which the objects to be identified are used. When a desired attribute is selected, distribution information of the selected attribute is displayed. In other words, it is possible to grasp the range over which the attributes of multiple objects to be identified are distributed, and therefore it is possible to easily grasp the properties of the objects to be identified.

[0207] In the display device of the present embodiment, the attributes include at least one of a major axis, a minor axis, and an area diameter of the object to be identified.

[0208] The attributes include at least one of the major axis, the minor axis, and the area diameter of the object to be identified. When the object to be identified is a powder, the shape characteristics of the powder can be easily grasped.

[0209] In the display device of the present embodiment, the processor displays all or some of the multiple classification targets in the output image in a identifiable manner.

[0210] Since all or some of the plurality of identification objects are displayed in an identifiable manner, the identification objects can be easily recognized.

[0211] In the display device of this embodiment, the processor displays the area to be identified.

[0212] Since the area of ​​the object to be identified is displayed, even if objects to be identified and non-objects are mixed, the range of the object to be identified can be easily grasped, which is useful for identifying the size and shape of the object to be identified.

[0213] In the display device of this embodiment, the processor displays the center of gravity of the identification object.

[0214] Since the center of gravity of the classification target is displayed, the position of each classification target can be easily grasped even when a plurality of classification targets are displayed.

[0215] In the display device of this embodiment, the processor displays adjacent classification objects in different display modes.

[0216] Since adjacent identification objects are displayed in different display modes, even if the identification objects are displayed together, they can be easily distinguished from one another.

[0217] In the display device of this embodiment, the processor displays a wave animation extending from the center of gravity of the object to be identified toward the boundary.

[0218] The system displays a wave animation from the center of gravity of the object to be identified toward the boundary, making it easy to grasp the position of the object to be identified and the range of its area.

[0219] The learning model generation device of this embodiment includes a processor that acquires a first image in which particles are photographed by a microscope, acquires a binary image corresponding to the pixels of the first image, generates a second image that identifies particles, boundaries and background with a ternary label to which particle boundaries are added for the acquired binary image, and generates a learning model that outputs an output image including particles, boundaries and background from an input image in which particles are photographed by a microscope based on the first image and the second image.

[0220] The computer program of this embodiment causes a computer to execute the following processes: acquiring a first image in which a particle is photographed by a microscope; acquiring a binary image corresponding to the pixels of the first image; generating a second image that identifies the particle, boundary, and background with a ternary label to which the particle boundary is added for the acquired binary image; and generating a learning model that outputs an output image including the particle, boundary, and background from an input image in which a particle is photographed by a microscope based on the first image and the second image.

[0221] The learning model generation method of this embodiment acquires a first image in which a particle is photographed by a microscope, acquires a binary image corresponding to the pixels of the first image, generates a second image that identifies the particle, boundary, and background with a ternary label to which the particle boundary is added for the acquired binary image, and generates a learning model that outputs an output image including the particle, boundary, and background from an input image in which the particle is photographed by the microscope based on the first and second images.

[0222] The first image is an input image for learning in which particles are captured by a microscope. The binary image is an image corresponding to the pixels of the first image, and can be, for example, a binarized image of particles and background. The second image is an image that identifies particles, boundaries, and background with ternary labels in which particle boundaries are added to the binary image.

[0223] The learning model is composed of a neural network structure including an input layer, an output layer, and multiple intermediate layers, as well as weight coefficients for each neuron. By inputting the first image and the second image, which is a ternary label image, into the learning model, the weight coefficients for the neurons are recalculated to generate the learning model.

[0224] With the above-described configuration, when an input image in which a particle is captured is input to the learning model, the learning model identifies whether each pixel in the input image is a particle, a boundary, or the background, so that the boundaries in the input image distinguish between particles and the background, enabling particles to be detected with high accuracy.

[0225] In the learning model generation device of this embodiment, the processor calculates at least one of the brightness variance and the frequency variance for each of multiple rectangular areas of an original image in which a particle is photographed by a microscope, and selects the first image based on the calculated at least one of the brightness variance and the frequency variance.

[0226] The processor calculates at least one of the brightness variance and the frequency variance for each of a plurality of rectangular regions of an original image in which particles are photographed by a microscope. The original image is an original image from which the first image is cut out, and has a size (resolution) larger than that of the first image. The rectangular regions are regions for determining a target region when scanning the original image, and can be the same size as the first image.

[0227] The luminance variance is the variance of the luminance values ​​of each pixel in a rectangular region, and for example, the variance value can be calculated by summing the squares of the differences between the luminance values ​​of each pixel in the rectangular region and the average luminance value of the rectangular region for each pixel, and dividing the sum by the number of pixels. The frequency variance is the variance of each frequency component of an image that has been frequency-converted by a Fourier transform or the like, and can be calculated in the same manner as the luminance variance.

[0228] The first image is selected based on at least one of the calculated luminance variance and frequency variance, thereby making it possible to select a first image in which an appropriate number of particles are present, are appropriately dense, and have a relatively large number of patterns and various characteristics, thereby making it possible to collect optimal learning data.

[0229] In the learning model generation device of this embodiment, the processor overlays a mask image on the first image, in which areas corresponding to particles are transparent and areas corresponding to boundaries and background are semi-transparent, accepts an editing operation to add boundaries to the mask image, and generates the second image with the accepted boundaries added.

[0230] The processor displays a mask image, which has regions corresponding to particles made transparent and regions corresponding to boundaries and background made semi-transparent, overlaid on the first image. This allows the particle parts of the first image to be displayed as is, and non-particle parts (boundaries and background) to be displayed semi-transparently, allowing the user to easily view the boundary parts.

[0231] The processor accepts an editing operation to add a boundary to the mask image. The editing operation may be, for example, an operation to fill in the boundary portion. The processor generates a second image to which the accepted boundary is added. This makes it possible to easily generate the second image, which is a ternary label image.

[0232] In the learning model generation device of this embodiment, the processor acquires an input image in which particles are photographed by a microscope, and a second image in which particles, boundaries, and background are identified by ternary labels in which particle boundaries are added to a binary image corresponding to the pixels of the input image, and re-learns the learning model based on the acquired input image and second image.

[0233] The processor acquires an input image of particles photographed by a microscope, and acquires a second image in which particles, boundaries, and a background are identified by ternary labels in which boundaries of particles are added to a binary image corresponding to pixels of the input image. The processor re-trains the learning model based on the acquired input image and second image.

[0234] With the above-described configuration, the learning model can be retrained, and the accuracy of particle detection by the learning model can be improved.

[0235] The particle identification device of this embodiment includes a processor, which acquires an input image of a particle photographed by a microscope, inputs the acquired input image into a learning model generated based on a first image of the particle photographed by the microscope and a second image in which particles, boundaries and background are identified by ternary labels in which particle boundaries are added to a binary image corresponding to the pixels of the first image, and outputs an output image including the particles, boundaries and background from the learning model.

[0236] The computer program of this embodiment causes a computer to execute the following processes: acquiring an input image of a particle photographed by a microscope; inputting the acquired input image into a learning model generated based on a first image of a particle photographed by a microscope and a second image in which particles, boundaries, and background are identified by ternary labels in which particle boundaries are added to a binary image corresponding to the pixels of the first image; and outputting an output image including the particles, boundaries, and background from the learning model.

[0237] The particle identification method of the present embodiment acquires an input image in which a particle is photographed by a microscope, inputs the acquired input image into a learning model generated based on a first image in which the particle is photographed by the microscope and a second image in which particles, boundaries and background are identified by ternary labels in which particle boundaries are added to a binary image corresponding to the pixels of the first image, and outputs an output image including the particles, boundaries and background from the learning model.

[0238] The input image is an image of particles captured by a microscope before the particles are detected. The learning model is generated based on a first image of particles captured by a microscope and a second image in which particles, boundaries, and background are identified by ternary labels in which boundaries of particles are added to binary images corresponding to pixels of the first image.

[0239] When an input image is input to the learning model, the learning model outputs an output image including particles, boundaries, and background. That is, the learning model can, for example, identify whether each pixel of the input image is a particle, a boundary, or a background. By overlapping the input image and the identification result, an output image including multiple particles can be output.

[0240] As a result, when an input image of a particle captured by a microscope is input into the learning model, the learning model identifies whether each pixel in the input image is a particle, a boundary, or the background. Therefore, even if the particle has a complex or irregular shape, the particle can be distinguished by its boundary, and particles in the input image can be detected with high accuracy.

[0241] In the particle identification device of this embodiment, the processor receives a second image in which particles, boundaries, and background are identified by ternary labels to which particle boundaries are added in a binary image corresponding to the pixels of the acquired input image, and outputs the acquired input image and the received second image.

[0242] The processor receives a second image in which particles, boundaries, and background are identified by ternary labels in which boundaries of particles are added to a binary image corresponding to pixels of the acquired input image. The processor outputs the acquired input image and the received second image. The output destination can be, for example, an external server or the like that generates or retrains a learning model based on the input image and the second image. This makes it possible to provide learning data for generating or retraining a learning model.

[0243] In the particle identification device of the present embodiment, the processor re-learns the learning model based on the acquired input image as the first image and the accepted second image.

[0244] The processor retrains the learning model based on the acquired input image (first image) and the accepted second image, thereby improving the particle detection accuracy of the learning model.

[0245] In the particle identification device of this embodiment, the processor overlays a mask image on the acquired input image, in which areas corresponding to particles are transparent and areas corresponding to boundaries and background are semi-transparent, accepts an editing operation to add boundaries to the mask image, and generates the second image with the accepted boundaries added.

[0246] The processor displays a mask image on the acquired input image, which has regions corresponding to particles made transparent and regions corresponding to boundaries and background made semi-transparent. This allows the particle parts of the input image to be displayed as is, and non-particle parts (boundaries and background) to be displayed semi-transparently, allowing the user to easily see the boundary parts.

[0247] The processor accepts an editing operation to add a boundary to the mask image. The editing operation may be, for example, an operation to fill in the boundary portion. The processor generates a second image to which the accepted boundary is added. This makes it possible to easily generate the second image, which is a ternary label image. [Explanation of symbols]

[0248] 100 Image Recognition Device 10 Control section 11 Input section 12 Control section 13 Detection processing section 14 Classification processing section 15 Editing section 16 Output Processing Section 17 Display Panel 20 Learning processing unit 21 Learning data input section 22 Detection parameter learning unit 23 Classification parameter learning unit 24 Model file output section 30 Storage section 31 Detection model files 32 Classification model files 33 Training Data

Claims

1. A processor is provided. The processor, A first image of an object to be identified is obtained; acquiring a second image in which an object to be identified is identified by at least a binary label corresponding to a pixel of the first image; generating a learning model for identifying an object to be identified based on at least the acquired first image and second image; replacing a pixel corresponding to a specific label in a second image output by the learning model based on a predetermined first image with a pixel in the first image corresponding to the pixel; Displaying the second image with the replaced pixels; Accepting an editing operation for the displayed second image; A learning model generation device that updates the learning model based on at least the specified first image and a second image that has received an editing operation.

2. The processor, The learning model generating device according to claim 1 , wherein the classification target is specified by at least three-value labels corresponding to pixels of the first image.

3. The processor, The learning model generation device according to claim 2, wherein a second image is acquired in which the object to be classified is identified using labels corresponding to pixels of the first image, the labels including a first label indicating that the area is not the object to be classified, a second label indicating that the area includes the boundary of the object to be classified, and a third label indicating that the area is the object to be classified.

4. The processor, The learning model generation device according to claim 3, further comprising: a second image that identifies an object to be identified using labels corresponding to pixels of the first image, the labels including at least two types of third labels that indicate an area to be identified.

5. The processor, The learning model generating device according to claim 2 , wherein a second image is acquired in which a region of the object to be classified is identified using at least three-value labels corresponding to pixels of the first image, and the type of the object to be classified is identified based on the shape of the object to be classified.

6. The processor, The learning model generating device according to claim 1 , wherein a second image is acquired that is identified as being outside a learning range by a label corresponding to a pixel of the first image.

7. The processor, obtaining a third image in which the classification object is classified; obtaining a fourth image for identifying a type of the object based on a shape of the object using labels corresponding to pixels of the third image; The learning model generating device according to claim 1 , further comprising: a learning model for identifying a type of an object to be identified based on at least the acquired third image and fourth image.

8. A processor is provided. The processor, Take an input image, A learning model generated based on a first image in which at least an object to be identified is captured, and a second image in which the object to be identified is identified by at least a binary label corresponding to a pixel of the first image, replacing a pixel corresponding to a specific label of a second image output by the learning model based on a predetermined first image with a pixel of the predetermined first image corresponding to the pixel, displaying a second image in which the pixel has been replaced, accepting an editing operation of the displayed second image, and inputting the acquired input image into the learning model updated based on at least the predetermined first image and the second image for which the editing operation has been accepted; A display device that displays an output image including a plurality of classification targets based on the classification results output by the learning model.

9. The processor, inputting the acquired input image into a learning model generated based on at least a third image in which the classification object is classified and a fourth image in which a type of the classification object is identified based on a shape of the classification object using a label corresponding to a pixel of the third image; The display device according to claim 8 , wherein the type of the classification object based on the shape of the classification object in the output image is displayed in a identifiable manner based on the classification result output by the learning model.

10. The processor, 10. The display device according to claim 8, wherein at least one of a major axis and a minor axis of the identification object of the output image is displayed.

11. The processor, The display device according to claim 8 , wherein a line segment indicating a major axis and a line segment indicating a minor axis of the identification object of the output image are displayed.

12. The processor, The display device according to claim 8 , further comprising a display device that displays a ratio of a major axis to a minor axis of the identification object of the output image.

13. The processor, Accepting an operation to select a desired attribute for a plurality of classification targets of the output image; The display device according to any one of claims 8 to 12, which displays distribution information of a selected attribute.

14. The display device according to claim 13 , wherein the attribute includes at least one of a major axis, a minor axis, and an area diameter of the object to be identified.

15. The processor, The display device according to claim 8 , wherein all or some of the plurality of classification targets of the output image are displayed in a identifiable manner.

16. The processor, The display device according to claim 15, wherein the display device displays the area to be identified.

17. The processor, 17. The display device according to claim 15, further comprising a display device that displays a center of gravity of the object to be identified.

18. The processor, The display device according to claim 15 , wherein adjacent objects to be identified are displayed in different display modes.

19. The processor, The display device according to claim 15 , wherein an animation of a wave motion from the center of gravity of the object to a boundary is displayed.

20. On the computer, A process of acquiring a first image in which an object to be identified is captured; obtaining a second image in which the object is identified by at least a binary label corresponding to pixels of the first image; A process of generating a learning model for identifying an object to be identified based on at least the acquired first image and second image; A process of replacing a pixel corresponding to a specific label of a second image output by the learning model based on a predetermined first image with a pixel of the first image corresponding to the pixel; displaying a second image with the replaced pixels; A process of accepting an editing operation for the displayed second image; updating the learning model based on at least the predetermined first image and a second image that has been subjected to an editing operation; A computer program that executes the following:

21. On the computer, obtaining an input image; A learning model generated based on a first image in which at least an object to be identified is captured, and a second image in which the object to be identified is identified by at least a binary label corresponding to a pixel of the first image, a process of replacing pixels corresponding to a specific label of a second image output by the learning model based on a predetermined first image with pixels of the predetermined first image corresponding to the pixels, displaying a second image in which the pixels have been replaced, accepting an editing operation of the displayed second image, and inputting the acquired input image into the learning model updated based on at least the predetermined first image and the second image for which the editing operation has been accepted; A process of displaying an output image including a plurality of classification targets based on the classification results output by the learning model. A computer program that executes the following:

22. A processor acquires a first image of an object to be identified; The processor acquires a second image in which the object is identified by at least a binary label corresponding to a pixel of the first image; The processor generates a learning model for identifying an object based on at least the acquired first image and the acquired second image; The processor replaces a pixel corresponding to a specific label in a second image output by the learning model based on a predetermined first image with a pixel in the first image corresponding to the pixel; The processor displays the second image with the replaced pixels; The processor accepts an editing operation for the displayed second image; A learning model generation method in which the processor updates the learning model based on at least the specified first image and a second image that has received an editing operation.

23. A processor acquires an input image; A learning model generated based on a first image in which at least an object to be identified is captured, and a second image in which the object to be identified is identified by at least a binary label corresponding to a pixel of the first image, The processor replaces a pixel corresponding to a specific label in a second image output by the learning model based on a predetermined first image with a pixel in the first image corresponding to the pixel; The processor displays the second image with the replaced pixels; The processor accepts an editing operation for the displayed second image; The processor inputs the acquired input image into the learning model updated based on at least the predetermined first image and the second image that has been edited; A display method in which the processor displays an output image including multiple classification targets based on the classification results output by the learning model.

24. A processor is provided. The processor, obtaining a first image of the particle by a microscope; obtaining a binary image corresponding to pixels of the first image; A second image is generated in which the grains, the boundaries, and the background are identified by a ternary label with the grain boundaries added to the acquired binary image; generating a learning model that outputs an output image including particles, boundaries, and a background from an input image of particles photographed by a microscope based on the first image and the second image; Calculating at least one of brightness variance and frequency variance for each of a plurality of rectangular regions in an original image of particles photographed by a microscope; A learning model generating device that selects the first image based on at least one of the calculated brightness variance and frequency variance.

25. A processor is provided. The processor, obtaining a first image of the particle by a microscope; obtaining a binary image corresponding to pixels of the first image; A second image is generated in which the grains, the boundaries, and the background are identified by a ternary label with the grain boundaries added to the acquired binary image; generating a learning model that outputs an output image including particles, boundaries, and a background from an input image of particles photographed by a microscope based on the first image and the second image; a mask image is superimposed on the first image, the mask image having transparent regions corresponding to particles and semi-transparent regions corresponding to boundaries and a background; accepting an editing operation to add a boundary to the mask image; A learning model generation device that generates the second image to which the accepted boundary is added.

26. The processor, a mask image is superimposed on the first image, the mask image having transparent regions corresponding to particles and semi-transparent regions corresponding to boundaries and a background; accepting an editing operation to add a boundary to the mask image; The learning model generation device according to claim 24, which generates the second image to which the accepted boundary is added.

27. The processor, Obtaining an input image of particles photographed by a microscope, and a second image in which particles, boundaries, and a background are identified by a ternary label in which boundaries of particles are added to a binary image corresponding to pixels of the input image; The learning model generating device according to claim 24 , further comprising: a processor configured to re-learn the learning model based on the acquired input image and the second image.

28. A processor is provided. The processor, Obtain an input image of a particle taken by a microscope; a first image of a particle photographed by a microscope, and a mask image, which is made transparent in an area corresponding to the particle and semi-transparent in an area corresponding to the boundary and background, are superimposed on the first image and displayed; an editing operation for adding a boundary to the mask image is received; and the acquired input image is input to a learning model generated based on a second image in which the particle, boundary, and background are identified by a ternary label to which the received particle boundary is added; A particle identification device that outputs an output image from the training model that includes particles, boundaries and background.

29. The processor, A second image is received in which particles, boundaries, and a background are identified by a ternary label in which boundaries of particles are added to a binary image corresponding to pixels of the acquired input image; 30. The apparatus of claim 28, further comprising: an input image acquisition unit configured to receive the second image and output the input image and the second image.

30. The processor, 30. The particle identification device according to claim 28 or 29, wherein the learning model is retrained based on the acquired input image as the first image and a received second image.

31. On the computer, obtaining a first image of the particle by a microscope; obtaining a binary image corresponding to pixels of the first image; a process of overlaying and displaying a mask image on the acquired binary image, in which regions corresponding to particles are made transparent and regions corresponding to boundaries and background are made semi-transparent, accepting an editing operation for adding boundaries to the mask image, and generating a second image that identifies the particles, boundaries, and background with ternary labels to which the accepted boundaries of the particles are added; A process of generating a learning model that outputs an output image including particles, boundaries, and a background from an input image in which particles are photographed by a microscope, based on the first image and the second image. A computer program that executes the following:

32. On the computer, obtaining an input image of a particle taken by a microscope; a process of displaying a first image of a particle photographed by a microscope and a mask image, which is superimposed on the first image and makes regions corresponding to the particle transparent and regions corresponding to the boundary and background semi-transparent, receiving an editing operation to add a boundary to the mask image, and inputting the acquired input image into a learning model generated based on a second image in which the particle, boundary, and background are identified by a ternary label to which the boundary of the received particle is added; outputting an output image from the learning model, the output image including particles, boundaries, and a background; A computer program that executes the following:

33. A processor acquires a first image of the particle taken by the microscope; The processor obtains a binary image corresponding to pixels of the first image; The processor displays a mask image on the acquired binary image, the mask image having transparent regions corresponding to particles and semi-transparent regions corresponding to boundaries and background; the processor accepts an editing operation to add a boundary to the mask image; generating a second image identifying the particles, the boundaries, and the background with ternary labels augmented with the boundaries of the accepted particles; A learning model generation method in which the processor generates a learning model that outputs an output image including particles, boundaries, and background from an input image in which particles are photographed by a microscope, based on the first image and the second image.

34. A processor acquires an input image of a particle captured by a microscope; The processor displays a first image of the particle captured by the microscope and a mask image superimposed on the first image, the mask image having transparent areas corresponding to the particle and semi-transparent areas corresponding to the boundary and background; the processor accepts an editing operation to add a boundary to the mask image; The processor inputs the acquired input image into a learning model generated based on a second image in which the particles, the boundaries, and the background are identified by ternary labels with the boundaries of the received particles; The method of particle identification, wherein the processor outputs an output image from the learning model that includes particles, boundaries and background.

Images