Analysis device, analysis method, and analysis program
The analysis device automates image processing by calculating local features and learning optimal methods based on their spatial positions, addressing inefficiencies in manual parameter adjustment and human intervention in microscopic image analysis.
Patent Information
- Application Number
- JP2021066803
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-04-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-04-09
AI Technical Summary
Existing image analysis methods for microscopic images require manual parameter adjustment and human intervention, making automation difficult due to image variability in sharpness and hue, leading to inefficiencies and increased man-hours.
An analysis device and method that calculates statistical local features, places them in a latent space, and learns image processing based on their positions to automate the analysis, using techniques like Neighborhood Gray Tone Difference Matrix, Gray Level Co-occurrence Matrix, and Gray Level Size Zone Matrix, with machine learning to determine optimal processing.
Automates image processing by eliminating individuality and reducing man-hours, ensuring consistent and efficient analysis of microscopic images.
Smart Images

Figure 0007593210000001 
Figure 0007593210000002 
Figure 0007593210000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an analysis device, an analysis method, and an analysis program. [Background technology]
[0002] Techniques for analyzing images captured by a microscope have been proposed (see Non-Patent Documents 1 and 2, etc.).
[0003] Non-Patent Document 1 proposes a technique for reducing noise in 3D CT images using wavelet transform and Sobel filter. Non-Patent Document 2 proposes a technique for extracting cracks from digital images using an interactive genetic algorithm. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Masaki Saito, Nobuyuki Nakamori, “Development of a Noise Reduction Method for Dental 3D CT Images Using Wavelet Transform and Sobel Filter”, [online], Internet<URL:http: / / www.fjt.info.gifu-u.ac.jp / mii-sci / pdfs / 24_1.pdf> [Non-Patent Document 2] Kei Kawamura, Fumio Miyamoto, Hideaki Nakamura, Ryo Sato, “Crack Extraction from Digital Images Using Interactive Genetic Algorithms”, [online], Internet<URL:http: / / library.jsce.or.jp / jsce / open / 00037 / 2003 / 742-0115.pdf> Summary of the Invention [Problem to be solved by the invention]
[0005] Deriving features (crystal grain size, distribution, etc.) from microscopic images required adjusting pre-processing parameters each time because there was variation in clarity and color between images, making it impossible to automate the analysis.
[0006] The technology disclosed in Non-Patent Document 1 requires manual advanced parameter adjustment and formula modification for each data, and the worker's desired noise reduction results depend heavily on the worker's intuition and the number of trial and error attempts. In the technology disclosed in Non-Patent Document 2, image processing parameters are automatically determined by a genetic algorithm, but a human being provides an evaluation value for each learning data on whether the processing matches the purpose, resulting in frequent human intervention.
[0007] The present invention has been made in consideration of the above-mentioned points, and has an object to provide an analysis device, an analysis method, and an analysis program that eliminate dependency on individuals and reduce the number of steps by automating processing. [Means for solving the problem]
[0008] An analysis device according to a first aspect of the present invention includes a feature calculation unit that calculates statistical local features of an entire image, a placement unit that arranges the local features calculated by the feature calculation unit in a latent space, a learning unit that learns to determine image processing according to positions of the local features arranged by the placement unit in the latent space, and an image processing unit that selects and executes image processing for a target image using a learning result by the learning unit.
[0009] In the analysis device according to the first aspect, a feature calculation unit calculates statistical local features of the entire image, an arrangement unit arranges the local features calculated by the feature calculation unit in a latent space, a learning unit learns to determine image processing according to the positions in the latent space of the local features arranged by the arrangement unit, and an image processing unit selects and executes image processing for a target image using a learning result by the learning unit. According to the analysis device according to the first aspect, by selecting and executing image processing for a target image using a learning result about image processing, it is possible to eliminate dependency on individual users and reduce man-hours by automating the processing.
[0010] An analysis device according to a second aspect of the present invention is the analysis device according to the first aspect, in which the image processing unit executes image processing such that local features of the image after image processing move to a range determined as a correct answer in the latent space. According to the analysis device according to the second aspect, correct image processing can be executed on the target image by executing image processing such that local features move to a range determined as a correct answer in the latent space.
[0011] An analysis device according to a third aspect of the present invention is the analysis device according to the first aspect, wherein the feature amount calculation unit calculates the local feature amount using one or more of a NeighborhoodGrayToneDifferenceMatrix (NGTDM), a GrayLevelCo-occurrenceMatrix (GLCM), and a GrayLevelSizeZoneMatrix (GLSZM). The analysis device according to the third aspect can appropriately calculate the local feature amount of an image to be analyzed.
[0012] In an analysis method according to a fourth aspect of the present invention, a processor calculates statistical local features of an entire image, arranges the calculated local features in a latent space, learns to determine image processing according to positions of the arranged local features in the latent space, and selects and executes image processing for a target image using a result of the learning.
[0013] According to the analysis method of the fourth aspect, the learning results about image processing are used to select and execute image processing on a target image, thereby eliminating dependency on individual users and reducing man-hours by automating the processing.
[0014] An analysis program according to a fifth aspect of the present invention causes a computer to execute processes of calculating statistical local features of an entire image, arranging the calculated local features in a latent space, learning to determine image processing according to positions of the arranged local features in the latent space, and selecting and executing image processing for a target image using a result of the learning.
[0015] According to the analysis program of the fifth aspect, the learning results about image processing are used to select and execute image processing on a target image, thereby eliminating dependency on individual users and reducing man-hours by automating the processing. Effect of the Invention
[0016] According to the present invention, it is possible to provide an analysis device, an analysis method, and an analysis program that eliminate dependency on individuals and reduce the number of steps by automating the process. [Brief description of the drawings]
[0017] [Figure 1] 1 is a diagram showing a schematic configuration of an analysis device according to an embodiment. [Diagram 2] FIG. 13 is a diagram illustrating an example of local features arranged in a latent space by an arrangement unit. [Diagram 3] 11 is a diagram illustrating a learning process performed by a learning unit. FIG. [Figure 4] 13 is a flowchart showing the flow of a learning process performed by the analysis device. [Diagram 5] 13 is a flowchart showing a flow of an analysis process performed by the analysis device. [Figure 6] 1 is a flowchart showing an example of a flow of analysis of an image captured by an electron microscope. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.
[0019] 1 is a diagram showing a schematic configuration of an analysis device according to the present embodiment. The analysis device 10 according to the present embodiment is a device for analyzing images, in particular, an image captured by an electron microscope.
[0020] Here, an example of the flow of analysis of an image captured by an electron microscope will be described. Fig. 6 is a flow chart showing an example of the flow of analysis of an image captured by an electron microscope. Analysis of an image captured by an electron microscope is performed, for example, as shown in Fig. 6, by carrying out preprocessing such as noise removal on the captured image (raw data) (step S11), binarizing the preprocessed image (step S12), analyzing the binarized image (step S13), and analyzing the analysis results according to needs (step S14).
[0021] The analysis of the binarized image and the analysis results for each need vary depending on the analysis method, and once the method is decided, there is no need to change it. On the other hand, the optimal method for preprocessing the raw data and binarizing the preprocessed image varies depending on the nature of the image to be analyzed. In other words, the optimal method varies depending on the device used for capturing the image, the object to be captured, the level of noise in the image, the color tone of the image, the level of image degradation, etc.
[0022] For example, even if the noise removal parameters are the same, the characteristics of the image after removal may differ depending on the image being analyzed. Consider the case of removing noise from an image by applying a Gaussian filter. If the size of the Gaussian filter is 5x5, the noise may be properly removed or may not be completely removed depending on the characteristics of the image being analyzed. Also, if the size of the Gaussian filter is 15x15, the effect may be too strong, causing contours to disappear or may not be completely removed depending on the characteristics of the image being analyzed. As such, the optimal image processing method differs depending on the characteristics of the image being analyzed, so in the past, it was necessary for people to repeatedly check the results of image processing and conduct trial and error.
[0023] There are techniques to roughly classify images based on the results of analyzing the images, such as UMAP (Uniform Manifold Approximation and Projection). However, there are many different images to analyze, and simply classifying images will be affected by the global characteristics of the image. There is also no clear guide to which technique should be applied to which image. There are many different types of noise in captured images, and even if the spatial coordinates of each type of noise can be identified, it is not clear which filter should be used for the image.
[0024] Therefore, the analysis device 10 according to the present embodiment calculates local features of an image, arranges the calculated local features in a latent space, and then calculates local features of an image after image processing and arranges the calculated local features in the latent space, thereby learning correct image processing parameters. The analysis device 10 according to the present embodiment is characterized in that by learning correct image processing parameters, it eliminates personal dependency and reduces man-hours by automating processing. Specifically, as described below, the analysis device 10 achieves automation of analysis processing by performing machine learning on the results of image processing and reflecting the results of machine learning in image processing.
[0025] The analysis device 10 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, and a display unit 16.
[0026] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program recorded in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an analysis program that executes image processing for analyzing an image.
[0027] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and stores various programs including an operating system, and various data.
[0028] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.
[0029] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.
[0030] When executing the above analysis program, analysis device 10 uses the above hardware resources to realize various functions. The functional configuration realized by analysis device 10 will be described.
[0031] As shown in FIG. 1, analysis device 10 functions such that CPU 11 has a feature calculation unit 101, a placement unit 102, a learning unit 103, and an image processing unit 104 by reading and executing an analysis program stored in ROM 12 or storage 14 by CPU 11.
[0032] The feature calculation unit 101 calculates local features of an image to be analyzed. The feature calculation unit 101 calculates local features of the image using at least one or more of the following analysis methods: Neighborhood Gray Tone Difference Matrix (NGTDM), Gray Level Co-occurrence Matrix (GLCM), Gray Level Size Zone Matrix (GLSZM), etc. GLCM is an analysis method that produces the frequency and distribution of each pixel value, and has the characteristic that pixels with low frequency are difficult to detect. NGTDM is an analysis method that looks at the difference with surrounding pixels, and has the characteristic that it has high detection power for pixel values that deviate from the overall distribution. GLSZM is an analysis method that evaluates the continuity of pixel values in the entire image. Of course, the analysis method used by the feature calculation unit 101 when calculating local features is not limited to the above-mentioned analysis methods. The feature calculation unit 101 may use existing pixel neighborhood matrix calculation algorithms such as NGTDM, GLCM, GLSZM, etc., or equivalents to these algorithms.
[0033] The arrangement unit 102 arranges the local features calculated by the feature calculation unit 101 in the latent space. When arranging the local features in the latent space, the arrangement unit 102 performs clustering using UMAP, for example. UMAP has the following characteristics: spatial coordinates are determined by global features and local features, it has high expressiveness, data can be added to a learned model, and it is unclear what the axes represent (the similarity of features becomes the distance).
[0034] Fig. 2 is a diagram showing an example of local features arranged in the latent space by the arrangement unit 102. As shown in Fig. 2, images similar to each type of noise are clustered at coordinates in the latent space. If it is assumed that the coordinates in the latent space are different for each type of noise in an image, it becomes possible to capture the coordinates in the latent space as shown in Fig. 2 as the state of the image.
[0035] The learning unit 103 performs a learning process on the latent space in which the local features are arranged by the arrangement unit 102. Specifically, the learning unit 103 learns appropriate image processing for each cluster in the latent space that is not a correct cluster. The correct cluster is, for example, a cluster in which clean images without noise are gathered. Which cluster is the correct cluster is specified by the user who performs the analysis.
[0036] The image processing unit 104 performs image processing on the image to be analyzed. The image processing unit 104 performs image processing on the image to be analyzed, such as processing to remove noise and processing to make the image clearer.
[0037] The learning unit 103 learns that, for an image that belongs to a cluster other than the correct cluster, if a local feature after image processing in the image processing unit 104 according to predetermined parameters is placed in a latent space and the local feature moves to a position to which the correct cluster belongs, then the image processing is appropriate.
[0038] On the other hand, for an image that belongs to a cluster that is not the correct cluster, if the local feature after image processing in the image processing unit 104 according to predetermined parameters is placed in the latent space, and the local feature does not move to a position to which the correct cluster belongs, the learning unit 103 learns that the image processing is not appropriate.
[0039] The learning unit 103 learns the change between coordinates in the latent space for each image processing. The image processing unit 104 can automatically select the optimal processing method or necessary parameters from the distance in the latent space coordinates between the input data and the target data by using the learning result of the learning unit 103.
[0040] Fig. 3 is a diagram for explaining the learning process by the learning unit 103. For example, consider the case where there are three clusters A, B, C, and D in the latent space, as shown in Fig. 3, and cluster D is the correct cluster.
[0041] In this case, suppose that an image whose local feature amount belongs to cluster A is subjected to image processing for flattening contrast and image processing for applying a Gaussian filter, and the local feature amount of the image after the image processing is placed again in the latent space, so that the image belongs to cluster D. The learning unit 103 learns that the image processing for flattening contrast and image processing for applying a Gaussian filter are correct image processing for an image whose local feature amount belongs to cluster A.
[0042] In this case, suppose that an image whose local feature amount belongs to cluster B is subjected to image processing for flattening contrast and image processing for applying a Gaussian filter, and the local feature amount of the image after the image processing is placed again in the latent space, so that the image now belongs to cluster A. The learning unit 103 learns that the image processing for flattening contrast and image processing for applying a Gaussian filter are not correct image processing for an image whose local feature amount belongs to cluster B.
[0043] The feature calculation unit 101 calculates the local feature of the analysis target, and the learning unit 103 repeats learning, so that the analysis device 10 can automatically determine appropriate image processing from the local feature of the image to be analyzed. By automatically determining appropriate image processing from the local feature of the image to be analyzed, the analysis device 10 can execute image processing that is free from personal dependency. Furthermore, by automatically determining appropriate image processing from the local feature of the image to be analyzed, the analysis device 10 can easily execute image processing on a large number of images, compared to a case where image processing is not automatically determined.
[0044] Next, the operation of the analysis device 10 will be described.
[0045] 4 is a flowchart showing the flow of the learning process by the analysis device 10. The CPU 11 reads out an analysis program from the ROM 12 or storage 14, loads it into the RAM 13, and executes it, thereby performing the learning process.
[0046] First, the CPU 11 calculates local feature amounts for the entire image to be analyzed (step S101). As described above, the feature amount calculation method includes NGTDM, GLCM, GLSZM, and the like.
[0047] Following step S101, the CPU 11 places the calculated local features in a latent space and clusters the local features (step S102).
[0048] Following step S102, the CPU 11 accepts the designation of a correct cluster from among the clustered features (step S103). The correct cluster is, for example, a cluster of clean images free of noise. Which cluster is the correct cluster is designated by the user who performs the analysis.
[0049] Following step S103, CPU 11 learns appropriate image processing for each cluster that is not the correct cluster, and generates a learned model (step S104). CPU 11 performs image processing on a cluster that is not the correct cluster, and if the feature amount after the image processing is arranged in the latent space and moves to a position to which the correct cluster belongs, CPU 11 determines that the image processing is appropriate.
[0050] The CPU 11 learns the changes between coordinates in the latent space for each image processing. The CPU 11 automatically selects the optimal processing method and necessary parameters based on the distance between the input data and the target data in the latent space coordinates. Even if the optimal method produces poor results, the CPU 11 can automatically adapt and reevaluate the method considered to be the next best.
[0051] 5 is a flowchart showing the flow of the analysis process by the analysis device 10. The analysis process is performed by the CPU 11 reading out the analysis program from the ROM 12 or the storage 14, loading it into the RAM 13, and executing it.
[0052] First, the CPU 11 calculates local feature amounts for the entire image to be analyzed (step S111). As described above, the feature amount calculation method includes NGTDM, GLCM, GLSZM, and the like.
[0053] Following step S111, the CPU 11 arranges the calculated local feature amounts of the entire image in a latent space (step S112).
[0054] Following step S112, CPU 11 executes image processing on the image to be analyzed according to the placement position in the latent space (step S113). The image processing executed in step S113 includes, for example, well-known image processing such as contrast flattening and Gaussian filtering. CPU 11 selects the image processing to be executed in step S113 based on the result of the learning process executed by CPU 11. When executing image processing, CPU 11 may select not only the type but also the strength of the processing.
[0055] Following step S113, CPU 11 determines whether the image processing for the image to be analyzed was appropriate (step S114). Specifically, CPU 11 calculates local features for the image after image processing, and, as a result of arranging the calculated local features in the same latent space, determines whether the image processing was appropriate based on whether the local features belong to the correct cluster.
[0056] If it is determined in step S114 that the image processing performed on the image to be analyzed was appropriate (step S114; Yes), CPU 11 ends the processing on that image. On the other hand, if it is determined in step S114 that the image processing performed on the image to be analyzed was inappropriate (step S114; No), CPU 11 selects another image processing and performs the image processing on the image to be analyzed (step S115). Selecting another image processing includes performing another image processing or changing the parameters of the image processing.
[0057] When the process of step S115 is completed, CPU 11 returns to the process of step S114 and repeats the image processing of the image to be analyzed until the image processing of the image to be analyzed is appropriate. Also, if it is determined in step S114 that the image processing of the image to be analyzed is not appropriate, CPU 11 updates the contents of the trained model generated as a result of the learning process.
[0058] CPU 11 calculates local feature amounts of an analysis target, and automatically selects and executes appropriate image processing based on the local feature amounts of an analysis target, thereby enabling analysis device 10 to execute image processing that is independent of individual users.
[0059] In the above embodiment, the analysis device 10 has been described as a standalone device, but the present invention is not limited to this example. For example, the functions of the CPU 11 may be executed in an external device such as a server.
[0060] In addition, the learning process and analysis process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. In addition, the learning process and analysis process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.
[0061] In addition, in each of the above embodiments, the learning process and analysis process programs are described as being stored (installed) in advance in a ROM or storage, but the present invention is not limited to this. The programs may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network. [Explanation of symbols]
[0062] 10 Analysis device 101 Feature Calculation Unit 102 Placement section 103 Learning Department 104 Image processing section
Claims
1. a feature amount calculation unit that calculates a statistical local feature amount of an entire image; an arrangement unit that arranges the local feature calculated by the feature calculation unit in a latent space; a learning unit that learns to determine image processing according to a position in the latent space of the local feature that is arranged by the arrangement unit; an image processing unit that selects and executes image processing on a target image using a learning result by the learning unit; An analysis device comprising:
2. The analysis device according to claim 1 , wherein the image processing unit executes image processing such that a local feature value of the image after the image processing moves into a range in the latent space that is determined as a correct answer.
3. The analysis device according to claim 1 , wherein the feature calculation unit calculates the local feature using one or more of a Neighborhood Gray Tone Difference Matrix (NGTDM), a Gray Level Co-occurrence Matrix (GLCM), and a Gray Level Size Zone Matrix (GLSZM).
4. The processor: Calculate statistical local features for the entire image, The calculated local feature amount is arranged in a latent space; learning to determine image processing according to positions of the arranged local features in the latent space; Using the results of the learning, select and execute image processing on the target image. Analysis method.
5. On the computer, Calculate statistical local features for the entire image, The calculated local feature amount is arranged in a latent space; learning to determine image processing according to positions of the arranged local features in the latent space; Using the results of the learning, select and execute image processing on the target image. An analysis program that performs the processing.
Citation Information
Patent Citations
Characteristic amount generation device, characteristic amount generation method, characteristic amount generation program, class discriminant device, class discriminant method and class discriminant program
JP2010271787A
Classifying apparatus, learning apparatus, classifying method, learning method, control program, and recording medium
JP2021039757A
