Information processing device, information processing method, and recording medium

Abstract

The present invention provides an information processing device, an information processing method and a recording medium that can easily judge the crystallinity of a material in an image captured by a transmission electron microscope (TEM). An acquisition unit (142) of a server (14) acquires an image captured by a transmission electron microscope. Next, a transformation unit (144) performs a two-dimensional Fourier transform on the image of each local area of ​​the acquired image. A clustering unit (146) clusters the frequency intensity obtained from the result of the two-dimensional Fourier transform based on the result of the two-dimensional Fourier transform of each local area. Then, a judgment unit (148) judges the regions of different crystallinity in the image based on the clustering result.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium having an information processing program recorded thereon. Background Art

[0002] A technique for analyzing images captured by a transmission electron microscope (TEM) (hereinafter referred to as "TEM images") has long been known. For example, HREMResearch Inc., "GPA for Digital Micrograph", <https: / / www.hremresearch.com / Eng / download / documents / gpa4dm.pdf> discloses a general image analysis technique for TEM images.

[0003] In addition, in "A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images" by Jacob Madsen et al., <https: / / onlinelibrary.wiley.com / doi / abs / 10.1002 / adts.201800037@10.1002 / (ISSN)25130390.Machine Learning>, a technology that uses deep neural networks to analyze the local structure of substances reflected in TEM images is disclosed. Summary of the invention

[0004] Problems to be solved by the invention

[0005] However, sometimes it is desirable to evaluate the crystallinity of a material as reflected in a TEM image. Figure 8 An example of a TEM image is shown in FIG. Figure 8 As shown, the portions like the patterns shown in the TEM images IM1 and IM2 are portions with high crystallinity, and the other portions are portions with low crystallinity.

[0006] In wanting to Figure 8 When evaluating the crystallinity of TEM images IM1 and IM2 as shown in FIG. Fig. 9As shown, it is considered to evaluate the crystallinity by the brightness value of the image.

[0007] However, if Fig. 9 As shown in FIG. 1 , there are many portions with approximately the same brightness in the portion L1 with high crystallinity and the portion L2 with low crystallinity in the TEM image IM, so that the crystallinity cannot be evaluated only by the brightness value of the image. When threshold processing is performed only on the brightness value of the TEM image IM, the following image is generated: Fig. 9 The image IMb shown. However, Fig. 9 Although only the portion with a higher brightness value is extracted in pixel units in the image IMb, the portion with different crystallinity is not extracted in region units.

[0008] Therefore, for example, only by image analysis such as that disclosed in "GPA for Digital Micrograph" by HREM Research Inc. mentioned above, it is impossible to determine regions with different crystallinity in a material reflected in a TEM image.

[0009] In addition to Jacob Madsen mentioned above, if you want to evaluate the crystallinity of the material reflected in the TEM image through a neural network such as the one disclosed in "A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images", you need to prepare a large amount of labeled learning data. The accuracy of the learning model of the neural network depends on the quality and quantity of the learning data. If you want to generate a learning model with good accuracy, you need to prepare a large amount of learning data, and such preparation will take a huge amount of time and effort.

[0010] Therefore, in addition to the above-mentioned HREM Research Inc., "GPA for Digital Micrograph", <https: / / www.hremresearch.com / Eng / download / documents / gpa4dm.pdf> and Jacob Madsen, the related technologies of "A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images" have the problem that the crystallinity of the material reflected in the TEM image cannot be simply judged.

[0011] The present disclosure takes the above-mentioned facts into consideration and simply determines the crystallinity of a material reflected in a TEM image.

[0012] Methods for solving problems

[0013] An information processing device of the first mode is an information processing device as follows, the information processing device includes a memory and a processor connected to the memory, wherein the processor is configured to obtain an image captured by a transmission electron microscope, perform a two-dimensional Fourier transform on the image of each local area of ​​the image, cluster frequency intensities obtained from the results of the two-dimensional Fourier transform based on the results of the two-dimensional Fourier transform of each local area, and judge regions of different crystallinity in the image based on the clustering results.

[0014] In the information processing device of the second mode, the processor is configured to cluster the plurality of local regions into a first cluster representing regions with higher crystallinity and a second cluster representing regions with lower crystallinity when performing the clustering. Thus, it is possible to determine whether each of the local regions is a region with higher crystallinity or a region with lower crystallinity.

[0015] In the information processing device of the third mode, the processor is configured to perform the two-dimensional Fourier transform for each of the local regions representing an image within a predetermined range centered on a central pixel when performing the two-dimensional Fourier transform, and when determining regions with different crystallinity in the image, determine whether the central pixel in the local region is a pixel with higher crystallinity based on the clustering result obtained based on the result of the two-dimensional Fourier transform calculated for each of the local regions, thereby determining whether the local region is a region with higher crystallinity. Thus, it is possible to determine whether each pixel in the image is a region with higher crystallinity or a region with lower crystallinity.

[0016] In the information processing device of the fourth aspect, the processor is configured to generate a crystallinity image showing the degree of crystallinity by assigning a first pixel value to a pixel at a position corresponding to the central pixel when the crystallinity is determined to be high, and assigning a second pixel value to a pixel at a position corresponding to the central pixel when the crystallinity is determined to be low, based on the judgment result of each of the central pixels. Thus, regions of different crystallinity of the material projected in the TEM image can be visualized.

[0017] In the fifth aspect of the information processing device, the processor is configured to, when performing the clustering, accumulate the frequency intensities obtained from the results of the two-dimensional Fourier transform and cluster the values ​​obtained by the accumulation. Thus, it is possible to determine regions with different crystallinity in consideration of the frequency intensities in the local regions.

[0018] In the information processing device of the sixth aspect, when performing the clustering, the processor compresses the high-dimensional vector, which is the value obtained by the accumulation, into a low-dimensional vector, which is a vector with a lower dimension than the high-dimensional vector, using a predetermined vector compression method, and clusters the compressed low-dimensional vector. Thus, since the amount of information is reduced, the processing required for the determination of crystallinity can be reduced.

[0019] In the seventh information processing method, the processor performs the following processing, namely: obtaining an image captured by a transmission electron microscope, performing a two-dimensional Fourier transform on the image of each local area of ​​the image, clustering the frequency intensity obtained from the result of the two-dimensional Fourier transform based on the result of the two-dimensional Fourier transform of each local area, and judging the areas with different crystallinity in the image based on the clustering result.

[0020] The recording medium of the eighth mode records an information processing program, which is used to cause the processor to perform the following processing, namely: obtain an image taken by a transmission electron microscope, perform a two-dimensional Fourier transform on the image of each local area in the obtained image, cluster the frequency intensity obtained from the result of the two-dimensional Fourier transform based on the result of the two-dimensional Fourier transform of each local area, and judge the areas with different crystallinity in the image based on the clustering result.

[0021] Effects of the Invention

[0022] As described above, according to the present disclosure, there is an effect that the crystallinity of a material reflected in a TEM image can be easily determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic block diagram of an information processing system according to an embodiment of the present invention.

[0024] Figure 2 A diagram for explaining a local area in a TEM image.

[0025] Figure 3 A diagram for explaining a method of obtaining frequency intensity from a result of two-dimensional Fourier transform.

[0026] Figure 4 A diagram illustrating clustering.

[0027] Figure 5 This is a diagram showing an example of a crystallinity image.

[0028] Figure 6 It is a diagram showing a configuration example of a user terminal and a server computer according to an embodiment.

[0029] Figure 7 This is a flowchart showing an example of information processing performed using a server according to an embodiment.

[0030] Figure 8 A diagram showing an example of a TEM image.

[0031] Fig. 9 This is a diagram showing an example of an image obtained by performing threshold processing on a TEM image. DETAILED DESCRIPTION

[0032] <Implementation Method>

[0033] Figure 11 is a block diagram showing an example of the functional structure of the information processing system 10 according to the embodiment. Figure 1 As shown, the information processing system 10 includes a user terminal 12 and a server 14 as an example of an information processing device. The user terminal 12 and the server 14 are connected via a network 13 such as the Internet.

[0034] The information processing system 10 involved in the embodiment analyzes the TEM image which is an image taken by a transmission electron microscope, and determines the crystallinity of the material reflected in the TEM image. Specifically, the server 14 of the information processing system 10 performs Fourier transform on each local area of ​​the TEM image sent from the user terminal 12, thereby obtaining the frequency intensity of each local area. In addition, the server 14 clusters the frequency intensity of each local area of ​​the TEM image, thereby determining the area with different crystallinity in the TEM image. Thus, the crystallinity of the material reflected in the TEM image can be easily determined.

[0035] The following is a specific description.

[0036] (User Terminal)

[0037] The user inputs a TEM image of a material to be analyzed to the user terminal 12 operated by the user. The TEM image is an image of a material captured by a transmission electron microscope.

[0038] The user terminal 12 transmits the TEM image input from the user to the server 14 via the network 13. The TEM image transmitted from the user terminal 12 is analyzed in the server 14 described later.

[0039] (server)

[0040] like Figure 1 As shown, the server 14 includes a transmitting and receiving unit 140 , an acquiring unit 142 , a converting unit 144 , a clustering unit 146 , a determining unit 148 , and an image generating unit 150 .

[0041] The transmitting and receiving unit 140 receives the TEM image transmitted from the user terminal 12 .

[0042] The acquisition unit 142 acquires the TEM image received by the transmission and reception unit 140 .

[0043] The transforming unit 144 performs a two-dimensional Fourier transform on the image of each local region of the TEM image acquired by the acquiring unit 142. Figure 2 , a diagram for explaining a local area in a TEM image is shown.

[0044] Specifically, if Figure 2 As shown, the conversion unit 144 sets a plurality of local regions R in the TEM image IM, the local region R representing an image within a predetermined range centered on the central pixel P. For example, the conversion unit 144 sets each of the local regions R by sequentially setting each pixel constituting the TEM image IM as the central pixel P. Furthermore, the conversion unit 144 performs a two-dimensional Fourier transform on each of the local regions.

[0045] The clustering unit 146 clusters the frequency intensities obtained from the results of the two-dimensional Fourier transform into two or more clusters using a known technique based on the results of the two-dimensional Fourier transform of each local area obtained by the transform unit 144. The frequency intensities are power spectra.

[0046] exist Figure 3 , a diagram is shown for explaining a method of obtaining frequency intensity from the result of a two-dimensional Fourier transform. Figure 3 As shown in FIG. 1 , a result If of a two-dimensional Fourier transform is obtained for a local area R. Each pixel value of the result If of the two-dimensional Fourier transform corresponds to the frequency intensity. In this case, Figure 3 As shown, the clustering unit 146 extracts the center x of the result If of the two-dimensional Fourier transform. f0 A straight line L is set toward the outside, and each of the frequency intensities located on the straight line L is obtained. Then, the clustering unit 146 generates a vector having the frequency intensities at the respective positions on the straight lines L as elements.

[0047] Next, the clustering unit 146 uses Figure 3 The straight line L shown in FIG. 1 is slightly rotated in the direction of the arrow mark r. Then, the clustering unit 146 obtains a vector whose elements are the frequency intensities located on the straight line L at this time. The clustering unit 146 obtains vectors representing the frequency intensities in various directions of the result If of the two-dimensional Fourier transform in this way, and accumulates these vectors to obtain a vector representing the frequency intensities in various directions of the result If of the two-dimensional Fourier transform. Figure 3 The high-dimensional vector of the accumulated value as shown in the graph Sp of the graph. In addition, the horizontal axis in the graph Sp represents the distance from the center x of the result If of the two-dimensional Fourier transform. f0 The vertical axis represents the frequency intensity S. Each value of the graph Sp becomes the value of each element of the high-dimensional vector. Therefore, a high-dimensional vector is obtained for a local area.

[0048] Next, the clustering unit 146 compresses the high-dimensional vector into a low-dimensional vector using principal component analysis, which is an example of a predetermined vector compression method. A low-dimensional vector is a vector with a lower dimension than a high-dimensional vector. Therefore, one local region is equivalent to one low-dimensional vector.

[0049] exist Figure 4 In FIG. 1 , a diagram for explaining clustering is shown. When the low-dimensional vector is a two-dimensional vector, a two-dimensional vector is equivalent to a local area, thus forming a graph such as Figure 4 The distribution of Q1 is shown.

[0050] For example, when clustering the low-dimensional vectors obtained by compression into two clusters, the clustering unit 146 clusters the low-dimensional vectors into a first cluster representing a region with high crystallinity and a second cluster representing a region with low crystallinity.

[0051] For example, in Figure 4 When the distribution of the two-dimensional vectors shown in Q1 is clustered into two clusters, a first cluster CL1 and a second cluster CL2 are obtained as clustering results based on the distribution of the two-dimensional vectors.

[0052] Figure 4 Q2 is a graph showing the density of the distribution of two-dimensional vectors, and its density is shown by contour lines. For example, C1 is the center of the first cluster corresponding to a local region with a higher degree of crystallinity, and C2 is the center of the second cluster corresponding to a local region with a lower degree of crystallinity.

[0053] The judging unit 148 judges the regions with different crystallinity in the TEM image based on the clustering result realized by the clustering unit 146. Specifically, the judging unit 148 judges whether the central pixel in the local region is a pixel with high crystallinity based on the clustering result. Thus, for each of the plurality of local regions, it is judged whether it is a region with high crystallinity.

[0054] For example, the determination unit 148 determines that the central pixel of the local area belonging to the first cluster is a pixel with high crystallinity. On the other hand, the determination unit 148 determines that the central pixel of the local area belonging to the second cluster is a pixel with low crystallinity.

[0055] The image generation unit 150 assigns a first pixel value (e.g., a pixel value representing white) to a pixel at a position corresponding to the central pixel when the central pixel is determined to have high crystallinity based on the respective determination results of the central pixels obtained by the determination unit 148. In addition, the image generation unit 150 assigns a second pixel value (e.g., a pixel value representing black) to a pixel at a position corresponding to the central pixel when the central pixel is determined to have low crystallinity based on the respective determination results of the central pixels obtained by the determination unit 148. In this way, the image generation unit 150 generates a crystallinity image representing the degree of crystallinity.

[0056] exist Figure 5, an example of a TEM image IM, a crystallinity image IMb generated by conventional techniques, and a crystallinity image IMc generated by processing of the server 14 of this embodiment is shown. The crystallinity image IMb generated by conventional techniques is an image obtained by performing threshold processing on the brightness value of the image and evaluating its crystallinity.

[0057] In the crystallinity image IMc generated by the processing of the server 14 of this embodiment, when the central pixel is determined to belong to the first cluster and has high crystallinity, white is assigned, and when the central pixel is determined to belong to the second cluster and has low crystallinity, black is assigned. Figure 5 As shown, the crystallinity image IMb obtained by the conventional method is an image obtained only by threshold processing based on pixel values, and although the portion with higher crystallinity is extracted in the form of pixel units, it is not extracted in the form of area units. It can be seen that the portion with higher crystallinity of the crystallinity image IMc is extracted in the form of area units with good accuracy.

[0058] The transmitting and receiving unit 140 transmits the crystallinity image generated by the image generating unit 150 to the user terminal 12.

[0059] The user terminal 12 receives the crystallinity image transmitted from the server 14. The user checks the crystallinity image displayed on the display unit of the user terminal 12.

[0060] The user terminal 12 and the server 14 can be connected by, for example, Figure 6 The computer 50 shown in FIG. 1 is implemented. The computer 50 that implements the user terminal 12 and the server 14 includes a CPU 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. In addition, the computer includes an input / output interface (I / F) 54 to which an input / output device or the like (not shown) is connected, and a read / write (R / W) unit 55 that controls the reading and writing of data relative to a recording medium 59. In addition, the computer includes a network I / F 56 connected to a network such as the Internet. The CPU 51, the memory 52, the storage unit 53, the input / output I / F 54, the R / W unit 55, and the network I / F 56 are connected to each other via a bus 57.

[0061] The storage unit 53 can be implemented by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. In the storage unit 53 as a storage medium, a program for making the computer function is stored. The CPU 51 reads the program from the storage unit 53 and expands it in the memory 52, thereby sequentially executing the processing of the program.

[0062] Next, the operation of the information processing system 10 according to the embodiment will be described.

[0063] When the TEM image is input to the user terminal 12, the user terminal 12 sends the TEM image to the server 14. When the server 14 receives the TEM image, the server 14 executes Figure 7 The information processing routine shown.

[0064] In step S100 , the CPU 51 functions as the acquisition unit 142 to acquire the TEM image received by the transmission / reception unit 140 .

[0065] In step S102, the CPU 51 as the conversion unit 144 sets a local area centered on the central pixel for the TEM image obtained in step S100. A plurality of local areas are set. Then, the CPU 51 as the conversion unit 144 performs a two-dimensional Fourier transform on the image of each of the plurality of local areas.

[0066] In step S104, the CPU 51 as the clustering unit 146 generates a high-dimensional vector having the frequency intensity obtained from the result of the two-dimensional Fourier transform as an element based on the result of the two-dimensional Fourier transform of each local area obtained in the above step S102. Specifically, the clustering unit 146 accumulates the vectors representing the frequency intensity in the local area and obtains the high-dimensional vector as described above. Figure 3 The high-dimensional vector representing the cumulative value is shown in the curve graph Sp. The high-dimensional vector is obtained for each local area.

[0067] In step S106 , the CPU 51 as the clustering unit 146 compresses the high-dimensional vector of each local region obtained in step S104 into a low-dimensional vector by using principal component analysis, thereby generating a low-dimensional vector for each local region.

[0068] In step S108 , the CPU 51 functions as the clustering unit 146 and clusters the low-dimensional vectors for each local region in step S106 .

[0069] In step S110 , the CPU 51 as the determination unit 148 determines regions having different crystallinity in the TEM image based on the clustering result obtained in step S108 .

[0070] In step S112, the CPU 51 as the image generating unit 150 assigns a first pixel value (e.g., a pixel value representing white) to the pixel at the position corresponding to the center pixel when the center pixel is determined to have a high crystallinity based on the determination result obtained in the above step S110. In addition, the CPU 51 as the image generating unit 150 assigns a second pixel value (e.g., a pixel value representing black) to the pixel at the position corresponding to the center pixel when the center pixel is determined to have a low crystallinity based on the determination result obtained in the above step S110. Thus, a crystalline image representing the degree of crystallinity is generated.

[0071] In step S114 , the CPU 51 functions as the transmission and reception unit 140 and transmits the crystalline image generated in step S112 as a result to the user terminal 12 .

[0072] As described above, the server 14 of the information processing system 10 involved in the embodiment performs a two-dimensional Fourier transform on each of the local areas of the TEM image, which is an image captured by a transmission electron microscope. The server 14 clusters the frequency intensity obtained from the result of the two-dimensional Fourier transform based on the result of the two-dimensional Fourier transform of each of the plurality of local areas. Moreover, the server 14 determines the regions with different crystallinity in the TEM image based on the clustering result. Thus, the crystallinity of the material reflected in the TEM image can be easily determined. More specifically, the regions with different crystallinity of the material reflected in the TEM image can be easily determined.

[0073] In addition, since there is no need to use a learning model such as a neural network, the crystallinity of each local area in the TEM image can be determined without preparing labeled learning data. In addition, the crystallinity of each local area in the TEM image can be determined without setting a threshold.

[0074] In addition, although the processing implemented by each device in the above-mentioned embodiment is described as software processing implemented by executing a program, it can also be set as processing implemented by hardware. Or, it can also be set as processing that combines both software and hardware. In addition, the following method can be adopted, that is, the program stored in the ROM is stored in various storage media and circulated.

[0075] In addition, the present disclosure is not limited to the above contents, and can of course be implemented with various modifications other than the above contents without departing from the gist of the present disclosure.

[0076] For example, although in the above-mentioned embodiment, the case where a plurality of local regions in a TEM image are clustered into a first cluster representing a region with higher crystallinity and a second cluster representing a region with lower crystallinity is described as an example, it is not limited thereto. The number of clusters used to judge regions with different crystallinity in a TEM image can be any number.

[0077] In addition, although the above-mentioned embodiment uses the case where the principal component analysis is used as a predetermined vector compression method as an example, it is not limited to this. Other compression methods may also be used. For example, UMAP (Uniform Manifold Approximation and Projection) which is a known compression method may also be used.

[0078] In addition, although the above embodiment uses the case of compressing a high-dimensional vector into a two-dimensional vector as an example, it is not limited to this, and a method of compressing a vector into more than two dimensions may also be adopted. Alternatively, clustering may also be performed without compressing the high-dimensional vector.

Claims

1. An information processing device comprising a memory and a processor connected to the memory, wherein: The processor is configured to: Obtain images taken using a transmission electron microscope. For each of the local regions of the image obtained, a two-dimensional Fourier transform is performed on the image of the local region, Based on the obtained result of the two-dimensional Fourier transform of each of the local regions, the frequency intensities obtained from the result of the two-dimensional Fourier transform are clustered, and Based on the clustering results, different crystallinity regions in the image are judged. When performing the two-dimensional Fourier transform, the two-dimensional Fourier transform is performed for each of the local areas representing an image within a predetermined range centered on a central pixel, When judging the regions with different crystallinity in the image, whether the central pixel in the local region is a pixel with higher crystallinity is judged according to the clustering result, thereby judging whether the local region is a region with higher crystallinity, wherein the clustering result is obtained based on the result of the two-dimensional Fourier transform calculated for each of the local regions.

2. The information processing device according to claim 1, wherein: The processor is configured to: When performing the clustering, the plurality of local regions are clustered into a first cluster indicating regions with high crystallinity and a second cluster indicating regions with low crystallinity.

3. The information processing device according to claim 1, wherein: The processor is configured to: Based on the judgment result of each of the central pixels, a first pixel value is assigned to the pixel at the position corresponding to the central pixel when it is judged that the crystallinity is high, and a second pixel value is assigned to the pixel at the position corresponding to the central pixel when it is judged that the crystallinity is low, thereby generating a crystallinity image showing the degree of crystallinity.

4. The information processing device according to any one of claims 1 to 3, wherein: The processor is configured to: When performing the clustering, the frequency intensities obtained from the results of the two-dimensional Fourier transform are accumulated, and the values ​​obtained by the accumulation are clustered.

5. The information processing device according to claim 4, wherein: When performing the clustering, the processor compresses the high-dimensional vector, which is the value obtained by the accumulation, into a low-dimensional vector, which is a vector with a lower dimension than the high-dimensional vector, using a predetermined vector compression method, and clusters the compressed low-dimensional vector.

6. An information processing method, wherein: The processor performs the following processing, namely: Obtain images taken using a transmission electron microscope. For each of the local regions of the image obtained, a two-dimensional Fourier transform is performed on the image of the local region, Based on the obtained result of the two-dimensional Fourier transform of each of the local regions, the frequency intensities obtained from the result of the two-dimensional Fourier transform are clustered, and Based on the clustering results, the regions with different crystallinity in the image are judged. When performing the two-dimensional Fourier transform, the two-dimensional Fourier transform is performed for each of the local areas representing an image within a predetermined range centered on a central pixel, When judging the regions with different crystallinity in the image, whether the central pixel in the local region is a pixel with higher crystallinity is judged based on the clustering result, thereby judging whether the local region is a region with higher crystallinity, wherein the clustering result is obtained based on the result of the two-dimensional Fourier transform calculated for each of the local regions.

7. A recording medium having an information processing program recorded thereon, wherein the information processing program is used to cause a processor to execute the following processing, namely: Obtain images taken using a transmission electron microscope. For each of the local regions of the image obtained, a two-dimensional Fourier transform is performed on the image of the local region, Based on the obtained result of the two-dimensional Fourier transform of each of the local regions, the frequency intensities obtained from the result of the two-dimensional Fourier transform are clustered, and Based on the clustering results, the regions with different crystallinity in the image are judged. When performing the two-dimensional Fourier transform, the two-dimensional Fourier transform is performed for each of the local areas representing an image within a predetermined range centered on a central pixel, When judging the regions with different crystallinity in the image, whether the central pixel in the local region is a pixel with higher crystallinity is judged according to the clustering result, thereby judging whether the local region is a region with higher crystallinity, wherein: The clustering result is obtained based on the result of the two-dimensional Fourier transform calculated for each of the local regions.

Images