Nanomaterial microstructure visual recognition method based on deep learning

Abstract

The method for visual recognition of nanomaterial micro-morphology based on deep learning belongs to the field of recognition technology and includes the following steps: S1: input nanomaterial morphology image file obtained by SEM, specifically, the morphology image file includes picture file and Meta metadata file; S2: pixel-level segmentation of picture file and key information extraction of Meta file; S3: select pre-trained model to perform pixel-level intelligent recognition of nanomaterial morphology image obtained by SEM; S4: record the result, and automatically convert the pixel-level result and the real size data; S5: merge the result of different nanomaterial morphology images obtained by SEM of the same sample. The present invention can automatically capture, analyze and calculate the content of microscopic images, and users can intuitively observe and obtain the data of the identification target object in the sample content, can distinguish stacked objects more accurately, and can quickly perform positioning verification through coding masks for the identification errors or invalid areas.

Description

Technical Field

[0001] The present invention relates to the field of recognition technology, and in particular to a method for visually recognizing nanomaterial microscopic morphology based on deep learning. Background Art

[0002] At present, image recognition technology has become a very powerful tool in the field of nanoscience. For example, the typical results of post-characterization techniques such as scanning electron microscopy (SEM) for identifying nanomaterials are presented in the form of images; scanning electron microscopy is a conventional technology for characterizing the structure of nanomaterials, and its spatial resolution can reach the nanometer level. However, although the scientific background of using scanning electron microscopy is very broad and it promotes the microscopic morphology recognition of nanomaterials, due to technical limitations, it does not have suitable tools to classify and archive the large amount of data generated by recognition. In addition, although scanning electron microscopy technology is widely used in the field of nanoscience, when using scanning electron microscopy to observe and find the identification target objects in the sample, due to the limited field of view, it is difficult to observe all the required identification targets and analyze their states, resulting in low detection efficiency, which greatly limits the availability of scanning electron microscope images and slows down the progress of scientific research.

[0003] With the development of science and technology, there are also solutions in the existing technology that use deep learning methods to identify the microscopic morphology images of nanomaterials obtained by scanning electron microscopes, which reduces the problems existing in the use of scanning electron microscopes alone. However, due to technical limitations, there are still the following specific problems that cannot be overcome. (1) For the stacked images of samples, it is impossible to distinguish the targets in the stacked area; (2) When the image content is densely distributed, it is impossible to accurately identify each object; (3) When the target is recognized incorrectly, it is impossible to remove the recognized incorrect area or invalid area through effective means. Summary of the invention

[0004] In order to overcome the existing methods of identifying nanomaterial microscopic morphology images obtained by scanning electron microscopes through deep learning methods, which have the disadvantages as described in the background technology due to technical limitations, the present invention provides a Vision Transformer model trained based on SEM images, which can automatically capture, analyze and calculate the content of microscopic images. Users can intuitively observe and obtain data on identified target objects in sample contents, thereby improving the detection efficiency and quality of the nanomaterial microscopic morphology visual recognition method based on deep learning.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A method for visual recognition of nanomaterial micro-morphology based on deep learning comprises the following steps: S1: inputting a nanomaterial morphology image file obtained by SEM, specifically, the morphology image file comprises a picture file and a Meta metadata file; S2: performing pixel-level segmentation on the picture file and extracting key information from the Meta file; S3: selecting a pre-trained model to perform pixel-level intelligent recognition on the nanomaterial morphology image obtained by SEM, specifically, loading the pre-trained model weight file for intelligent recognition of the image, and the intelligent recognition comprises the following steps: (1) automatically encoding and masking the recognition area of ​​the nanomaterial morphology image, and saving the mask image and the mask-coded image for calibration of the result; (2) perform pixel-level calculations on the masked area, including the following: masked area code, masked area area, masked area perimeter, masked area circumscribed maximum rectangle, masked area circumscribed minimum rectangle, and masked area equivalent diameter. This step segments the objects in the stacked area at the pixel level, which can accurately identify the outline of the stacked objects and the contents of the stacked area. It can also achieve the purpose of accurately identifying multiple object targets in complex scenes, as well as manually correcting existing recognition errors. S4: record the results, and automatically convert the pixel-level results and the real size data. S5: merge the results of different nanomaterial morphology images obtained by SEM of the same sample.

[0007] Furthermore, in S1, the image information obtained from the SEM and the corresponding Meta information are named consistently. In addition, in addition to the tif / tiff formats, jpg / jpeg / png / bmp image formats are also supported.

[0008] Furthermore, in S2, when performing pixel-level segmentation on the image file, the scale information in the SEM image is removed, the key parameters in the Meta metadata file are extracted, and the image is renamed in the format of (magnification#(*file name)). Specifically, the key parameters include magnification and unit information.

[0009] Furthermore, in S4, the result obtained in S3 is written into a local csv file, and each column of data is divided by the magnification factor to obtain the real size data.

[0010] Furthermore, in S5, when the results of different SEM images of the same sample are merged, a significant identification symbol should be included in the file name.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: under the action of the corresponding technical scheme, the present invention reduces the difficulty of identifying the target object and analyzing its size information, and greatly improves the detection efficiency; through numerical conversion, the pixel-level recognition results are automatically converted into the real size, avoiding cumbersome unit conversion; no need for users to manually select targets, reducing the number of times users need to manually select when the SEM image content is complex; the generated result file contains statistically diverse result information files, and comprehensively counts nanomaterial information such as perimeter, area, length, aspect ratio, etc.; avoids users switching back and forth and manually merging in multiple result files; supports the recognition of multiple image formats; can distinguish stacked objects more accurately, and can quickly perform positioning verification through coding masks for recognition errors or invalid areas. In summary, the present invention has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic block diagram of the process of the present invention.

[0013] Figure 2 These are different types of SEM images and corresponding Meta metadata files for the implementation cases involved in the present invention.

[0014] Figure 3 This is the SEM image recognition UI interface and 5 modules involved in the present invention.

[0015] Figure 4 The present invention relates to the operation content and steps of extracting key parameters from the Meta metadata file and renaming the image.

[0016] Figure 5 This is the output result of the "file renaming" module involved in the present invention.

[0017] Figure 6 The present invention involves removing the scale information of the SEM image and retaining only the operation content and steps of the object information.

[0018] Figure 7 It is the output result of the “image cropping” module involved in the present invention.

[0019] Figure 8 The present invention involves the use of a pre-trained AI model to perform content segmentation, recognition and calculation on SEM images.

[0020] Fig. 9 It is the output result of the “segmentation AI” module involved in the present invention.

[0021] Fig.10The present invention relates to the operation contents and steps of converting the pixel-level calculation results into the actual size.

[0022] Fig.11 This is the recognition result of implementation case 2 involved in the present invention.

[0023] Fig.12 This is the recognition result of implementation case three involved in the present invention. DETAILED DESCRIPTION

[0024] Figure 2 In the figure, there are file information of three implementation cases of the present invention, including SEM images and Meta (metadata) data files, and the image formats are bmp / jpg / tif respectively; Figure 3 The UI interface designed by the present invention is included.

[0025] Figure 1-12As shown, the method for visual recognition of nanomaterial micromorphology based on deep learning includes the following steps: Step 1: Input the nanomaterial morphology image file obtained by SEM, the morphology image file includes a picture file and a Meta metadata file; specifically, the image information obtained from the SEM and the corresponding Meta information are named consistently, and in addition to tif / tiff formats, jpg / jpeg / png / bmp image formats are also supported. Step 2: Perform pixel-level segmentation on the image file in step 1 and extract key information from the Meta file; specifically, perform pixel-level segmentation on the image file, remove the scale information in the SEM image, extract the key parameters in the Meta metadata file, and rename the image in the format of (magnification#(*file name)). Specifically, (# is a simple # symbol with no special meaning. It is only used to concatenate the original file names. For example, if the original file name is "test" and the method magnification is 100, the new file name is "100#test"). The key parameters include magnification and unit information. This step has two functions: first, remove the scale part of the SEM image to prevent recognition interference; second, extract the magnification parameter from the meta file for subsequent unit conversion, because the image calculation is at the pixel level, pixel value / magnification = actual size). Step 3: Select a pre-trained model (the model is the pth file of the trained vision transformer, which is actually the weight parameters of the convolutional neural network) to perform pixel-level intelligent recognition on the nanomaterial morphology image obtained by SEM; specifically, the pre-trained model weight file is loaded to perform intelligent recognition on the image. The recognition includes the following steps: (1) Automatically encode and mask the recognition area of ​​the nanomaterial morphology image, and save the mask image and the mask encoding image to verify the result; (2) Perform pixel-level calculation on the mask area, and the calculation content includes but is not limited to: mask area code, mask area area, mask area perimeter, mask area circumscribed maximum rectangle, mask area circumscribed minimum rectangle, mask area equivalent diameter; Step 3 has three functions: first, the objects in the stacking area are segmented at the pixel level, so that the outline of the stacking objects can be accurately identified, thereby achieving the purpose of accurately identifying the content of the stacking area; second: it can achieve the purpose of accurately identifying multiple object targets in complex scenes; third: marking the mask area can achieve the purpose of manually correcting possible recognition errors. Step 4: Record the results of step 3, and automatically convert the pixel-level results and the real-size data. Specifically, write the results obtained in step 3 into a local csv file, and divide each column of data by the magnification to obtain the real-size data. The main function of this step is to convert the pixel size into the real size.Step 5: Merge the results of nanomaterial morphology images obtained by different SEMs of the same sample. Specifically, when merging the results of different SEM images of the same sample, a significant identification symbol should be included in the file name. This step is used because there may be multiple electron microscope images for the same sample. In order to facilitate the viewing of relevant personnel, the results can be merged.

[0026] In the present invention, the first embodiment is to identify the sparse irregular block microstructure of nanomaterials. The specific process is as follows: (1) First, the user needs to put the SEM image file and the Meta metadata file in a folder and open it in the UI (interface) Figure 4 Specify the input path in the interface shown in 1.1. Figure 4 Specify the output path in the interface shown in 1.2. Figure 4 The Meta metadata is captured in the interface shown in 1.3, and the image is renamed as "magnification#file name". The conversion result is as follows Figure 5 (2) Click Figure 6 In the image cropping module interface, through Figure 6 The interface shown in 2.1 in the figure specifies the output file path of (1) as the new input path. Figure 6 The interface shown in 2.2 in the figure specifies the output path of the cropped image. Figure 6 The image is cropped in the interface shown in 2.3. This step will remove the scale information of the SEM image to ensure the integrity of the image content. The cropping result is as follows: Figure 7 (3) Click Figure 8 In the AI ​​segmentation module interface, through Figure 8 The interface shown in 3.1 in the figure specifies the output file path of (2) as the new input path by Figure 8 The interface shown in 3.2 in the figure specifies the save path of AI segmentation and recognition results. Figure 8 Select the pre-trained AI model in the interface shown in 3.3. Figure 8 The image is automatically recognized in the interface shown in 3.4. The recognition results are in three folders: the comparison of the original image and the mask image, the encoded mask image and the calculation results, as shown in Fig. 9 As shown in Table 1 below. (4) Click Fig.10 The unit conversion module in Fig.10 In the interface shown in 4.1, the output file csv folder of (3) is specified as the new input path. Fig.10The interface shown in 4.2 in the figure specifies the path for saving the converted result. The unit conversion is performed through the interface shown in 4.3 in the figure 10. The conversion rule is the method multiple extracted from each column of data / Meta metadata file, and the real size data can be obtained. The result is shown in Table 2 below. Through the above steps, the intelligent recognition, segmentation and calculation of the SEM image can be completed, and the processing process, processing results and statistical information are clear and traceable. In addition, the present invention can also better segment the boundaries of stacked irregular particles.

[0027] Table 1

[0028]

[0029] Table 2

[0030]

[0031] In the present invention, Example 2 is to identify the microscopic morphology of dense spherical particles of nanomaterials. All specific processes are the same as those of Example 1. The processing results are as follows: Fig.11 As shown, it can be found that even when faced with complex and dense spherical particle images, the present invention can still perform relatively accurate and complete recognition.

[0032] In the present invention, Example 3 is to identify the microscopic morphology of one-dimensional nanomaterials. The specific process is the same as that of Example 1. The processing result diagram is as follows: Fig.12 As shown, it can be found that there are still deficiencies in the recognition of some areas. However, since the mask areas are all encoded accordingly, the misidentified areas can be manually removed from the CSV file through manual verification.

[0033] Figure 1-12As shown, the present invention has the following advantages through the above technical scheme. (1) According to the input nanomaterial image file and Meta file, the image content can be automatically recognized and the calculation results can be performed by loading the pre-trained model, which greatly improves the detection efficiency in addition to reducing the difficulty of identifying the target object and analyzing its size information required by the user. (2) By parsing the Meta file, the scale information of the SEM is automatically obtained, and the pixel-level recognition results are automatically converted into real sizes through numerical conversion, avoiding the user from performing tedious unit conversion. (3) It can effectively automatically recognize SEM image objects of various forms such as block, spherical and irregular shapes, without the need for users to manually select targets, reducing the number of manual selections required by users when the SEM image content is complex. (4) The generated result file contains a statistically diverse result information file, and a comprehensive statistics of nanomaterial information such as perimeter, area, length, and aspect ratio is performed. (5) For different SEM images of the same object, unit result merging is supported, avoiding users from switching back and forth and manually merging in multiple result files. (6) Supports recognition of multiple image formats such as jpg / jpeg / png / bmp / tif / tiff, making the application more flexible and diverse. (7) Can distinguish stacked objects more accurately, and can quickly locate and verify the recognition errors or invalid areas through coding masks. In general, the Vision Transformer model (deep learning model based on attention mechanism) trained on SEM images in the present invention can automatically capture, analyze and calculate the content of microscopic images. Users can intuitively observe and obtain the data of the identified target objects in the sample contents, saving a lot of computing time and computing costs for the statistical calculation of the microscopic size of nanomaterials. It has excellent recognition effects for complex and dense spherical images and irregular block materials, and improves detection efficiency and quality.

[0034] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention.

[0035] In addition, it should be understood that although the present specification is described according to the implementation mode, the implementation mode does not only include an independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in the embodiments can also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for visual recognition of nanomaterial microstructure based on deep learning, characterized in that: The following steps are involved: S1: Input the nanomaterial morphology image file obtained by SEM. Specifically, the morphology image file includes a picture file and a Meta metadata file; S2: Pixel-level segmentation of the image file and key information extraction of the Meta metadata file; S3: Select a pre-trained nanomaterial morphology image model to perform pixel-level intelligent recognition on the nanomaterial morphology image acquired by SEM. Specifically, the pre-trained nanomaterial morphology image model weight file is loaded and the image is intelligently recognized. The intelligent recognition includes the following steps: (1) Automatically encode and mask the recognition area of ​​the nanomaterial morphology image, and save the mask image and the mask encoding image to verify the result; (2) Pixel-level calculation of the mask area. The calculation content includes the following: mask area code, mask area area, mask area perimeter, maximum rectangle circumscribed by the mask area, minimum rectangle circumscribed by the mask area, and equivalent diameter of the mask area. This step can accurately identify the outline of the stacked object and achieve the effect of accurately identifying the content of the stacked area. It can also achieve the purpose of accurately identifying multiple object targets in complex scenes and manually correct existing recognition errors; S4: Record the results and automatically convert the pixel-level results and the real size data; S5: Merge the results of different nanomaterial morphology images obtained by SEM of the same sample.

2. The method for visual recognition of nanomaterial microstructure based on deep learning according to claim 1, characterized in that: In S1, the image information obtained from SEM and the corresponding Meta information are named consistently. In addition to tif / tiff formats, jpg / jpeg / png / bmp image formats are also supported.

3. The method for visual recognition of nanomaterial microstructure based on deep learning according to claim 1, characterized in that: In S2, when performing pixel-level segmentation on the image file, the scale information in the SEM image is removed, the key parameters in the Meta metadata file are extracted, and the image is renamed. Specifically, the key parameters include magnification and unit information.

4. The method for visual recognition of nanomaterial microstructure based on deep learning according to claim 1, characterized in that: In S4, the results obtained in S3 are written into a local csv file, and each column of data is divided by the magnification factor to obtain the actual size data.

5. The method for visual recognition of nanomaterial microstructure based on deep learning according to claim 1, characterized in that: In S5, when merging the results of different SEM images of the same sample, a prominent identification symbol should be used in the file name.

Images