Data processing method and system for images of analytical materials

Abstract

This invention provides a data processing method and system for images of analytical materials that can accurately remove specific unwanted substances present in the images of the analytical materials. [Solution] In a data processing method for images of the cross-section of an analytical material, in which knife marks N caused by material cutting are removed as predetermined unwanted objects present in the original image D1 of the cross-section Sa of the analytical material, a predictive model 9 is constructed by performing machine learning using a dataset 8 as training data, which is a collection of many images of the cross-section of the same type of material as the analytical material, consisting of damaged images 8a in which knife marks N are present and undamaged images 8b in which knife marks N are not present. The predictive model 9 stored in the computing device 2 is then used to perform data processing on the original image D1 to obtain a processed image D2 from which the knife marks N have been removed from the original image D1.

Description

Technical Field

[0001] The present invention relates to a data processing method and system for an image of an analysis material, and more particularly to a data processing method and system for an image of an analysis material that can accurately remove predetermined unnecessary objects existing in the image of the analysis material.

Background Art

[0002] When analyzing the internal structure of a rubber material or the like, for example, observing the cross-sectional image of the material is performed. This cross-sectional image has unnecessary scratches caused by material cutting. These unnecessary scratches become noise when observing the cross-sectional image to accurately grasp the internal structure.

[0003] As described in Patent Document 1, generally, for noise removal in image data, noise removal filters such as Gaussian filters and median filters are used (see paragraph 0025). However, in data processing using these noise removal filters, the above-mentioned unnecessary scratches (noise) cannot be accurately removed. Therefore, there is room for improvement in accurately removing predetermined unnecessary objects such as unnecessary scratches (noise) in the analysis material.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] An object of the present invention is to provide a data processing method and system for an image of an analysis material that can accurately remove predetermined unnecessary objects existing in the image of the analysis material.

Means for Solving the Problems

[0006] The present invention provides a data processing method for images of analytical materials that achieves the above objectives, which involves removing predetermined unwanted substances from an original image of an analytical material. The method is characterized by constructing a predictive model by performing machine learning using a dataset as training data, which consists of a large number of images of the same type of material as the analytical material, including images with the predetermined unwanted substances present and images without the predetermined unwanted substances present compared to the images with the unwanted substances. The method then uses the predictive model stored in a computing device to perform data processing on the original image, thereby obtaining a processed image from which the predetermined unwanted substances have been removed.

[0007] The present invention relates to a data processing system for images of analytical materials, comprising a calculation unit for removing predetermined unwanted substances present in the original image of the analytical material, and an output device, wherein the calculation unit has a storage unit that stores a prediction model constructed by machine learning using a dataset as training data, which is a collection of images of the same type of material as the analytical material, including images with the predetermined unwanted substances present and images without the predetermined unwanted substances that do not exist in the images with the unwanted substances; a calculation unit; and an input unit, wherein the calculation unit performs data processing on the original image input to the calculation unit through the input unit, removing the predetermined unwanted substances using the prediction model, and the processed image is output to the output device. [Effects of the Invention]

[0008] According to the present invention, in the prediction model constructed using the image with unwanted objects and the image without unwanted objects, the predetermined unwanted objects are treated as noise, and the pixel values ​​of each pixel constituting the portion of the original image other than the predetermined unwanted objects are analyzed and interpreted to obtain the processed image as the prediction result. Therefore, the processed image has the predetermined unwanted objects from the original image removed with high accuracy.

[0009] Furthermore, obtaining an image from which the predetermined unwanted substances have been removed eliminates the need for cumbersome tasks such as meticulously setting processing conditions for the analytical material or extracting areas from the original image where the predetermined unwanted substances are absent. This is advantageous for accurately and efficiently understanding the internal structure of the analytical material. [Brief explanation of the drawing]

[0010] [Figure 1] This is an explanatory diagram illustrating an example of a data processing system for images of analytical materials. [Figure 2] This is an explanatory diagram illustrating the original image. [Figure 3] This flowchart illustrates the procedure for processing image data of analytical materials. [Figure 4] This is an explanatory diagram illustrating a dataset. [Figure 5] This is an explanatory diagram illustrating a predictive model. [Figure 6] This is an explanatory diagram illustrating the processed image. [Figure 7] This is an explanatory diagram illustrating extracted images. [Figure 8] This is an explanatory diagram illustrating the dataset for the first modified example. [Figure 9] This flowchart illustrates the steps involved in the data processing method described in Modification Example 1. [Figure 10] This is an explanatory diagram illustrating the selected image for example 1. [Figure 11] This is an explanatory diagram illustrating the creation of a model. [Figure 12] This is an explanatory diagram illustrating an example of a damaged image after conversion. [Figure 13] This is an explanatory diagram illustrating the original image for the first modified example. [Figure 14] This is an explanatory diagram illustrating the image after processing in the modified example 1. [Figure 15] This is an explanatory diagram illustrating the original image for variation 2. [Figure 16] This flowchart illustrates the steps involved in the data processing method for Modification Example 2. [Figure 17]It is an explanatory diagram illustrating the selected image of Modification 2. [Figure 18] It is an explanatory diagram illustrating the dataset of Modification 2. [Figure 19] It is an explanatory diagram illustrating an image with unwanted objects. [Figure 20] It is an explanatory diagram illustrating the image after processing of Modification 2. [Figure 21] It is an explanatory diagram illustrating another dataset. [Figure 22] It is an explanatory diagram illustrating another original image. [Figure 23] It is an explanatory diagram illustrating another processed image. [Figure 24] It is an explanatory diagram illustrating the original images of multiple samples, the processed images and the extracted images obtained using the prediction model of Example 1. [Figure 25] It is a graph diagram illustrating the evaluation of the prediction model of Example 1. [Figure 26] It is an explanatory diagram illustrating the original image used in Example 2. [Figure 27] It is an explanatory diagram illustrating the processed image of Example 2. [Figure 28] It is an explanatory diagram illustrating another processed image of Example 2. [Figure 29] It is an explanatory diagram illustrating the original image used in Example 3. [Figure 30] It is an explanatory diagram illustrating the dataset used in Example 3. [Figure 31] It is an explanatory diagram illustrating the processed image of Example 3.

Mode for Carrying Out the Invention

[0011] Hereinafter, a data processing method and system for an image of an analysis material of the present invention will be described based on the embodiments shown in the drawings.

[0012] The embodiment of the data processing system 1 illustrated in Figure 1 comprises a computing device 2 and an output device 3. Using this data processing system 1, an embodiment of the data processing method for images of analytical materials illustrated in Figure 3 is implemented. This data processing method is performed as a preparatory step for understanding the internal structure of the analytical material, and uses the original image D1, which is a cross-sectional image of a sample S taken from the desired analytical material. In the procedure of this data processing method, the original image D1 acquired by the imaging device C is input to the computing device 2 (S110). Next, a prediction model 9 is constructed by performing machine learning using a dataset 8, which is an accumulation of images with defects 8a and images without defects 8b, as training data (S120, S130). Next, by performing data processing on the original image D1 using the prediction model 9 with the computing device 2, a processed image D2 is obtained from the original image D1, from which unwanted defects (hereinafter referred to as knife marks N) caused by material cutting are removed (S140). Finally, the processed image D2 is output to the output device 3, and the internal structure of the analytical material is quantified based on the output processed image D2.

[0013] In this embodiment, knife marks N correspond to the predetermined unwanted material of the present invention. That is, the scratched image 8a corresponds to the image with unwanted material, and the scratch-free image 8b corresponds to the image without unwanted material. The predetermined unwanted material of the present invention is not limited to knife marks N, but may also be noise such as Poisson noise or luminance noise (such as overexposure), or the remaining components of the analytical material after excluding the predetermined object of observation. For example, if the analytical material is vulcanized rubber or unvulcanized rubber, the original image is an image from an electron microscope, and the object of observation is particles of a filler such as carbon black, the predetermined unwanted material may include the rubber components (polymers) of the vulcanized or unvulcanized rubber, and various types of noise in the image (such as Poisson noise or luminance noise).

[0014] For example, various known vulcanized rubbers and unvulcanized rubbers containing at least a filler can be used as analytical materials. Vulcanized and unvulcanized rubbers generally contain one or more types of polymers and additives such as fillers, vulcanization accelerators, antioxidants, and anti-aging agents. Various known fillers such as carbon black, silica, and calcium carbonate can be used as fillers. In this embodiment, vulcanized rubber containing carbon black as a filler is used. The analytical materials are not limited to vulcanized or unvulcanized rubber; for example, resins and metals can also be used.

[0015] As illustrated in Figure 1, a sample S is taken from the analytical material using a cutting tool such as a knife or microtome to obtain the original image D1. On the cross-section Sa of sample S, particles of compounding components such as carbon black and antioxidants are visible. Therefore, it is preferable to use a sample S that shows a typical distribution state of the compounding component to be observed. For example, if vulcanized rubber containing carbon black is used as the analytical material and the carbon black is to be observed, multiple samples S should be taken from the vulcanized rubber, and a sample S in which the carbon black appearing on the cross-section Sa shows a typical distribution state should be selected from among the multiple samples S.

[0016] The imaging device C acquires the original image D1 of the cross-section Sa of the sample S. Details of the original image D1 will be described later. Various known imaging devices can be used in this imaging device C, depending on the analytical material and the object to be observed. Examples of imaging devices C include optical microscopes with cameras, focused ion beam (FIB) systems, electron microscopes (TEM, SEM), and atomic force microscopes (AFM).

[0017] The arithmetic unit 2 receives and stores various data, and performs data processing using this data. The arithmetic unit 2 can use various known computers. The arithmetic unit 2 has an arithmetic unit (CPU) 4, a main memory unit (memory) 5, an auxiliary storage unit (HDD, etc.) 6, and an input unit (various interfaces) 7. The auxiliary storage unit 6 corresponds to the storage unit of the present invention and stores the dataset 8 and the prediction model 9. The input unit 7 can be any interface that can input the original image D1 acquired by the imaging device C. The input unit 7 can include data input devices such as scanners and camera devices that digitize and capture the original image D1, an input port to which the imaging device C is directly connected or to which a USB memory containing the original image D1 is connected, and an external storage device that reads storage media such as CD-ROMs.

[0018] Output device 3 is connected to arithmetic unit 2 and outputs the results of data processing by arithmetic unit 2. Output device 3 can use a known display, printer, or the like.

[0019] The original image D1 shown in Figure 2 was acquired using a camera-equipped optical microscope as the imaging device C and is stored in the auxiliary storage unit 6 of the computing device 2. In this original image D1, the black dots represent carbon black particles P, the black diagonal lines represent knife marks N, and the remaining gray areas represent rubber components (polymers, etc.). The X and Y directions in Figure 2 represent the horizontal and vertical directions, respectively, in the original image D1, and are orthogonal to each other. In Figure 2, R represents the size (particle size) of the particles P, B represents the width of the knife marks N (the width dimension in the direction perpendicular to the extension direction of the knife marks N), and L represents the distance between adjacent knife marks N.

[0020] Knife marks N are formed on the cut surface Sa when a sample S is taken from the analytical material using a cutting tool such as a knife or microtome (material cutting). The chipped portion of the cutting tool's blade or deposits attached to the blade cut into the cut surface Sa. Knife marks N are linear in shape, extending in the direction of cutting by the cutting tool, and in Figure 2, they extend diagonally from the upper left to the lower right. The numerous knife marks N present in the original image D1 generally have the same direction of extension, but other factors such as width B, brightness (pixel value of each pixel constituting the knife mark N), and spacing L between knife marks N differ slightly. The state of knife marks N in the original image D1 differs slightly depending on the type of cutting tool, the cutting conditions when the sample S was taken, and the internal structure of the analytical material (distribution state of particles P). For example, the particles P of carbon black are harder and more difficult to cut than other parts (e.g., polymers), so the knife marks N differ subtly in areas where these particles P are dense and areas where they are sparse. In other words, even if samples S are taken from the same type of vulcanized rubber using the same cutting tool and cutting conditions, if the distribution of particles P at each cut surface Sa is different, the state of knife marks N in the original image D1 will also be different. Therefore, although knife marks N are a highly regular type of noise that forms a linear shape extending in the cutting direction, their state in the original image D1 differs depending on various factors such as the type of cutting tool, the cutting conditions when taking the sample S, and the internal structure of the analytical material (size and distribution of particles P). Consequently, knife marks N cannot be accurately removed by noise reduction filters such as Gaussian filters and median filters commonly used in image processing.

[0021] The content of each step (S110~S140) in the data processing procedure for the image of the analytical material illustrated in Figure 3 will be explained below.

[0022] In step S110, the original image D1 acquired by the imaging device C is input to the processing unit 2. The original image D1 is input to the processing unit 2 via the input unit 7 from data input devices such as scanners and camera devices including the imaging device C, or from external storage devices such as USB memory or CD-ROMs.

[0023] In step S120, a dataset 8 is created by combining images with and without damage (8a and 8b). The dataset 8 can be created using data selected from a large amount of pre-prepared data, or, as in this embodiment, by simulation.

[0024] The dataset 8 illustrated in Figure 4 is created in step S120 and stored in the auxiliary storage unit 6 of the arithmetic unit 2. This dataset 8 is used as training data in step S130, which will be described later. The dataset 8 is a collection of images with and without scratches 8a for each sample (1, 2, ..., n) shown in the leftmost column of the table. In the same sample, the image with scratches 8a and the image without scratches 8b are paired images, differing only in the presence or absence of knife marks N. A pair means that an explanatory variable (condition) and an objective variable (correct answer or label) are set together. That is, in the same sample, the image with scratches 8a and the image without scratches 8b differ only in the presence or absence of unwanted objects such as knife marks N, and they are similar enough that the size and distribution of the observed objects can be considered to be roughly the same. In this dataset 8, in the same sample, the image with scratches 8a is the explanatory variable and the image without scratches 8b is the objective variable. The dataset 8 is further a collection of scratched images 8c for each sample. The damaged image 8c corresponds to the unwanted object image of the present invention. Since this damaged image 8c is used to generate the damaged image 8a, it can be omitted in samples where the damaged image 8a already exists.

[0025] Image 8a with scratches is a cross-sectional image of the same type of material as the material to be analyzed, and image 8b without scratches is a cross-sectional image of the same type of material as image 8a, but without the knife marks N. "Same type of material as the material to be analyzed" means, for example, that when vulcanized rubber is used as the material to be analyzed, vulcanized rubber is used; when unvulcanized rubber is used as the material to be analyzed, unvulcanized rubber is used; and when resin is used as the material to be analyzed, resin is used. More preferably, when vulcanized rubber is used as the material to be analyzed, vulcanized rubber of the same specifications as the vulcanized rubber is used; when unvulcanized rubber is used as the material to be analyzed, unvulcanized rubber of the same specifications as the unvulcanized rubber is used; and when resin is used as the material to be analyzed, resin of the same specifications as the resin is used.

[0026] Those involved in the manufacturing and development of analytical materials accumulate vast amounts of data similar to the original image D1, including data obtained during the manufacturing and development process and the results of computer simulations. Therefore, this vast amount of data can be used for the damaged image 8a. Then, by visually identifying the knife marks N from the damaged image 8a and manually selecting and removing the knife marks N, the undamaged image 8b can be generated. Thus, the dataset 8 can use actual cross-sectional images obtained in the same way as the original image D1, but creating the undamaged image 8b from actual cross-sectional images requires considerable effort. Therefore, in this embodiment, the damaged image 8a and the undamaged image 8b are created by simulation using the computing device 2. Specifically, in this simulation, the undamaged image 8b and the damaged image 8c are created using predetermined features extracted from the original image D1, and then the damaged image 8a is created by combining these created images. The damaged image 8c is an image in which only the knife marks N are present.

[0027] The creation of the undamaged image 8b, the damaged image 8c, and the synthesis of the created undamaged image 8b and damaged image 8c are performed using simulations based on predetermined features extracted from the original image D1. Various known simulations can be used for this simulation, such as simulations that model the extracted predetermined features, and simulations that use a predictive model constructed using those features and the original image D1 as training data. In addition, simulations based on features extracted from the vast amount of accumulated data mentioned above can also be used to create these images. Extracting the features used in the simulation from the original image D1 makes each created image approximate the original image D1, which is advantageous for improving the accuracy of the predictive model 9 described later. Furthermore, extracting the features used in the simulation from the vast amount of accumulated data makes each created image a diverse cross-sectional image, which is advantageous for improving the versatility of the predictive model 9 described later.

[0028] The predetermined features can be arbitrarily selected from among the many features present in the original image D1, but it is preferable to select features related to the object being observed and the knife marks N. For example, when carbon black in vulcanized rubber is the object being observed, the predetermined features include the size R and distribution state of carbon black particles P present in the original image D1, the width B of the knife marks N, the spacing L between knife marks N, the brightness of the knife marks N, and the relative position between the particles P and the knife marks N. The particle size R, the width B of the knife marks N, and the spacing L can be measured by selecting representative particles P and knife marks N from the original image D1. For the distribution state of particles P, a histogram showing the approximate number of particles P for each size R can be used. For the relative position between particles P and knife marks N, the degree of proximity or overlap of the knife marks N to the location of particles P in the original image D1 can be used, such as the presence of more particles P on the line of the knife marks N. Furthermore, the particle size R and distribution state of P vary depending on the composition of the analytical material (the proportions of several components of the vulcanized rubber) and the manufacturing conditions (such as the mixing conditions when forming the unvulcanized rubber and the vulcanization conditions when vulcanizing that unvulcanized rubber), so these can also be used as feature quantities. The width B, brightness, and spacing L of knife marks N vary depending on the type of cutting tool and cutting conditions, so these can also be used as feature quantities.

[0029] For example, to create the undamaged image 8b, the size R and distribution state of particle P are used as features. To create the damaged image 8c, the width B, brightness, and spacing L between knife marks N are used as features. To combine the undamaged image 8b and the damaged image 8c, the relative positions of particle P and knife marks N are used as features. In the simulation, random numbers within the range of possible features are input as these features, and images are created and combined randomly. In addition, the simulation uses molecular simulations based on the composition and manufacturing conditions of the analytical materials to create and combine images.

[0030] The simulation generates at least one undamaged image 8b, and may contain multiple undamaged images. Similarly, the simulation generates at least one damaged image 8c for each undamaged image 8b, and may contain multiple damaged images. By applying image processing such as inversion and rotation to a single image, approximately a dozen duplicate images based on that image can be created. The number of samples (n) in dataset 8, including the images created by the simulation and their duplicates, is preferably 1000 or more, and more preferably 4000 or more. To create multiple images, simulations can be repeated under the same conditions or with different conditions. For the same conditions, features extracted from the original image D1 can be used. For different conditions, these features can be used as a baseline, and features that differ within an acceptable range from this baseline can be used. The acceptable range is such that the generated undamaged image 8b is roughly similar to the cross-section image from which the noise marks N have been removed from the original image D1, and the damaged image 8c is roughly similar to the noise marks N in the original image D1. For example, in simulations with different conditions, values ​​are used that differ from the representative values ​​of the features in the original image D1 within an acceptable range of 30% to 300%. The damaged image 8a, which is created by combining multiple undamaged images 8b and damaged images 8c generated by repeatedly performing simulations under the same conditions, approximates the cross-sectional image of the material being analyzed, which is advantageous for improving the prediction accuracy of the prediction model 9 described later. Furthermore, the damaged image 8a, which is created by combining multiple undamaged images 8b and damaged images 8c generated by repeatedly performing simulations with different conditions, shows a wider variety of distribution states of particles P and knife marks N, which is advantageous for improving the versatility of the prediction model 9 described later.

[0031] In the synthesis of the created undamaged image 8b and damaged image 8c (creation of damaged image 8a), the appropriate position of damaged image 8c can be simulated by rotating, translating, or inverting it relative to the undamaged image 8b. The relative positions of the particles P and knife marks N in the original image D1 can be used to appropriately position damaged image 8c. For example, in a cross-section image, multiple particles P may exist on the line of the knife marks N, and it is desirable to synthesize damaged image 8a such that noise marks N overlap with the locations where multiple particles P are lined up on such a line.

[0032] Dataset 8 is created using the damaged image 8a and the undamaged image 8b, which are generated by the simulation. Creating dataset 8 using simulation in this way makes it possible to prepare a large number of samples (1, 2, ..., n) for dataset 8, which is advantageous for improving the accuracy of the prediction model 9 in machine learning, as described later. In addition, by creating the undamaged image 8b and the damaged image 8c in the simulation and then combining these images to create the damaged image 8a, dataset 8 can be created more simply than by manually removing the knife marks N present in the damaged image 8a to create the undamaged image 8b.

[0033] The created dataset 8 is stored in the auxiliary storage unit 6 of the computing unit 2. The auxiliary storage unit 6 should ideally store a large number of datasets 8, distinguished by the type of analytical material, the type of cutting tool, and the cutting conditions. If a large number of datasets 8 are stored, the appropriate dataset 8 can be selected from among the many datasets 8 according to the type of analytical material, the type of cutting tool, and the cutting conditions. Furthermore, when using multiple samples S taken from the same analytical material using the same cutting tool and under the same cutting conditions, the dataset 8 is created by a simulation in which feature quantities showing similar characteristics are extracted from the original image D1 of each sample S. For example, when using three samples S taken from the same vulcanized rubber, the size R and distribution state of the particle P, the width B and brightness of the knife mark N, the distance L between knife marks N, and the relative position of the particle P and knife mark N can be extracted as feature quantities from each original image D1, and a common dataset 8 can be created by a simulation using these respective feature quantities.

[0034] The original image D1 obtained during the process of implementing the data processing method of this embodiment may be added to dataset 8 as a damaged image 8a, and the processed image D2, described later, may be added as a damaged image 8b. In other words, each time this data processing method is repeated, the number of samples in dataset 8 increases, and dataset 8 is expanded.

[0035] It is preferable that at least some of the samples in dataset 8 include damaged image 8a and undamaged image 8b created by simulation using predetermined features extracted from the original image D1. For example, if a dataset pre-stored in the auxiliary storage unit 6 is used as dataset 8, it is preferable to add at least one pair of damaged image 8a and undamaged image 8b created by simulation using predetermined features extracted from the original image D1 as samples to that dataset. The greater the degree of similarity between the samples in dataset 8 and the original image D1, the more advantageous it is to construct a prediction model 9 that is optimal for the original image D1.

[0036] In step S130, data processing is performed to construct a prediction model 9 by performing machine learning using dataset 8 as training data on the computing unit 2. The constructed prediction model 9 is stored in the auxiliary storage unit 6 of the computing unit 2. This prediction model 9 is a type of computer program that obtains a processed image D2 based on the input original image D1. The prediction model 9 is constructed using supervised machine learning with damaged image 8a of dataset 8 as the explanatory variable and undamaged image 8b as the target variable. Various known semantic segmentation models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers can be used for the prediction model 9. A semantic segmentation model labels each pixel of an image, identifies the location and shape of a desired object from the image, and reconstructs the identified desired object. That is, in a semantic segmentation model constructed using dataset 8, knife marks N are detected as irregular regions (noise) by annotation and are therefore not reconstructed. Then, with particle P as the desired object and the rubber component as the background, the location and shape of particle P are identified and reconstructed. As a result, the knife marks N present in the original image D1 are removed, and a processed image D2 is obtained in which the carbon black particles P are clearly visible.

[0037] The prediction model 9 illustrated in Figure 5 is a type of convolutional neural network called U-NET. In prediction model 9, pairs of damaged images 8a and undamaged images 8b are associated, differing in the presence or absence of knife marks N. In other words, in this prediction model 9, the knife marks N in the original image D1 are treated as noise, and the pixel values ​​of each pixel constituting the parts other than the knife marks N are analyzed and processed, with particles P being treated as the target for reconstruction. Therefore, this prediction model 9 can create a processed image D2 from which the knife marks N have been removed by extracting the location and shape of particles P in the input original image D1. The degree to which knife marks N are removed in this prediction model 9 is sufficient if the particles P present in the original image D1 can be clearly distinguished, and it is not necessary for all knife marks N present in the original image D1 to be removed.

[0038] The prediction model 9 in this embodiment is created by simulation using features extracted from the original image D1 in the dataset 8, and is therefore specialized for the original image D1. In other words, this prediction model 9 has high prediction accuracy even with a small number of training iterations (epochs). Therefore, it is advantageous in reducing the computational load on the computing unit 2 in machine learning. In addition, it is less susceptible to overfitting, and as a result, it can avoid situations where prediction results that deviate significantly from the original image D1 are output.

[0039] The constructed predictive model 9 can be applied to cross-sectional images of other samples S taken from the same analytical material, provided there are no changes in the cutting tool or cutting conditions; therefore, steps S120 and S130 can be omitted. However, when using multiple vulcanized rubbers with different formulations (e.g., carbon black content) as analytical materials, it is advisable to create a dataset 8 each time by extracting the features of each sample S taken from each vulcanized rubber, and to construct a separate predictive model 9 for each dataset 8. Similarly, when multiple samples S are taken from the same analytical material using different types of cutting tools or cutting conditions, it is advisable to create a dataset 8 each time by extracting the features of each sample S, and to construct a separate predictive model 9 for each dataset 8.

[0040] Furthermore, the prediction model 9 is updated (reconstructed) when the original image D1 obtained during the process of implementing the data processing method of this embodiment is added to the dataset 8 as a damaged image 8a, and the processed image D2, described later, is added as a clean image 8b.

[0041] In step S140, data processing is performed in which the processing unit 2 uses the prediction model 9 to obtain a processed image D2 from which the knife marks N have been removed from the original image D1. In step S140, the original image D1 is input to the prediction model 9, and the processing unit 2 processes the data to output the processed image D2. The processed image D2 is stored in the auxiliary storage unit 6 of the processing unit 2 and also output to the output device 3.

[0042] The processed image D2, illustrated in Figure 6, shows the removal of knife marks N from the original image D1, illustrated in Figure 2. In Figure 6, the black dots represent carbon black particles P, while the other areas represent rubber components. In this processed image D2, the knife marks N present in the original image D1 have been largely removed, making the shape and location of each carbon black particle P clearer.

[0043] As described above, according to this embodiment, the prediction model 9 is constructed using a dataset 8, which is an aggregate of images with and without damage (8a and 8b), as training data. Therefore, in this prediction model 9, knife marks N (unwanted objects in the original image D1) are treated as noise, and the pixel values ​​of each pixel constituting the parts of the original image D1 other than the knife marks N are analyzed and interpreted to obtain the processed image D2 as the prediction result. Consequently, the processed image D2 output to the output device 3 has the knife marks N from the original image D1 removed with high accuracy.

[0044] Furthermore, according to this embodiment, it is possible to eliminate the need for cumbersome tasks such as setting detailed cutting conditions for the analytical material or extracting areas where knife marks N are not present from the analytical material's cross-sectional image in order to obtain a cross-sectional image from which knife marks N have been removed. Therefore, it is advantageous for accurately and efficiently understanding the internal structure of the analytical material.

[0045] While it is possible to quantify the internal structure of the analytical material using the processed image D2 exemplified in Figure 6 above, it is preferable to perform binarization processing on the processed image D2 using the computing device 2. By performing binarization processing on the processed image D2 in this way, the boundary between the particles P and the rest of the image becomes clearer, which is advantageous for accurately quantifying the size R and distribution state of the particles P.

[0046] The extracted image D3 shown in Figure 7 is extracted from the original image D1 by applying a binarization process to the processed image D2 shown in Figure 6, which represents the region occupied by carbon black particles P. The region enclosed by the white line in Figure 7 indicates the region occupied by carbon black particles P. This extracted image D3 may be used to quantify the internal structure of the analytical material. Furthermore, this extracted image D3 allows us to understand the relationship between the distribution state of carbon black particles P and the knife marks N. For example, in Figure 7, there are several places where the knife marks N and multiple particles P overlap. Therefore, using the extracted image D3 is advantageous for understanding the relative positions of particles P and knife marks N.

[0047] Next, we will describe a modified example 1 of the data processing method and system for images of analytical materials.

[0048] Modification 1 differs in that the dataset 8A shown in Figure 8 is created based on the dataset 8 shown in Figure 4, but the configuration of the data processing system 1 is the same. The dataset 8A in Modification 1 contains transformed damaged images 8d, which are created by transforming the domain of the damaged image 8a. In other words, in Modification 1, this dataset 8A is used to create a machine learning prediction model 9, and the transformed damaged image 8d is used as the damaged image 8a.

[0049] The dataset 8A illustrated in Figure 8 is stored in the auxiliary storage unit 6 of the arithmetic unit 2. This dataset 8A is created in steps S150 and S160 of Figure 9, described later, based on the dataset 8 illustrated in Figure 4. This dataset 8A is then used as training data in step S130. The dataset 8A only needs to contain the undamaged image 8b and the transformed damaged image 8d for each sample (···, 4, ···, n) shown in the leftmost column of the table, and the damaged image 8a and damaged image 8c used in the process of creating the transformed damaged image 8d can be omitted. In the same sample, the undamaged image 8a and undamaged image 8b before transformation are paired images, so the transformed damaged image 8d and undamaged image 8b can also be considered as paired images. Furthermore, in the same sample, the converted damaged image 8d approximates the extension direction of the knife marks N and the spacing between them to the simulated damaged image 8a, but overall it is a more realistic image. Details of the converted damaged image 8d will be described later.

[0050] The data processing method for the images of the analytical material in Modification 1 is performed according to the procedure illustrated in Figure 9. This data processing method has steps S150 and S160 added to the procedure for the data processing method illustrated in Figure 2 described above. In the procedure of Modification 1, when dataset 8 is created (S120), the converted damaged image 8d is created (S150), and dataset 8A is created by accumulating the converted damaged image 8d and the undamaged image 8b (S160). Therefore, in step S130, the created dataset 8A is used. The details of each step S150 and S160 are described below.

[0051] In step S150, the arithmetic unit 2 performs data processing to transform the domain of the damaged image 8a by referring to the selected image D4, which has a domain that can be considered identical to the domain of the original image D1. This data processing creates a transformed damaged image 8d in which the domain of the damaged image 8a is transformed to the same domain as the selected image D4.

[0052] The selected image D4 illustrated in Figure 10 is selected from images that have a domain that can be considered identical to the domain of the original image D1, and is pre-stored in the auxiliary storage unit 6 of the arithmetic unit 2. A domain is a set of data that exhibits a specific characteristic. When the domains of different images can be considered generally identical, it means that the characteristic parts of the different images are similar. For example, if the different images are paintings of different subjects created by the same artist, the domain is the painting style; if the different images are animal photographs of the same species of animal, the domain is the species of animal. In other words, this selected image D4 is a cross-sectional image of the same type of material as the material to be analyzed, taken by the same type of imaging device as imaging device C, and it is sufficient that carbon black particles P, knife marks N, and rubber components (polymers, etc.) are present. The selected image D4 and the damaged image 8a do not need to be paired images. Not paired means that the domains are different. For example, if the damaged image 8a is a simulation image, it is preferable that the selected image D4 is an image actually taken by the imaging device. The selected image D4 can be chosen from the vast amount of data accumulated by those who manufacture and develop analytical materials, and can be an image that has a domain that can be considered identical to the domain of the original image D1. However, the original image D1 can also be used.

[0053] The number of pre-prepared selection images D4 is, for example, between 1 and 20. By applying image processing such as inversion and rotation to one selection image D4, a dozen or so duplicate images can be created. If there are multiple original images D1, a larger total number of selection images D4 and their duplicates is desirable. This total number is, for example, between a dozen and several hundred, and at most between one thousand and two thousand. Since the damaged image 8a and the selection image D4 are unpaired images, the same selection image D4 can be used in different samples. Therefore, the number of selection images D4 can be as small as one, and significantly less than the number of damaged images 8a.

[0054] Various known domain transformation methods can be used for data processing to transform the domain of the damaged image 8a. These domain transformation methods include, for example, a transformation method using histogram matching that references the selected image D4, and a transformation method using a transformation model 10 constructed by unsupervised machine learning with the selected image D4 and the damaged image 8a as unpaired training data.

[0055] In the transformation method using histogram matching, the domains of the damaged image 8a are transformed by histogram matching referencing the selected image D4, and a transformed damaged image 8d is created. In this histogram matching, the pixel values ​​of each pixel corresponding to the particle P, knife mark N, and rubber component in the damaged image 8a are transformed to approximate the pixel values ​​of each pixel constituting the particle P, knife mark N, and rubber component in the selected image D4.

[0056] In the transformation method using transformation model 10, the transformation model 10 is constructed using unsupervised machine learning with the selected image D4 and the damaged image 8a as unpaired training data. Then, the damaged image 8a is input to the transformation model 10, and data processing is performed by the processing unit 2 to output the transformed damaged image 8d. This transformation model 10 is a type of computer program that transforms the input damaged image 8a into the transformed damaged image 8d. The transformation model 10 is constructed using unsupervised machine learning with the damaged image 8a and the selected image D4 from dataset 8A as unpaired images from different domains. Various known generative adversarial networks that can use images from different domains as training data, such as CycleGAN and StarGAN, can be used to construct the transformation model 10.

[0057] Figure 11 shows an example of the procedure for constructing the transformation model 10. CycleGAN, a type of generative adversarial network, is used to construct the transformation model 10. CycleGAN has two sets of neural networks: one with a generator 11a and a discriminator 12a, and another with a generator 11b and a discriminator 12b. In unsupervised machine learning with CycleGAN, the transformation and restoration of domains, indicated by the solid arrows in Figure 11 and the dashed arrows, are repeated. In Figure 11, X represents a sample consisting of multiple damaged images 8a, y represents a fake sample where the domain of sample X has been transformed by the generator 11a, and x* represents a restored sample where the domain of fake sample y has been restored by the generator 11b. Similarly, Y represents a sample consisting of multiple selected images D4, x represents a fake sample where the domain of sample Y has been transformed by the generator 11b, and y* represents a restored sample where the domain of fake sample x has been restored by the generator 11a.

[0058] The discrimination unit 12a determines the truth value of the fake sample y and the reconstructed sample y* for sample Y. Based on the truth value determination result of the discrimination unit 12a, the generation unit 11a and the discrimination unit 12a are trained (their weights are changed) through backpropagation. Similarly, the discrimination unit 12b determines the truth value of the fake sample x and the reconstructed sample x* for sample X. Based on the truth value determination result of the discrimination unit 12b, the generation unit 11b and the discrimination unit 12b are trained (their weights are changed) through backpropagation. In this way, the two sets of neural networks compete to construct the transformation model 10 so as to minimize both adversarial loss and cycle consistency loss.

[0059] Figure 12 shows a magnified view of the converted damaged image 8d from Figure 8 described above. In this converted damaged image 8d, the shape and distribution of knife marks N and particles P can be considered to be approximately the same as in the original damaged image 8a, and the pixel values ​​of each pixel can be considered to be approximately the same as in the selected image D4. Thus, in the conversion model 10, the unpaired damaged image 8a and the selected image D4, which have different domains, are associated in a way that allows for bidirectional conversion of their respective domains. Therefore, the conversion model 10 can create a converted damaged image 8d that more closely approximates the captured image by referring to the selected image D4 and converting the domain of the simulated damaged image 8a to the same domain as the selected image D4.

[0060] Each damaged image 8a from the dataset 8 is input to the constructed transformation model 10, and each transformed image 8d is created. The constructed transformation model 10 can be used any number of times as long as the domains of the damaged images 8a and the original image D1 are the same. Therefore, when creating transformed images 8d of a different sample S taken from the same analytical material, steps S150 and S160 can be omitted.

[0061] In step S160, the computing unit 2 performs data processing to create a dataset 8A in which the undamaged image 8b and the converted damaged image 8d are combined. Next, steps S130 and S140 described above are performed to use dataset 8A as training data for machine learning.

[0062] Figure 13 shows the original image D1A, and Figure 14 shows the processed image D2A obtained by inputting the original image D1A into a prediction model 9 constructed using the dataset 8A exemplified in Figure 8 as training data. In the processed image D2A, the knife marks N and components other than carbon black particles P (e.g., rubber components) present in the original image D1A have been largely removed. Therefore, in this processed image D2A, the shape and location of each carbon black particle P are clearer compared to the processed image D2 exemplified in Figure 7 above. In addition, since the prediction model 9 can remove knife marks N and components other than particles P with higher accuracy, it is possible to avoid the unnecessary removal of particles P.

[0063] As described above, according to Modification 1, the domain of the transformed damaged image 8d used to construct the prediction model 9 is transformed to the same domain as the original image D1. Therefore, the prediction model 9 constructed using the transformed damaged image 8d can remove the knife marks N in the original image D1 with higher accuracy and has less impact on the particles P compared to the prediction model 9 constructed using the damaged image 8a before domain transformation. This is advantageous for more accurately understanding the actual shape and distribution of particles P.

[0064] The construction of the prediction model 9 requires a large number of pairs of images, each containing an explanatory variable (condition) and a target variable (ground truth or label), and it is desirable that the domain of the image used as the target variable is the same as the domain of the original image D1. However, creating a flawless image 8b by removing noise such as knife marks N from a flawed image 8a obtained by an imaging device C or the like requires considerable effort. According to Modification 1, if at least one selected image D4 is prepared in advance, the domain of the flawed image 8a created by simulation can be converted to the same domain as the original image D1. Therefore, increasing the number of samples in the dataset 8A by simulation and converting the domain of the image used as the explanatory variable to the same domain as the original image D1 by referring to the pre-prepared selected image D4 is advantageous for constructing a prediction model 9 with improved prediction accuracy. In this way, even if the number of suitable images that can be prepared in advance is limited, highly accurate image analysis adapted to the internal structure of the analytical material becomes possible, thus greatly contributing to material development.

[0065] In the first modification, the undamaged image 8b can be input into a transformation model constructed using unsupervised machine learning with the selected image D4 and the undamaged image 8b as unpaired training data to obtain the transformed damaged image 8d. In this transformation model, unwanted elements such as knife marks N in the selected image D4 are included in the domain. Therefore, this transformation model can create a transformed damaged image 8d that more closely approximates the captured image by referencing the selected image D4 and transforming the domain of the simulated undamaged image 8b into a domain that includes the same unwanted elements such as knife marks N as the selected image D4.

[0066] Next, we will describe a modified example of the data processing method and system for images of analytical materials.

[0067] In Modification 2, an electron microscope is used as the imaging device C, and the original image D1B, exemplified in Figure 15 described later, is different from the original image D1A, exemplified in Figure 13 described above, but the configuration of the data processing system 1 is the same. In Modification 2, only the image 8e without unwanted objects (corresponding to the previously described image 8b without defects) is created by simulation, and the image 8f with unwanted objects (corresponding to the previously described image 8a with defects and the transformed image 8d with defects) is created by transforming the domain of the image 8e without unwanted objects. That is, in Modification 2, the dataset 8B, which is an aggregate of the image 8e without unwanted objects and the image 8f with unwanted objects, is used to construct the machine learning prediction model 9.

[0068] The original image D1B illustrated in Figure 15 was acquired using an electron microscope as the imaging device C and is stored in the auxiliary storage unit 6 of the computing device 2. The sample S is taken from vulcanized rubber containing multiple types of rubber components (polymers) G1 and G2, with carbon black as a filler. In this original image D1B, the black areas represent carbon black particles P, the lighter white areas represent one of the rubber components G1, and the lighter gray areas represent the other rubber component G2.

[0069] In the original image D1B, rubber components G1 and G2 are irregularly mixed throughout, making it difficult to understand the size and distribution of carbon black particles P as part of the internal structure of the analytical material. The degree of mixing of rubber components G1 and G2 varies depending on the vulcanized rubber compound and manufacturing conditions, and also differs between different locations even within the same sample S. When trying to understand the size and distribution of carbon black particles P as part of the internal structure of the analytical material, the multiple types of rubber components G1 and G2 present throughout the original image D1B are unwanted elements. In addition, the original image D1B also contains Poisson noise and luminance noise (blown-out highlights) N1, which are unwanted elements throughout the image. Removing all of these unwanted elements from the original image D1B would require a great deal of effort.

[0070] In the data processing method procedure of Modification 2 illustrated in Figure 16, steps S210 to S230 are added in place of step S120 in the data processing method procedure illustrated in Figure 2 described above. In the procedure of Modification 2, an image 8e without unwanted objects is created (S210), and the domain of the image 8e without unwanted objects is transformed to create an image 8f with unwanted objects (S220). Next, a dataset 8B is created by aggregating the image 8e without unwanted objects and the image 8f with unwanted objects (S230). Therefore, in the construction of the prediction model 9 using machine learning in step S130, the created dataset 8B is used. The details of each step S210 to S230 are described below.

[0071] In step S210, data processing is performed by the computing unit 2 to create an image 8e free of unwanted objects using a simulation that utilizes predetermined features extracted from the original image D1. This image 8e free of unwanted objects corresponds to the previously described image 8b free of scratches and is created by a simulation similar to the simulation used to create the image 8b free of scratches in step S120, as illustrated in Figure 3 above.

[0072] In step S220, the arithmetic unit 2 performs data processing to transform the domain of the unwanted image 8e by referring to the selected image D4B, which has a domain that can be considered identical to the domain of the original image D1B. As a result of this data processing, an image 8f with unwanted objects is created in which the domain of the unwanted image 8e is transformed to the same domain as the selected image D4B.

[0073] The selected image D4B illustrated in Figure 17 is selected from images that have domains that can be considered identical to the domains of the original image D1B, and is stored in advance in the auxiliary storage unit 6 of the computing unit 2. For example, the selected image D4B is an image of the same type of material as the analytical material, taken by an electron microscope, and contains carbon black particles P and multiple types of rubber components G1 and G2. The selected image D4B can be selected from the vast amount of data accumulated by those who manufacture and develop analytical materials, and can be images that have domains that can be considered identical to the domains of the original image D1B, but the original image D1B can also be used. Furthermore, it is preferable to select an image D4B that has noise (such as Poisson noise or luminance noise N1) that is similar to the noise present in the original image D1B, as unwanted material. The number of selected images D4B to be prepared in advance is, for example, between 1 and 20.

[0074] The data processing for transforming the domain of the unwanted image 8e includes, for example, a transformation method using histogram matching that references the selected image D4B, and a transformation method using a transformation model 10 constructed by unsupervised machine learning with the selected image D4B and the unwanted image 8e as unpaired training data. Since this data processing for transforming the domain is the same as step S150 illustrated in Figure 9 above, a detailed explanation is omitted.

[0075] In step S230, the processing unit 2 performs data processing to create a dataset 8B in which images 8e without unwanted objects and images 8f with unwanted objects are accumulated. Dataset 8B is used as training data in step S130.

[0076] The dataset 8B illustrated in Figure 18 is stored in the auxiliary storage unit 6 of the arithmetic unit 2. The dataset 8B is a collection of images 8e without unwanted objects and images 8f with unwanted objects for each sample (1, 2, ..., m) shown in the leftmost column of the table. For the same sample, the image 8e without unwanted objects and the image 8f with unwanted objects can be considered as a pair of images.

[0077] Figure 19 shows a magnified view of image 8f containing unwanted particles. Image 8f with unwanted particles approximates the size and distribution of carbon black particles P to the simulation image 8e without unwanted particles, which is the image before conversion. However, it contains various unwanted particles similar to those in selected image D4B, such as rubber components G1 and G2, Poisson noise, and luminance noise N1.

[0078] Next, in step S130, data processing is performed to construct a prediction model 9 by performing machine learning using dataset 8B as training data on the computing device 2. Then, in step S140, data processing is performed to acquire the processed image D2B using the prediction model 9 on the computing device 2.

[0079] The processed image D2B, illustrated in Figure 20, has had unwanted elements such as rubber components G1 and G2, Poisson noise, and luminance noise N1 present in the original image D1B removed. Thus, in the processed image D2B, in addition to unwanted noise such as Poisson noise and luminance noise N1, the remaining components of the analytical material, excluding the carbon black particles P which are the object of observation, have been removed as unwanted elements.

[0080] As described above, according to Modification 2, in the prediction model 9, in addition to noise such as knife marks N, Poisson noise, and luminance noise N1, the remaining components of the analytical material, excluding the desired object of observation, are also treated as unwanted. Therefore, in the processed image D2B obtained using the prediction model 9, the size and distribution of carbon black particles P, which are the object of observation, become clearer, which is advantageous for understanding the internal structure of the analytical material.

[0081] In the embodiments and modifications 1 and 2 described above, the training data used to construct the prediction model 9 can be an image without unwanted objects 8e (image with defects 8b) or an image with unwanted objects 8f (image with defects 8a, converted image with defects 8d) in which the contours of carbon black particles P, which are the object of observation, have been extracted (enhanced). In the prediction model 9 constructed using the image from which the contours of particles P have been extracted, the pixel values ​​of each pixel constituting the carbon black particles P are reconstructed, and a processed image is obtained in which particles P in a state from which their contours have been extracted exist.

[0082] The dataset 8C illustrated in Figure 21 is a collection of images 8e without unwanted objects and images 8f with unwanted objects, in which the contours of carbon black particles P, the object of observation, are extracted (enhanced). In the image 8e without unwanted objects, each particle P is surrounded by a white line, and its contour is extracted (enhanced). This image 8e without unwanted objects may be created by applying contour extraction image processing to an image created by simulation, or it may be created directly by simulation. When the image 8e without unwanted objects is created directly by simulation, an image with a white contour and a black interior should be used as the image of the particle P. The contour of each particle P should preferably be white, and the pixel value of each pixel constituting the part corresponding to this contour is, for example, 255 in a grayscale image.

[0083] Figure 22 shows the original image D1C, and Figure 23 shows the processed image D2C obtained by applying the prediction model 9, constructed using the dataset 8C exemplified in Figure 21 as training data, to the original image D1C. In the processed image D2C, unwanted objects from the original image D1C have been removed. In addition, the contours of the carbon black particles P, which are the object of observation, have been extracted in the processed image D2C.

[0084] Thus, in the prediction model 9 constructed using the image 8e without unwanted objects and the image 8f with unwanted objects, from which the contour of particle P has been extracted, the pixel values ​​of each pixel constituting particle P in the original image are reconstructed and converted into an image of particle P from which the contour has been extracted. As a result, by inputting the original image D1C into the prediction model 9, a processed image D2C containing particle P from which the contour has been extracted can be obtained. Therefore, by using an image from which the contour of particle P has been extracted as training data used to construct the prediction model 9, the effort of extracting the contour of particle P can be omitted.

[0085] The embodiments and modifications 1 and 2 described above are not limited to vulcanized rubber as the analytical material, but are also applicable to unvulcanized rubber, resins, metals, etc. Furthermore, the internal structure (object of observation) that can be grasped using the post-processed images D2, D2A, and D2B obtained using the prediction model 9 is not limited to the distribution state of carbon black, but may also be the distribution state of other compounding components that appear as particles in the original images D1, D1A, and D1B. Moreover, it may also be the distribution state of molecular chains constituting the network structure of the vulcanized rubber that appear in the original image acquired using an electron microscope as the imaging device C. [Examples]

[0086] Using samples A to C, taken from the same analytical material at different cutting points, we evaluated the predictive model 9 constructed using the same procedure as the data processing method exemplified in Figure 3 above.

[0087] Figure 24 shows the original image D1, processed image D2, and extracted image D3 for samples A to C. The analytical material used was vulcanized rubber with a carbon black content of 30 parts by mass per 100 parts by mass of polymer. Samples A to C, each less than 1 mm thick, were collected from the analytical material sample, which was several centimeters in both length and width, using a razor. The imaging device C was an optical microscope with a camera and a magnification of 100x.

[0088] A common dataset 8 for samples A-C was created by simulation using predetermined features extracted from each original image D1 acquired by imaging device C, and a predictive model 9 was constructed using this dataset 8 as training data. The predetermined features used were the size and distribution state of particles P, the width B and brightness of knife marks N, the spacing L between knife marks N, and the relative position of particles P and knife marks N, extracted from each original image D1. Simulation parameters were set based on these features. In the simulation, 1000 grayscale (8-bit) images with scratches 8a and 1000 images without scratches 8b were created with an image size of 600x600 and a white background color (grayscale value 255). Image without scratches 8b and image with scratches 8c were used to create image with scratches 8a. For the simulation parameters of image without scratches 8b, particles P were treated as circles, and the number of circles was randomly selected from the range of 1 to 200, and the brightness of the circles was randomly selected from the grayscale value range of 0 to 20. Furthermore, the circles were selected according to a log-normal distribution with a mean radius of 2 pixels and a standard deviation of 0.5. The position of the circles was randomized. For the simulation parameters of the scratch image 8c, the angle of the knife mark N was randomly selected from within the range of 0° to 180°, the brightness from within the range of grayscale values ​​of 0 to 100, the width B from either 1 pixel or 2 pixels, and the interval L from the width B up to a maximum of 20 pixels. The processed image D2 was obtained using the constructed prediction model 9. Based on the images obtained by applying binarization to each processed image D2, the extracted image data D3 was obtained.

[0089] Figure 25 shows the evaluation results of prediction model 9 using a loss function as the evaluation function, with the horizontal axis representing the number of epochs (number of training iterations) and the vertical axis representing the loss. In Figure 25, the solid line graph shows the result of the loss function (Training Loss) for dataset 8, and the dashed line graph shows the result of the loss function (Validation Loss) for samples A to C. As illustrated in Figure 25, the prediction model 9 in the example shows that the loss decreases as training progresses, and the prediction accuracy increases. Therefore, it can be seen that the knife marks N in the original image D1 are removed with high accuracy in the processed image D2 obtained using this prediction model 9.

[0090] Furthermore, this prediction model 9 has a loss of 0.05 or less even with only one training iteration (number of epochs). Therefore, it can be seen that the prediction model 9, constructed using machine learning with a dataset 8 created by simulation using predetermined features extracted from each original image D1, can achieve high prediction accuracy with a small number of training iterations. [Examples]

[0091] Using the same source image D1, the degree to which the observed object disappears was compared between the data processing method exemplified in Figure 3 and the data processing method of Modification 1 exemplified in Figure 9.

[0092] Figure 26 shows the original image D1 of Example 2. This original image D1 was obtained using the same imaging device C as in Example 1, using a sample taken from the same type of vulcanized rubber as in Example 1. The data processing method exemplified in Figure 3 above was performed on this original image D1 using the dataset 8 constructed in Example 1 to obtain the processed image D2 exemplified in Figure 27. Furthermore, the data processing method exemplified in Figure 9 above was performed on this original image D1 based on the dataset 8 to obtain the processed image D2A exemplified in Figure 28. The selected image D4 was the image exemplified in Figure 10 above.

[0093] In Figures 26-28, P1 and P2 represent the same carbon black particles. Comparing the shape and size of particles P1 and P2 in each image, it can be seen that in processed image D2A, exemplified in Figure 28, the degree of disappearance of carbon black particles P due to the removal of knife marks N is smaller compared to processed image D2, exemplified in Figure 27. Therefore, it can be seen that in processed image D2A, exemplified in Figure 28, the knife marks N have been removed with higher precision compared to processed image D2, exemplified in Figure 27. [Examples]

[0094] Using the procedure for the data processing method of Modification 2 illustrated in Figure 16 above, the molecular chains constituting the network structure of vulcanized rubber were observed.

[0095] Figure 29 shows the original image D1B of Example 3. In Figure 29, the white area represents the molecular chain P3 of the vulcanized rubber, and the black area represents the embedding resin P4. This original image D1B was acquired using a transmission electron microscope as imaging device C. As a sample, a vulcanized rubber specimen prepared by vulcanizing isoprene rubber with sulfur was used. The specimen was swollen with styrene and a curing agent for 48 hours, and then heated for 24 hours to polymerize and cure. Ultrathin sections of 30 nm to 70 nm were prepared from the specimen using an ultramicrotome and stained with osmium tetroxide. A transmission electron microscope was used as imaging device C. The transmission electron microscope was set to an acceleration voltage of 200 [kV] and a magnification of 60,000x, and the prepared ultrathin sections were photographed to obtain the original image D1B of the ultrathin section exemplified in Figure 29.

[0096] Figure 30 shows a portion of the dataset 8B from Example 3. Image 8e, which is free of unwanted substances, is a simulation image and contains an image that can be considered to be the molecular chain P3. In image 8f, which contains unwanted substances, the domain of image 8e is transformed into a domain that can be considered identical to the domain of the original image D1B exemplified in Figure 29 above. By performing machine learning using this dataset 8B as training data, the processed image D2B exemplified in Figure 31 was obtained using the constructed prediction model 9.

[0097] The processed image D2B, illustrated in Figure 31, shows a clearer boundary between the vulcanized rubber molecular chain P3 and the embedding resin P4 compared to the original image D1B, illustrated in Figure 29, with unwanted noise removed. Thus, the present invention demonstrates that, when the shape of the object to be observed in the analytical material is known and approximates a specific shape, even a simple simulation that creates an image mimicking that shape can effectively reduce unwanted noise.

[0098] The present invention is not limited to any particular embodiment, and various modifications and changes are possible within the scope of the gist of the invention.

[0099] This disclosure encompasses the following inventions: Invention 1: A data processing method for images of the cross-section of an analytical material, which removes unwanted scratches caused by material cutting present in the original image of the cross-section of the analytical material, A predictive model is constructed by performing machine learning using a dataset as training data, which consists of numerous cross-sectional images of the same type of material as the aforementioned analytical material, including images with unwanted defects and images without defects that do not have the unwanted defects. A data processing method for cross-sectional images of analytical materials, wherein a processed image is obtained from which unwanted scratches have been removed from the original image by performing data processing on the original image using the prediction model stored in the computing device. Invention 2: A data processing method for cross-sectional images of an analytical material according to Invention 1, wherein at least some of the damaged images and the undamaged images are images generated by a simulation using predetermined feature quantities extracted from the original image. Invention 3: The data processing method for cross-sectional images of analytical materials according to Invention 2, wherein the simulation generates an image without defects and an image with defects where only unwanted defects are present, and then generates the image with defects by combining the image without defects and the image with defects. Invention 4: A data processing method for a cross-sectional image of an analytical material according to Invention 2 or 3, wherein the predetermined feature quantities include the size and distribution state of particles present in the original image, the width of the unwanted scratches, the brightness of the scratches, the spacing between the scratches, and the relative positions of the particles and the unwanted scratches. Invention 5: A data processing method for cross-sectional images of an analytical material according to any one of Inventions 1 to 4, wherein the original image and the processed image are used as the damaged image and the undamaged image, respectively, to update the prediction model. Invention 6: A data processing method for cross-sectional images of an analytical material according to any one of Inventions 1 to 5, using a semantic segmentation model as the prediction model. Invention 7: A data processing method for a cross-sectional image of an analytical material according to any one of inventions 1 to 6, wherein the processed image is subjected to binarization processing by the computing device. Invention 8: A method for processing data of a cross-sectional image of an analytical material according to any one of Inventions 1 to 7, wherein the analytical material is vulcanized rubber or unvulcanized rubber containing at least a filler. Invention 9: A data processing system for images of the cross-section of an analytical material, comprising a calculation device for removing unwanted scratches caused by material cutting present in the original image of the cross-section of the analytical material, and an output device, The computing device has a storage unit that stores a predictive model constructed by machine learning using a dataset as training data, which is a collection of cross-sectional images of the same type of material as the analytical material, consisting of images with defects where the unwanted defects are present and images without defects where the unwanted defects are absent compared to the images with defects; a computing unit; and an input unit. A data processing system for cross-sectional images of analytical materials, wherein the original image input to the arithmetic unit through the input unit is subjected to data processing by the arithmetic unit using the prediction model to remove unwanted scratches, and the processed image is output to the output unit. Invention 10: A data processing method for images of analytical materials, which removes predetermined unwanted substances present in the original image of the analytical material, A predictive model is constructed by performing machine learning using a dataset as training data, which consists of numerous images of the same type of material as the aforementioned analytical material, including images with the predetermined unwanted material present and images without the predetermined unwanted material that do not exist in the images with the unwanted material. A data processing method for images of analytical material, wherein data processing is performed on the original image using the prediction model stored in the computing device, thereby obtaining a processed image from which predetermined unwanted objects have been removed from the original image. Invention 11: The data processing method for images of analytical materials according to Invention 11, wherein at least some of the images with unwanted substances and the images without unwanted substances are images created by simulation using predetermined feature quantities extracted from the original images. Invention 12: The data processing method for images of analytical materials according to Invention 11, wherein the simulation creates an image without unwanted substances and an image containing only the predetermined unwanted substances, and then creates an image with unwanted substances by combining the image without unwanted substances and the image containing unwanted substances. Invention 13: The data processing method for images of analytical materials according to invention 11 or 12, wherein, in the simulation, after creating the image without unwanted substances, the calculation device performs data processing to convert the domain of the image without unwanted substances to the same domain as the selected image by referring to a selected image having a domain that can be considered identical to the domain of the original image, thereby creating the image with unwanted substances. Invention 14: In the above simulation, after creating the image with unwanted objects and the image without unwanted objects, the arithmetic unit performs data processing to convert the domain of the image with unwanted objects to the same domain as the selected image by referring to a selected image that can be considered to have the same domain as the original image, thereby creating a converted image with unwanted objects. The data processing method for images of analytical materials according to invention 11 or 12, wherein the converted image containing unwanted substances is used as the image containing unwanted substances in the construction of the prediction model. Invention 15: The data processing method for images of analytical material according to claim 13 or 14, wherein the data processing for transforming the domain uses a transformation model constructed by a generative adversarial network. Invention 16: The data processing method for images of analytical materials according to Invention 15, which uses CycleGAN as the generative adversarial network. Invention 17: A data processing method for images of analytical materials according to any one of inventions 11 to 16, using a semantic segmentation model as the prediction model. Invention 18: A data processing method for an image of an analytical material according to any one of inventions 11 to 17, wherein the processed image is subjected to binarization processing by the computing device. Invention 19: A data processing method for an image of an analytical material according to any one of Inventions 11 to 18, wherein the predetermined unwanted matter is the remaining unwanted matter and / or noise obtained by removing a predetermined object to be observed from the original image to the processed image. Invention 20: A data processing method for an image of an analytical material according to any one of Inventions 11 to 19, wherein the original image is an image of the cross-section of the analytical material, and the predetermined unwanted material is an unwanted scratch caused by cutting the material. Invention 21: A method for processing images of an analytical material according to any one of inventions 11 to 20, wherein the analytical material is vulcanized rubber or unvulcanized rubber containing at least a filler. Invention 22: A data processing system for images of analytical materials, comprising a calculation device for removing predetermined unwanted substances present in the original image of the analytical material, and an output device, The computing device has a storage unit that stores a predictive model constructed by machine learning using a dataset as training data, which is a collection of images of the same type of material as the analytical material, in which the predetermined unwanted material is present and the predetermined unwanted material is absent in the images with the unwanted material. It also has a computing unit and an input unit. A data processing system for images of analytical materials, wherein the original image input to the arithmetic unit through the input unit is subjected to data processing by the arithmetic unit using the prediction model to remove predetermined unwanted objects, and the processed image is output to the output device. [Explanation of Symbols]

[0100] 1. Data Processing System 2 Arithmetic unit 3. Output device 4 Arithmetic section 5 Main memory 6 Auxiliary storage section (memory section) 7 Input section 8, 8A, 8B, 8C datasets 8a Damaged image (image with unwanted items) 8b Image without damage (image without unwanted objects) 8c Damage image (image of unwanted item) 8d Image with scratches after conversion (Image with unwanted objects after conversion) 8e Image without unwanted elements Image 8f contains unwanted items 9. Predictive Models C Imaging device D1, D1A, D1B, D1C Original Images Images after processing D2, D2A, D2B, and D3C. D3 Extracted Image D4, D4A Selected Images N knife mark P particles N1 Luminance noise

Claims

1. A data processing method for images of analytical materials, which removes predetermined unwanted substances present in the original image of the analytical material, A predictive model is constructed by performing machine learning using a dataset as training data, which consists of numerous images of the same type of material as the aforementioned analytical material, including images with the predetermined unwanted material present and images without the predetermined unwanted material that do not exist in the images with the unwanted material. A data processing method for images of analytical material, wherein data processing is performed on the original image using the prediction model stored in the computing device, thereby obtaining a processed image from which predetermined unwanted objects have been removed from the original image.

2. The data processing method for images of analytical materials according to claim 1, wherein at least some of the images with unwanted substances and the images without unwanted substances are images created by a simulation using predetermined feature quantities extracted from the original images.

3. The data processing method for images of analytical materials according to claim 2, wherein in the simulation, an image without unwanted substances and an image containing only the predetermined unwanted substances are created, and then the image with unwanted substances is created by combining the image without unwanted substances and the image containing unwanted substances.

4. The data processing method for an image of an analytical material according to claim 2, wherein, in the simulation, after creating the image without unwanted objects, the arithmetic unit performs data processing to convert the domain of the image without unwanted objects to the same domain as the selected image by referring to a selected image having a domain that can be considered identical to the domain of the original image, thereby creating the image with unwanted objects.

5. In the above simulation, after creating the image with unwanted objects and the image without unwanted objects, the arithmetic unit performs data processing to convert the domain of the image with unwanted objects to the same domain as the selected image by referring to a selected image that can be considered to have the same domain as the original image, thereby creating a converted image with unwanted objects. The data processing method for images of analytical materials according to claim 2, wherein in the construction of the prediction model, the converted image containing unwanted substances is used as the image containing unwanted substances.

6. The data processing method for images of analytical material according to claim 4 or 5, wherein the data processing for transforming the domain uses a transformation model constructed by a generative adversarial network.

7. The data processing method for images of analytical materials according to claim 6, wherein CycleGAN is used in the generative adversarial network.

8. The data processing method for images of analytical materials according to any one of claims 1 to 5, wherein a semantic segmentation model is used as the prediction model.

9. A data processing method for an image of an analytical material according to any one of claims 1 to 5, wherein the processed image is subjected to binarization processing by the computing device.

10. The data processing method for an image of an analytical material according to any one of claims 1 to 5, wherein the predetermined unwanted matter is the remaining unwanted matter and / or noise obtained by removing a predetermined object to be observed from the original image to the processed image.

11. The data processing method for an image of an analytical material according to any one of claims 1 to 5, wherein the original image is an image of the cross-section of the analytical material, and the predetermined unwanted material is an unwanted scratch caused by cutting the material.

12. A method for processing images of an analytical material according to any one of claims 1 to 5, wherein the analytical material is vulcanized rubber or unvulcanized rubber containing at least a filler.

13. A data processing system for images of analytical materials, comprising a calculation device for removing predetermined unwanted substances present in the original image of the analytical material, and an output device, The computing device has a storage unit that stores a predictive model constructed by machine learning using a dataset as training data, which is a collection of images of the same type of material as the analytical material, in which the predetermined unwanted material is present and the predetermined unwanted material is absent in the images with the unwanted material. It also has a computing unit and an input unit. A data processing system for images of analytical materials, wherein the original image input to the arithmetic unit through the input unit is subjected to data processing by the arithmetic unit using the prediction model to remove predetermined unwanted objects, and the processed image is output to the output device.

Images