Measurement system

Abstract

A measurement system 1 is provided with: an optical imaging device 11 that captures an image of a sample, that calculates features of frequency components by applying an image filter to the captured image of the sample, and that calculates the degree of blurring of the captured image on the basis of the features of the frequency components; an optical measurement device 13 that measures a molecular state of the sample at the same optical focus as the optical focus of the imaging device; and a processing device 19 that controls one or more of the position of a sample stage on which the sample is disposed and the optical focus and that associates the captured image of the sample, molecular state information of the sample, and the degree of blurring of the captured image with one another.

Description

Measurement System

[0001] The present disclosure relates to measurement systems.

[0002] Spectra that indicate chemical properties of materials (identification of chemical composition, total amount, etc.) are used to estimate the chemical properties of the materials. For example, to investigate the chemical properties of substances present on the surface of a material, there is laser Raman spectroscopy, which measures two-dimensional Raman spectra of the material surface using a spectrometer equipped with an optical microscope (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2001-215193

[0004] In laser Raman spectroscopy, laser light is spatially focused for measurement. Therefore, in the case of materials with uneven surfaces, it is difficult to precisely align the focal point of the laser light with the surface position of the material, and the spectral information can be degraded due to a misalignment between the focal point of the laser light and the surface position of the material.

[0005] Therefore, it was difficult to distinguish whether the observed spectral change was a true spectral change resulting from the mixture or an unintended spectral change due to a shift in the focal point, which reduces reliability when identifying the type of material.

[0006] The present disclosure has been made in consideration of the above circumstances, and an object of the present disclosure is to provide a technology that can easily add the quality of characteristic information to characteristic information of a material.

[0007] A measurement system according to one aspect of the present disclosure includes an optical imaging device that images a sample, calculates frequency component features by applying an image filter to the image of the sample, and calculates the degree of blur of the image based on the frequency component features; an optical measurement device that measures the molecular state of the sample at the same optical focus as the optical focus of the imaging device; and a processing device that controls one or more of the position of a sample stage on which the sample is placed and the optical focus, and associates the image of the sample, molecular state information of the sample, and the degree of blur of the image.

[0008] According to the present disclosure, a technique can be provided that can easily add the quality of characteristic information to the characteristic information of a material.

[0009] Fig. 1 is a diagram showing an example of the configuration of a measurement system. Fig. 2 is a diagram showing optical microscope images of samples at different heights on the same plane. Fig. 3 is a diagram showing Raman spectroscopy spectra of samples at different heights on the same plane. Fig. 4 is a diagram showing an image of a data set. Fig. 5 is a diagram showing an example of the operation flow of the measurement system.

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0011] [Summary of the Disclosure] The disclosure relates to a technology for the purpose of material analysis of a material, which performs imaging of the material and spectrum measurement showing the chemical properties of the material, and easily adds quality information of the measured spectrum to the spectrum.

[0012] Specifically, a measurement system is constructed in which the optical system of an optical microscope and the laser optical system of Raman spectroscopy are combined so that their optical foci coincide with each other. Then, the optical microscope measures geometrical shape information of the material, and the Raman spectroscopy measures the Raman spectrum as information on the chemical properties of the material.

[0013] At this time, the focal position of the optical system, i.e., the distance between the objective lens of the optical microscope and the stage where the material is placed, is changed minutely while the material is imaged and its spectrum is measured. In some of the measurements where the focal position is changed, the focal position shifts because the focal position and the material surface position are misaligned, and the optical microscope image is measured as a blurred image.

[0014] This blurring of the optical microscope image is thought to contain information about the height of the structure on the surface of the material, i.e., the three-dimensional shape of the material. In other words, it is highly likely that the Raman spectrum at the position where the blurring of the optical microscope image occurs has a spectral change resulting from a shift in the focal position.

[0015] Therefore, the present disclosure utilizes the blurriness of an optical microscope image as quality information of the spectrum. The blurriness of the optical microscope image is appropriately calculated, and the blurriness is associated with the spectrum as quality information of the spectrum, and stored together with the three-dimensional position information of the material. The three-dimensional positions of the material are associated with each other during measurement of the optical microscope image and measurement of the spectrum of the material.

[0016] 1 is a diagram showing an example of the configuration of a measurement system 1 according to this embodiment. The measurement system 1 is configured by combining the optical system of an optical microscope and the laser optical system of Raman spectroscopy measurement so that their optical focal points coincide with each other.

[0017] The measurement system 1 includes an imaging device (optical microscope) 11, a probe laser light source 12, a measurement device (Raman spectrometer) 13, a mirror 14, a first half mirror 15, a second half mirror 16, an objective lens 17 with a white light source, a sample stage 18, a processing device 19, and a machine learning system 20.

[0018] The optical system of the optical microscope is arranged so that white light is irradiated from the objective lens 17 at an appropriate intensity onto a sample (material) 100 placed on a sample stage 18, and an image of the sample 100 is formed on the light-receiving element of the imaging device 11. The imaging device 11 images the sample 100 and calculates the degree of blur of the optical microscope image of the sample 100 from the imaging information.

[0019] Furthermore, a laser optical system for Raman spectroscopy measurement is arranged so that the laser light from the probe laser light source 12 is focused on the sample 100. The Raman scattered light from the sample 100 is guided to a spectrometer in the measurement device 13, and the scattered light intensity (Raman spectrum) appropriately separated into wavelengths by a grating in the spectrometer is measured by a photoreceiver in the measurement device 13.

[0020] In addition, the processing device 19 acquires optical microscope image data of the sample 100, acquires blur data of the optical microscope image of the sample 100, acquires Raman scattering spectrum data of the sample 100, acquires three-dimensional position information of the sample 100, and calculates and saves the data set.

[0021] The optical system of the optical microscope and the laser optical system for Raman spectroscopy can be switched appropriately at the timings of imaging the sample 100 and measuring the Raman scattering spectrum.

[0022] An example of the configuration of the measurement system 1 will be described in detail below.

[0023] The mirror 14, the first half mirror 15, the second half mirror 16, and the objective lens 17 are arranged in a line in that order with respect to the sample 100, and are shared by the optical system of the optical microscope and the laser optical system for Raman spectroscopy measurement.

[0024] The mirror 14 is disposed so as to be inclined at 45 degrees relative to the sample 100 toward the imaging device 11. The first half mirror 15 is disposed so as to be inclined at 45 degrees relative to the sample 100 toward the measuring device 13. The second half mirror 16 is disposed so as to be inclined at 45 degrees relative to the sample 100 toward the probe laser light source 12.

[0025] The imaging device 11 is arranged so that a light receiving element therein faces the mirror 14. The imaging device 11 is an optical shape imaging and measurement device that has an imaging function for imaging the sample 100 and a calculation function for calculating feature amounts of two-dimensional frequency components by applying an image filter to an optical microscope image (captured image) of the sample 100, and for calculating the degree of blur of the optical microscope image based on the feature amounts of the two-dimensional frequency components. The imaging device 11 is, for example, an optical microscope.

[0026] The probe laser light source 12 is arranged such that a laser light output unit in the probe laser light source 12 faces the second half mirror 16. The probe laser light source 12 is a light source device that outputs laser light to the sample 100 via the second half mirror 16 and the objective lens 17.

[0027] The measuring device 13 is arranged such that a light receiving element therein faces the first half mirror 15. The measuring device 13 is an optical molecular state measuring device that measures a two-dimensional Raman spectroscopy spectrum (molecular state) of the surface of the sample 100 at the same optical focus as the optical focus of the imaging device 11. The probe laser light source 12 and the measuring device 13 are, for example, a Raman scattering spectrum spectrometer.

[0028] The objective lens 17 is disposed immediately in front of the sample 100. The objective lens 17 is provided with a white light source. The objective lens 17 is generally a component of an optical microscope.

[0029] The processing device 19 is connected to the imaging device 11, the probe laser light source 12, the measuring device 13, the objective lens 17, and the sample stage 18 so as to be able to control each of them. The processing device 19 is connected to the imaging device 11 and the measuring device 13 so as to be able to communicate with each of them. The processing device 19 includes a control unit 191, a calculation unit 192, an identification unit 193, and a storage unit 194.

[0030] The control unit 191 has a function of controlling the operation of the imaging device 11, the operation of the probe laser light source 12, the operation of the measuring device 13, the position of the objective lens 17, the operation of the white light source in the objective lens 17, and the position of the sample stage 18. For example, the control unit 191 controls the position of the objective lens 17 and the position of the sample stage 18, thereby controlling the optical focal position.

[0031] The calculation unit 192 has the function of receiving from the imaging device 11 an optical microscope image (captured image) of the sample 100, the degree of blur of the optical microscope image of the sample 100, and three-dimensional position information of the sample 100 at the time of imaging.

[0032] The calculation unit 192 has a function of receiving, from the measurement device 13, the Raman spectrum (molecular state information) of the sample 100 and three-dimensional position information of the sample 100 at the time of measurement.

[0033] The calculation unit 192 has a function of storing, in the storage unit 194, an optical microscope image (captured image) of the sample 100, a Raman spectrum (molecular state information) of the sample 100, the blurriness of the optical microscope image of the sample 100, and three-dimensional position information of the sample 100, all associated with one another for the same three-dimensional position. At this time, the calculation unit 192 associates the blurriness of the optical microscope image as quality information of the Raman spectrum.

[0034] The identification unit 193 has the function of inputting the Raman spectroscopy spectrum (molecular state information) of the sample 100 to identify the molecular species, etc. of the sample 100, and outputting the identification results of the molecular species, etc. of the sample 100 together with the degree of blur of the optical microscope image.

[0035] The identification unit 193 has the function of identifying the molecular species, etc. of the sample 100 using the Raman spectroscopy spectrum (molecular state information) of the sample 100 and the blurriness of the optical microscope image as input, and outputting the identification results of the molecular species, etc. of the sample 100 together with the blurriness of the optical microscope image.

[0036] The processing device 19 is, for example, a computer including a CPU, a memory, a storage, a communication device, an input device, and an output device. The control unit 191, the calculation unit 192, and the identification unit 193 may be separate computers.

[0037] The machine learning system 20 is communicatively connected to the processing device 19. The machine learning system 20 is an autoencoder machine learning system that uses a combination of two or more of an optical microscope image (captured image) of the sample 100, a Raman spectroscopy spectrum (molecular state information) of the sample 100, and a blur degree of the optical microscope image of the sample 100 as input and output, and extracts desired features in an intermediate layer between the input layer and the output layer.

[0038] [Example of operation of measurement system] In the measurement system 1 shown in Figure 1, optical microscope image capturing and Raman spectrum measurement are performed sequentially while minutely changing the focal position of the optical system, i.e., the distance (height in the Z-axis direction) between the objective lens 17 and the sample stage 18 on which the sample 100 is placed.

[0039] In capturing some optical microscope images during multiple measurements with different focal positions, the distance between the optical focal position and the sample surface shifts, causing a shift in the optical focus, which is measured as a blurred image as shown in Figure 2. Figure 2 shows optical microscope images of samples 100 at different heights on the same plane (X-Y plane).

[0040] This blurring of the optical microscope image is considered to contain height information of the sample surface structure, i.e., the three-dimensional shape. In other words, it is highly likely that the Raman spectrum at the position where the blurring of the optical microscope image occurs has undergone spectral changes due to a shift in the focal position. Therefore, the degree of blurring of the optical microscope image is used as information on the quality of the spectrum.

[0041] An appropriate method can be selected from a variety of methods to evaluate the degree of blur in an optical microscope image. For example, a feature amount corresponding to two-dimensional frequency components can be calculated by convolution of the image with an appropriate image filter, and the resulting feature amount is used as a power distribution from low-frequency components to high-frequency components contained in the image. When the degree of blur is a continuous value, the power distribution can be used as is. When the degree of blur is expressed as a binary value representing the presence or absence of blur, the condition can be set as the high-frequency component power being equal to or greater than a threshold value based on the power distribution rate.

[0042] 2, an area of ​​a predetermined size (area of ​​interest) is set in the image, and the degree of blur of the image for the area of ​​interest P is calculated using the above evaluation index. Raman spectroscopy measurement is performed at multiple locations across a two-dimensional plane including the observation target range, and the area of ​​interest P is set at each point with the focal point of the laser light as the center, and the degree of blur of the optical microscope image is calculated using the above evaluation index.

[0043] Furthermore, when the Raman spectrum is measured while changing the distance between the objective lens 17 and the sample stage 18, the focal position of the laser light from the probe laser light source 12 and the sample surface position are shifted, and as a result, as shown in Figure 3, the spectral shape may differ even when the two-dimensional plane position is the same.

[0044] Therefore, the Raman spectrum is measured by focusing laser light on a two-dimensional plane of the sample 100 while moving in the height direction. Similarly, a blur index of the image is calculated from two-dimensional optical microscope images captured while moving in the height direction. Then, for the same three-dimensional position, the image blur index is associated with the Raman spectrum as quality information of the Raman spectrum.

[0045] By this processing, as shown in FIG. 4, a data set can be obtained that includes optical microscope images I1 to I3 of sample 100 at each point in three-dimensional space, Raman spectroscopy spectra D1 to D3 of sample 100, and associated Raman spectroscopy quality information Q1 to Q3.

[0046] Often, a spectral database of Raman spectra previously measured for known materials or compounds is referenced, and the substances contained in the object of observation are identified from the measured Raman spectrum.

[0047] Therefore, when a single Raman spectrum is input and the identification process result is output, quality information of the single Raman spectrum (image blur index) is presented to the user as the reliability of the identification process, which helps the user make a decision.

[0048] Furthermore, when performing identification processing using multiple Raman spectra as input, quality information for each Raman spectrum is also included in the input, which can be used as reliability information for each Raman spectrum.

[0049] In addition, environmental conditions such as temperature and acidity at the time of sample creation may be estimated from the optical microscope image and Raman spectrum of the sample surface. In this case, a machine learning system is used that inputs the optical microscope image of the sample and the Raman spectrum and outputs the environmental conditions to be estimated.

[0050] In the training process of such machine learning systems, there is a data augmentation method in which training is performed using a large amount of similar data set that has been expanded by slightly modifying the training data. This data augmentation method is known to be effective in improving the generalization ability of machine learning (an index that indicates how accurately predictions can be made about unknown data that is not included in the training data).

[0051] In this embodiment, a dataset is constructed that includes a combination of an optical microscope image of a sample, a Raman spectrum of the sample, and quality information about the Raman spectrum at each point in three-dimensional space. Each piece of data in this dataset is similar to the others, but with minor changes due to misalignment between the sample surface and the optical focal position. Therefore, this dataset is ideal for use as a data augmentation method, since it is a large set of similar data with minor changes that can actually be measured. Therefore, this dataset can be used to train a machine learning system using data augmentation.

[0052] The machine learning system 20 is a machine learning system that trains an autoencoder, which is composed of a deep neural network, using the above-mentioned dataset. The autoencoder is a machine learning system that aims to acquire a condensed information representation in the intermediate layer of the network by progressing through training so that input information and output information become identical (reproduced). The partial network of the autoencoder that has completed training can then be used as a feature extractor as part of a classifier for another classification task, such as estimating environmental information.

[0053] The machine learning system 20 selects any combination of an optical microscope image of the sample, a Raman spectrum of the sample, and quality information of the Raman spectrum from the dataset, sets these as input values ​​for the autoencoder, and proceeds with learning until the output value becomes equal to the input value. After learning is complete, the partial network of the autoencoder, in many cases from the data input layer to the intermediate layer, which is a contracted representation, is used as a feature extractor, and the network configuration information is saved.

[0054] FIG. 5 is a diagram showing an example of an operation flow of the measurement system 1.

[0055] Step S1: First, the control unit 191 of the processing device 19 turns on the white light source of the objective lens 17 to irradiate the sample 100 with white light. Thereafter, the imaging device 11 measures an optical microscope image of the sample 100.

[0056] Step S2: Next, the imaging device 11 calculates the degree of blur of the optical microscope image from the optical microscope image of the sample 100 measured in step S1.

[0057] Step S3: Next, the control unit 191 of the processing device 19 turns off the white light source of the objective lens 17 and turns on the probe laser light source 12. The Raman scattered light scattered on the surface of the sample 100 is collected by the objective lens 17 and input to the measuring device 13. Thereafter, the measuring device 13 separates the input Raman scattered light into wavelengths using a grating and measures the Raman spectrum of the sample 100.

[0058] Step S4: Next, the calculation unit 192 of the processing device 19 receives the measurement data from the imaging device 11 and the measuring device 13, and stores, for the same three-dimensional position, the optical microscope image of the sample 100, the Raman spectrum of the sample 100, the three-dimensional position information of the sample 100, and the degree of blur of the optical microscope image of the sample 100 in association with one another in the storage unit 194. At this time, the calculation unit 192 regards the degree of blur of the optical microscope image as quality information of the Raman spectrum.

[0059] Step S5: Next, the processing device 19 or the user determines whether or not to continue the measurement.

[0060] Step S6: If the measurement is to be continued, the control unit 191 of the processing device 19 changes the distance between the objective lens 17 and the sample 100 by finely adjusting the position of the sample stage 18. Then, the process returns to step S1. Steps S1 to S4 are repeated until the distance reaches the predetermined distance.

[0061] Step S7: If the measurement is to be ended, the processing is ended. If necessary, the identification unit 193 of the processing device 19 identifies the molecular species of the sample 100 using the Raman spectrum of the sample 100 or the blurriness of the optical microscope image, and outputs the identification results of the molecular species of the sample 100 and the blurriness of the optical microscope image to a monitor screen or the like. At this time, the identification unit 193 outputs the blurriness of the optical microscope image as quality information of the Raman spectrum.

[0062] [Modification] In the present embodiment, a Raman scattering spectrometer is used as the measuring device 13, but an infrared, visible, and ultraviolet reflection / absorption spectrometer may also be used.

[0063] [Effects] According to this embodiment, the measurement system 1 includes an optical imaging device 11 that images the sample 100, calculates feature quantities of frequency components by applying an image filter to the optical microscope image of the sample 100, and calculates the degree of blur of the optical microscope image based on the feature quantities of the frequency components; an optical measuring device 13 that measures the Raman spectrum of the sample 100 at the same optical focus as the optical focus of the imaging device 11; and a processing device 19 that controls one or more of the position and optical focus of a sample stage 18 on which the sample 100 is placed, and associates the optical microscope image of the sample 100, the Raman spectrum of the sample 100, and the degree of blur of the optical microscope image of the sample 100 with one another. Therefore, it becomes possible to easily add spectrum quality information to each of a large number of measured three-dimensional Raman spectra by the simple method of calculating the degree of blur of the optical microscope image measured simultaneously with the measurement of the Raman spectrum.

[0064] Furthermore, according to this embodiment, the processing device 19 inputs the Raman spectroscopy spectrum of the sample 100 to identify the molecular species of the sample 100, and outputs the identification results of the molecular species of the sample 100 and the blur degree of the optical microscope image, so that the validity of the identification results can be presented to the user.

[0065] Furthermore, according to this embodiment, the processing device 19 further inputs the degree of blur of the optical microscope image to identify the molecular species of the sample 100, and outputs the identification result of the molecular species of the sample 100 and the degree of blur of the optical microscope image, which can be used as reliability information of each Raman spectrum.

[0066] REFERENCE SIGNS LIST 1 Measurement system 11 Imaging device 12 Probe laser light source 13 Measurement device 14 Mirror 15 First half mirror 16 Second half mirror 17 Objective lens 18 Sample stage 19 Processing device 191 Control unit 192 Calculation unit 193 Identification unit 194 Memory unit 20 Machine learning system 100 Sample

Claims

1. An optical imaging device that images a sample, calculates a feature amount of a frequency component by applying an image filter to the captured image of the sample, and calculates a blur degree of the captured image based on the feature amount of the frequency component; an optical measurement device that measures a molecular state of the sample at the same optical focus as the optical focus of the imaging device; and a processing device that controls one or more of a position of a sample stage on which the sample is disposed and the optical focus, and associates the captured image of the sample, the molecular state information of the sample, and the blur degree of the captured image with each other. A measurement system comprising:

2. The processing device according to claim 1, wherein the processing device inputs the molecular state information of the sample to identify the molecular species of the sample, or inputs the molecular state information of the sample and the blur degree of the optical focus to identify the molecular species of the sample, and outputs the identification result of the molecular species of the sample and the blur degree of the optical focus. The measurement system described.

3. The measurement system according to claim 1, further comprising a machine learning system of an autoencoder that inputs and outputs any combination of two or more of the captured image of the sample, the molecular state information of the sample, and the blur degree of the captured image.

4. The imaging device is an optical microscope, and the measurement device is a Raman scattering spectrum measurement device or an infrared-visible-ultraviolet reflection absorption spectrum measurement device. The measurement system according to any one of claims 1 to 3.

Images