Image processing methods, programs, image processing devices, and imaging mass spectrometers

Abstract

The image processing method disclosed herein includes: a step of acquiring first image data obtained by measuring a first sample using a first measurement means (S10); and a step of setting a candidate group of a first region of interest based on the first image data (S12). Furthermore, the image processing method includes: a step of acquiring second image data obtained by measuring the first sample using a second measurement means different from the first measurement means, reflecting a physical quantity different from the first image data (S14); and a step of setting a region of interest group in the first image data or the second image data based on the candidate group of the first region of interest and the second image data (S16).

Description

Technical Field

[0001] This disclosure relates to image processing methods, programs, image processing apparatus, and imaging mass spectrometers, and more specifically, to image processing methods for defining regions of interest (ROIs) in image data. Background Technology

[0002] Currently, regions of interest are defined for image data obtained from measuring biological samples, and statistics are calculated for these regions of interest.

[0003] As a method for acquiring image data from biological samples, as disclosed in Japanese Patent Application Publication No. 2010-261882 (Patent Document 1), there is a method using an imaging mass spectrometer. Imaging mass spectrometry analysis is a method that obtains mass spectrometry information at various locations on the surface of a biological sample and depicts the peak intensity distribution, thereby obtaining information on the distribution of substances with a specific mass-to-charge ratio. For example, for an image obtained through imaging mass spectrometry analysis of a biological sample, by setting a region of interest using the ion intensity of substances specifically accumulated in the tumor as an indicator, the size of the tumor can be investigated based on the area of ​​the region of interest.

[0004] As a method for defining regions of interest (ROIs) in image data obtained from biological samples, there are known methods that automatically define ROIs based on predetermined conditions. By automatically defining ROIs, the time required for user-defined ROI settings can be saved.

[0005] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2010-261882 Summary of the Invention The technical problem that the invention aims to solve Due to the complex structure of biological samples, methods for automatically defining regions of interest (ROIs) based on predetermined conditions sometimes inadvertently designate regions that should not be considered as ROIs. This can affect the statistical quantities calculated based on those ROIs. Therefore, techniques to improve the accuracy of ROI definition in image data are needed.

[0006] This disclosure is made in view of such practical situations, and its purpose is to provide a technique for improving the accuracy of setting regions of interest in sample analysis image data.

[0007] Solution to the above technical problems The image processing method of the first aspect of this disclosure includes the steps of: acquiring first image data obtained by measuring a first sample using a first measurement means; and setting a candidate group of a first region of interest based on the first image data. Furthermore, the image processing method includes the steps of: acquiring second image data obtained by measuring the first sample using a second measurement means different from the first measurement means, reflecting a physical quantity different from the first image data; and setting a region of interest group in the first image data or the second image data based on the candidate group of the first region of interest and the second image data.

[0008] The image processing apparatus of the second aspect of this disclosure acquires first image data obtained by measuring a first sample using a first measurement means, and sets a candidate group of a first region of interest based on the first image data. The image processing apparatus acquires second image data obtained by measuring the first sample using a second measurement means different from the first measurement means, reflecting a physical quantity different from the first image data, and sets a region of interest group in the first image data or the second image data based on the candidate group of the first region of interest and the second image data.

[0009] The imaging mass spectrometer of the third aspect of this disclosure is an imaging mass spectrometer that generates ions by irradiating a sample with a laser and performs mass spectrometric analysis on the ions. The imaging mass spectrometer includes a laser irradiation unit, a detection unit, a generation unit, an imaging unit, an image analysis unit, and a display unit. The laser irradiation unit emits a laser beam onto the sample. The detection unit detects the ions generated by the laser. The generation unit generates a mass spectrometry imaging image based on the signal detected by the detection unit. The imaging unit acquires an optical microscopic image of the sample. The image analysis unit acquires and analyzes the mass spectrometry imaging image and the optical microscopic image. The display unit displays the analysis results of the image analysis unit. The image analysis unit sets a candidate group of regions of interest based on the optical microscopic image, and sets a region of interest group in the optical microscopic image or the mass spectrometry imaging image based on the candidate group of regions of interest and the mass spectrometry imaging image.

[0010] Invention Effects According to this disclosure, the accuracy of setting the region of interest in the analytical image data of a sample can be improved. Attached Figure Description

[0011] 【 Figure 1 [Illustration 1] is a schematic diagram of the imaging mass spectrometer of Embodiment 1.

[0012] 【 Figure 2 The diagram is used to illustrate the method of setting the region of interest.

[0013] 【 Figure 3 This is an example of an image displayed when a region of interest is defined for an image obtained from a biological sample.

[0014] 【 Figure 4The flowchart shows an example of region-of-interest determination processing performed by an imaging mass spectrometer.

[0015] 【 Figure 5 The figure shown is an example of the region of interest determination method in Implementation 2.

[0016] 【 Figure 6 The figure shown is an example illustrating the region of interest determination method in Implementation 2.

[0017] 【 Figure 7 [Image] is a flowchart of an example of the region of interest determination process performed by the imaging mass spectrometer in Implementation 2.

[0018] 【 Figure 8 [Image] is a flowchart of an example of the region of interest determination process performed by the imaging mass spectrometer in Implementation 3.

[0019] 【 Figure 9 The figure shown is used to illustrate the region of interest selection method involved in the variation example. Detailed Implementation

[0020] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Furthermore, identical or equivalent parts in the drawings are labeled with the same symbols, and their descriptions will not be repeated.

[0021] (Implementation Method 1) [Overall structure of the imaging mass spectrometer] An embodiment of the imaging mass spectrometer involved in this invention is described with reference to the accompanying drawings. Figure 1 This is an overall configuration diagram of the imaging mass spectrometer 100 in this embodiment. The imaging mass spectrometer 100 uses atmospheric pressure matrix-assisted laser desorption / ionization or atmospheric pressure laser desorption / ionization as its ionization method. Figure 1 As shown, the imaging mass spectrometer 100 includes an airtight chamber 1, a vacuum chamber 2, a control device 3, an input device 4, and a display device 5. The imaging mass spectrometer 100 determines the region of interest based on optical image data and mass spectrometry imaging data obtained by measuring the sample S. Figure 1 In this design, the direction of the irradiating laser is set as the Z-axis, which is also the height direction. Furthermore, the direction in which ions pass through vacuum chamber 2 is set as the X-axis, and the direction perpendicular to the XZ plane is set as the Y-axis.

[0022] The airtight chamber 1 is a shell whose interior is maintained at approximately atmospheric pressure, and includes a sample stage 11, a guide rail 12, a drive mechanism 13, an imaging unit 14, an illumination unit 15, a laser irradiation unit 16, and an ion transmission tube 17. The sample S, which is to be measured, is placed on the sample stage 11.

[0023] The drive mechanism 13 can move the sample stage 11 along the guide rail 12 in the X direction. Figure 1 The position of the sample stage 11, indicated by the dashed line, corresponds to the position P1 where the optical image of the sample S is obtained. Figure 1 The position of the sample stage 11, indicated by the solid line, corresponds to the position P2 where the mass spectrometry imaging image is acquired. The sample stage 11 is configured to move in the X direction, as well as in the Y and Z directions, via the drive mechanism 13.

[0024] An imaging unit 14 for acquiring optical image data of sample S is disposed inside the airtight chamber 1. The imaging unit 14 is positioned above position P1. The imaging unit 14 includes an optical system such as a lens and an imaging device. The imaging device is implemented, for example, by a CCD (Charge-Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging device generates image data by converting light incident from the sample S placed on the sample stage 11 into an electrical signal.

[0025] An illumination unit 15 is disposed inside the airtight chamber 1 and opposite to the imaging unit 14. When the sample stage 11 is in position P1, light emitted from the illumination unit 15 illuminates the sample S through an opening 11a formed on the sample stage 11, and the imaging unit 14 captures an image of the sample formed by the transmitted light. The image data generated by the imaging device of the imaging unit 14 is sent to the control device 3.

[0026] A laser irradiation unit 16 is provided above position P2 to irradiate the surface of sample S with laser light. By irradiating sample S with a laser focused into a small diameter, the material on the surface of sample S is ionized. Inside the airtight chamber 1, the collection port of the ion transmission tube 17, which is used to transport the generated ions to the vacuum chamber 2, is configured to be opposite to sample S.

[0027] Vacuum chamber 2 is a shell maintained at approximately a vacuum level, and includes ion transport optical systems 21 and 22, an ion trap 23, a mass spectrometer 24, and a detector 25. Ions delivered into vacuum chamber 2 are focused by ion transport optical systems 21 and 22 and transported to the next stage, where they are temporarily held in ion trap 23. Subsequently, the ions are separated by mass spectrometer 24 based on their time of flight and detected by detector 25.

[0028] The signal detected by detector 25 is sent to control device 3 and converted into mass-to-charge ratio based on the flight time of each ion. Then, mass spectra at various measurement points on the surface of sample S are created.

[0029] The control device 3 includes a processor 31, an input / output port 32, and a memory 33.

[0030] Processor 31 is an example of a circuit that controls the operation of control device 3 by executing a given program. The program executed by processor 31 can be stored in memory 33 or in a storage device external to control device 3. The processor is, for example, a CPU (Central Processing Unit).

[0031] The memory 33 stores data. The data stored in the memory 33 includes an optical image data acquisition program 331, a mass spectrometry imaging image data acquisition program 332, and a region of interest setting program 333. The memory 33 includes volatile memory (e.g., RAM (Random Access Memory)) and non-volatile memory (e.g., ROM (Read Only Memory), hard disk drives, and solid-state drives).

[0032] The optical image data acquisition program 331 is a program used to acquire optical image data from the sample S. The optical image data acquisition program 331 can be stored in the memory 33 or in an external storage device accessible to the processor 31. The control device 3 reads the optical image data acquisition program 331 and controls each part of the imaging mass spectrometer 100 to acquire the optical image data of the sample S.

[0033] The mass spectrometry imaging image data acquisition program 332 is a program used to acquire mass spectrometry imaging image data from sample S. The mass spectrometry imaging image data acquisition program 332 can be stored in memory 33 or in an external storage device accessible to processor 31. Control device 3 reads the mass spectrometry imaging image data acquisition program 332 and controls various parts of the imaging mass spectrometer 100 to acquire mass spectrometry imaging image data of sample S.

[0034] The region of interest setting program 333 is a program used to set the region of interest based on the acquired optical image data and mass spectrometry imaging image data. The region of interest setting program 333 can be stored in the memory 33 or in an external storage device accessible to the processor 31.

[0035] Input device 4 receives information input to control device 3. This information may include, for example, predetermined conditions used when setting candidate groups of regions of interest and predetermined conditions used when determining groups of regions of interest. Input device 4 may consist of, for example, a mouse and a keyboard.

[0036] The display device 5 displays information according to the instructions of the processor 31. This information may include, for example, an optical image of sample S, a mass spectrometry image of sample S, a candidate group of regions of interest, and a group of regions of interest. The display device 5 may be, for example, a liquid crystal display.

[0037] [Previous image processing methods] Currently, statistics are calculated based on image data obtained from measuring biological samples. These statistics are used for diagnosis and research.

[0038] When calculating statistics from image data, a region of interest (ROI) is sometimes defined, and statistics are calculated for that ROI. For example, in mass spectrometry imaging data of biological samples, the ROI is defined by using the ion intensity of ions from substances specifically accumulated in the tumor as an indicator, and the size of the tumor can be investigated from the area of ​​that ROI.

[0039] As a method for defining regions of interest (ROIs) in image data obtained from biological samples, there are known methods for automatically defining ROIs based on predetermined conditions. By automatically defining ROIs, especially when multiple image data points exist as objects, the time required for user-defined ROI settings can be saved.

[0040] However, when automatically setting the region of interest (ROI), sometimes the ROI is not set as the user expects. For example, when setting the ROI based on brightness in optical image data, the region without biological samples might be set as the ROI if the brightness of the area in the image data is the same as the brightness of the part of the biological sample that is intended to be set as the ROI. In such cases, if an area that should not be set as the ROI is determined to be the ROI, it can sometimes affect the statistics calculated based on the ROI. Therefore, techniques to improve the accuracy of ROI setting are needed.

[0041] Therefore, the imaging mass spectrometer of Embodiment 1 selects regions of interest (ROIs) from a candidate group of ROIs defined based on optical image data, using mass spectrometry imaging data. The imaging mass spectrometer acquires image data for a single sample using two different measurement methods. Since the physical quantities reflected in these image data are different, by using both image data, conditions that cannot be set using only one image data can be used to determine the ROI, thereby improving the accuracy of ROI setting.

[0042] In Embodiment 1, the use of optical image data and mass spectrometry imaging image data as image data for setting the region of interest is described. Furthermore, the imaging mass spectrometer disclosed herein can process image data that is not limited to optical image data and mass spectrometry imaging image data, but may also include image data obtained by Raman spectroscopy, image data obtained by scanning probe microscopy, image data of stained samples, and image data obtained by detecting fluorescence.

[0043] [Image processing methods for determining regions of interest] This describes a method for determining the region of interest using an imaging mass spectrometer 100. Figure 2 This diagram illustrates a method for setting a region of interest based on optical image data obtained by imaging unit 14 and mass spectrometry data detected by detector 25. Figure 2 In the image, image 200 represents optical image data obtained by capturing sample S through imaging unit 14.

[0044] <1. Acquisition of Optical Image Data> This describes the optical image data acquisition process performed by the imaging mass spectrometer 100 during the optical image data acquisition procedure 331, executed by the control device 3. After the user places the sample S on the sample stage 11, the control device 3 sends a signal to the drive mechanism 13 to move the sample stage 11 to position P1. After the sample stage 11 moves to position P1, the imaging unit 14 captures an image of the sample formed by light emitted from the illumination unit 15 and passing through the sample S. The imaging device of the imaging unit 14 sends the captured signal to the control device 3, and the control device 3 acquires an image 200 of the sample S.

[0045] <2. Setting the Candidate Group of Regions of Interest Based on Optical Image Data> This section describes the region of interest (ROI) candidate group setting process performed by control device 3 when reading the ROI setting procedure 333 after acquiring optical image data. For the obtained optical image, ROI candidates are set based on predetermined conditions. These predetermined conditions may be, for example, the brightness of the pixels in the optical image. There is no single ROI candidate; when multiple ROI candidates satisfy the predetermined conditions, they are referred to as a ROI candidate group.

[0046] exist Figure 2 In this process, candidate groups of regions of interest are defined based on the pixel values ​​of each pixel contained in image 200. Furthermore, the pixel value of each pixel corresponds to the brightness in the optical image. Figure 2 In this process, the condition that the pixel value is above a predetermined size is used as a predetermined condition to set the candidate region of interest. Regions R1 to R4 that meet this condition are set as candidate regions of interest.

[0047] <3. Acquisition of mass spectrometry imaging data> This section describes the optical image data acquisition process performed by the imaging mass spectrometer 100 during the reading of mass spectrometry imaging image data acquisition program 332 by the control device 3. The control device 3 sends a signal to the drive mechanism 13 to move the sample stage 11 to position P2. After the sample stage 11 moves to position P2, the control device 3 controls the laser irradiation unit 16 to irradiate the sample S with laser light. Ions generated by the irradiated laser are detected by the detector 25, and the detection signal is converted into mass spectrometry data. Based on the mass spectrometry data obtained by irradiating the sample S with laser light at various positions, a mass spectrometry imaging image of the sample S is generated. The mass spectrometry imaging image is, for example, an image representing the ion intensity at a predetermined mass-to-charge ratio at each position on the sample S as the pixel value of the corresponding position. The mass spectrometry imaging image data includes the mass spectrometry data at each position on the sample S.

[0048] <4. Determining the Region of Interest> The description explains that after acquiring optical image data and mass spectrometry imaging image data, a region of interest (ROI) group setting process is performed by the control device 3 using the region of interest setting procedure 333. The control device 3 uses information acquired through mass spectrometry imaging to select ROIs from candidate ROIs based on predetermined conditions. These predetermined conditions include, for example, the average ion intensity of a predetermined mass-to-charge ratio among the candidate ROIs, and the median ion intensity of a predetermined mass-to-charge ratio among the candidate ROIs.

[0049] exist Figure 2 In the defined candidate regions of interest, regions R1 and R4, where the average ionic intensity of ions with a mass-to-charge ratio of 1500 is above a predetermined value, are represented in gray. When regions with an average ionic intensity of ions with a mass-to-charge ratio of 1500 above a predetermined value are selected from the candidate regions of interest, regions R1 and R4 are determined as regions of interest, while regions R2 and R3, which do not meet this condition, are not selected as regions of interest.

[0050] <5. Display of Region of Interest> The control device 3 outputs an instruction display to show the determined region of interest (ROI) group. Alternatively, optical images, mass spectrometry images, and ROI candidate groups can also be displayed together with the determined ROI group.

[0051] In addition, the determined region of interest is used when calculating the predetermined statistics.

[0052] [Method for determining the region of interest (ROI) in mouse testicular sections] Figure 3 This is a diagram illustrating a method for determining a group of regions of interest based on optical images and mass spectrometry images obtained from mouse testicular slices as samples. Image 300 represents a candidate group of regions of interest set based on optical images acquired by imaging mass spectrometer 100.

[0053] Figure 3 (a) represents a candidate group of regions of interest defined based on optical image data. Furthermore, by mass spectrometry imaging, candidate groups of regions of interest whose signals within the region satisfy predetermined conditions are represented in gray. Figure 3 (b) represents the final group of regions of interest. For example... Figure 3 As shown, regions of interest are selected from a candidate group of regions of interest set based on optical image data, based on data containing physical quantities that are different from those in the optical image data.

[0054] Furthermore, as a condition for setting candidate groups of regions of interest, when using brightness in optical image data as an indicator, such as in... Figure 3 As shown by the shaded area R5 in (a), regions along the sample edge are sometimes designated as candidates for regions of interest. Region R5 represents the area without a sample (the background only), which should not originally be designated as a region of interest. When determining the region of interest, it is sometimes difficult to exclude such regions from the region of interest if only optical images are used.

[0055] Since the imaging mass spectrometer 100 shown in Embodiment 1 uses both optical image data and mass spectrometry imaging image data to determine the region of interest, it can eliminate regions of interest that are not intended by the user when determining the region of interest based on data obtained by a measurement method. In other words, the accuracy of the region of interest setting can be improved. Figure 3 As shown in (b), the region R5 is ultimately determined to be absent from the region group of the region of interest.

[0056] [Processing for determining the region of interest] Figure 4 This is a flowchart illustrating an example of the processing performed to determine a group of regions of interest using optical image data and mass spectrometry imaging image data. In one implementation example, when the processor 31 of the control device 3 executes a given program, Figure 4 The processing is invoked and executed from the main program. Since the control device 3 determines the region of interest based on the acquired image data, it can be said that it functions as an image processing device.

[0057] In step S10, the control device 3 acquires optical image data of sample S.

[0058] In step S12, the control device 3 sets a candidate group of regions of interest based on the optical image data obtained in step S10.

[0059] In step S14, the control device 3 acquires a mass spectrometry image of sample S.

[0060] In step S16, the control device 3 selects regions from the candidate regions of interest set in step S12 that meet predetermined criteria in the data obtained through mass spectrometry analysis and determines them as regions of interest.

[0061] In step S18, the control device 3 causes the display device 5 to display the region of interest determined in step S16. Afterwards, the control device 3 terminates the subroutine involved in determining the region of interest and returns the processing to the main program.

[0062] According to the region of interest determination method disclosed herein, in a candidate group of regions of interest set based on physical quantities obtained by a first measurement means, a physical quantity different from the physical quantity used to set the candidate group of regions of interest is used as an indicator to select the candidate group of regions of interest, thereby improving the accuracy of region of interest setting.

[0063] By using data obtained from measuring different physical quantities to define the region of interest, users can more easily prevent unintended regions from being designated as regions of interest compared to using data obtained from measuring only one physical quantity. Therefore, users can easily set conditions for determining the regions they desire as regions of interest.

[0064] Furthermore, in Embodiment 1 described above, the range of mass spectrometry imaging data acquired is not limited. Mass spectrometry imaging can be performed only on the range designated as candidate regions of interest, or it can be performed on the entire range of acquired optical image data. When mass spectrometry imaging is performed only on the range designated as candidate regions of interest, the measurement time required for mass spectrometry imaging can be shortened, and the computational load can be reduced since the amount of data can be suppressed. On the other hand, by performing mass spectrometry imaging on the entire range of acquired optical image data, even if the conditions for setting the candidate group of regions of interest change after mass spectrometry imaging, the region of interest can be determined without performing mass spectrometry imaging again, thus shortening the operation time required for remeasurement of mass spectrometry imaging.

[0065] Furthermore, although optical image data and mass spectrometry imaging image data are used in Embodiment 1 to determine the region of interest, the image data used in the image processing method disclosed herein is not limited to a combination of these image data. Image data obtained by Raman spectroscopy or by scanning probe microscopy may also be used. In addition, the optical image data may be optical image data of the stained sample, image data of fluorescence emitted from the sample, or image data obtained by phase contrast observation.

[0066] Furthermore, in Implementation 1, although optical image data is used to set up a candidate group of regions of interest and mass spectrometry imaging image data is used to determine the region of interest, it is also possible to use mass spectrometry imaging image data to set up a candidate group of regions of interest and optical image data to determine the region of interest. The order in which various image data are used is not limited.

[0067] (Implementation Method 2) In Embodiment 1 described above, the results of mass spectrometry imaging are used to select a candidate group of regions of interest. In this case, the region of interest is selected from the regions designated as region of interest candidates. In Embodiment 2, in addition to the region of interest candidate group set based on optical image data, a region of interest candidate group is also set based on the results of mass spectrometry imaging data. In Embodiment 2, the region of interest is determined based on both types of region of interest candidate groups. Furthermore, in Embodiment 2, components that are identical to those described in Embodiment 1 are marked with the same symbols and will not be described in detail again. Moreover, the contents described in Embodiment 1 can be combined within the scope that does not contradict Embodiment 2.

[0068] [A method for determining the region of interest group when two candidate regions of interest are set] This section describes the method for determining the region of interest (ROI) group when two candidate ROI groups are set. Figure 5 This diagram illustrates the region of interest (ROI) determination method when two ROI candidate groups are defined. Figure 5 In the image 400, two candidate groups of regions of interest are set.

[0069] First, a candidate group of regions of interest (the first candidate group of regions of interest) is established based on the acquired optical image data. Figure 5 In the example shown in region R6, the region enclosed by the solid line corresponds to the first candidate region of interest.

[0070] Next, a candidate group of regions of interest (a second candidate group of regions of interest) is established based on the acquired mass spectrometry imaging data. This establishment is independent of the first candidate group of regions of interest. Figure 5 In the example shown in region R7, the region enclosed by the dashed line corresponds to the second region of interest candidate group.

[0071] Among the obtained first and second candidate regions of interest, the region selected from both is determined as the region of interest. Figure 5In the diagram, as shown by the shaded area, the region R8, which overlaps with region R6, is determined as the region of interest. In this case, since any region satisfying any condition is determined as the region of interest, the region determined as the region of interest tends to be smaller compared to embodiment 1.

[0072] For example, the above-described region of interest determination method is effective when it is desired to designate a region in the cell nucleus containing substances with a mass-to-charge ratio of 1000 as the region of interest. Specifically, in an image obtained by staining the cell nucleus, the stained region is designated as a first region of interest candidate group, and in mass spectrometry imaging, the region where the ion intensity of ions with a mass-to-charge ratio of 1000 is above a predetermined value is designated as a second region of interest candidate group. By using the overlapping regions in the designated first and second region of interest candidate groups as the region of interest, the region in the nucleus containing substances with a mass-to-charge ratio of 1000 can be determined as the region of interest.

[0073] On the other hand, the region selected by at least one of the first and second region of interest candidate groups can also be determined as the region of interest. Figure 6 This is used to illustrate the situation where two candidate groups of regions of interest are set. Figure 5 The method described is different depending on the group of regions of interest, which determines the method's graph. Figure 6 In the image, image 500 represents an image with two candidate regions of interest set.

[0074] First, a candidate group of regions of interest (the first candidate group of regions of interest) is established based on the acquired optical image data. Figure 6 In the example shown in region R9, the region enclosed by the solid line corresponds to the first candidate region of interest.

[0075] Next, a candidate group of regions of interest (a second candidate group of regions of interest) is established based on the acquired mass spectrometry imaging data. This establishment is performed separately from the first candidate group of regions of interest. Figure 6 In the example shown in region R10, the region enclosed by the dashed line corresponds to the second region of interest candidate group.

[0076] Among the obtained first and second region of interest candidate groups, the region in the candidate group that is set as a region of interest in at least one of them is determined as the region of interest. Figure 6 In the diagram, as shown by the shaded areas, regions defined only in region R9, regions defined only in region R10, and regions defined in both region R9 and region R10 are determined as regions of interest. In this case, since regions satisfying any of the conditions are determined as regions of interest, the regions determined as regions of interest tend to be larger compared to embodiment 1.

[0077] For example, the above-described region of interest determination method is effective when it is desired to define a region containing at least one of the substances before and after the conversion (which is metabolized into a substance with a mass-to-charge ratio of 1200) as a region of interest. Specifically, in an image obtained by staining the substance before conversion, the stained region is defined as a first region of interest, and in mass spectrometry imaging, the region where the ion intensity of the ion with a mass-to-charge ratio of 1200 is above a predetermined value is defined as a second region of interest, thereby determining a region containing at least one of the substances before and after conversion as a region of interest.

[0078] [Processing for determining the region of interest in Implementation 2] Figure 7 This is a flowchart illustrating an example of the processing performed to determine a group of regions of interest using optical image data and mass spectrometry imaging image data. In one implementation example, when the processor 31 of the control device 3 executes a given program, Figure 7 The processing is invoked and executed from the main program. Additionally, in Figure 7 In China, for the sake of Figure 4 The flowcharts described herein use the same symbols and do not require further detailed explanation.

[0079] In step S20, the control device 3 sets a region of interest candidate group that is different from the region of interest candidate group set in step S12, based on the mass spectrometry imaging data obtained in step S14.

[0080] In step S22, the control device 3 determines the region of interest group based on the region of interest candidate group set in step S12 and the region of interest candidate group set in step S20.

[0081] According to the above method for determining the region of interest, since two candidate groups of regions of interest are used to determine the group of regions of interest, the accuracy of the region of interest setting can be improved compared with the case where the group of regions of interest is determined based on a physical quantity obtained by a measurement method.

[0082] Furthermore, unlike Implementation 1, since each candidate group of regions of interest is set based on the physical quantities obtained by the two measurement methods, the regions of interest reflecting the two physical quantities at each measurement point can be determined.

[0083] (Implementation Method 3) Because biological samples have complex structures, the optimal conditions for defining regions of interest (ROIs) may differ from sample to sample. However, sometimes it is desirable to use the same ROI definition conditions for image data from one sample with ROIs from other samples. For example, consider a tissue yielding multiple slices and image data for each slice. In this case, if the ROI definition conditions vary from slice to slice, the relative position or shape of the ROI will no longer be consistent across slices, causing inconvenience, for instance, when constructing 3D data of the ROI.

[0084] In Embodiment 3, the control device 3 uses setting conditions for a candidate region of interest (ROI) or region of interest group determined for image data obtained from one sample to determine the ROI or region of interest group for image data obtained from other samples. Furthermore, in Embodiment 3, components identical to those described in Embodiments 1 and 2 are labeled with the same symbols and will not be described in detail again. Additionally, the contents described in Embodiments 1 and 2 can be combined within the scope that does not contradict Embodiment 3.

[0085] [Processing for determining the region of interest in Implementation 3] Figure 8 This is a flowchart illustrating an example of the processing performed to determine a group of regions of interest using optical image data and mass spectrometry imaging image data. In one implementation example, when the processor 31 of the control device 3 executes a given program, Figure 8 The processing is invoked and executed from the main program.

[0086] In this embodiment, we envision a scenario where two samples are obtained from an tissue, and a region of interest is defined for each sample under the same conditions. One of the two samples is referred to as the first sample, and the other as the second sample.

[0087] In step S24, the control device 3 acquires the optical image data of the first sample.

[0088] In step S26, the control device 3 sets a first region of interest group based on the optical image data of the first sample obtained in step S24.

[0089] In step S28, the control device 3 acquires optical image data of the second sample.

[0090] In step S30, the control device 3 uses the region of interest setting conditions used in step S26 to set a second region of interest group based on the optical image data of the second sample.

[0091] In step S32, the control device 3 causes the display device 5 to display the first and second regions of interest. Afterwards, the control device 3 terminates the subroutine involved in determining the regions of interest and returns the processing to the main program.

[0092] Based on the image processing method described above, regions of interest (ROIs) can be defined for multiple image data sets under the same conditions, avoiding significant differences in the position and size of ROIs between different image data sets. As a result, users can easily construct 3D data based on multiple image data sets.

[0093] Furthermore, although the case of two samples was envisioned in the above embodiment 3, the number of samples is not limited to two, and may be more than two.

[0094] Furthermore, in Embodiment 3 described above, although only optical image data is used for defining the region of interest, mass spectrometry imaging data can also be used simultaneously. Specifically, when multiple samples exist, optical image data and mass spectrometry imaging data are acquired for each sample, and a region of interest group is determined for one sample using the method described in Embodiment 1 or Embodiment 2. Then, the region of interest groups for the remaining samples are determined using the same conditions used to determine the region of interest group for one sample. According to this method, regions of interest can be determined for multiple samples under the same conditions.

[0095] (Variation example) The user can also confirm the displayed regions of interest (ROI) and further select ROIs based on user-specified criteria. By displaying automatically generated ROIs and allowing the user to select from them, it is possible to prevent unintended regions from being identified as ROIs.

[0096] This describes an example of a display screen shown on the display device 5 when the imaging mass spectrometer 100 has determined a group of regions of interest. Figure 9 This is an example of a display screen.

[0097] Display screen 600 includes windows 610 and 620.

[0098] Window 610 displays the region of interest group determined by the imaging mass spectrometer 100.

[0099] Figure 9 (a) represents an example of the screen displayed before the user specifies the conditions. Figure 9 In (a), window 610 displays 13 regions, including region R11 and region R12 determined by control device 3.

[0100] Window 620 displays sliders for the user to select a region as the region of interest, using the number of pixels constituting the determined region as an indicator. Window 620 includes a slider 621 for determining the minimum number of pixels for a region selected as the region of interest, and a slider 622 for determining the maximum number of pixels for a region selected as the region of interest.

[0101] exist Figure 9 In (a), slider 621 indicates excluding 0 items from the region of interest in ascending order, and slider 622 indicates retaining 13 items from the region of interest in descending order. That is to say, in Figure 9 In state (a), the displayed region of interest is not excluded.

[0102] like Figure 9 As shown in (b), when the user operates the slider 621 via the input device 4 and specifies a predetermined number of pixels, areas consisting of pixels smaller than the specified predetermined number of pixels are excluded from the regions of interest. As a result, 10 areas in the regions displayed in window 610 are no longer displayed in ascending order. That is, the three regions displayed in window 610, including region R12, are selected as regions of interest.

[0103] Based on the above variation, users can visually confirm the automatically determined area group and select the region of interest. Therefore, the region of interest can be easily set as the user expects.

[0104] In addition, in the above variation, the user selected the region of interest based on the size of the region, but the indicator used is not limited to the size of the region, and can also be the pixel value of the image data.

[0105] Furthermore, in the above variations, although the user selects regions for those regions that have been determined as regions of interest, the user can also select regions for those regions that have been set as candidate regions of interest.

[0106] [plan] Those skilled in the art should understand that the above-described exemplary embodiments are specific examples of the following scheme.

[0107] (Item 1) An image processing method may include: the steps of obtaining first image data obtained by measuring a first sample using a first measurement means; the steps of setting a candidate group of a first region of interest based on the first image data; the steps of obtaining second image data obtained by measuring the first sample using a second measurement means different from the first measurement means, reflecting a physical quantity different from the first image data; and the steps of setting a region of interest group in the first image data or the second image data based on the candidate group of the first region of interest and the second image data.

[0108] The image processing method described in item 1 can improve the accuracy of region of interest setting.

[0109] (Item 2) According to the image processing method described in Item 1, the first image data may be an optical image, and the second image data may be a mass spectrometry imaging image.

[0110] The image processing method described in item 2 can improve the accuracy of setting regions of interest based on optical images and mass spectrometry imaging images.

[0111] (Item 3) According to the image processing method described in Item 1 or Item 2, the step of setting the region of interest group may include: setting the region in the candidate group of the first region of interest whose signal strength of the second image data corresponding to the candidate of the first region of interest meets a predetermined standard as the region of interest group.

[0112] According to the image processing method described in item 3, in a candidate group of regions of interest set based on first image data reflecting physical quantities obtained by the first measurement means, a physical quantity different from the physical quantity obtained by the first measurement means is used to select regions of interest, thereby improving the accuracy of region of interest setting.

[0113] (Item 4) The image processing method according to any one of items 1 to 3 may further include the step of setting a candidate group of a second region of interest based on the second image data. The step of setting the region of interest group may include the step of setting the region of interest group based on the candidate group of the first region of interest and the candidate group of the second region of interest.

[0114] According to the image processing method described in item 4, the region of interest is determined using a region of interest candidate group set based on first image data reflecting a physical quantity obtained by a first measurement means, and a region of interest candidate group set based on second image data reflecting a physical quantity different from the physical quantity obtained by the first measurement means.

[0115] (Item 5) According to the image processing method described in Item 4, the step of setting the region of interest group may include: setting the common region in the candidate group of the first region of interest and the candidate group of the second region of interest as the region of interest group.

[0116] (Item 6) According to the image processing method of Item 4, the step of setting the group of regions of interest may include: setting the region of interest as a candidate region from at least one of the candidate group of the first region of interest and the candidate group of the second region of interest.

[0117] (Item 7) According to the image processing method of any one of items 1 to 6, the second image data can be obtained by the second measurement means for the candidate group of the first region of interest.

[0118] According to the image processing method described in item 7, a candidate group of first regions of interest is defined before implementing the second measurement method. Then, the measurement by the second measurement method is performed only on the regions defined by the candidate group of first regions of interest. Since the regions that are the objects of measurement by the second measurement method are limited, the amount of data in the second image data can be reduced, thereby reducing the computational processing load required to determine the region of interest group.

[0119] (8) The image processing method according to any one of the first to seventh items may further include: the step of displaying a candidate group of the first region of interest; the step of receiving a first predetermined condition from a user; and the step of excluding at least one region included in the candidate group of the first region of interest from the candidate group of the first region of interest based on the first predetermined condition.

[0120] According to the image processing method described in item 8, the user can easily identify the candidate group of the automatically set first region of interest (ROI) and confirm whether there are any regions that the user did not expect to be set as candidates for the first ROI. Furthermore, if an unexpected region is set as a candidate for the first ROI, that region can be excluded based on the user's operation, thereby improving the accuracy of the ROI setting.

[0121] (Item 9) According to the image processing method of Item 8, the first predetermined condition may include at least one of the size of the region contained in the candidate group of the first region of interest and the pixel value of the pixel of the region contained in the candidate group of the first region of interest.

[0122] (Item 10) The image processing method according to any one of items 1 to 9 may further include: the step of displaying the group of regions of interest; the step of receiving a second predetermined condition from a user; and the step of excluding at least one region included in the group of regions of interest from the group of regions of interest based on the second predetermined condition.

[0123] According to the image processing method described in item 10, the user can easily identify automatically set regions of interest (ROIs) and confirm whether any unexpected regions have been set as ROIs. Furthermore, if an unexpected region is set as an ROI, that region can be excluded based on the user's actions, thereby improving the accuracy of ROI setting.

[0124] (Item 11) According to the image processing method of Item 10, the second predetermined condition may be the size of the region contained in the group of regions of interest.

[0125] (Item 12) The image processing method according to any one of items 1 to 11 may further include: the step of obtaining third image data obtained by measuring a second sample different from the first sample using the first measurement means; and the step of setting a candidate group of a third region of interest in the third image data based on the third image data, provided that a candidate group of the first region of interest has been set.

[0126] According to the image processing method described in item 12, it is possible to set the region of interest under the same conditions for image data obtained from different samples. Therefore, it is possible to prevent regions containing different physical quantities in image data obtained from different samples from being set as regions of interest.

[0127] (Item 13) The image processing method according to any one of items 1 to 12 may further include: a step of obtaining third image data obtained by measuring a second sample different from the first sample using the first measurement means; a step of obtaining fourth image data obtained by measuring the second sample using the second measurement means; and a step of setting a region of interest group in the third image data or the fourth image data based on the third image data and the fourth image data, under the same conditions as setting a region of interest group in the first image data or the second image data based on the first image data and the second image data.

[0128] (Item 14) A program according to an embodiment, which is executed by a processor mounted on a computer, can cause the computer to perform the image processing method of any one of items 1 to 13.

[0129] (Item 15) An image processing apparatus according to a scheme can acquire first image data obtained by measuring a first sample by a first measurement means, set a candidate group of a first region of interest based on the first image data, acquire second image data obtained by measuring the first sample by a second measurement means different from the first measurement means, reflecting a physical quantity different from the first image data, and set a region of interest group in the first image data or the second image data based on the candidate group of the first region of interest and the second image data.

[0130] (Item 16) An imaging mass spectrometer according to one embodiment, which generates ions by irradiating a sample with a laser and performs mass spectrometry analysis on the ions, may include: a laser irradiation unit that emits the laser to the sample; a detection unit that detects the ions generated by the laser; a generation unit that generates a mass spectrometry imaging image based on the signal detected by the detection unit; an imaging unit that acquires an optical microscopic image of the sample; an image analysis unit that acquires and analyzes the mass spectrometry imaging image and the optical microscopic image; and a display unit that displays the analysis results of the image analysis unit, wherein the image analysis unit sets a candidate group of regions of interest based on the optical microscopic image, and sets a region of interest group in the optical microscopic image or the mass spectrometry imaging image based on the candidate group of regions of interest and the mass spectrometry imaging image.

[0131] The imaging mass spectrometer described in item 16 can improve the accuracy of region of interest setting.

[0132] The embodiments disclosed herein should be considered illustrative in all respects and not restrictive. The scope of this disclosure is defined not by the description of the embodiments above but by the scope of the claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of the claims. Furthermore, the techniques in the embodiments are intended to be implemented individually or, where possible, in combination with other techniques in the embodiments as needed.

[0133] Explanation of reference numerals in the attached figures 1. Airtight chamber; 2. Vacuum chamber; 3. Control device; 4. Imaging unit; 5. Laser irradiation unit; 6. Input device; 7. Display device; 11. Sample stage; 12. Guide rail; 13. Drive mechanism; 14. Illumination unit; 15. Ion transmission tube; 16. Detection unit; 21, 22. Ion transmission optical system; 23. Ion trap; 24. Mass spectrometer; 25. Detector; 31. Processor; 32. Input / output port; 33. Memory; 100. Imaging mass spectrometer; 331. Image data acquisition program; 332. Region of interest setting program; 3311. Sample stage drive unit; 3312. Optical image processing unit; 3313. Laser drive unit; 3314. Analysis and control unit; 3315. Imaging image processing unit; 3321. Region of interest candidate setting unit; 3322. Region of interest determination unit.

Claims

1. An image processing method, comprising: The step of obtaining first image data by measuring a first sample using a first measuring means; The step of setting a candidate group for the first region of interest based on the first image data; The step of obtaining second image data that reflects a physical quantity different from the first image data, obtained by measuring the first sample using a second measurement method different from the first measurement method; as well as The step of setting a region of interest group in the first image data or the second image data based on the candidate group of the first region of interest and the second image data.

2. The image processing method according to claim 1, characterized in that, The first image data is an optical image, and the second image data is a mass spectrometry imaging image.

3. The image processing method according to claim 1 or claim 2, characterized in that, The step of setting the region of interest group includes: setting the region in the candidate group of the first region of interest whose signal strength of the second image data corresponding to the candidate of the first region of interest meets a predetermined standard as the region of interest group.

4. The image processing method according to claim 1 or claim 2, characterized in that, The method further includes the step of setting a candidate group for a second region of interest based on the second image data. The step of setting the region of interest group includes: setting the region of interest group based on the candidate group of the first region of interest and the candidate group of the second region of interest.

5. The image processing method according to claim 4, characterized in that, The step of setting the region of interest group includes: setting the common region in the candidate group of the first region of interest and the candidate group of the second region of interest as the region of interest group.

6. The image processing method according to claim 4, characterized in that, The step of setting the region of interest group includes: setting the region of interest as a candidate region from at least one of the candidate groups of the first region of interest and the candidate groups of the second region of interest.

7. The image processing method according to claim 1 or claim 2, characterized in that, The second image data is obtained using the second measurement method for the candidate group of the first region of interest.

8. The image processing method according to claim 1 or claim 2, characterized in that, Further includes: The step of displaying the candidate group of the first region of interest; The step of receiving the first predetermined conditions from the user; as well as The step of excluding at least one region included in the candidate group of the first region of interest from the candidate group of the first region of interest based on the first predetermined condition.

9. The image processing method according to claim 8, characterized in that, The first predetermined condition includes at least one of the following: the size of the region contained in the candidate group of the first region of interest, and the pixel value of the pixel in the region contained in the candidate group of the first region of interest.

10. The image processing method according to claim 1 or claim 2, characterized in that, Further includes: The steps for displaying the group of regions of interest; The step of receiving the second predetermined condition from the user; as well as The step of excluding at least one region contained in the region of interest from the region of interest group based on the second predetermined condition.

11. The image processing method according to claim 10, characterized in that, The second predetermined condition is the size of the region included in the group of regions of interest.

12. The image processing method according to claim 1 or claim 2, characterized in that, Further includes: The step of obtaining third image data obtained by measuring a second sample that is different from the first sample using the first measurement method; as well as The step of setting a candidate group for a third region of interest based on the third image data, under the condition of setting a candidate group for the first region of interest.

13. The image processing method according to claim 1 or claim 2, characterized in that, Further includes: The step of obtaining third image data obtained by measuring a second sample that is different from the first sample using the first measurement method; The step of obtaining fourth image data obtained by measuring the second sample using the second measurement means; as well as The step of setting a region of interest group in the third image data or the fourth image data based on the third image data and the fourth image data, under the same conditions as setting a region of interest group in the first image data or the second image data based on the first image data and the second image data.

14. A program executed by a processor mounted on a computer, characterized in that, The computer is made to perform the image processing method according to claim 1 or claim 2.

15. An image processing apparatus, characterized in that: Obtain first image data by measuring the first sample using a first measurement method; A candidate group for the first region of interest is defined based on the first image data; Obtain second image data that reflects a physical quantity different from the first image data, obtained by measuring the first sample using a second measurement method different from the first measurement method; and Based on the candidate group of the first region of interest and the second image data, a region of interest group is set in the first image data or the second image data.

16. An imaging mass spectrometer, which generates ions by irradiating a sample with a laser and performs mass spectrometry analysis on the ions, characterized in that, have: A laser irradiation unit that emits the laser beam toward the sample; A detection unit for detecting ions generated by the laser; A generation unit that generates a mass spectrometry imaging image based on the signal detected by the detection unit; Imaging unit for obtaining optical microscopic images of the sample; An image analysis unit that acquires and analyzes the mass spectrometry imaging image and the optical microscopy image; as well as A display unit that displays the analysis results of the image analysis unit. The image analysis unit sets a candidate group of regions of interest based on the optical microscopic image. Based on the candidate group of the region of interest and the mass spectrometry imaging image, a region of interest group is set in the optical microscopy image or the mass spectrometry imaging image.

Images