Image processing method, program, image processing device, and imaging mass spectrometry device

Abstract

An image processing method according to the present disclosure includes a step (S10) for acquiring first image data obtained by measuring a first sample according to a first measurement method and a step (S12) for setting a first region-of-interest candidate group on the basis of the first image data. The image processing method further comprises: a step (S14) for acquiring second image data reflecting a physical quantity different from that of the first image data, the second image data being obtained by measuring the first sample according to a second measurement method different from the first measurement method; and a step (S16) for setting a region-of-interest group with respect to the first image data or the second image data on the basis of the first region-of-interest candidate group and the second image data.

Description

Image processing method, program, image processing device, and imaging mass spectrometer

[0001] The present disclosure relates to an image processing method, a program, an image processing device, and an imaging mass spectrometer, and more particularly to an image processing method for setting a region of interest in image data.

[0002] A region of interest is set for image data obtained by measuring a biological sample, and statistics are calculated for the region of interest.

[0003] One method for acquiring image data from a biological sample uses an imaging mass spectrometer, as disclosed in Japanese Patent Application Laid-Open No. 2010-261882 (Patent Document 1). Imaging mass spectrometry is a technique for acquiring mass spectral information at various positions on the surface of a biological sample and obtaining distribution information for substances with a predetermined mass-to-charge ratio by plotting a peak intensity distribution. For example, by setting a region of interest in an image obtained by imaging mass spectrometry of a biological sample using the ion intensity of ions derived from substances that specifically accumulate in tumor areas as an index, the size of the tumor can be determined from the area of ​​the region of interest.

[0004] A method for automatically setting a region of interest based on predetermined conditions is known as a method for setting a region of interest in image data obtained from a biological sample. By automatically setting a region of interest, the user can save time on setting the region of interest.

[0005] JP 2010-261882 A

[0006] Biological specimens have complex structures, and methods for automatically setting regions of interest based on predetermined conditions may set regions of interest that should not be included in the original target region. Setting regions of interest that should not be included in the original target region may affect statistics calculated based on the regions of interest. Therefore, there is a need for a technology that improves the accuracy of setting regions of interest in image data.

[0007] The present disclosure has been devised in view of the above circumstances, and its purpose is to provide a technique for improving the accuracy of setting a region of interest in analytical image data of a sample.

[0008] An image processing method according to a first aspect of the present disclosure includes the steps of acquiring first image data obtained by measuring a first sample using a first measurement technique, and setting a first set of region of interest candidates based on the first image data. The image processing method further includes the steps of acquiring second image data obtained by measuring the first sample using a second measurement technique different from the first measurement technique, the second image data reflecting physical quantities different from those of the first image data, and setting a region of interest group in the first image data or the second image data based on the first set of region of interest candidates and the second image data.

[0009] An image processing device according to a second aspect of the present disclosure acquires first image data obtained by measuring a first sample using a first measurement technique, and sets a group of first region of interest candidates based on the first image data. The image processing device acquires second image data obtained by measuring the first sample using a second measurement technique different from the first measurement technique, and reflects physical quantities different from those of the first image data. The image processing device sets a group of regions of interest in the first image data or the second image data based on the group of first region of interest candidates and the second image data.

[0010] An imaging mass spectrometer according to a third aspect of the present disclosure is an imaging mass spectrometer that generates ions by irradiating a sample with laser light and performs mass analysis of the ions. The imaging mass spectrometer includes a laser irradiation unit, a detection unit, a generation unit, an imaging unit, an image analysis unit, and a display unit. The laser irradiation unit emits laser light toward the sample. The detection unit detects ions generated by the laser light. The generation unit generates a mass analysis image based on a signal detected by the detection unit. The imaging unit acquires an optical microscope image of the sample. The image analysis unit acquires and analyzes the mass analysis image and the optical microscope image. The display unit displays the analysis results of the image analysis unit. The image analysis unit sets a group of candidate regions of interest based on the optical microscope image, and sets a group of regions of interest in the optical microscope image or the mass analysis image based on the group of candidate regions of interest and the mass analysis image.

[0011] According to the present disclosure, it is possible to improve the accuracy of setting a region of interest in analytical image data of a sample.

[0012] FIG. 1 is a schematic configuration diagram of an imaging mass spectrometer according to embodiment 1. FIG. 2 is a diagram for explaining a method for setting a region of interest. FIG. 3 is an example of an image displayed when setting a region of interest in an image obtained from a biological sample. FIG. 4 is a flowchart of an example of a region of interest determination process performed by an imaging mass spectrometer. FIG. 5 is a diagram for explaining an example of a region of interest determination method in embodiment 2. FIG. 6 is a diagram for explaining an example of a region of interest determination method in embodiment 2. FIG. 7 is a flowchart of an example of a region of interest determination process performed by an imaging mass spectrometer according to embodiment 2. FIG. 8 is a flowchart of an example of a region of interest determination process performed by an imaging mass spectrometer according to embodiment 3. FIG. 9 is a diagram for explaining a method for selecting a region of interest according to a modified example.

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail with reference to the accompanying drawings, in which the same or corresponding parts are designated by the same reference numerals and will not be described repeatedly.

[0014] (Embodiment 1) [Overall Configuration of Imaging Mass Spectrometer] An embodiment of an imaging mass spectrometer according to the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing the overall configuration of an imaging mass spectrometer 100 according to this embodiment. This imaging mass spectrometer 100 uses atmospheric pressure matrix-assisted laser desorption / ionization (MALDI) or atmospheric pressure laser desorption / ionization (LADI) as its ionization method. As shown in FIG. 1, the imaging mass spectrometer 100 comprises an airtight chamber 1, a vacuum chamber 2, a control device 3, an input device 4, and a display device 5. The imaging mass spectrometer 100 determines a region of interest (ROI) based on optical image data and mass analysis imaging data obtained by measuring a sample S. In FIG. 1, the direction in which the laser light is irradiated is defined as the Z-axis direction, which is the height direction. Furthermore, the direction in which ions pass through the vacuum chamber 2 is defined as the X-axis direction, and the direction perpendicular to the XZ plane is defined as the Y-axis direction.

[0015] The airtight chamber 1 is a housing whose interior is maintained at approximately atmospheric pressure, and includes a sample stage 11, a guide 12, a drive mechanism 13, an imaging unit 14, an illumination unit 15, a laser irradiation unit 16, and an ion transport tube 17. A sample S to be measured is placed on the sample stage 11.

[0016] The driving mechanism 13 can move the sample stage 11 in the X direction along the guide 12. The position of the sample stage 11 shown by the dotted line in Fig. 1 corresponds to position P1 where an optical image of the sample S is acquired. The position of the sample stage 11 shown by the solid line in Fig. 1 corresponds to position P2 where a mass spectrometry imaging image is acquired. The sample stage 11 is configured so that it can be moved by the driving mechanism 13 not only in the X direction but also in the Y and Z directions.

[0017] An imaging unit 14 for acquiring optical image data of the sample S is installed inside the airtight chamber 1. The imaging unit 14 is located above position P1. The imaging unit 14 includes an optical system such as a lens and an imaging element. The imaging element is realized by, for example, a charge coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor. The imaging element generates image data by converting light incident from the sample S placed on the sample stage 11 into an electrical signal.

[0018] An illumination unit 15 is installed inside the airtight chamber 1 so as to face the imaging unit 14. When the sample stage 11 is at position P1, light emitted from the illumination unit 15 is irradiated onto the sample S through an opening 11a formed in the sample stage 11, and the imaging unit 14 captures an image of the sample using the transmitted light. Image data generated by the imaging element of the imaging unit 14 is sent to the control device 3.

[0019] Above position P2, a laser irradiation unit 16 is installed, which irradiates the surface of the sample S with laser light. When the sample S is irradiated with the laser light, which has been narrowed to a very small diameter, the material on the surface of the sample S is ionized. Inside the airtight chamber 1, a collection port of an ion transport pipe 17, which transports the generated ions to the vacuum chamber 2, is arranged facing the sample S.

[0020] The vacuum chamber 2 is a housing whose interior is maintained at a substantially vacuum state, and includes ion transport optical systems 21 and 22, an ion trap 23, a mass analyzer 24, and a detector 25. Ions transported into the vacuum chamber 2 are focused by the ion transport optical systems 21 and 22 and transported to subsequent stages, where they are temporarily held in the ion trap 23. The ions are then separated by the mass analyzer 24 according to their time of flight, and detected by the detector 25.

[0021] The signal detected by the detector 25 is sent to the control device 3, where the flight time of each ion is converted into a mass-to-charge ratio. Thereafter, a mass spectrum at each measurement point on the surface of the sample S is created.

[0022] The control device 3 includes a processor 31, an input / output port 32, and a memory 33. The processor 31 is an example of an electric circuit, and controls the operation of the control device 3 by executing a given program. The program executed by the processor 31 may be stored in the memory 33, or may be stored in a storage device external to the control device 3. The processor is, for example, a CPU (Central Processing Unit).

[0023] The memory 33 stores data. The data stored in the memory 33 includes an optical image data acquisition program 331, a mass spectrometry imaging image data acquisition program 332, and a region of interest setting program 333. The memory 33 includes volatile memory (e.g., RAM (Random Access Memory)) and non-volatile memory (e.g., ROM (Read Only Memory), a hard disk drive, and a solid state drive).

[0024] The optical image data acquisition program 331 is a program for acquiring optical image data from the sample S. The optical image data acquisition program 331 may be stored in the memory 33, or may be stored in an external storage device accessible by the processor 31. The control device 3 reads out the optical image data acquisition program 331, controls each part of the imaging mass spectrometer 100, and acquires optical image data of the sample S.

[0025] The mass analysis imaging image data acquisition program 332 is a program for acquiring mass analysis imaging image data from the sample S. The mass analysis imaging image data acquisition program 332 may be stored in the memory 33, or may be stored in an external storage device accessible by the processor 31. The control device 3 reads out the mass analysis imaging image data acquisition program 332, controls each part of the imaging mass spectrometer 100, and acquires mass analysis imaging image data of the sample S.

[0026] The region-of-interest setting program 333 is a program for setting a region of interest based on the acquired optical image data and mass spectrometry imaging image data. The region-of-interest setting program 333 may be stored in the memory 33 or in an external storage device accessible by the processor 31.

[0027] The input device 4 accepts information input to the control device 3. The information is, for example, predetermined conditions used when setting a group of region of interest candidates and predetermined conditions used when determining a group of regions of interest. The input device 4 is, for example, configured with a mouse and a keyboard.

[0028] The display device 5 displays information in accordance with instructions from the processor 31. The information is, for example, an optical image of the sample S, a mass spectrometry image of the sample S, a set of candidate regions of interest, and a set group of regions of interest. The display device 5 is, for example, configured by a liquid crystal display.

[0029] [Conventional Image Processing Methods] Statistics are calculated based on image data obtained by measuring biological samples, and these statistics are used for diagnosis and research.

[0030] When calculating statistics from image data, a region of interest may be set and statistics calculated for that region of interest. For example, in mass spectrometry imaging image data of a biological sample, a region of interest may be set using the ion intensity of ions derived from a substance that specifically accumulates in tumor areas as an index, and the size of the tumor can be determined from the area of ​​the region of interest.

[0031] A method for automatically setting a region of interest based on predetermined conditions is known as a method for setting a region of interest in image data obtained from a biological sample. By automatically setting a region of interest, it is possible to reduce the time required for the user to set the region of interest, especially when there are multiple pieces of image data to be targeted.

[0032] However, when a region of interest is automatically set, it may not be set as intended by the user. For example, when setting a region of interest in optical image data based on brightness, if the brightness of a region in the image data where no biological sample is present is the same as the brightness of a portion of the biological sample that is to be set as the region of interest, the region where no biological sample is present may be set as the region of interest. In this way, if a region that should not be set as the region of interest is determined as the region of interest, it may affect the statistics calculated based on the region of interest. Therefore, a technology that improves the accuracy of setting a region of interest is desired.

[0033] Therefore, the imaging mass spectrometer according to the first embodiment selects a region of interest (ROI) based on mass analysis imaging data from a group of candidate regions of interest defined based on optical image data. The imaging mass spectrometer acquires image data for a single sample using two different measurement techniques. Because the physical quantities reflected in these image data are different, using these two sets of image data allows the region of interest to be determined using conditions that cannot be set using only one set of image data, thereby improving the accuracy of defining the region of interest.

[0034] In the first embodiment, a case will be described in which optical image data and mass analysis imaging image data are used as image data for setting a region of interest. Note that the image data that can be processed by the imaging mass spectrometer according to the present disclosure is not limited to optical image data and mass analysis imaging image data, and may include image data obtained by Raman spectroscopy, image data obtained by a scanning probe microscope, image data of a stained sample, and image data obtained by detecting fluorescence.

[0035] [Image Processing Method for Determining a Region of Interest] A method for determining a region of interest using the imaging mass spectrometer 100 will now be described. Fig. 2 is a diagram for explaining a method for setting a region of interest based on optical image data obtained by the imaging unit 14 and mass spectrum data detected by the detector 25. In Fig. 2, an image 200 represents optical image data acquired by photographing the sample S with the imaging unit 14.

[0036] 1. Acquisition of Optical Image Data The following describes the optical image data acquisition process executed by the imaging mass spectrometer 100 when the control device 3 reads out the optical image data acquisition program 331. After the user places the sample S on the sample stage 11, the control device 3 sends a signal to the drive mechanism 13 to move the sample stage 11 to position P1. After the sample stage 11 has moved to position P1, the imaging unit 14 captures an image of the sample using light that has been emitted from the illumination unit 15 and passed through the sample S. The image sensor of the imaging unit 14 sends a signal obtained by the image capture to the control device 3, and the control device 3 acquires an image 200 of the sample S.

[0037] 2. Setting a group of candidate regions of interest based on optical image data The following describes the process of setting a group of candidate regions of interest executed by the control device 3 when the control device 3 reads the region-of-interest setting program 333 after acquiring optical image data. Candidate regions of interest are set for the acquired optical image based on predetermined conditions. The predetermined conditions are, for example, the brightness of the pixels in the optical image. The number of candidate regions of interest is not limited to one; when there are multiple candidate regions of interest that satisfy the predetermined conditions, they are referred to as a group of candidate regions of interest.

[0038] 2, a group of candidate regions of interest is set based on the pixel values ​​of each pixel included in image 200. The pixel values ​​of each pixel correspond to brightness in the optical image. In FIG. 2, a predetermined condition that the pixel values ​​are equal to or greater than a predetermined magnitude is used to set candidate regions of interest, and regions R1 to R4 that satisfy this condition are set as candidate regions of interest.

[0039] 3. Acquisition of Mass Analysis Imaging Data The following describes the process of acquiring optical image data executed by the imaging mass spectrometer 100 when the control device 3 reads out the mass analysis imaging image data acquisition program 332. The control device 3 sends a signal to the drive mechanism 13 to move the sample stage 11 to position P2. After the sample stage 11 moves to position P2, the control device 3 controls the laser irradiation unit 16 to irradiate the sample S with laser. Ions generated by the laser irradiation are detected by the detector 25, and the detection signal is converted into mass spectrum data. A mass analysis imaging image of the sample S is generated from the mass spectrum data obtained by irradiating each position on the sample S with the laser. The mass analysis imaging image represents, for example, the ion intensities of a predetermined mass-to-charge ratio at each position on the sample S as pixel values ​​at the corresponding positions. The mass analysis imaging image data includes mass spectrum data at each position on the sample S.

[0040] 4. Determining the Region of Interest After acquiring optical image data and mass spectrometry imaging image data, the control device 3 executes a process for setting a group of regions of interest using the region of interest setting program 333. The control device 3 uses information acquired by mass spectrometry imaging to select candidates for the region of interest based on predetermined conditions. The predetermined conditions include, for example, the average ion intensity of a predetermined mass-to-charge ratio in the candidate region of interest and the median ion intensity of a predetermined mass-to-charge ratio in the candidate region of interest.

[0041] 2, among the set candidate regions of interest, regions R1 and R4 in which the average ion intensity of ions with a mass-to-charge ratio of 1500 is equal to or greater than a predetermined value are shown in gray. When regions in which the average ion intensity of ions with a mass-to-charge ratio of 1500 is equal to or greater than a predetermined value are selected as regions of interest from among the candidate regions of interest, regions R1 and R4 are determined as regions of interest, and regions R2 and R3, which do not satisfy this condition, are not selected as regions of interest.

[0042] 5. Display of Regions of Interest The control device 3 outputs an instruction to display the determined group of regions of interest on the display device 5. Note that, together with the determined group of regions of interest, an optical image, a mass spectrometry imaging image, and a group of candidate regions of interest may also be displayed.

[0043] The determined group of regions of interest is used when calculating predetermined statistics. [Method for determining a group of regions of interest for a mouse testis slice] Figure 3 is a diagram illustrating a method for determining a group of regions of interest based on an optical image and a mass spectrometry imaging image obtained using a mouse testis slice as a sample. Image 300 shows a group of candidate regions of interest set based on an optical image acquired by imaging mass spectrometer 100.

[0044] 3A shows a group of candidate regions of interest established based on optical image data. Furthermore, the group of candidate regions of interest whose signals satisfy predetermined conditions through mass spectrometry imaging are shown in gray. FIG. 3B shows the group of candidate regions of interest that are finally determined. As shown in FIG. 3, regions of interest are selected from the group of candidate regions of interest established based on optical image data based on data containing physical quantities different from those of the optical image data.

[0045] Furthermore, when brightness in optical image data is used as an index for setting a group of candidate regions of interest, a region along the edge of the sample may be set as a candidate region of interest, such as region R5 shown by hatching in Figure 3(a). Region R5 represents an area where there is no sample (a portion containing only the background), and is a region that should not be set as a region of interest. When determining a region of interest using only optical images, it may be difficult to exclude such a region from the region of interest.

[0046] The imaging mass spectrometer 100 described in the first embodiment determines a region of interest using not only optical image data but also mass analysis imaging image data. This allows for the elimination of regions of interest that are not intended by the user when determining a region of interest based on data obtained by a single measurement technique. This improves the accuracy of region of interest determination. As shown in Figure 3(b), region R5 is not included in the group of regions ultimately determined as regions of interest.

[0047] [Process for Determining Regions of Interest] Figure 4 is a flowchart showing an example of a process performed to determine a group of regions of interest using optical image data and mass spectrometry imaging image data. In one implementation example, the process in Figure 4 is called from a main routine and executed when the processor 31 of the control device 3 executes a given program. Since the control device 3 determines regions of interest based on the acquired image data, it can be said to function as an image processing device.

[0048] In step S10, the control device 3 acquires optical image data of the sample S. In step S12, the control device 3 sets a group of candidate regions of interest based on the optical image data acquired in step S10.

[0049] In step S14, the control device 3 acquires a mass spectrometry imaging image of the sample S. In step S16, the control device 3 selects, from the candidate regions of interest set in step S12, those that satisfy predetermined criteria in the data obtained by mass spectrometry, and determines them as regions of interest.

[0050] In step S18, the control device 3 causes the region of interest determined in step S16 to be displayed on the display device 5. Thereafter, the control device 3 ends the subroutine related to determining the region of interest, and returns the process to the main routine.

[0051] According to the method for determining a region of interest of the present disclosure, a group of region of interest candidates set based on physical quantities obtained by the first measurement technique is selected using a physical quantity different from the physical quantity used to set the group of region of interest candidates as an index, thereby improving the accuracy of setting the region of interest.

[0052] By setting a region of interest using data obtained by measuring different physical quantities, the user can more easily prevent a region unintended by the user from being set as a region of interest than when setting a region of interest using data obtained by measuring a single physical quantity. Therefore, the user can easily set conditions for determining the region intended by the user as a region of interest.

[0053] Furthermore, in the first embodiment described above, the range from which mass spectrometry imaging data is obtained is not limited. Mass spectrometry imaging may be performed only on the range set as the candidate region of interest, or on the entire range from which optical image data was obtained. When mass spectrometry imaging is performed only on the range set as the candidate region of interest, the measurement time required for mass spectrometry imaging can be shortened and the amount of data can be reduced, thereby reducing the amount of calculation. On the other hand, by performing mass spectrometry imaging on the entire range from which optical image data was obtained, even if the conditions for setting the candidate region of interest are changed after mass spectrometry imaging has been performed, the region of interest can be determined without performing mass spectrometry imaging again, thereby reducing the work time required for re-measurement of mass spectrometry imaging.

[0054] In the first embodiment, the region of interest is determined using optical image data and mass spectrometry imaging image data, but the image data used in the image processing method according to the present disclosure is not limited to this combination of image data. Image data obtained by Raman spectroscopy or image data obtained by scanning probe microscopy may also be used. Furthermore, the optical image data may be optical image data of a stained sample, image data of fluorescence emitted from the sample, or image data obtained by phase contrast observation.

[0055] Furthermore, in the first embodiment, a group of candidate regions of interest is set using optical image data, and a region of interest is determined using mass spectrometry imaging image data, but a group of candidate regions of interest may be set using mass spectrometry imaging image data, and a region of interest may be determined using optical image data. The order in which the various types of image data are used is not limited.

[0056] (Embodiment 2) In the above-described embodiment 1, the results of mass spectrometry imaging were used to select a group of region-of-interest candidates. In this case, a region of interest is selected from regions set as region-of-interest candidates. In embodiment 2, a group of region-of-interest candidates is set based on the results of mass spectrometry imaging data, separate from the group of region-of-interest candidates set based on optical image data. In embodiment 2, a region of interest is determined based on two types of region-of-interest candidate groups. Note that in embodiment 2, components that are the same as those of the imaging mass spectrometer 100 described in embodiment 1 are assigned the same reference numerals, and detailed description thereof will not be repeated. Furthermore, the contents described in embodiment 1 can be combined with embodiment 2 to the extent that they do not contradict.

[0057] [Method for determining a region of interest group when two types of region of interest candidate groups are set] A method for determining a region of interest group when two types of region of interest candidate groups are set will be described. Figure 5 is a diagram for explaining a method for determining a region of interest group when two types of region of interest candidate groups are set. In Figure 5, image 400 shows an image in which two types of region of interest candidate groups have been set.

[0058] First, a group of candidate regions of interest (first group of candidate regions of interest) is set based on the acquired optical image data. In Figure 5, the region surrounded by a solid line, as shown as region R6, corresponds to the first group of candidate regions of interest.

[0059] Next, a group of candidate regions of interest (second group of candidate regions of interest) is set based on the acquired mass spectrometry imaging image data, independently of the first group of candidate regions of interest. In Figure 5, the region surrounded by a dashed line, as shown by region R7, corresponds to the second group of candidate regions of interest.

[0060] The region selected from both the first group of region-of-interest candidates and the second group of region-of-interest candidates is determined as the region of interest. As shown by the hatched area in Figure 5, region R8, which is the overlapping region of regions R6 and R7, is determined as the region of interest. In this case, the region that satisfies both conditions is determined as the region of interest, so the region determined as the region of interest tends to be smaller than in the first embodiment.

[0061] For example, the above-described region of interest determination method is effective when it is desired to set a region in the nucleus of a cell where a substance with a mass-to-charge ratio of 1000 is present as a region of interest. Specifically, in an image obtained by staining the nucleus, the stained region is set as a first group of region of interest candidate, and regions in mass spectrometry imaging where the ion intensity of ions with a mass-to-charge ratio of 1000 is equal to or greater than a predetermined value are set as a second group of region of interest candidate. By setting the overlapping region of the set first group of region of interest candidate and the second group of region of interest candidate as the region of interest, it is possible to determine the region in the nucleus where a substance with a mass-to-charge ratio of 1000 is present as the region of interest.

[0062] Alternatively, a region selected in at least one of the first group of region-of-interest candidates and the second group of region-of-interest candidates may be determined as the region of interest. Figure 6 is a diagram for explaining a method of determining a region of interest group that differs from the method described in Figure 5 when two types of region-of-interest candidate groups are set. In Figure 6, image 500 shows an image in which two types of region-of-interest candidate groups have been set.

[0063] First, a group of candidate regions of interest (first group of candidate regions of interest) is set based on the acquired optical image data. In FIG. 6, the region surrounded by a solid line, as shown as region R9, corresponds to the first group of candidate regions of interest.

[0064] Next, a group of candidate regions of interest (second group of candidate regions of interest) is set based on the acquired mass spectrometry imaging image data, independently of the first group of candidate regions of interest. In Figure 6, the region R10 surrounded by a dashed line corresponds to the second group of candidate regions of interest.

[0065] Regions that are set as region of interest candidates in at least one of the first and second region of interest candidate groups are determined as regions of interest. As shown by the shaded regions in Figure 6, a region set only by region R9, a region set only by region R10, and a region set in both region R9 and region R10 are determined as regions of interest. In this case, a region that satisfies any one of the conditions is determined as a region of interest, so the regions determined as regions of interest tend to be larger than in embodiment 1.

[0066] For example, the above-described region of interest determination method is effective when it is desired to set as a region of interest at least one of the substances before and after conversion of a substance that is converted upon metabolism into a substance with a mass-to-charge ratio of 1200. Specifically, by setting the stained region in an image obtained by staining the substance before conversion as the first region of interest and setting as the second region of interest a region in which the ion intensity of ions with a mass-to-charge ratio of 1200 is equal to or greater than a predetermined value in mass spectrometry imaging, it is possible to determine as the region of interest at least one of the substances before and after conversion.

[0067] [Process for Determining Regions of Interest in Embodiment 2] Figure 7 is a flowchart showing an example of a process performed to determine a group of regions of interest using optical image data and mass analysis imaging image data. In one implementation example, the process in Figure 7 is called from a main routine and executed when the processor 31 of the control device 3 executes a given program. Note that in Figure 7, the same components as those in the flowchart described in Figure 4 are designated by the same reference numerals, and detailed description thereof will not be repeated.

[0068] In step S20, the control device 3 sets a group of region of interest candidates that is different from the group of region of interest candidates set in step S12, based on the mass spectrometry imaging data acquired in step S14.

[0069] In step S22, the control device 3 determines a group of regions of interest based on the group of region of interest candidates set in step S12 and the group of region of interest candidates set in step S20.

[0070] According to the above-described method for determining the region of interest, the region of interest group is determined using two types of region of interest candidate groups, thereby improving the accuracy of setting the region of interest compared to when the region of interest group is determined based on physical quantities obtained by a single measurement method.

[0071] Furthermore, unlike embodiment 1, each candidate region of interest group is set according to the physical quantities obtained by the two measurement techniques, so that a region of interest that reflects two types of physical quantities at each measurement point can be determined.

[0072] (Embodiment 3) Because biological samples have complex structures, the optimal conditions for setting a region of interest may differ for each sample. However, it may be desirable to use the same region of interest setting conditions for image data obtained from one sample without changing them when setting a region of interest for image data obtained from another sample. For example, this may occur when multiple sections are produced from a single tissue and image data for each section is obtained. In such a case, if the conditions for setting a region of interest differ for each section, the relative positions and shapes of the regions of interest may not match between the sections, which may cause problems, for example, when constructing three-dimensional data of the regions of interest.

[0073] In the third embodiment, the control device 3 determines a group of candidate regions of interest or a group of regions of interest for image data obtained from one sample, using setting conditions for the group of candidate regions of interest or the group of regions of interest determined for image data obtained from another sample. Note that in the third embodiment, the same components as those of the imaging mass spectrometer 100 described in the first and second embodiments are denoted by the same reference numerals, and detailed description thereof will not be repeated. Furthermore, the contents described in the first and second embodiments can be combined with the third embodiment to the extent that they do not contradict each other.

[0074] [Processing for Determining Regions of Interest in Embodiment 3] Fig. 8 is a flowchart showing an example of processing performed to determine a group of regions of interest using optical image data and mass spectrometry imaging image data. In one implementation example, the processing in Fig. 8 is called from a main routine and executed when the processor 31 of the control device 3 executes a given program.

[0075] In this embodiment, it is assumed that two samples are obtained from one tissue and that a group of regions of interest is set under the same conditions for both samples, one of which is referred to as the first sample and the other as the second sample.

[0076] In step S24, the control device 3 acquires optical image data of the first sample. In step S26, the control device 3 sets a first group of regions of interest based on the optical image data of the first sample acquired in step S24.

[0077] In step S28, the control device 3 acquires optical image data of the second sample. In step S30, the control device 3 sets a second group of regions of interest based on the optical image data of the second sample, using the region of interest setting conditions used in step S26.

[0078] In step S32, the control device 3 causes the first and second groups of regions of interest to be displayed on the display device 5. Thereafter, the control device 3 ends the subroutine related to determining the regions of interest, and returns the process to the main routine.

[0079] The image processing method described above makes it possible to set regions of interest for multiple sets of image data under the same conditions, thereby preventing the positions and sizes of regions of interest from being set significantly differently between sets of image data. As a result, users can easily construct three-dimensional data based on multiple sets of image data.

[0080] In the third embodiment described above, it is assumed that there are two samples, but the number of samples is not limited to two, and may be two or more.

[0081] Furthermore, in the above-described third embodiment, only optical image data is used to set the regions of interest, but mass spectrometry imaging data may also be used in combination with optical image data. Specifically, when there are multiple samples, optical image data and mass spectrometry imaging data are acquired for each sample, and a region of interest group is determined for one of the samples using the method described in the first or second embodiment. Next, a region of interest group is determined for the remaining samples under the same conditions as those used to determine the region of interest group for one sample. This method allows regions of interest to be determined for multiple samples under the same conditions.

[0082] (Modification) The user may confirm the displayed group of regions of interest and further select a group of regions of interest based on conditions specified by the user. By displaying a group of automatically determined regions of interest and allowing the user to select from among them, it is possible to prevent regions unintended by the user from being determined as regions of interest.

[0083] An example of a display screen that the imaging mass spectrometer 100 displays on the display device 5 when it determines a group of regions of interest will be described below. Fig. 9 is a diagram showing an example of the display screen.

[0084] The display screen 600 includes a window 610 and a window 620. The window 610 displays the group of regions of interest determined by the imaging mass spectrometer 100.

[0085] 9A shows an example of a display screen before the user specifies conditions, in which 13 regions including regions R11 and R12 determined by the control device 3 are displayed in a window 610.

[0086] Window 620 displays a bar used by the user to determine an area to be selected as a region of interest using the number of pixels that make up the determined area as an index. Window 620 includes bar 621 for determining the minimum number of pixels in the area to be selected as a region of interest, and bar 622 for determining the maximum number of pixels in the area to be selected as a region of interest.

[0087] In Figure 9(a), bar 621 indicates that 0 regions of interest are to be removed from the region of interest, starting with the smallest region, and bar 622 indicates that 13 regions of interest are to be left in the region of interest, starting with the largest region. In other words, in the state of Figure 9(a), no displayed regions of interest are to be removed.

[0088] 9B, when the user operates bar 621 with input device 4 to specify a predetermined number of pixels, regions consisting of fewer than the specified number of pixels are excluded from the region of interest. As a result, the smallest 10 regions displayed in window 610 are no longer displayed. In other words, the three regions displayed in window 610, including region R12, are selected as the region of interest.

[0089] According to the above-described modified example, the user can visually check the automatically determined region group and select a region of interest, thereby easily setting the region of interest as intended by the user.

[0090] In the above-described modified example, the user selected the region of interest using the size of the region as an index, but the index used is not limited to the size of the region and may also be the pixel value of the image data.

[0091] In addition, in the above-described modified example, the user selected regions from among the regions determined as a group of regions of interest, but the user may also select regions from among the regions set as a group of candidate regions of interest.

[0092] Aspects It will be understood by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

[0093] (Clause 1) An image processing method according to one aspect may include the steps of acquiring first image data obtained by measuring a first sample using a first measurement technique, setting a group of first region of interest candidates based on the first image data, acquiring second image data obtained by measuring the first sample using a second measurement technique different from the first measurement technique, the second image data reflecting a different physical quantity from the first image data, and setting a group of regions of interest in the first image data or the second image data based on the group of first region of interest candidates and the second image data.

[0094] According to the image processing method described in paragraph 1, it is possible to improve the accuracy of setting the region of interest.

[0095] (Item 2) In the image processing method described in item 1, the first image data may be an optical image, and the second image data may be a mass spectrometry imaging image.

[0096] According to the image processing method described in the second paragraph, it is possible to improve the accuracy in setting a region of interest based on an optical image and a mass spectrometry imaging image.

[0097] (Clause 3) In the image processing method described in paragraph 1 or paragraph 2, the step of setting the group of regions of interest may include a step of setting as the group of regions of interest those from the group of candidates for the first region of interest whose signal intensities of the second image data corresponding to the candidates for the first region of interest satisfy a predetermined criterion.

[0098] According to the image processing method described in paragraph 3, a region of interest is selected from a group of candidate regions of interest set based on first image data that reflects physical quantities obtained by the first measurement method, using physical quantities different from the physical quantities obtained by the first measurement method, thereby improving the accuracy of setting the region of interest.

[0099] (Clause 4) The image processing method described in any one of clauses 1 to 3 may further include a step of setting a group of candidates for a second region of interest based on the second image data, and the step of setting the group of regions of interest may include a step of setting the group of regions of interest based on a group of candidates for the first region of interest and a group of candidates for the second region of interest.

[0100] According to the image processing method described in paragraph 4, a region of interest is determined using a group of region of interest candidates set based on first image data that reflects physical quantities obtained by a first measurement technique, and a group of region of interest candidates set based on second image data that reflects physical quantities different from the physical quantities obtained by the first measurement technique.

[0101] (Clause 5) In the image processing method described in Clause 4, the step of setting the group of regions of interest may include a step of setting a region common to the group of candidates for the first region of interest and the group of candidates for the second region of interest as the group of regions of interest.

[0102] (Clause 6) In the image processing method described in Clause 4, the step of setting the group of regions of interest may include a step of setting, as the region of interest, an area set as a candidate region of interest in at least one of the first group of candidate regions of interest and the second group of candidate regions of interest.

[0103] (Clause 7) In the image processing method described in any one of clauses 1 to 6, the second image data may be acquired by the second measurement technique for a group of candidates for the first region of interest.

[0104] According to the image processing method described in paragraph 7, a group of candidates for a first region of interest is set before the second measurement method is performed. Then, measurement by the second measurement method is performed only on the regions set in the group of candidates for the first region of interest. Because the regions to be measured by the second measurement method are limited, the amount of data in the second image data can be reduced, and the burden of calculation processing required to determine the group of regions of interest can be reduced.

[0105] (Clause 8) The image processing method described in any one of clauses 1 to 7 may further include the steps of displaying a group of candidates for the first region of interest, accepting first predetermined conditions from a user, and excluding at least one region included in the group of candidates for the first region of interest from the group of candidates for the first region of interest based on the first predetermined conditions.

[0106] According to the image processing method described in paragraph 8, the user can easily recognize the automatically set candidate first regions of interest and can check whether a region not intended by the user has been set as a candidate first region of interest. Furthermore, if a region not intended by the user has been set as a candidate first region of interest, the region can be excluded based on a user operation, thereby improving the accuracy of setting the region of interest.

[0107] (Clause 9) In the image processing method described in Clause 8, the first predetermined condition may include at least one of the size of an area included in the candidate group of the first region of interest and the pixel value of a pixel of the area included in the candidate group of the first region of interest.

[0108] (Clause 10) The image processing method described in any one of clauses 1 to 9 may further include the steps of displaying the group of regions of interest, accepting second predetermined conditions from a user, and excluding at least one region included in the group of regions of interest from the group of regions of interest based on the second predetermined conditions.

[0109] According to the image processing method described in paragraph 10, the user can easily recognize the automatically set region of interest group and can check whether a region unintended by the user has been set as a region of interest group. Furthermore, if a region unintended by the user has been set as a region of interest group, the region can be excluded based on a user operation, thereby improving the accuracy of setting the region of interest.

[0110] (Item 11) In the image processing method described in item 10, the second predetermined condition may be the size of the region included in the group of regions of interest.

[0111] (Clause 12) The image processing method described in any one of clauses 1 to 11 may further include a step of acquiring third image data obtained by measuring a second sample different from the first sample using the first measurement technique, and a step of setting a group of candidates for a third region of interest in the third image data based on the third image data under conditions for setting a group of candidates for the first region of interest.

[0112] According to the image processing method described in paragraph 12, it is possible to set regions of interest under the same conditions for image data obtained from different samples, thereby preventing regions containing different physical quantities from being set as regions of interest in image data obtained from different samples.

[0113] (Clause 13) The image processing method described in any one of clauses 1 to 12 may further include the steps of: acquiring third image data obtained by measuring a second sample different from the first sample using the first measurement technique; acquiring fourth image data obtained by measuring the second sample using the second measurement technique; and setting a group of regions of interest in the third image data or the fourth image data based on the third image data and the fourth image data under the same conditions as those under which a group of regions of interest was set in the first image data or the second image data based on the first image data and the second image data.

[0114] (14th paragraph) A program according to one aspect is a program executed by a processor installed in a computer, and may cause the computer to execute the image processing method described in any one of the first to thirteenth paragraphs.

[0115] (Clause 15) An image processing device according to one aspect may acquire first image data obtained by measuring a first sample using a first measurement method, set a group of candidates for a first region of interest based on the first image data, acquire second image data obtained by measuring the first sample using a second measurement method different from the first measurement method, and reflect a physical quantity different from that of the first image data, and set a group of regions of interest in the first image data or the second image data based on the group of candidates for the first region of interest and the second image data.

[0116] (Item 16) An imaging mass spectrometer according to one aspect is an imaging mass spectrometer that generates ions by irradiating a sample with laser light and performs mass analysis of the ions, and includes a laser irradiation unit that emits the laser light toward the sample, a detection unit that detects ions generated by the laser light, a generation unit that generates a mass analysis imaging image based on a signal detected by the detection unit, an imaging unit that acquires an optical microscope image of the sample, an image analysis unit that acquires and analyzes the mass analysis imaging image and the optical microscope image, and a display unit that displays the analysis results of the image analysis unit, and the image analysis unit may set a group of candidate regions of interest based on the optical microscope image, and set a group of regions of interest in the optical microscope image or the mass analysis imaging image based on the group of candidate regions of interest and the imaging image.

[0117] According to the imaging mass spectrometer described in Item 16, it is possible to improve the accuracy of setting the region of interest.

[0118] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the description of the above-described embodiments, and is intended to include all modifications within the meaning and scope of the claims. Furthermore, it is intended that each technique in the embodiments can be implemented alone or, if necessary, in combination with other techniques in the embodiments to the extent possible.

[0119] 1 Airtight chamber, 2 Vacuum chamber, 3 Control device, 4 Imaging unit, 5 Laser irradiation unit, 6 Input device, 7 Display device, 11 Sample stage, 12 Guide, 13 Drive mechanism, 14 Illumination unit, 15 Ion transport tube, 16 Detection unit, 21, 22 Ion transport optical system, 23 Ion trap, 24 Mass analyzer, 25 Detector, 31 Processor, 32 Input / output port, 33 Memory, 100 Imaging mass spectrometer, 331 Image data acquisition program, 332 Region of interest setting program, 3311 Sample stage drive unit, 3312 Optical image processing unit, 3313 Laser drive unit, 3314 Analysis control unit, 3315 Imaging image processing unit, 3321 Region of interest candidate setting unit, 3322 Region of interest determination unit.

Claims

1. An image processing method comprising the steps of: acquiring first image data obtained by measuring a first sample using a first measurement method; setting a first group of candidate regions of interest based on the first image data; acquiring second image data obtained by measuring the first sample using a second measurement method different from the first measurement method, the second image data reflecting a different physical quantity than the first image data; and setting a group of regions of interest in the first image data or the second image data based on the first group of candidate regions of interest and the second image data.

2. The image processing method according to claim 1, wherein the first image data is an optical image and the second image data is a mass spectrometry imaging image.

3. An image processing method as described in claim 1 or claim 2, wherein the step of setting the group of regions of interest includes a step of setting as the group of regions of interest those from the group of candidates for the first region of interest whose signal intensities of the second image data corresponding to the candidates for the first region of interest satisfy a predetermined criterion.

4. An image processing method as described in claim 1 or claim 2, further comprising a step of setting a group of candidates for a second region of interest based on the second image data, wherein the step of setting the group of regions of interest includes a step of setting the group of regions of interest based on a group of candidates for the first region of interest and a group of candidates for the second region of interest.

5. The image processing method according to claim 4, wherein the step of setting the group of regions of interest includes a step of setting a common region between the first group of candidate regions of interest and the second group of candidate regions of interest as the group of regions of interest.

6. The image processing method of claim 4, wherein the step of setting the group of regions of interest includes a step of setting, as the region of interest, a region set as a candidate region of interest in at least one of the first group of candidate regions of interest and the second group of candidate regions of interest.

7. The image processing method according to claim 1 or 2, wherein the second image data is acquired by the second measurement technique for a group of candidates for the first region of interest.

8. An image processing method as described in claim 1 or claim 2, further comprising the steps of: displaying a group of candidates for the first region of interest; accepting first specified conditions from a user; and excluding at least one region included in the group of candidates for the first region of interest from the group of candidates for the first region of interest based on the first specified conditions.

9. The image processing method of claim 8, wherein the first predetermined condition includes at least one of the size of the area included in the candidate group of the first region of interest and the pixel values ​​of the pixels of the area included in the candidate group of the first region of interest.

10. An image processing method as described in claim 1 or claim 2, further comprising the steps of: displaying the group of regions of interest; accepting second specified conditions from a user; and excluding at least one region included in the group of regions of interest from the group of regions of interest based on the second specified conditions.

11. The image processing method according to claim 10, wherein the second predetermined condition is the size of the areas included in the group of areas of interest.

12. An image processing method as described in claim 1 or claim 2, further comprising the steps of: acquiring third image data obtained by measuring a second sample different from the first sample using the first measurement method; and setting a third group of candidate regions of interest in the third image data based on the third image data under conditions for setting a group of candidate regions of interest for the first region of interest.

13. An image processing method according to claim 1 or claim 2, further comprising the steps of: acquiring third image data obtained by measuring a second sample different from the first sample using the first measurement method; acquiring fourth image data obtained by measuring the second sample using the second measurement method; and setting a group of regions of interest in the third image data or the fourth image data based on the third image data and the fourth image data under the same conditions as those under which a group of regions of interest was set in the first image data or the second image data based on the first image data and the second image data.

14. A program executed by a processor installed in a computer, causing the computer to execute the image processing method according to claim 1 or 2.

15. An image processing device that acquires first image data obtained by measuring a first sample using a first measurement method, sets a first group of candidate regions of interest based on the first image data, acquires second image data that reflects a different physical quantity from the first image data and is obtained by measuring the first sample using a second measurement method different from the first measurement method, and sets a group of regions of interest in the first image data or the second image data based on the first group of candidate regions of interest and the second image data.

16. An imaging mass spectrometer that generates ions by irradiating a sample with laser light and performs mass analysis on the ions, comprising: a laser irradiation unit that emits the laser light toward the sample; a detection unit that detects ions generated by the laser light; a generation unit that generates a mass analysis imaging image based on a signal detected by the detection unit; an imaging unit that acquires an optical microscope image of the sample; an image analysis unit that acquires and analyzes the mass analysis imaging image and the optical microscope image; and a display unit that displays the analysis results of the image analysis unit, wherein the image analysis unit sets a group of candidate regions of interest based on the optical microscope image, and sets a group of regions of interest in the optical microscope image or the mass analysis imaging image based on the group of candidate regions of interest and the mass analysis imaging image.

Images