Device, program, and method for inspecting biological tissue

Abstract

The present invention achieves automatic inspection of a biological tissue, the inspection enabling highly accurate determination. In an inspection image (Tim), a feature quantity is associated with a combination of a first coordinate, a second coordinate and a specific physical quantity. The first coordinate and the second coordinate specify a position in a focal plane when a biological tissue is imaged. The specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at the position specified by the first coordinate and the second coordinate. The feature quantity includes the intensity of light having a specific physical quantity. With respect to the inspection image, an inference model (Md) extracts information related to an abnormality from the inspection image (Tim) by performing on the feature quantity a three-dimensional convolution operation, in which the first coordinate, the second coordinate, and the specific physical quantity are set as a first dimension, a second dimension, and a third dimension, respectively.

Description

Apparatus, program, and method for examining biological tissue

[0001] The present disclosure relates to devices, programs, and methods for examining biological tissue.

[0002] Conventionally, configurations for examining biological tissue are known. For example, JP 2023-551913 A (Patent Document 1) discloses a method and system for predicting a subject's diagnostic status regarding a biological disease or disorder. According to this method and system, the subject's diagnostic status regarding the disease or disorder can be predicted by acquiring a Raman image of the sample, analyzing the Raman spectrum spatially across the sample, generating a temporal Raman profile, and processing the data using a trained model.

[0003] Special Publication No. 2023-551913

[0004] Qingqing Hong, Xinyi Zhong, Weitong Chen, Zhenghua Zhang, Bin Li, "Hyperspectral Image Classification Network Based on 3D Octave Convolution and Multiscale Depthwise Separable Convolution", ISPRS Int. J. Geo-Inf. 2023, 12(12), 505; https: / / doi.org / 10.3390 / ijgi12120505.Siqi Wei, Yafei Liu, Mengshan Li, Haijun Huang, Xin Zheng, Lixin Guan, "DCCaps-UNet: A U-Shaped Hyperspectral Semantic Segmentation Model Based on the Depthwise Separable and Conditional Convolution Capsule Network", Remote Sens. 2023, 15(12), 3177; https: / / doi.org / 10.3390 / rs15123177.Si-Jie Hao, Yuan Wan, Yi-Qiu Xia, Xin Zou, Si-Yang Zheng, "Size-based separation methods of circulating tumor cells", in Advanced Drug Delivery Reviews 125 (2018), pp. 3-20.

[0005] Cancer cells are an example of a cause of biological disease. In an initial cancer diagnosis, a test (cytology) is typically performed in which cell tissue (specimen) collected from a patient is stained with hematoxylin eosin (HE), and the stained specimen (sample) is observed under a microscope to determine the presence or absence of abnormal cells, etc. Depending on the results of the cytology test, tissue is subsequently collected and a histology (needle biopsy) is performed for a more accurate diagnosis. In recent years, in order to obtain rapid and accurate results, histology is often performed without cytology. In cytology / histology, a laboratory technician typically stains the specimen collected by the attending physician with HE to prepare a sample, and a pathologist then observes the sample to determine the presence or absence of abnormal cells, etc.

[0006] In recent years, the shortage of pathologists has become a problem. Under these circumstances, if pathologists are always required to make the final decision on cytology / histology, the burden on them may become excessive. Furthermore, it is difficult to obtain the results of cytology / histology on the same day; it usually takes about one to two weeks for the results to be obtained. The fact that it takes a certain amount of time to obtain cytology results can increase the burden on hospital staff and can also increase the physical and mental burden on patients.

[0007] In this regard, it is possible to shorten the time required to obtain cytology results by introducing automatic analysis processing etc. to images of biological tissues. However, as an alternative to the judgments made by pathologists, automatic examination of biological tissues requires high accuracy.

[0008] The present disclosure has been made to solve the above-mentioned problems, and its purpose is to realize automatic examination of biological tissue that allows highly accurate determination.

[0009] According to one aspect of the present disclosure, an apparatus includes a memory unit and an inference unit. The memory unit stores an inspection image of biological tissue. The inference unit uses an inference model to determine whether the inspection image contains an abnormality. In the inspection image, a feature is associated with a combination of a first coordinate, a second coordinate, and a specific physical quantity. The first coordinate and the second coordinate identify a position on a focal plane when the biological tissue is imaged. The specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at a position identified by the first coordinate and the second coordinate. The feature includes the intensity of light having the specific physical quantity. The inference model extracts information related to the abnormality from the inspection image by performing a three-dimensional convolution operation on the feature, with the first coordinate, the second coordinate, and the specific physical quantity being the first dimension, the second dimension, and the third dimension, respectively.

[0010] According to another aspect of the present disclosure, an apparatus includes a storage unit and a learning unit. The storage unit stores training images labeled with abnormalities in biological tissue. The learning unit performs machine learning on an inference model using the training images. In the training images, a feature is associated with a combination of a first coordinate, a second coordinate, and a specific physical quantity. The first coordinate and the second coordinate identify a position on a focal plane when the biological tissue is imaged. The specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at a position identified by the first coordinate and the second coordinate. The feature includes the intensity of light having the specific physical quantity. The learning unit constructs an inference model that determines information related to an abnormality from the training images by performing a three-dimensional convolution operation on the feature, with the first coordinate, the second coordinate, and the specific physical quantity being the first dimension, the second dimension, and the third dimension, respectively, on the training images.

[0011] A program according to another aspect of the present disclosure, when executed by a processor, causes the processor to acquire an inspection image of biological tissue and determine whether the inspection image contains an abnormality using an inference model. In the inspection image, a feature is associated with a combination of a first coordinate, a second coordinate, and a specific physical quantity. The first coordinate and the second coordinate identify a position on the focal plane when the biological tissue is imaged. The specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at the position identified by the first coordinate and the second coordinate. The feature includes the intensity of light having the specific physical quantity. The inference model extracts information related to the abnormality from the inspection image by performing a three-dimensional convolution operation on the feature, with the first coordinate, the second coordinate, and the specific physical quantity being the first dimension, the second dimension, and the third dimension, respectively.

[0012] A method according to another aspect of the present disclosure includes acquiring an inspection image of biological tissue and using an inference model to determine whether the inspection image contains an abnormality. In the inspection image, a feature is associated with a combination of a first coordinate, a second coordinate, and a specific physical quantity. The first coordinate and the second coordinate identify a position on a focal plane when the biological tissue is imaged. The specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at the position identified by the first coordinate and the second coordinate. The feature includes the intensity of light having the specific physical quantity. The inference model extracts information related to the abnormality from the inspection image by performing a three-dimensional convolution operation on the feature, with the first coordinate, the second coordinate, and the specific physical quantity being the first dimension, the second dimension, and the third dimension, respectively.

[0013] According to the device, program, and method disclosed herein, an inference model performs a three-dimensional convolution operation on the test image to extract information about abnormalities from the test image, thereby enabling automatic testing of biological tissue with high accuracy.

[0014] 5 is a block diagram showing an example of the configuration of an information processing device 10 according to an embodiment; FIG. 6 is a diagram showing an example of Raman spectra when Raman spectroscopy is performed on abnormal biological tissue containing cancer cells and normal biological tissue not containing cancer cells; FIG. 7 is a diagram showing a three-dimensional representation of an examination image, which is an example of an image acquired by Raman imaging; FIG. 8 is a diagram showing an example of a two-dimensional image for each wavenumber included in the examination image of FIG. 3; FIG. 9 is a diagram showing an overview of a convolution operation according to a comparative example; FIG. 10 is a diagram showing an overview of a three-dimensional convolution operation according to an embodiment; FIG. 11 is a diagram comparing indices related to cancer determination for each block when the block size of FIGS. 5 and 6 is changed between an embodiment and a comparative example; FIG. 12 is a diagram comparing indices related to cancer determination for examination images when the block size of FIGS. 5 and 6 is changed between an embodiment and a comparative example; FIG. 13 is a flowchart showing an example of the flow of a cancer determination process executed by a processor that executes the image analysis program of FIG. 1;

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals, and their description will not be repeated in principle.

[0016] 1 is a block diagram showing an example of the configuration of an information processing device 10 according to an embodiment. The information processing device 10 is an example of a device for examining biological tissue.

[0017] The information processing device 10 includes a processor 11 (inference unit), a memory 12 (storage unit), an input unit 13, an output unit 14, and a bus 15. The processor 11, the memory 12, the input unit 13, and the output unit 14 are communicably connected to one another via the bus 15. The information processing device 10 includes, for example, a PC (Personal Computer) or a workstation.

[0018] The processor 11 includes, for example, a central processing unit (CPU) and may include a graphics processing unit (GPU). The processor 11 executes programs stored in the memory 12 and realizes each function of the information processing device 10.

[0019] The memory 12 includes a non-volatile memory (e.g., a hard disk) and a volatile memory (e.g., a random access memory (RAM)). The memory 12 stores an inference model Md, an image analysis program Pg1, a machine learning program Pg2, an inspection image Tim, and a training image Lim. Although not shown in FIG. 1, the memory 12 may also store an operating system (OS), data required to execute each program, and the processing results of each program.

[0020] The test image Tim and the learning image Lim are acquired by Raman imaging of a sample including a patient's biological tissue (e.g., breast tissue). In Raman imaging, a Raman spectrum based on Raman spectroscopy is acquired for each coordinate (pixel) on a focal plane (a plane normal to the optical axis and including the focal point) on which the sample is placed. In Raman spectroscopy, monochromatic light (incident light) from a laser or the like is irradiated onto the focal plane, and Raman scattered light scattered by a substance present at the irradiated position of the incident light is acquired.

[0021] In a Raman spectrum, the intensity (feature quantity) of Raman scattered light is associated with a wavenumber (specific physical quantity), which is the reciprocal of the wavelength of the Raman scattered light. Raman scattered light is defined as the change in wavelength of scattered light relative to incident light in terms of energy, which is a value obtained by multiplying Planck's constant by the wavenumber. Because the wavelength of Raman scattered light scattered by a substance depends on the molecular vibration of the substance, a Raman spectrum often exhibits a peak at a wavenumber specific to the substance. In other words, a Raman spectrum contains information that can identify the substance. Note that the specific physical quantity is not limited to the wavenumber of Raman scattered light, and may be any physical quantity related to light from a substance in biological tissue present at specific coordinates (e.g., light emitted from the substance).

[0022] The training image Lim is an image labeled with an abnormality in the biological tissue. The label includes, for example, information regarding the presence or absence of cancer cells determined by a pathologist who analyzed the biological tissue. In each of the test image Tim and the training image Lim, the intensity of Raman scattered light is associated with a combination of an X coordinate (first coordinate), a Y coordinate (second coordinate), and a wavenumber. That is, each coordinate in each of the test image Tim and the training image Lim is associated with the Raman spectrum of the substance located at that coordinate. Each of the test image Tim and the training image Lim is an image obtained by superimposing (merging) two-dimensional images parallel to the XY plane, each composed of the Raman spectrum intensity for each wavenumber within the wavenumber range of Raman spectroscopy.

[0023] The inference model Md extracts information about abnormalities (e.g., cancer cells) contained in the test image Tim from the test image Tim by performing a three-dimensional convolution operation (see Non-Patent Documents 1 and 2) on the intensity of the test image Tim, where the X coordinate (first coordinate), Y coordinate (second coordinate), and wavenumber on the focal plane are the first, second, and third dimensions, respectively. That is, the inference model Md includes a 3D-CNN (convolutional neural network). The three-dimensional convolution operation considers not only information about the substance itself located at each coordinate, but also information about what substances are present around the substance when determining whether or not an abnormality exists at a certain coordinate in the test image Tim.

[0024] The processor 11 (inference unit) executing the image analysis program Pg1 uses the inference model Md to determine whether or not an abnormality is contained in the inspection image Tim. The processor 11 (learning unit) executing the machine learning program Pg2 performs machine learning (e.g., supervised learning) on ​​the inference model Md using the training images Lim. The processor 11 executing the machine learning program Pg2 constructs the inference model Md that determines information related to an abnormality from the training images Lim by performing a three-dimensional convolution operation on the training images Lim. Note that machine learning includes deep learning.

[0025] The input unit 13 receives input from the user (for example, GUI (Graphical User Interface) operations, gestures, CUI (Character User Interface) input, command input, or voice input). The input unit 13 includes a mouse, a keyboard, a touch panel, a camera, and a microphone. The output unit 14 outputs the processing results of the processor 11 to the user. The output unit 14 includes, for example, a touch panel, a display, and a speaker.

[0026] 2 shows an example of Raman spectra obtained by Raman spectroscopy on abnormal tissue containing cancer cells and normal tissue containing no cancer cells. The solid line shows the Raman spectrum of the abnormal tissue, and the dotted line shows the Raman spectrum of the normal tissue. As shown in FIG. 2, the wavenumber range of the Raman spectra is 600 cm －１ ~3050cm －１ Since the Raman spectrum of abnormal biological tissue and the Raman spectrum of normal biological tissue are different from each other, it is possible to distinguish between abnormal biological tissue and normal biological tissue by using this information.

[0027] FIG. 3 is a three-dimensional representation of an inspection image Tim1, which is an example of an image acquired by Raman imaging. As shown in FIG. 3, the X-axis, Y-axis, and wavenumber axis are orthogonal to one another. The XY plane corresponds to the focal plane. The wavenumber axis extends from the origin of the XY plane as an axis orthogonal to the XY plane. The data contained in inspection image Tim1 is, in three-dimensional space, in the range of 0 μm to 150 μm on the X-axis, the range of 0 μm to 45 μm on the Y-axis, and the range of 600 cm on the wavenumber axis. －１ ~3050cm －１ It exists in the range of , and forms a rectangular parallelepiped area.

[0028] 4 is a diagram showing examples of two-dimensional images for each wave number included in the inspection image Tim1 in FIG. 3. As shown in FIG. 4, two-dimensional images Tim11, Tim12, Tim13, Tim14, and Tim15 are two-dimensional images superimposed on the inspection image Tim1, each of which is 750 cm －１ , 1246 cm －１ , 1520cm－１ , 1655cm －１ , 2855 cm －１ Based on the principle of Raman spectroscopy, in which the wavelength of Raman scattered light scattered by a substance depends on the molecular vibration of the substance, the two-dimensional images Tim11, Tim12, Tim13, Tim14, and Tim15 mainly depict mitochondria, collagen, carotenoids, proteins, and lipids, respectively.

[0029] Below, a comparison is made between the configuration according to the embodiment and a comparative example that performs one-dimensional convolution calculations on Raman spectra in terms of accuracy in determining whether or not cancer cells are contained in the test image Tim (cancer determination).

[0030] FIG. 5 is a diagram illustrating an overview of a convolution operation according to a comparative example. As shown in FIG. 5, an inspection image Tim is divided into multiple blocks Bc. The length of each block Bc in the X-axis direction (box size), the length of each block Bc in the Y-axis direction (box size), and the length of each block Bc in the wavenumber direction are B, B, and R, respectively. A surface of each block Bc on the XY plane includes multiple coordinates. In the comparative example, an average Raman spectrum Sav is derived, in which the average value of the intensities of multiple Raman spectra associated with each of the multiple coordinates is associated with each wavenumber. The average Raman spectrum Sav is input to a 1D-CNN, and a one-dimensional convolution operation is performed on the average Raman spectrum Sav. Information regarding abnormalities contained in the blocks Bc is extracted from the average Raman spectrum Sav by the 1D-CNN, and a cancer node value (cancer index) and a normal node value are calculated in the output layer of the 1D-CNN. The cancer index is an example of an abnormal value indicating the degree of abnormality of the block Bc.

[0031] If the value of the cancer node is greater than the value of the normal node, the block Bc is determined to contain cancer cells. Furthermore, if the average value of the cancer indexes of multiple blocks Bc exceeds a threshold, the test image Tim is determined to contain cancer cells. The value compared with the threshold in cancer determination may be any statistical value of the cancer indexes of multiple blocks Bc, such as the median, maximum, or minimum value. Furthermore, the threshold can be appropriately determined through experiments or simulations.

[0032] The average Raman spectrum Sav does not include position information of the Raman spectrum for each of the multiple coordinates included in the block Bc. Therefore, cancer diagnosis based on the average Raman spectrum Sav does not take into account information about what substances are present around the substances located at each coordinate.

[0033] FIG. 6 is a diagram illustrating an overview of a three-dimensional convolution operation according to an embodiment. The test image Tim is the same as the test image Tim shown in FIG. 5. As shown in FIG. 6, in this embodiment, the block Bc is input directly to the 3D-CNN, and a three-dimensional convolution operation is performed on the block Bc. The 3D-CNN extracts information about abnormalities contained in the block Bc, and calculates values ​​for cancer nodes and normal nodes in the output layer of the 3D-CNN. In this embodiment as well, if the average value of the cancer index exceeds a threshold, it is determined that the test image Tim contains cancer cells. In cancer determination according to this embodiment, information about what substances are present around the substance located at each coordinate is taken into consideration.

[0034] FIG. 7 is a diagram comparing indices related to cancer detection for each block Bc when the block size B in FIGS. 5 and 6 is changed between the embodiment and the comparative example. In FIG. 7, for each of the indices, accuracy, sensitivity, and specificity, which are examples of the indices, a broken line is shown, along with an error band surrounding the broken line. The error band represents the distribution of the indices when cancer detection is performed multiple times for a certain block size B. That is, the lower end of the error band represents the minimum value of the indices, and the upper end of the error band represents the maximum value of the indices, with a dot representing the median between the lower and upper ends. The plotted data is based on the results of cancer detection for 1.5 μm, 3 μm, 4.5 μm, 6 μm, 7.5 μm, 9 μm, 12 μm, 15 μm, 18 μm, 22.5 μm, and 30 μm. The broken lines labeled 1D and 3D are based on cancer detection by the comparative example and the embodiment, respectively. The same applies to FIG. 8, which will be described next to FIG. 7.

[0035] As shown in Figure 7, the accuracy of cancer detection according to the embodiment exceeds 0.8 for all block sizes B, and is also higher than the accuracy of cancer detection according to the comparative example. The accuracy of cancer detection according to the embodiment increases in the block size B range of 1.5 μm to 9 μm, remains almost constant in the block size B range of 9 μm to 15 μm, and decreases above 15 μm. The range of 9 μm to 15 μm corresponds to the size of cells contained in the test image Tim (see Non-Patent Document 3). By adjusting the block size B to the size of cells contained in the test image Tim, the accuracy of cancer detection for each block size B can be improved.

[0036] 8 is a diagram comparing the indices for cancer detection of the test image Tim between the embodiment and the comparative example when the block size B in FIGS. 5 and 6 is changed. As shown in FIG. 8, the accuracy of cancer detection by the embodiment exceeds the accuracy of cancer detection by the comparative example for all block sizes B. The accuracy of cancer detection by the embodiment exceeds 0.9 in the range of block sizes B from 3 μm to 22.5 μm.

[0037] Fig. 9 is a flowchart showing an example of the flow of the cancer detection process executed by the processor 11 that executes the image analysis program Pg1 in Fig. 1. In the following, steps will be simply abbreviated as S.

[0038] 9 , in S101, the processor 11 acquires an inspection image Tim and proceeds to S102. In S102, the processor 11 calculates cancer indices for multiple blocks Bc included in the inspection image Tim using the inference model Md and proceeds to S103. In S103, the processor 11 calculates an average value of the cancer indices and proceeds to S104. In S104, the processor 11 determines whether the average value of the cancer indices is greater than a threshold value.

[0039] If the average value of the cancer index is greater than the threshold value (YES in S104), the processor 11 determines in S105 that the test image Tim contains cancer cells and proceeds to S107. If the average value of the cancer index is equal to or less than the threshold value (NO in S104), the processor 11 determines in S106 that the test image Tim does not contain cancer cells and proceeds to S107. In S107, the processor 11 outputs the result of the cancer determination to the output unit 14 and ends the process.

[0040] As described above, the device, program, and method according to the embodiment can realize automatic examination of biological tissue, which allows highly accurate determination.

[0041] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims.

[0042] 10 Information processing device, 11 Processor, 12 Memory, 13 Input unit, 14 Output unit, 15 Bus, B Block size, Bc Block, Lim Learning image, Md Inference model, Pg1 Image analysis program, Pg2 Machine learning program, Sav Average Raman spectrum, Tim, Tim1 Inspection image, Tim11 to Tim15 Two-dimensional images.

Claims

1. An apparatus comprising: a memory unit for storing test images of biological tissue; and an inference unit for using an inference model to determine whether the test images contain abnormalities; wherein, in the test images, features are associated with combinations of first coordinates, second coordinates, and specific physical quantities, the first coordinates and the second coordinates identify a position on a focal plane when the biological tissue is photographed, the specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at a position identified by the first coordinates and the second coordinates, and the feature includes the intensity of the light having the specific physical quantity; and the inference model extracts information related to the abnormality from the test images by performing a three-dimensional convolution operation on the feature, with the first coordinates, the second coordinates, and the specific physical quantity being the first dimension, second dimension, and third dimension, respectively.

2. The device described in claim 1, wherein the information regarding the abnormality is calculated by calculating multiple abnormality values ​​each indicating the degree of abnormality of multiple blocks contained in the inspection image, and the inference unit determines whether the inspection image contains an abnormality based on statistical values ​​of the multiple abnormality values.

3. The device according to claim 2, wherein the size of each of the plurality of blocks corresponds to the size of a cell contained in the biological tissue.

4. An apparatus according to any one of claims 1 to 3, wherein the inspection image is obtained by Raman imaging, the light includes Raman scattered light, and the specific physical quantity includes a physical quantity related to the wavelength of the Raman scattered light.

5. An apparatus comprising: a memory unit that stores training images labeled with information related to abnormalities in biological tissue; and a learning unit that performs machine learning on an inference model using the training images, wherein in the training images, a feature is associated with a combination of a first coordinate, a second coordinate, and a specific physical quantity, the first coordinate and the second coordinate specify a position on a focal plane when the biological tissue is photographed, the specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at a position specified by the first coordinate and the second coordinate, and the feature includes the intensity of the light having the specific physical quantity, and the learning unit constructs the inference model that determines information related to the abnormality from the training images by performing a three-dimensional convolution operation on the feature, with the first coordinate, the second coordinate, and the specific physical quantity being the first dimension, the second dimension, and the third dimension, respectively.

6. A program that, when executed by a processor, causes the processor to acquire an examination image of biological tissue and determine whether the examination image contains an abnormality using an inference model, wherein in the examination image, a feature is associated with a combination of a first coordinate, a second coordinate, and a specific physical quantity, the first coordinate and the second coordinate specify a position on a focal plane when the biological tissue is photographed, the specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at a position specified by the first coordinate and the second coordinate, and the feature includes the intensity of the light having the specific physical quantity, and the inference model extracts information related to the abnormality from the examination image by performing a three-dimensional convolution operation on the feature, with the first coordinate, the second coordinate, and the specific physical quantity being the first dimension, the second dimension, and the third dimension, respectively.

7. A method comprising the steps of: acquiring an examination image of biological tissue; and using an inference model to determine whether the examination image contains an abnormality; wherein in the examination image, a feature is associated with a combination of a first coordinate, a second coordinate, and a specific physical quantity; the first coordinate and the second coordinate specify a position on a focal plane when the biological tissue is photographed; the specific physical quantity includes a physical quantity related to light from a substance of the biological tissue present at a position specified by the first coordinate and the second coordinate; the feature includes the intensity of the light having the specific physical quantity; and the inference model extracts information related to the abnormality from the examination image by performing a three-dimensional convolution operation on the feature, with the first coordinate, the second coordinate, and the specific physical quantity being the first dimension, the second dimension, and the third dimension, respectively.

Images