Biological structure identification device for displaying biological structure and gene expression in biological structure together, biological structure identification method, and biological structure identification computer program

Abstract

When a three-dimensional structure is constructed in advance from a sample before section creation and gene expression information measured by the section is associated with the three-dimensional structure, it was found that there is a problem where the gene information cannot be accurately associated with the three-dimensional structure created in advance because deformation from the three-dimensional structure exists due to section creation processing. It has been found that gene expression data is associated in an accurate three-dimensional structure by aligning section image data and three-dimensional image data, adjusting the section image data in accordance with the three-dimensional image data on the basis of the alignment, and adjusting the gene expression data in accordance with the section image data.

Description

A biological structure identification device, a biological structure identification method, and a computer program for biological structure identification that display both the biological structure and gene expression within the biological structure.

[0001] The present invention relates to a biological structure recognition device, a biological structure recognition method, and a computer program for biological structure recognition that display the three-dimensional structure of a living organism and gene expression together.

[0002] Techniques for acquiring biodata representing the three-dimensional structure of living organisms or biological samples are known, such as computed tomography (CT) or magnetic resonance imaging (MRI) (see, for example, Non-Patent Documents 1-3). An observer can understand the structure of an object represented in such biodata by observing it, for example, displayed on a display device. Apparatuses and methods are known that can identify each tissue and / or cell included in biodata representing a three-dimensional structure, identify the biological structure in three dimensions from the biodata, and display the three-dimensional structure at the cell and / or tissue level (Patent Document 1: Patent No. 7387340). While methods for constructing three-dimensional models at the cell and / or tissue level have been very useful for analyzing biological tissues, especially skin structures, in terms of shape and distribution, information on shape and distribution alone has limitations from the perspective of biological tissue research.

[0003] On the other hand, techniques have been developed to perform observation of tissue sections and nucleic acid sequencing extracted from each spot on the tissue sections in parallel, making it possible to record hundreds to tens of thousands of gene expressions and DNA information for each spot in formalin-fixed, paraffin-embedded samples.

[0004] Brenner, D. J., & Hall, E. J., Computed tomography - an increasing source of radiation exposure, New England Journal of Medicine, 357(22), 2277 - 2284, 2007 Agatston, A. S., Janowitz, W. R., Hidner, F. J., Zusmer, N. R., Viamonte, M., & Detrano, R., Quantification of coronary artery calcium using ultrafast computed tomography, Journal of the American College of Cardiology, 15(4), 827 - 832, 1990 Ogawa, S., Lee, T. M., Kay, A. R., & Tank, D. W., Brain magnetic resonance imaging with contrast dependent on blood oxygenation, Proceedings of the National Academy of Sciences, 87(24), 9868 - 9872, 1990

[0005] Based on the gene expression information and section images at each spot of the tissue section, an attempt was made to construct a three - dimensional structure associated with the gene expression information. However, due to the deformation caused by the operations during section preparation, there was a problem that the three - dimensional structure associated with the gene expression information was deformed from the three - dimensional structure of the actual sample. Also, it was a problem that spatial information such as the position of the excised tissue piece in the original tissue was lost.

[0006] Therefore, an idea was conceived to pre - construct a three - dimensional structure from the sample before section preparation instead of from the three - dimensional structure constructed from the section, and to associate the gene expression information measured by the section with such a three - dimensional structure. However, due to the deformation from the three - dimensional structure and the loss of position information in the three - dimensional space caused by section preparation, a problem was found that the gene information could not be accurately associated with the pre - constructed three - dimensional structure.

[0007] To solve this problem using a machine learning model, we found that by aligning section image data and 3D image data, adjusting the section image data to match the 3D image data based on the alignment, and then adjusting the gene expression data to match the section image data, we can accurately associate gene expression data with the 3D structure.Therefore, the present invention relates to the following: [1] A method for identifying a biological structure, comprising: a position identification step of identifying the position in the 3D image data corresponding to a biological component from 3D image data of a biological sample and section image data of the biological sample; an image adjustment step of adjusting the section image data to match the 3D image data based on the section image data and the position in the 3D image data corresponding to the biological component in the section image data; a gene expression data adjustment step of adjusting gene expression data associated with the section image data in the same manner as the adjustment of the section image data by the image adjustment step; and a step of displaying the adjusted gene expression data adjusted by the gene expression data adjustment step in association with the 3D image data. [2] A biological structure identification device comprising an input unit, a storage unit, a processor, and a display unit, wherein the processor comprises the following functional blocks: a position identification unit that identifies the position in the three-dimensional image data corresponding to a component of the biological organism in the section image data, based on three-dimensional image data of a biological organism or biological sample stored in the storage unit and section image data of the biological organism or biological sample; an image adjustment unit that adjusts the section image data to match the identified position in the three-dimensional image data; a gene expression data adjustment unit that adjusts gene expression data associated with the section image data in the same manner as the adjustment of the section image data by the image adjustment unit; and a display control unit that displays the adjusted gene expression data adjusted by the gene expression data adjustment unit in association with the three-dimensional image data; and a biological structure identification device that displays the adjusted gene expression data in association with the three-dimensional image data in the display unit in response to input from the input unit.[3] The biological structure identification device according to item 2, wherein the storage unit receives three-dimensional image data of a living organism or biological sample and section image data of one or more sections of the living organism or biological sample, and stores a first learning model that identifies the position in the three-dimensional image data corresponding to the biological components in the section image data, and the position identification unit identifies the position in the three-dimensional image data corresponding to the biological components in the section image data by inputting the three-dimensional image data of the living organism or biological sample and section image data of the living organism or biological sample into the first learning model. [4] The biological structure identification device according to item 2 or 3, wherein the storage unit receives section image data and the position in the three-dimensional image data corresponding to the biological components in the section image data, and stores a second learning model that adjusts the section image data to match the three-dimensional image data, and the image adjustment unit adjusts the section image data to match the three-dimensional image data by inputting the section image data and the position in the three-dimensional image data corresponding to the biological components in the section image data into the second learning model. [5] The biological structure identification device according to item 2 or 3, wherein the display control unit displays gene expression data of one or more types of genes in association with the three-dimensional image data. [6] The biological structure identification device according to item 2 or 3, wherein the storage unit includes a third learning model that identifies biological components in the section image data when the section image data is input, the processor further includes a first component identification unit, the first component identification unit inputs the section image data to the third learning model to identify biological components in the section image data, and the position identification unit uses identified section image data instead of the section image data to identify the position in the three-dimensional image data corresponding to the biological components in the identified section image data. [7] The biological structure identification device according to item 6, wherein the display control unit further displays biological components in the identified section image data.[8] The biological structure identification device according to item 6, wherein the storage unit, upon input of three-dimensional image data of a living organism or biological sample, further stores a fourth learning model for identifying the components of a living organism in the three-dimensional image data, the processor further includes a second component identification unit, the second component identification unit inputs three-dimensional image data of a living organism or sample into the fourth learning model to identify the components of a living organism in the three-dimensional image data, and the position identification unit, instead of the three-dimensional image data, uses the identified three-dimensional image data to identify the position in the three-dimensional image data that corresponds to the components of a living organism in the section image data. [9] The biological structure identification device according to item 8, wherein the display control unit displays adjusted gene expression data of one or more types of genes in association with the components of a living organism in the identified three-dimensional image data.

[10] A computer program for identifying biological structures, comprising a storage unit, an input unit, a processor, and a display unit, wherein the computer executes the following commands: causing the processor to identify the position in the three-dimensional image data corresponding to the biological components in the section image data, based on three-dimensional image data of a biological sample input from the input unit and section image data of a section of the biological sample; causing the processor to adjust the section image data to match the three-dimensional image data based on the section image data and the position in the three-dimensional image data corresponding to the biological components in the section image data; causing the processor to adjust the gene expression data associated with the section image data in the same manner as the adjustment of the section image data by the image adjustment step; and the display unit to display the adjusted gene expression data adjusted by the gene expression data adjustment step in association with the three-dimensional image data.

[0008] The biological structure recognition device according to the present invention can automatically associate gene expression data with three-dimensional image data of a biological sample, using the sample's three-dimensional image data, section image data, and gene expression information.

[0009] Figure 1 shows the hardware configuration diagram of the biostructure identification device. Figure 2 shows the functional block diagram of the processor of the biostructure identification device. Figure 3 is the operation flowchart of the processing in the biostructure identification device. Figure 4 shows the functional block diagram of the processor of the biostructure identification device in a modified example. Figure 5 shows the functional block diagram of the processor of the biostructure identification device in a modified example. Figure 6 shows the functional block diagram of the processor of the biostructure identification device in a modified example.

[0010] The biological structure identification device, biological structure identification method, and biological structure identification computer program according to the present invention adjust section image data (hereinafter simply referred to as section image data) and gene expression information in biological data (hereinafter referred to as 3D image data of a biological organism or biological sample, or simply 3D image data) that represents the three-dimensional structure of a living organism or biological sample in a three-dimensional orthogonal coordinate system, and display the adjusted gene expression data in association with the 3D image data.

[0011] The biological data subject to biological structure recognition processing may be obtained by any method, but for example, it can be a three-dimensional representation of a part (e.g., internal organs or skin tissue, etc.) or the whole of a human or animal body obtained by CT or MRI. The biological data may be three-dimensional image data acquired in a living organism or three-dimensional images acquired in a biological sample. From the viewpoint of acquisition from a biological sample, it is preferable to acquire it by CT, particularly X-ray CT.

[0012] Section image data is image data obtained from sections prepared from biological samples, and is usually fixed and stained. Gene expression is examined for each spot in the section and is associated with the section image data. That is, by selecting a predetermined position in the section image data, it is possible to refer to the gene expression data at that position. Sections are deformed due to the section preparation and fixation processes. The position identification unit 11 of the biological structure identification device 1 according to the present invention can identify the position of components included in the section image data in the biological or biological 3D image data, even in deformed section image data. Section image data may differ in appearance from 3D image data because staining has been performed, but the position identification unit 11 recognizes this difference and identifies the position of the components.

[0013] Figure 1 is a hardware configuration diagram of a biological structure identification device in one embodiment. As shown in Figure 1, the biological structure identification device 1 has a communication interface 2, an input device 3, a display device 4, a storage unit 6, and a processor 7, which are connected via a bus 5. Each component may be located externally, and in particular, the input device 3 and the display device 4 may be located externally. Processing in the biological structure identification device 1 is performed collaboratively by the communication interface 2, the input device 3, the display device 4, the memory 5, the storage medium access device 6, the processor 7, etc., which are connected by a bus and are hardware resources.

[0014] The communication interface 2 has a communication interface and control circuit for connecting to a communication network conforming to a communication standard such as Ethernet (registered trademark) or the Internet. The communication interface 2 receives various information or data from other devices (not shown) connected via the communication network and stores it in the storage unit 6. The data received by the communication interface 2 may include at least one data selected from the group consisting of three-dimensional image data, section image data, and gene expression data associated with the section image data that are the targets of biological structure identification processing. The communication interface 2 may also output gene information associated with the three-dimensional image data, and data concerning components associated with the three-dimensional image data, obtained as a result of processing received from the processor 7, to other devices via the communication network 2.

[0015] The input device 3 includes, for example, a keyboard and a pointing device such as a mouse. The input device 3 generates operation signals in response to user operations, such as selecting a 3D image to be processed, instructing the start of processing, or displaying genetic information associated with 3D image data or data related to components associated with 3D image data on the display device 4, and outputs these operation signals to the processor 7.

[0016] The display device 4 has, for example, a liquid crystal display or an organic EL display. The display device 4 displays three-dimensional data, which can be displayed by superimposing or separately displaying display data received from the processor 7, such as three-dimensional image data, genetic information associated with the three-dimensional image data, and data relating to components associated with the three-dimensional image data. The genetic information associated with the three-dimensional image data includes positional information in the three-dimensional image data and genetic information associated with the positional information. The data relating to components associated with the three-dimensional image data includes positional information in the three-dimensional image data and information relating to the components associated with the positional information. By displaying them separately, only the genetic information or components can be displayed in three dimensions.

[0017] The input device 3 and the display device 4 may be an integrated device, such as a touch panel display.

[0018] The storage unit 6 includes memory devices such as RAM, ROM, and flash memory, fixed disk devices such as hard disk drives and SSDs, or portable storage devices such as flexible disks and optical disks. The storage unit 6 may store data input from the communication interface 2 and instructions input from the input device 3. In addition to the results of calculations performed by the processor 7, the storage unit 6 may store programs and databases used for various computer processes, and may also store the learning model program. Within the storage unit, the memory may store 3D image data, section image data and gene expression information that are the target of processing by the processor 7, as well as the results of processing, such as location identification, adjusted section image data, adjusted gene expression data, identified section image data, identified 3D image data, gene information associated with the 3D image data, and components associated with the 3D image data, for processing by the processor. In one embodiment, the storage unit 6 may store a learning model. The learning model may include a first learning model and a second learning model, and in a modified example, a third learning model may be included. In yet another modified example, a fourth learning model may be included. In yet another variation, the system may include a first learning model, a second learning model, a third learning model, and a fourth learning model. Depending on the processing in the processor 7, the stored learning models are read out. The position identification unit 11, which is a functional block in the processor 7, reads out the first learning model, the image adjustment unit 12 reads out the second learning model, and the first component identification unit 15 reads out the third learning model. The second component identification unit 16 reads out the fourth learning model. The computer program may be installed, for example, on a computer-readable recording medium such as a CD-ROM or DVD-ROM, or via the Internet. The computer program is installed in the storage unit 6 using a known setup program or the like.

[0019] The processor 7 is an example of a processing unit and, for example, has one or more CPUs and their peripheral circuits. Furthermore, the processor 7 may have arithmetic circuits for numerical calculations and arithmetic circuits for logical calculations. The processor 7 controls the entire biological structure recognition device 1. The processor 7 also performs biological structure recognition processing on the target 3D image data, section image data, and gene expression information. The processing result data after each process in the processor 7 is stored in the storage unit 6. Furthermore, the processing result data may be output to other devices via the communication interface 2.

[0020] Figure 2 is a functional block diagram of the processor 7. As shown in Figure 2, the processor 7 includes a location identification unit 11, an image adjustment unit 12, a gene expression data adjustment unit 13, and a display control unit 14. Each of these units of the processor 7 is, for example, a functional module realized by a computer program executed on the processor 7. Alternatively, each of these units of the processor 7 may be a dedicated arithmetic circuit provided on the processor 7. In a modified example, the processor 7 may further include a first component identification unit 15 and / or a second component identification unit 16 (Figures 4-6).

[0021] The position identification unit 11 identifies the position in the 3D image data corresponding to the components of the living organism in the section image data, based on the 3D image data of the living organism or biological sample and the section image data of the living organism or biological sample. This process may be performed by an algorithm or by machine learning. As an example of algorithmic processing, feature points are recognized in both the section image data and the 3D image data, and the optimal cross-section plane that minimizes the distance between these feature points is selected from among countless cross-section planes in the 3D image data, thereby determining the distance between the feature points in the section image data and the corresponding feature points in the 3D image data. More specifically, the correspondence between multiple feature points in the section image data and multiple corresponding feature points in the 3D image data is found by nearest neighbor search, the deformation parameters are found by minimizing the squared error between the corresponding points, and the above process is repeated until convergence is achieved. Here, the feature points are components of the living organism or biological sample, and the position of the component in the section image data is determined by the position identification. Algorithms such as SIFT, SURF, and AKAZE can also be used for the position identification. In modified versions, alignment can be performed so that the distance between feature points is directly reduced within the 3D image data. For example, this may be done using methods such as Interative Closest Point (ICP) or Coherent Point Drift (CPD). Feature points may be identified using conventional feature recognition algorithms, or by recognizing biological components and defining them as feature points.

[0022] When the process is performed using machine learning, the positioning unit 11 reads 3D image data, section image data, and a first learning model from the storage unit 6, inputs the 3D image data and section image data into the first learning model, and identifies the position in the 3D image data that corresponds to the biological components in the section image data. For example, the positioning unit 11 identifies the position in the 3D coordinate system of the 3D image data that corresponds to the identified biological components in the section image data and stores it in the storage unit 6. Component refers to a region that includes some or all of the feature parts contained in the section image data, and may, for example, be any tissue or cell contained in a biological sample. Section image data may be represented in either a 2D or 3D coordinate system, but is usually image data in a 2D coordinate system. The positioning unit 11 identifies the position in the 3D image data that corresponds to the biological components in the section image data, and since the section image is image data of a section having a predetermined thickness, the position of the identified component is located on the cross-section in the 3D image data, and it is possible to identify which cross-section of the 3D image data the section image data corresponds to. In a modified example, the positioning unit 11 may identify the cross-section in the three-dimensional image data corresponding to the section image data. The section image data includes deformation due to the sectioning process, and the positions of the components of the section image data on the three-dimensional image do not perfectly coincide with the cross-section in the three-dimensional image data, but the closest cross-section is identified. In a modified example, the positioning unit 11 may use identified section image data and / or identified three-dimensional image data with identified components as input data, respectively. Identification of components means that the types of tissues and cells in the image data are identified.

[0023] The first learning model is pre-trained to identify the position in the 3D image data corresponding to the biological components in the section image data, when inputted as 3D image data of a living organism or biological sample and section image data of one or more sections of the living organism. In a modified example, the first learning model is pre-trained to identify the position in the 3D image data corresponding to the biological components in the identified section image data, when inputted as identified section image data in which components have been identified on the section image data, instead of section image data (Figure 4). In yet another modified example, the model is pre-trained to identify the position in the identified 3D image data corresponding to the biological components in the section image data, when inputted as identified 3D image data in which components have been identified on the 3D image data, instead of 3D image data (Figure 5). In yet another modified example, the model is pre-trained to identify the position in the identified 3D image data corresponding to the biological components in the section image data, when inputted as identified section image data and identified 3D image data, instead of section image data and 3D image data (Figure 6).

[0024] The image adjustment unit 12 reads section image data and the positions in the 3D image data corresponding to the biological components in the section image data from the storage unit 6, and adjusts the section image data to match the 3D image data. Adjustment refers to deformation processing such as enlargement, reduction, rotation, and translation. This processing may be performed by an algorithm or by machine learning. As an example of algorithmic processing, after identifying the position where the distance between feature points is minimized by alignment, the feature points in the section image data are adjusted to match the feature points in the 3D image data. The position of the section image data may be adjusted at the optimal cross-section in the 3D image data selected during alignment, or the position may be adjusted without selecting a cross-section. Homography matrices and the like can be used as algorithms for these adjustments. The position adjustment parameters are stored and used later for adjusting gene expression data. Through image adjustment processing, deformations caused by the processing of the section are adjusted so that it can match the 3D structure of the original biological organism or biological sample. Section image data adjusted to match 3D image data is called adjusted section image data and includes 3D coordinate data in the 3D image data. The adjusted section image data may also store the details of the adjustments made from the section image data, i.e., the details of transformation processes such as enlargement, reduction, rotation, and translation. In a modified example, the image adjustment unit 12 may use identified section image data in which the constituent elements have been identified.

[0025] When image adjustment is performed by machine learning processing, the section image data, the position in the 3D image data corresponding to the biological components in the section image data, and the second learning model are read from the storage unit 6. When the section image data and the position in the 3D image data corresponding to the biological components in the section image data are input to the second learning model, the section image data is adjusted to match the 3D image data.

[0026] The second learning model is pre-trained to adjust the section image data to match the 3D image data when it is input the section image data and the corresponding positions in the 3D image data for the biological components in the section image data. In a modified example, the second learning model is pre-trained to adjust the identified section image data to match the 3D image data when it is input, instead of section image data, for identified section image data in which the components have been identified.

[0027] The gene expression data adjustment unit 13 reads section image data, gene expression data associated with the section image data, and adjusted section image data from the storage unit 6. It adjusts the gene expression data associated with the section image data in the same way as the adjustment of the section image data to obtain adjusted gene expression data. The section image data and the adjusted section image data are compared, and the adjustments made on the 3D image are also applied to the gene expression data. As a result, the adjusted gene expression data includes 3D coordinate data in the 3D image data.

[0028] The display control unit 14 can read the adjusted gene expression data from the storage unit 6 and display it in association with the three-dimensional image data. Displaying in association with the three-dimensional image data includes reconstructing the adjusted gene expression data into three-dimensional coordinate data in the three-dimensional image data and displaying it. Since the gene expression data includes multiple types of gene expression, it is possible to select and display the expression of one or more types of genes. As an example of multiple types of genes, the expression of a specific related gene group can be displayed as a heat map. In this case, the gene expression of the specific gene group can be statistically processed, for example, averaged, and displayed on the heat map.

[0029] In a modified example, the display control unit 14 may read the adjusted gene expression data and 3D image data from the storage unit 6 and display the adjusted gene expression data superimposed on the 3D image data. Furthermore, the display control unit 14 may read the adjusted section image data from the storage unit 6 and display the adjusted section image data superimposed on the adjusted gene expression data.

[0030] In a further modification, the display control unit 14 may read the adjusted gene expression data and the identified 3D image data and / or identified section image data from the storage unit 6, and display the adjusted gene expression data overlaid on specific components of the identified 3D image data and / or specific components of the identified section image data.

[0031] The display control unit 14 may set the viewpoint direction for the subject represented in the biological data according to the user's operation input from the input device 3. The display control unit 14 may then display the adjusted gene expression data on the display device 4 as viewed from the set viewpoint direction. Alternatively, the display control unit 14 may set a cross-section of the subject represented in the biological data according to the user's operation input from the input device 3. The display control unit 14 may then display the adjusted gene expression data corresponding to the set cross-section of the biological data on the display device 4. In this case, the adjusted gene expression data can be switched or superimposed on each other. These processes can be implemented using 3D modeling techniques. In this way, the display control unit 14 makes it easier for the user to understand the structure and physiological activity of the biological body of the subject represented in the biological data by displaying gene expression data and / or components visible from any viewpoint on the display device 4. Furthermore, by specifying a particular location in the 3D image, gene expression at that location can be comprehensively displayed, and by specifying multiple specific locations, comparisons and differences can be displayed.

[0032] In this modified version, the processor 7 further includes a first component identification unit 15 (Figure 4). The first component identification unit reads section image data and a third learning model from the storage unit 6, and when it inputs the section image data to the third learning model, it identifies the components in the section image data. Section image data containing information about the identified components is called identified section image data. In this modified version, the positioning unit 11 of the processor 7 reads the first learning model, 3D image data, and identified section image data, and when it inputs the 3D image data and identified section image data to the first learning model, it identifies the position of the components. In this modified version, the first learning model is pre-trained to identify the position in the 3D image data corresponding to the biological components in the section image data when it receives identified section image data and 3D image data in which components have been identified on the section image data. By pre-identifying the components in the section image data, the positioning accuracy of the positioning unit 11 is improved.

[0033] The third learning model is pre-trained to identify constituent elements when given section image data as input. Component identification is performed using so-called semantic segmentation, where the constituent elements represented by pixels are identified. When using 3D image data of skin tissue as section image data, the identified constituent elements include, for example, tissues such as the epidermis, dermis, subcutaneous fat, hair follicles, sebaceous glands, arrector pili muscles, sweat glands, blood vessels, lymphatic vessels, and nerves. The types of cells that make up these tissues may also be identified. For example, in the case of epidermal tissue, it may be classified into the basal layer, spinous layer, granular layer, and stratum corneum, and in addition to keratinocytes that make up each layer, melanocytes, Langerhans cells, and Merkel cells may also be identified. In the case of dermal tissue, fibroblasts, mast cells, histiocytes, plasma cells, etc., may each be identified, and intracellular structures such as the intercellular matrix may also be identified.

[0034] In this modified version, the processor 7 further includes a second component identification unit 16 (Figure 5). The second component identification unit 16 reads 3D image data and a fourth learning model from the storage unit 6, and when the 3D image data is input to the fourth learning model, it identifies the components in the 3D image data. The 3D image data containing information about the identified components is called identified 3D image data. In this modified version, the positioning unit 11 of the processor 7 reads the first learning model, the identified 3D image data, and the intersection image data, and when the identified 3D image data and the adjusted intersection image data are input to the first learning model, it identifies the position of the components. The first learning model used in this modified version is pre-trained to identify the position in the 3D image data corresponding to the biological components in the intersection data when identified 3D image data in which components have been identified on the 3D image data and intersection image data are input. By pre-identifying the components in the 3D image data, the positioning accuracy of the positioning unit 11 is improved.

[0035] The fourth learning model is pre-trained to identify constituent elements when given 3D image data as input. Component identification is performed by so-called semantic segmentation, where elements represented by voxels are identified. As an example of 3D image data of a living organism or biological sample, when using 3D image data of skin tissue, the components of the living organism or biological sample to be identified include, for example, tissues such as the epidermis, dermis, subcutaneous fat, hair follicles, sebaceous glands, arrector pili muscles, sweat glands, blood vessels, lymphatic vessels, and nerves. The types of cells that make up these tissues may also be identified. For example, in the case of epidermal tissue, it can be classified into the basal layer, spinous layer, granular layer, and stratum corneum, and in addition to keratinocytes that make up each layer, melanocytes, Langerhans cells, and Merkel cells may also be identified. In the case of dermal tissue, fibroblasts, mast cells, histiocytes, plasma cells, etc., may each be identified.

[0036] In a modified example, the processor 7 further includes a component identification unit 15 and a component identification unit 16 (Figure 6). In this modified example, the position identification unit 11 of the processor 7 reads a first learning model, identified 3D image data, and identified section image data, inputs the identified 3D image data and identified section image data into the first learning model, and identifies the position of the component. The first learning model used in this modified example is pre-trained to identify the position in the identified 3D image data corresponding to the biological component in the identified section image data when it is input identified 3D image data in which the component has been identified on the 3D image data and identified section image data in which the component has been identified on the section image data. By pre-identifying the component in the 3D image data and the component in the section image data, the position identification accuracy of the position identification unit 11 is improved.

[0037] The learning model can be, for example, a convolutional neural network (CNN) having an input layer, an output layer, and a plurality of hidden layers connected between the input and output layers. The input layer and at least one of the plurality of hidden layers can be a 3D convolutional layer. A 3D convolutional layer is a convolutional layer that takes input data as input to the layer and outputs output data (hereinafter, for the sake of explanation, referred to as a 3D convolutional operation). Furthermore, in the classifier, it is preferable that the input layer and multiple or all of the plurality of hidden layers are 3D convolutional layers.

[0038] Specifically, the learning model can use a CNN, for example, a CNN in which one, several, or all of the convolutional layers of a Fully Convolutional Network (FCN), SegNet, DeepLab, or RefineNet are 3D convolutional layers. Furthermore, the learning model may have one or more deconvolutional layers that perform interpolation processing on the output side of each convolutional layer. These deconvolutional layers may also perform interpolation processing in 3D.

[0039] Figure 3 is an operation flowchart of the biological structure recognition process. The processor 7 should execute the biological structure recognition process for each biological data to be identified according to the operation flowchart below.

[0040] The processor 7 reads the 3D image data, section image data, and gene expression data associated with the section image data from the storage unit 6 (step S101). Alternatively, the processor 7 may obtain the 3D image data, section image data, and gene expression data associated with the section image data from another device via the communication interface 2. The position identification unit 11 reads the 3D image data and section image data and identifies the position in the 3D image data corresponding to the components of the living organism (step S102). Position identification may be performed by algorithmic processing, or by inputting the read 3D image data and section image data into the first learning model. In a modified example, before processing in the position identification unit 11, the process may include a step of identifying components in the section image data using a first component identification unit, and / or a step of identifying components in the 3D image data using a second component identification unit. The positions in the 3D image data corresponding to the components of the living organism in the identified section image data are written to the storage unit 6, particularly the memory, and stored (step S103). The image adjustment unit 12 reads the section image data and the positions of the constituent elements from the storage unit 6 and adjusts the section image data to match the 3D image data (step S104). Image adjustment may be performed by algorithmic processing, or by inputting the section image data and the positions of the constituent elements into a second learning model. The image adjustment unit 12 writes the section image data adjusted to match the 3D image data into memory (step S105). The gene expression data adjustment unit 13 reads the adjusted section image data and gene expression data and inputs them into a third learning model to adjust the gene expression data associated with the section image data in the same way as the section image data adjustment (step S106). The gene expression data adjustment unit 13 writes the adjusted gene expression data into memory (step S107), and the display control unit 14 reads the adjusted gene expression data and displays it on the display unit (S108).

[0041] Furthermore, if the biological structure identification result data is displayed by another device, or if display of the identification result data is not necessary, the display control unit 14 may be omitted in the processor. Alternatively, a processor in a separate computer from the biological structure identification device 1 may perform the processing of the display control unit 14 according to the above embodiment. Similarly, a processor in a separate computer from the biological structure identification device 1 may perform the processing of the first to fourth learning models.

[0042] Furthermore, a computer program that enables a computer to implement the functions of each part of the processor 7 of the biological structure identification device 1 according to the above embodiment or modification may be provided in the form of a recording medium that can be read by a computer. The recording medium that can be read by a computer can be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0043] 1. Biological structure identification device 2. Communication interface 3. Input device 4. Display device 5. Bus 6. Storage unit 7. Processor 11. Location identification unit 12. Image adjustment unit 13. Gene expression data adjustment unit 14. Display control unit 15. First component identification unit 16. Second component identification unit

Claims

1. A method for identifying a biological structure, comprising: a position identification step of identifying the position in the three-dimensional image data corresponding to a biological component from three-dimensional image data of a biological sample and section image data of the biological sample; an image adjustment step of adjusting the section image data to match the three-dimensional image data based on the section image data and the position in the three-dimensional image data corresponding to a biological component in the section image data; a gene expression data adjustment step of adjusting gene expression data associated with the section image data in the same manner as the adjustment of the section image data by the image adjustment step; and a step of displaying the adjusted gene expression data adjusted by the gene expression data adjustment step in association with the three-dimensional image data.

2. A biological structure identification device comprising an input unit, a storage unit, a processor, and a display unit, wherein the processor comprises the following functional blocks: a position identification unit that identifies the position in the three-dimensional image data corresponding to a component of the biological organism in the section image data, based on three-dimensional image data of a biological organism or biological sample stored in the storage unit and section image data of the biological organism or biological sample; an image adjustment unit that adjusts the section image data to match the identified position in the three-dimensional image data; a gene expression data adjustment unit that adjusts gene expression data associated with the section image data in the same manner as the adjustment of the section image data by the image adjustment unit; and a display control unit that displays the adjusted gene expression data adjusted by the gene expression data adjustment unit in association with the three-dimensional image data; and a biological structure identification device that displays the adjusted gene expression data in association with the three-dimensional image data in the display unit in response to input from the input unit.

3. The biological structure identification device according to claim 2, wherein the storage unit, upon input of three-dimensional image data of a living organism or biological sample and section image data of one or more sections of the living organism or biological sample, stores a first learning model that identifies the position in the three-dimensional image data corresponding to the components of the living organism in the section image data, and the position identification unit identifies the position in the three-dimensional image data corresponding to the components of the living organism in the section image data by inputting the three-dimensional image data of the living organism or biological sample and section image data of the living organism or biological sample into the first learning model.

4. The biological structure identification device according to claim 2 or 3, wherein the storage unit stores a second learning model for adjusting the intersection image data to match the three-dimensional image data when it receives intersection image data and the positions in the three-dimensional image data corresponding to the biological components in the intersection image data, and the image adjustment unit adjusts the intersection image data to match the three-dimensional image data by inputting the intersection image data and the positions in the three-dimensional image data corresponding to the biological components in the intersection image data to the second learning model.

5. The biological structure identification device according to claim 2 or 3, wherein the display control unit displays gene expression data of one or more types of genes in association with the three-dimensional image data.

6. The biological structure identification device according to claim 2 or 3, wherein the storage unit includes a third learning model that identifies biological components in the section image data when the section image data is input, the processor further includes a first component identification unit, the first component identification unit inputs the section image data to the third learning model to identify biological components in the section image data, and the position identification unit uses the identified section image data instead of the section image data to identify the position in the three-dimensional image data corresponding to the biological components in the identified section image data.

7. The biological structure identification device according to claim 6, wherein the display control unit further displays biological components in the identified section image data.

8. The biological structure identification device according to claim 6, wherein the storage unit, upon input of three-dimensional image data of a living organism or biological sample, further stores a fourth learning model for identifying the components of a living organism in the three-dimensional image data; the processor further includes a second component identification unit, the second component identification unit inputs three-dimensional image data of a living organism or sample into the fourth learning model to identify the components of a living organism in the three-dimensional image data; and the position identification unit, instead of the three-dimensional image data, uses the identified three-dimensional image data to identify the position in the three-dimensional image data corresponding to the components of a living organism in the section image data.

9. The biological structure identification device according to claim 8, wherein the display control unit displays adjusted gene expression data of one or more types of genes in association with biological components in the identified three-dimensional image data.

10. A computer program for identifying biological structures, comprising a storage unit, an input unit, a processor, and a display unit, wherein the computer executes the following commands: causing the processor to identify the position in the three-dimensional image data corresponding to the biological components in the section image data, based on three-dimensional image data of a biological sample input from the input unit and section image data of a section of the biological sample; causing the processor to adjust the section image data to match the three-dimensional image data based on the section image data and the position in the three-dimensional image data corresponding to the biological components in the section image data; causing the processor to adjust the gene expression data associated with the section image data in the same manner as the adjustment of the section image data by the image adjustment step; and the display unit to display the adjusted gene expression data adjusted by the gene expression data adjustment step in association with the three-dimensional image data.

Images