Laser beam attainment degree learning inference system, laser beam attainment degree learning system, laser beam attainment degree inference system, laser beam attainment degree learning method, laser beam attainment degree inference method and computer program

Abstract

To quickly infer by a computer whose specification is not high, a level of intensity of attainment of a laser beam radiated to a tissue of a living body such as a human being, to each portion in the tissue.SOLUTION: A laser beam attainment degree learning inference system is configured to: collect, as a data set 50, a three-dimensional tissue model 51 of a specific tissue of a living body, and a light fluence distribution 52 which is an attainment degree of a laser beam in each position on the three-dimensional tissue model 51 when radiating the laser beam to the three-dimensional tissue model 51; perform machine learning using the three-dimensional tissue model 51 and the light fluence distribution 52 as input data and correct answer data, respectively, for generating a learned neural network 67; input a three-dimensional tissue model 54 of the specific tissue of a subject, to the learned neural network 67, for calculating a light fluence distribution 55 on each position of the three-dimensional tissue model 54 when radiating the laser beam to the three-dimensional tissue model 54.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for estimating the degree to which irradiated laser light reaches each position in tissue of a living body. [Background technology]

[0002] In recent years, PDT (Photodynamic Therapy) has been attracting attention as a treatment method for removing tumors. PDT is a treatment method written as "photodynamic therapy" in Japanese, and is a tumor-selective, minimally invasive treatment method that utilizes the accumulation of a photosensitizer (PS), i.e., a photosensitive drug, in tumors and its photochemical reaction to light of a specific wavelength. When PS is administered into the body, it accumulates inside tumor cells. When it is irradiated with a laser light of a specific wavelength, a photochemical reaction occurs in PS, generating singlet oxygen. The singlet oxygen then kills the tumor cells.

[0003] These characteristics enable PDT to induce necrosis of tumor cells while minimizing the effects on surrounding healthy cells.

[0004] By optimizing treatment conditions such as the irradiation position or angle of the laser light in PDT, it is expected that the risk of damage to tissues such as organs irradiated with the laser light can be reduced.

[0005] In order to achieve optimization, it is necessary to understand the light distribution in tissue, i.e., how the irradiated laser light reaches inside the tissue. Since it is difficult to grasp the light distribution in tissue experimentally, it is effective to use simulation.

[0006] For example, the application software described in Non-Patent Document 1 makes it possible to simulate the intensity of laser light irradiated onto tissue such as the human brain, which will reach each part of the tissue. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "Monte Carlo Simulation of Photon Migration in 3D Turbid Media Accelerated by Graphics Processing Units", Qianqian Fang & David Boas, 2009 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the existing simulation method described in the above-mentioned Non-Patent Document 1 requires high-spec resources such as a large-scale parallel computer, so the environment in which it can be used is limited. Although it is technically possible to realize a similar method using a computer with low specifications, the calculation time is long. Therefore, it is difficult to use it in situations that require real-time performance, such as in the field of surgery.

[0009] In view of these problems, the present invention aims to infer, faster than ever before, using a computer with low specifications, the intensity at which laser light irradiated onto the tissue of a living body such as a human being will reach each part within the tissue. [Means for solving the problem]

[0010] A laser light reachability learning and inference system according to one embodiment of the present invention includes an acquisition means for acquiring, as a dataset, a learning three-dimensional model which is a three-dimensional model of a specific tissue of a living body, and a learning reachability which is the reachability of the laser light at each position of the learning three-dimensional model when laser light is irradiated onto each of the learning three-dimensional models; a learning means for generating a trained model by performing machine learning using the learning three-dimensional model and the learning reachability of the dataset as input data and correct answer data, respectively; and an inference means for inputting a subject three-dimensional model which is a three-dimensional model of the specific tissue of a subject into the trained model, thereby calculating the inferred reachability of the laser light at each position of the test three-dimensional model when the test three-dimensional model is irradiated with the laser light. Effect of the Invention

[0011] According to the present invention, it is possible to infer, faster than ever before, with a computer having low specifications, what intensity of laser light irradiated onto the tissue of a living body such as a human being will reach each part within the tissue. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration of a learning inference system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning inference device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning inference device. [Figure 4] FIG. 1 is a diagram illustrating an example of a three-dimensional tissue model. [Diagram 5] FIG. 13 is a diagram illustrating an example of light fluence. [Figure 6] FIG. 13 is a diagram illustrating an example of a light fluence distribution. [Figure 7] FIG. 1 is a diagram illustrating an example of a neural network. [Figure 8] FIG. 13 is a diagram showing an example of a three-dimensional model with double zero padding. [Figure 9]FIG. 13 is a diagram showing an example of how a three-dimensional model is enlarged 2×2×2 times. [Figure 10] 11 is a flowchart illustrating an example of an overall processing flow by a learning inference program. [Figure 11] FIG. 1 is a diagram illustrating an example of a mini-batch. [Figure 12] FIG. 13 is a diagram showing examples of attributes of four regions of a voxel group. [Figure 13] FIG. 13 is a diagram showing examples of attributes of five regions of a voxel group. [Figure 14] FIG. 13 is a diagram illustrating an example of splitting a laser beam. [Figure 15] FIG. 13 is a diagram illustrating an example of splitting a laser beam. [Figure 16] FIG. 13 is a diagram illustrating an example of splitting a laser beam. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [1. Overall system configuration] Fig. 1 is a diagram illustrating an example of an overall configuration of a learning inference system 1. Fig. 2 is a diagram illustrating an example of a hardware configuration of a learning inference device 2. Fig. 3 is a diagram illustrating an example of a functional configuration of the learning inference device 2.

[0014] The learning inference system 1 shown in Fig. 1 is a system that infers how laser light emitted from a light source reaches inside living tissue by using AI (Artificial Intelligence) technology, and is composed of a learning inference device 2 and a three-dimensional measurement device 3. Below, an example of inferring how laser light reaches each position in the human brain will be described.

[0015] The learning and inference device 2 generates a trained model by machine learning and infers how the irradiated laser light reaches the brain based on the trained model. Hereinafter, an example will be described in which a personal computer is used as the learning and inference device 2.

[0016] As shown in FIG. 2, the learning inference device 2 is composed of a main processor 20, a RAM (Random Access Memory) 21, a ROM (Read Only Memory) 22, an auxiliary storage device 23, a network interface 24, a serial interface 25, a wireless communication device 26, a display 27, a keyboard 28, and a pointing device 29.

[0017] In addition to an operating system, computer programs such as a learning and inference program 40 are installed in the ROM 22 or the auxiliary storage device 23. As the auxiliary storage device 23, a hard disk or an SSD (Solid State Drive) is used.

[0018] 3, functions such as a three-dimensional model generating unit 401, a light fluence calculating unit 402, a data set registering unit 403, a data set storing unit 404, a machine learning unit 405, a learned model storing unit 406, a light fluence inference unit 407, and an inference result output unit 408 are realized. The functions of each unit will be described later in order.

[0019] The RAM 21 is the main memory of the learning and inference device 2. Computer programs stored in the ROM 22 or the auxiliary storage device 23 are loaded into the RAM 21 as appropriate.

[0020] The main processor 20 executes a computer program loaded into the RAM 21. As the main processor 20, a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit) is used.

[0021] The network interface 24 communicates with other devices using protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol). As the network interface 24, a NIC (Network Interface Card) or a communication device for Wi-Fi is used.

[0022] The serial interface 25 communicates with peripheral devices by a serial communication method, and is implemented as a board conforming to standards such as USB (Universal Serial Bus).

[0023] The wireless communication device 26 communicates with peripheral devices by short-range wireless communication. A board conforming to standards such as Bluetooth is used as the wireless communication device 26.

[0024] The serial interface 25 and the wireless communication device 26 are particularly used for communicating with the three-dimensional measuring device 3. The serial interface 25 and the wireless communication device 26 are used depending on the communication standard of the three-dimensional measuring device 3.

[0025] The display 27 displays a screen for inputting commands or data, a screen showing the results of calculations performed by the main processor 20, and the like.

[0026] The keyboard 28 and the pointing device 29 are input devices for an operator to input commands, data, and the like.

[0027] The three-dimensional measuring device 3 measures the three-dimensional shape of a living body. In this embodiment, in particular, the three-dimensional shape of the whole or a part of the brain is measured. Then, data indicating the measured three-dimensional shape is transmitted to the learning and inference device 2.

[0028] Below, the functions of each part shown in Figure 3 will be explained, broadly divided into the dataset preparation phase, the machine learning phase, and the inference phase.

[0029] 2. Preparation of the Data Set Fig. 4 is a diagram showing an example of a three-dimensional tissue model 51. Fig. 5 is a diagram showing an example of light fluence. Fig. 6 is a diagram showing an example of light fluence distribution 52.

[0030] The three-dimensional model generating unit 401, the light fluence calculating unit 402, and the data set registering unit 403 execute the process of preparing a data set for machine learning as follows.

[0031] The three-dimensional model generating unit 401 generates a large number of three-dimensional tissue models 51 that imitate a part of the surface of the human brain as shown in FIG. 4. For example, 40,000 to 100,000 models are generated. X ×B Y ×B Z voxels and is divided into two layers: a tissue layer 51f and an air layer 51g. The tissue layer 51f represents the brain tissue. The air layer 51g represents the outside of the brain, i.e., the air.

[0032] The boundary surface between the two layers is generally parallel to the XY plane and has small projections and recesses. The three-dimensional tissue model 51 is generated, for example, by the following method.

[0033] The three-dimensional model generating unit 401 generates a three-dimensional model for each voxel on the XY plane (i.e., B X ×B Y voxels), an integer value D between D1 and D2 inclusive XY Then, the Z axis is parallel to the (X,Y,0) to (X,Y,(D0+D XY Each voxel up to (X, Y, (D0+D XY +1)) to (X,Y,B Z ) are defined as air layers 51g. Note that D0, D1, and D2 are all constants.

[0034] For example, for a voxel with X=12 and Y=7, the integer value D XY "1" is given as B Zis "31" and "15" is defined as D0, then the voxels (12,7,0) through (12,7,16) are defined as voxels in the air layer 51g, and the voxels (12,7,17) through (12,7,31) are defined as voxels in the tissue layer 51f.

[0035] Below, B X =B Y =B Z The processing of each unit will be described by taking as an example a case where D1=31, the voxel is a cube with each side of 0.1 mm, D1=0, D2=1, and D0=15. Therefore, the three-dimensional model generating unit 401 generates a three-dimensional tissue model 51 having a boundary surface between a tissue layer 51f having an unevenness of 0.1 mm and an air layer 51g at a depth of approximately 1.5 mm. In this embodiment, a case where 100,000 different three-dimensional tissue models 51 are generated will be described as an example.

[0036] The light fluence calculation unit 402 randomly extracts 80,000 from the 100,000 generated by the three-dimensional model generation unit 401. For each of the extracted 80,000 three-dimensional tissue models 51, the light fluence of each voxel of each three-dimensional tissue model 51 is calculated when the laser light 43 used in PDT (Photodynamic Therapy) is irradiated under predetermined conditions. The light fluence is the intensity of the laser light 43 in a voxel, and is the amount of laser light 43 that reaches a voxel per unit time. It can also be said to be equivalent to illuminance.

[0037] The optical fluence is calculated by a known method such as the Monte Carlo method. For example, it may be calculated using MCX (Monte Carlo eXtreme). MCX is software developed by Qianqian Fang, and its algorithm is described in the above-mentioned Non-Patent Document 1. In addition, technical information such as its source code is disclosed at http: / / mcx.space / .

[0038] In this embodiment, the predetermined conditions are set such that the incident angle of the laser light 43 is 90 degrees with respect to the surface of the tissue layer 51f, i.e., the XY plane, that the laser light 43 is incident from the center (16,16,0) of the surface of the tissue layer 51f, and that the radius of the laser light 43 is 0.1 mm. Furthermore, the scattering coefficient, absorption coefficient, and scattering anisotropy when the laser light 43 is irradiated to human brain tissue are obtained in advance, and these are input into the learning and inference device 2 by the operator.

[0039] When the laser light 43 is incident on the tissue layer 51f of the three-dimensional tissue model 51 under such conditions, the light fluence appears as shown in Fig. 5, for example. Hereinafter, the 32 x 32 x 32 light fluences as shown in Fig. 6 calculated by the light fluence calculation unit 402 will be referred to as "light fluence distribution 52."

[0040] The dataset registration unit 403 combines the i-th three-dimensional tissue model 51 and the light fluence distribution 52 as one set of dataset 50, and stores the combination in the dataset storage unit 404. As a result, 80,000 sets of datasets 50 are stored in the dataset storage unit 404.

[0041] [3. Machine Learning] Fig. 7 is a diagram showing an example of a neural network 60. Fig. 8 is a diagram showing an example of a three-dimensional model that has been double-zero padded. Fig. 9 is a diagram showing an example of a three-dimensional model being enlarged 2x2x2 times.

[0042] The machine learning unit 405 performs machine learning based on the dataset 50 stored in the dataset storage unit 404, and generates a trained model for inferring how the irradiated laser light 43 reaches the brain. Hereinafter, the machine learning method will be described with reference to Figs. 7 to 9 and the like.

[0043] In the machine learning unit 405, a neural network 60 as shown in FIG.

[0044] The neural network 60 is a U-Net type deep neural network, and is composed of an encoder 61 and a decoder 62. U-Net is one of the fully convolution networks (FCNs) and is a semantic segmentation model developed by Olaf et al. for biomedical applications. Details of U-Net are described at https: / / arxiv.org / abs / 1505.04597.

[0045] The machine learning phase mainly includes a training step and a validation step. In the training step, the machine learning unit 405 trains the neural network 60 as follows.

[0046] The encoder 61 is composed of convolutional layers 611-618 and pooling layers 61A-61C, etc. The decoder 62 is composed of convolutional layers 621-627 and upward convolutional layers 62A-62C, etc. Each process, such as convolution, pooling, upward convolution, feature map (channel) expansion, and feature map (channel) compression, which is appropriately performed in these layers, is performed by a known method.

[0047] Below, the processing at each layer will be explained using an example in which the neural network 60 is trained using the first data set 50.

[0048] The convolutional layer 611 also functions as an input layer, to which the three-dimensional tissue model 51 of the first data set 50 is input.

[0049] Then, the convolution layer 611 performs double zero padding on the input three-dimensional tissue model 51. Double zero padding is a process of adding two layers of "0" voxels to each of the six faces of the three-dimensional tissue model and adding 2×2×2 "0" voxels to the outside of the eight vertices as shown in Fig. 8. This process changes the size of the three-dimensional tissue model 51 from 32×32×32 to 36×36×36.

[0050] The convolutional layer 611 generates 16 feature maps 5A by convolving the zero-padded three-dimensional tissue model 51 with 16 different filters and activating it with an activation function. The 16 feature maps 5A are then sent to the convolutional layer 612. For example, ReLU is used as the activation function. Below, an example will be described in which ReLU is used as the activation function in the neural network 60.

[0051] All of the 16 filters are 3×3×3 filters. Moreover, convolution is performed with one stride. Therefore, the size of each of the 16 feature maps 5A is 34×34×34. In the convolution layers 612 to 614, convolution is also performed with one stride by 3×3×3 filters.

[0052] The convolutional layer 612 performs double zero padding on each of the 16 feature maps 5A. As a result, the size of each feature map 5A becomes 38×38×38 instead of 34×34×34. The convolutional layer 612 convolves each of the 16 zero-padded feature maps 5A with two different filters and activates them with an activation function to generate 32 feature maps 5B. The convolutional layer 612 then sends the 32 feature maps 5B to the pooling layer 61A and the convolutional layer 625.

[0053] The pooling layer 61A generates 32 feature maps 5C by performing 3 × 3 × 3 max pooling with one stride on each of the 32 feature maps 5B, and sends them to the convolution layer 613.

[0054] Like the convolutional layer 612, the convolutional layers 613 and 614 perform double zero padding on the feature maps sent from the previous layer, convolve the feature maps with one stride using two different filters, and activate the feature maps using an activation function to generate new feature maps. The generated feature maps are then sent to the next layer. As a result, 64 feature maps 5D are generated in the convolutional layer 613 and sent to the convolutional layer 614. Also, 128 feature maps 5E are generated in the convolutional layer 614 and sent to the pooling layer 61B and the convolutional layer 623.

[0055] The pooling layer 61B generates 128 feature maps 5F by performing 3 × 3 × 3 max pooling with 2 strides on each of the 128 feature maps 5E, and sends them to the convolution layer 615.

[0056] In both convolutional layers 615 and 616, a new feature map is generated by performing double zero padding on the feature map sent from the previous layer, convolving the feature map with two different 3×3×3 filters with two strides, and activating the feature map with an activation function. The generated feature map is then sent to the next layer. As a result, 256 feature maps 5G are generated in the convolutional layer 615 and sent to the convolutional layer 616. Also, 512 feature maps 5H are generated in the convolutional layer 616 and sent to the pooling layer 61C and the convolutional layer 621.

[0057] The pooling layer 61C, like the pooling layer 61B, performs 3×3×3 max pooling with 2 strides on each of the feature maps sent from the previous layer, i.e., the 512 feature maps 5H, and sends them to the next layer, i.e., the convolutional layer 617. As a result, 512 feature maps 5J are generated and sent to the convolutional layer 617.

[0058] The convolutional layer 617, like the convolutional layers 615 and 616, performs double zero padding on the feature map sent from the previous layer, convolves it with two different 3x3x3 filters with two strides, and activates it with an activation function to generate a new feature map. The generated feature map is then sent to the next layer. As a result, 1024 feature maps 5K are generated and sent to the convolutional layer 618.

[0059] The convolutional layer 618 performs double zero padding on the 1024 feature maps 5K, convolves them with a 3×3×3 filter with 2 strides, and activates them with an activation function to generate 1024 feature maps 5L. The feature maps 5L are then sent to the ascending convolutional layer 62A.

[0060] The ascending convolution layer 62A generates 1024 feature maps 5M by performing upward convolution on the 1024 feature maps 5L as follows, and sends the 1024 feature maps 5M to the convolution layer 621. Here, an example of a method of upward convolution will be described.

[0061] The ascending convolution layer 62A enlarges one feature map 5L by 2×2×2 times. That is, as shown in FIG. 9, the eight voxels of the feature map 5L are copied 2×2×2 times each. The enlarged feature map 5L is double zero-padded. Then, it is convolved with a 3×3×3 filter with one stride. As a result, one feature map 5M is generated. The remaining 1023 feature maps 5L are processed in the same way, and 1023 feature maps 5M are generated.

[0062] The convolutional layer 621 generates a feature map 5N by combining the k-th feature map 5M and the roundup(k / 2)-th feature map 5H among the 1024 feature maps 5M sent from the ascending convolutional layer 62A and the 512 feature maps 5H sent from the convolutional layer 616. Note that roundup(k / 2) is a value obtained by dividing k by 2 and rounding up the decimal point. Therefore, when k=1, 2, 3, 4, 5, ..., roundup(k / 2)=1, 1, 2, 2, 3, ....

[0063] Then, the convolution layer 621 performs 6-fold zero padding on the (2j-1)th feature map 5N and the 2jth feature map 5N, performs convolution with one stride using a 3×3×3 filter, and activates them using an activation function. This generates 512 feature maps 5P. These feature maps 5P are sent to the convolution layer 622.

[0064] In the convolutional layer 622, similar to the convolutional layer 621, the (2j-1)th feature map 5P and the 2jth feature map 5P are subjected to six-fold zero padding, convolved with a 3x3x3 filter at one stride, and activated by an activation function. This generates 256 feature maps 5Q. These feature maps 5Q are sent to the ascending convolutional layer 62B.

[0065] The ascending convolutional layer 62B enlarges each of the 256 feature maps 5Q by 2×2×2 and performs double zero padding. Then, it convolves each of them with a 5×5×5 filter with 2 strides to generate 256 feature maps 5R. These feature maps 5R are then sent to the convolutional layer 623.

[0066] The convolutional layer 623 generates a feature map 5S by combining the k-th feature map 5R and the roundup(k / 2)-th feature map 5R among the 256 feature maps 5R sent from the ascending convolutional layer 62B and the 128 feature maps 5E sent from the convolutional layer 614. Then, the (2j-1)-th feature map 5S and the 2j-th feature map 5S are double-padding with zeros, convolved with one stride by a 3×3×3 filter, and activated by an activation function. As a result, 128 feature maps 5T are generated. These feature maps 5T are sent to the convolutional layer 624.

[0067] In the convolutional layer 624, similar to the convolutional layer 623, the (2j-1)th feature map 5T and the 2jth feature map 5T are double-padding with zeros, convolved with a 3x3x3 filter at one stride, and activated by an activation function. As a result, 64 feature maps 5U are generated. These feature maps 5U are sent to the ascending convolutional layer 62C.

[0068] The ascending convolutional layer 62C enlarges each of the 64 feature maps 5U by 2×2×2 and performs 14-fold zero padding. Then, it convolves each of them with a 5×5×5 filter with a stride of 3 to generate 64 feature maps 5V. These feature maps 5V are then sent to the convolutional layer 625.

[0069] The convolutional layer 625 generates a feature map 5W by combining the k-th feature map 5V and the roundup(k / 2)-th feature map 5B among the 64 feature maps 5V sent from the ascending convolutional layer 62C and the 32 feature maps 5B sent from the convolutional layer 612. Then, the (2j-1)-th feature map 5W and the 2j-th feature map 5W are double-padding with zeros, convolved with a 9×9×9 filter at one stride, and activated by an activation function. As a result, 32 feature maps 5X are generated. These feature maps 5X are sent to the convolutional layer 626.

[0070] The convolution layer 626 performs double zero padding on the (2j-1)th feature map 5X and the 2jth feature map 5X, performs convolution with a 7x7x7 filter with one stride, and activates them with an activation function. This generates 16 feature maps 5Y. These feature maps 5Y are sent to the convolution layer 627.

[0071] The convolutional layer 627 performs double zero padding on the 16 feature maps 5Y, convolves them with a 3x3x3 filter at one stride, and activates them with an activation function. This generates one feature map 5Z. The convolutional layer 627 then outputs this feature map 5Z as prediction data.

[0072] The machine learning unit 405 calculates the error between the light fluence distribution 52 of the first data set 50, i.e., the correct data, and the feature map 5Z, i.e., the predicted data, using a loss function. Then, the calculated error is input to an optimization function, and the weight coefficient and bias of the neural network 60 are updated. This completes training based on the first data set 50. Note that, in order to evaluate the absolute error as the error, for example, MSELoss is used as the loss function. Also, Adam (Adaptive Momentum Estimation) is used as the optimization function.

[0073] Then, the machine learning unit 405 trains the neural network 60 by using the second and subsequent data sets 50 in the same manner. However, the learning rate η t is decayed for each learning. η t = η min +(η max -η min )(1+cos((T cur / T cur )π)) / 2 …(1)

[0074] If it is verified in the verification step that the trained neural network 67 has a certain level of accuracy, the machine learning unit 405 stores the trained neural network 60 as the trained neural network 67 in the trained model storage unit 406.

[0075] However, if a certain level of accuracy is not achieved, the machine learning unit 405 prepares a new data set 50 and performs training and validation steps until a certain level of accuracy is achieved.

[0076] [4. Reasoning] Once the machine learning is complete, the light fluence distribution when the brain of a patient (subject) is irradiated with laser light in actual PDT can be inferred using the trained neural network 67. Below, the inference process will be explained using as an example a case where PDT is performed using laser light with the same conditions (radius, intensity, wavelength, etc.) as those in machine learning, i.e., laser light 43.

[0077] A medical staff member such as a doctor measures the three-dimensional shape of the part of the patient's brain that is to be treated using a three-dimensional measuring device 3. This results in a three-dimensional tissue model 54 of that part. The three-dimensional tissue model 54 is then transmitted from the three-dimensional measuring device 3 to the learning and inference device 2.

[0078] The three-dimensional tissue model 54 has the same size as the three-dimensional tissue model 51. In this embodiment, the three-dimensional tissue model 54 is composed of 32×32×32 voxels, and each voxel is a cube with each side measuring 0.1 mm. Alternatively, the three-dimensional measuring device 3 may obtain a three-dimensional model larger than the three-dimensional tissue model 51, and the learning and inference device 2 may trim the three-dimensional model to the above size to obtain the three-dimensional tissue model 54.

[0079] The light fluence inference unit 407 infers the light fluence distribution of the three-dimensional tissue model 54 by inputting the three-dimensional tissue model 54 to the input layer of the trained neural network 67 and obtaining an output value from the output layer of the trained neural network 67. Hereinafter, this output value is referred to as "light fluence distribution 55."

[0080] The inference result output unit 408 colors each voxel of the three-dimensional tissue model 54 with a color or density according to the value (light fluence) of each voxel shown in the light fluence distribution 55. This results in a light distribution model 56. Then, a rendering image 571 obtained by rendering the light distribution model 56 in accordance with a direction specified by the medical staff is displayed on the display 27. Alternatively, a cross-sectional image 572 obtained by cross-sectioning the light distribution model 56 at an arbitrary position may be displayed.

[0081] Medical staff can perform PDT while viewing the light distribution model 56 displayed on the display 27.

[0082] 5. Overall Processing Flow and Effects of This Embodiment FIG. 10 is a flowchart illustrating an example of the overall processing flow by the learning inference program 40.

[0083] Next, the overall process flow of the learning inference device 2 will be described with reference to a flowchart.

[0084] The learning and inference device 2 executes the process of constructing an AI and the process of inferring how the laser light 43 reaches the brain of a patient based on the learning and inference program 40 in the procedure shown in FIG.

[0085] In the phase of preparing a dataset, the learning and inference device 2 generates a dataset 50 (#11 in FIG. 10) and stores it (#12). By performing machine learning based on the dataset 50, the learning and inference device 2 generates a trained neural network 67 as a trained model (#13) and stores it (#14).

[0086] Then, in the PDT, when the learning and inference device 2 receives the three-dimensional shape of the patient's brain, i.e., the three-dimensional tissue model 54, from the three-dimensional measurement device 3 (#15), it inputs the three-dimensional tissue model 54 into the trained neural network 67 to calculate the light fluence distribution 55 (#16). This allows the way in which the laser light 43 reaches the patient's brain to be inferred.

[0087] Furthermore, the learning and inference device 2 generates a light distribution model 56 based on the light fluence distribution 55 (#17), and renders or cross-sections the light distribution model 56 according to the specified orientation to display a rendered image 571 or a cross-sectional image 572 (#18).

[0088] According to this embodiment, it is possible to infer, faster than ever before, with a computer that does not have high specifications, what intensity of laser light irradiated onto the human brain will reach each part of the brain.

[0089] In addition, a learned model is generated by training a U-Net such as neural network 60, which is computationally efficient and enables inference at the speed of light.

[0090] 6. Modifications and Applications Fig. 11 is a diagram showing an example of a mini-batch 58. Fig. 12 is a diagram showing an example of attributes of each region when a voxel group is divided into four regions. Fig. 13 is a diagram showing an example of attributes of each region when a voxel group is divided into five regions.

[0091] In this embodiment, in the machine learning phase, the neural network 60 is trained by updating parameters such as weight coefficients and biases for each set of data sets 50. However, according to this method, even if a certain data set 50 is an outlier, noise, or abnormal value, the data set 50 is used as is, which may adversely affect the neural network 60. In other words, this method is vulnerable to outliers, etc. Also, since the number of updates is large, training takes time.

[0092] To overcome these disadvantages, machine learning may be performed using mini-batch learning. When the inventors trained the neural network 60 by setting the batch size to "4" and "32", respectively, they found that the former trained the neural network 60 better than the latter.

[0093] Therefore, the learning inference system 1 may perform machine learning by mini-batch learning with a batch size of "4". That is, the learning inference system 1 randomly groups 80,000 data sets 50 into groups of four to generate 20,000 mini-batches 58 as shown in FIG. 11. For each mini-batch 58, the gradient of the three-dimensional tissue model 51 of each of the four data sets 50 constituting it is calculated, and the parameters of the neural network 60 are updated by the average of the four gradients. Therefore, the number of epochs is 20,000, and the neural network 60 is trained 20,000 times.

[0094] Incidentally, in estimating the effect of PDT treatment, the intensity of voxels (intensity of light irradiation) that are considered to be strongly irradiated with laser light in the three-dimensional tissue model 51 is important. However, such voxels are limited to a very small portion of the whole, and therefore the intensity of such voxels is likely to be evaluated as an outlier.

[0095] To solve this problem, the learning inference system 1 may divide the voxels used to calibrate the three-dimensional tissue model 51 into several groups according to their intensities, calculate the error for each group, and calculate the sum of these errors as the error for the entire three-dimensional tissue model 51. For example, a loss function such as Loss_fnc in the following equation (2) may be prepared, and the three-dimensional tissue model 51 may be divided into four groups, a first group, a second group, a third group, and a fourth group, to calculate the error. Loss_fnc=α4×MAPE4+α1×MSE1+α2×MSE2+α3×MSE3…(2)

[0096] As shown in FIG. 12, the first group is a group whose common logarithm of intensity (Fluence) is equal to or greater than "1". The second group is a group whose common logarithm of intensity is equal to or greater than "-1" and less than "1". The third group is a group whose common logarithm of intensity is equal to or greater than "-3" and less than "-1". The fourth group is a group whose common logarithm of intensity is less than "-3". The unit of intensity is "J / mm 3 " α1 to α4 are weighting coefficients for the first to fourth groups, respectively, and α3>α2>α1>α4.

[0097] "MSE1", "MSE2", and "MSE3" are the mean squared errors of the first, second, and third groups, respectively. "MAPE4" is the mean absolute percentage error of the fourth group. Thus, "10 -3 The error can be calculated more precisely by taking into account small intensities below 0.01. Also, the fourth group of errors can be better evaluated by applying MAPE (Mean Absolute Percentage Error) instead of MSE (Mean Squared Error).

[0098] The numbers of voxels in each of the first to fourth groups shown in FIG. 12 are an example of the number of voxels belonging to each group in a certain three-dimensional tissue model 51. As can be seen from this example, the number of voxels in the third group accounts for the majority at 76% of the total, the number of voxels in the second group is approximately 22% of the total, and the number of voxels in the first group is less than 1% of the total. In consideration of such differences in share rate, MSE3, MSE2, and MSE1 are multiplied by weighting coefficients in that order. For example, α3=10 8 , α2=10 6 , α1=10 3 It is set as follows.

[0099] In addition, the number of voxels in the fourth group is less than 1% of the total. In consideration of the fact that MAPE is used instead of MSE as the error of the fourth group, the share of the number of voxels in the fourth group, and the importance of the fourth group, a coefficient smaller than α1, for example "1", is set as the weighting coefficient α4 of MAPE4.

[0100] The weighting coefficients α1, α2, α3, and α4 may be adjusted according to the share rates of the first group, the second group, the third group, and the fourth group, respectively.

[0101] Alternatively, the range of the common logarithm of the intensity of each of the first group, the second group, the third group, and the fourth group is not limited to the above example. For example, the range of the common logarithm of each intensity may be set to a range of "0.8" or more, a range of "-1.2" or more and less than "0.8", a range of "-3.2" or more and less than "-1.2", or a range less than "-3.2". The range of each group may be set according to the target share rate of each group. In the example of FIG. 12, the share rates of the first to fourth groups are about 0.07%, about 22.9%, about 76.3%, and about 0.72%, respectively.

[0102] Alternatively, the voxels of the three-dimensional tissue model 51 may be divided into three groups, or into five or more groups.

[0103] For example, as shown in FIG. 13, the three-dimensional tissue model 51 may be divided into five groups, a first group, a second group, a third group, and an error may be calculated based on a loss function such as Loss_fnc in the following equation (3). Loss_fnc=β1×MAPE5+β2×MSE6+β3×MSE7+β4×MSE8 +β5×MSE9…(3)

[0104] As can be seen by comparing this example with the previous example (see FIG. 12) in which the group was divided into four groups, this example divides the second group in the previous example further into two groups using the threshold value "0."

[0105] That is, the first group is a group whose common logarithm of intensity (Fluence) is "1" or more. The second group is a group whose common logarithm of intensity is "0" or more and less than "1". The third group is a group whose common logarithm of intensity is "-1" or more and less than "0". The fourth group is a group whose common logarithm of intensity is "-3" or more and less than "-1". The fifth group is a group whose common logarithm of intensity is less than "-3". The unit of intensity is "J / mm 3 " β1 to β5 are weighting coefficients for the first to fifth groups, respectively, and β4>β3>β2>β1>β5.

[0106] "MSE5", "MSE6", "MSE7", and "MSE8" are the mean squared errors of the first, second, third, and fourth groups, respectively. "MAPE9" is the mean absolute percentage error of the fifth group.

[0107] 13 is an example of the number of voxels belonging to each group in a certain three-dimensional tissue model 51. In addition, when the weighting coefficient is, for example, β1=10 3 , β2=10 5 , β3=10 6 , β4=10 10 , β5=1. The range of each group may be set according to the target share rate of each group. In the example of Fig. 13, the share rates of the first to fifth groups are approximately 0.09%, approximately 1.69%, approximately 24.4%, approximately 73.5%, and approximately 0.36%, respectively.

[0108] Fig. 13 is a diagram showing an example of splitting of laser light 44. Fig. 14 is a diagram showing an example of splitting of laser light 45. Fig. 15 is a diagram showing an example of splitting of laser light 47.

[0109] In this embodiment, the learning inference system 1 performs each process on the assumption that the laser light 43 is used in both the machine learning phase and the inference phase. However, it is also possible to infer a light distribution model 56 in the inference phase when a laser light having a radius larger than that of the laser light 43 is used.

[0110] Specifically, a laser beam with a radius larger than 0.1 mm is divided into multiple laser beams (segments) with a radius of 0.1 mm. For each segment, a light fluence distribution 55 is calculated based on a trained neural network 67. Then, the sum of the multiple calculated light fluence distributions 55 is adopted as the inference result, i.e., the target light fluence distribution.

[0111] For example, when a laser beam 44 with a radius of 0.3 mm is used, the learning inference system 1 divides the laser beam 44 into seven segments (laser beams 43) as shown in Fig. 14. That is, the laser beam 43 is regarded as a bundle of eight laser beams 43. An optical fluence distribution 55 obtained when each of the eight laser beams 43 is irradiated onto a three-dimensional tissue model 54 is calculated based on the trained neural network 67. Then, the sum of the calculated eight optical fluence distributions 55 is adopted as the inference result, i.e., the target optical fluence distribution.

[0112] It can be said that the cross-sectional area of ​​the laser light 44 is approximately equal to, but not completely equal to, the sum of the cross-sectional areas of the eight laser lights 43. Therefore, even with the above-mentioned method, it is possible to obtain a light fluence distribution that is sufficiently useful for PDT.

[0113] In addition to the trained neural network 67 corresponding to the laser light 43, i.e., the laser light having a radius of 0.1 mm, trained neural networks 67 corresponding to each of several laser lights having smaller radii (e.g., laser lights of 0.03 mm, 0.05 mm, 0.07 mm, etc.) may be generated in the above manner. Then, the learning inference system 1 may divide a laser light having a large radius such as the laser light 44 into several laser lights having smaller radii, and infer the light fluence distribution using each trained neural network 67.

[0114] For example, as shown in Fig. 15, a laser light 45 with a radius of 0.2 mm is regarded as one laser light 43 and a bundle of nine laser lights 46 with a radius of 0.05 mm. The learning inference system 1 calculates an optical fluence distribution 55 when one laser light 43 is irradiated based on a trained neural network 67 for the laser light 43. Furthermore, the learning inference system 1 calculates an optical fluence distribution 55 when each of nine laser lights 46 is irradiated based on a trained neural network 67 for the laser light 46. Then, the sum of the calculated ten optical fluence distributions 55 is adopted as the inference result, i.e., the target optical fluence distribution.

[0115] Alternatively, the laser light used in PDT may be divided into laser light (segments) having a rectangular cross section. For example, the laser light 47 shown in FIG. 16 is divided into laser light 48 (segments) having a square cross section. When divided in this manner, there are some parts that are less than one segment. If each of these parts is replaced with a segment, the sum of the segments will obviously be greater than the laser light 47. Therefore, among all the segments, those for which the distance between a predetermined position of the segment itself (for example, the lower left vertex) and the center of the laser light 47 exceeds the radius of the laser light 47 may be excluded. In the example of FIG. 16, the segments shown by hatching are excluded.

[0116] Moreover, a trained neural network 67 for the laser light 48 is generated in advance in the above manner.

[0117] Then, for the remaining segments, i.e., 115 of the 137 segments shown in FIG. 16, the light fluence distribution 55 is calculated based on the trained neural network 67, and the sum of the calculated light fluence distributions 55 is adopted as the inference result, i.e., the desired light fluence distribution.

[0118] Also, for laser light having different conditions such as output intensity and wavelength as well as radius, trained neural networks 67 may be generated and used separately. In particular, trained neural networks 67 with a plurality of sets of output intensity and radius may be generated and used separately for each laser light having a wavelength corresponding to each PS (photosensitizer).

[0119] 3 is provided in the learning and inference device 2, they may be provided in a plurality of devices. For example, the three-dimensional model generating unit 401, the light fluence calculating unit 402, and the data set registering unit 403 may be provided in a preparation computer, the data set storing unit 404 and the machine learning unit 405 may be provided in a machine learning computer, and the trained model storing unit 406, the light fluence inference unit 407, and the inference result output unit 408 may be provided in the inference computer. In this case, these computers may be connected by a communication line such as the Internet, a LAN (Local Area Network), or a public line, and the data set 50, the trained neural network 67, and the like may be exchanged between the computers.

[0120] The preparation computer and the machine learning computer may be realized by cloud computing, and the inference computer may be realized by a personal computer. The preparation computer and the machine learning computer require a large amount of calculations, so high specifications are required, but the inference computer does not require a large amount of calculations, so a computer with low specifications may be used. Therefore, it can be installed in a medical site such as an operating room and easily used by medical staff during PDT. Also, the inference computer may be installed at each medical site.

[0121] In this embodiment, the learning inference system 1 generates the trained neural network 67 by training the neural network 60 shown in Fig. 7, but the trained neural network 67 may be generated by training a U-Net with another configuration. Alternatively, the trained neural network 67 may be generated by training a neural network other than the U-Net.

[0122] In this embodiment, the learning inference system 1 generates a trained neural network 67 for the human brain and infers the light fluence distribution 55, but the above-mentioned method can also generate trained neural networks for other human tissues (e.g., gastrointestinal tract, small intestine, large intestine, kidney, liver, ...) and infer light fluence distributions. Also, trained neural networks can be generated for each tissue of other animals and infer light fluence distributions.

[0123] In addition, the configuration of the entire or each part of the learning inference system 1 and the learning inference device 2, the content of the processing, the order of processing, the configuration of the neural network, etc. can be modified as appropriate in accordance with the spirit of the present invention. [Explanation of symbols]

[0124] 1 Learning Inference System 27 Display (display means) 401 Three-dimensional model generation unit (acquisition means) 402 Light fluence calculation unit (acquisition means) 405 Machine Learning Unit (Learning Method) 407 Light fluence distribution inference unit (inference means) 408 Inference result output unit (display means) 43 Laser light (reference laser light) 44 Laser Light 45 Laser Light 50 Datasets 51 3D tissue model (3D model for learning) 52 Light Fluence Distribution (Learning Attainment Level) 54 Three-dimensional tissue model (three-dimensional test model) 55 Light fluence distribution (inference reach) 571 Rendered Images 572 Cross-sectional images 60 Neural Network (U-Net) 67 Trained Neural Networks (Trained Models)

Claims

1. an acquisition means for acquiring, as a data set, a learning three-dimensional model which is a three-dimensional model of a specific tissue of a living body, and a learning reach degree which is a reach degree of the laser light at each position of the learning three-dimensional model when the learning three-dimensional model is irradiated with the laser light; A learning means for generating a trained model by performing machine learning using the training three-dimensional model and the training achievement level of the data set as input data and correct answer data, respectively; an inference means for inputting a subject three-dimensional model, which is a three-dimensional model of the specific tissue of a subject, into the trained model, and calculating an inferred reach of the laser light at each position of the subject three-dimensional model when the subject three-dimensional model is irradiated with the laser light; A laser light arrival degree learning and inference system comprising:

2. an acquisition means for acquiring, as a data set, a learning three-dimensional model which is a three-dimensional model of a specific tissue of a living body, and a learning reach degree which is a reach degree of the laser light at each position of the learning three-dimensional model when the learning three-dimensional model is irradiated with the laser light; A learning means for generating a trained model by performing machine learning using the training three-dimensional model and the training achievement level of the data set as input data and correct answer data, respectively; A laser light reachability learning system comprising:

3. The learning means generates the learned model by training a convolutional neural network.

3. The laser light reachability learning system according to claim 2.

4. The learning means generates the learned model by training a U-Net as the convolutional neural network.

4. The laser light reachability learning system according to claim 3.

5. the learning means calculates an error between a feature map obtained by inputting the learning three-dimensional model into the convolutional neural network and the learning achievement level, and trains the convolutional neural network so as to reduce the error; 5. A laser light reaching degree learning system according to claim 3 or 4.

6. the learning means classifies each position of the training three-dimensional model into one of a plurality of groups according to the magnitude of the degree of achievement of the position indicated in the training achievement degree, calculates an error as an individual error for each group using a different loss function or weighting coefficient, and trains the convolutional neural network so as to reduce the sum of the individual errors; 6. A laser light reachability learning system according to claim 5.

7. The plurality of groups includes a top-level group, the learning means classifies the positions of the learning three-dimensional model into the highest group based on a threshold value that is set so that positions of the learning three-dimensional model whose reach level is equal to or greater than a predetermined value are classified into the highest group; 7. A laser light reachability learning system according to claim 6.

8. The plurality of groups further includes a lowest group, the learning means classifies the positions of the learning 3D model into the lowest group based on a second threshold value that is set so that a position of the learning 3D model whose degree of reach is less than a predetermined value is classified into the lowest group; 8. A laser light reachability learning system according to claim 7.

9. The plurality of groups further includes a first intermediate group and a second intermediate group; the learning means calculates, as the individual error of each of the top group, the first intermediate group, and the second intermediate group, a value obtained by multiplying the mean square error of each of the top group, the first intermediate group, and the second intermediate group by a respective weighting factor, and calculates, as the individual error of the lowest group, a value obtained by multiplying the mean absolute percentage error of the lowest group by the weighting factor of the lowest group.

9. A laser light reachability learning system according to claim 8.

10. the plurality of groups further includes a first intermediate group, a second intermediate group, and a third group; the learning means calculates, as the individual error for each of the top group, the first intermediate group, the second intermediate group, and the third group, a value obtained by multiplying the mean square error by a respective weighting factor, and calculates, as the individual error for the lowest group, a value obtained by multiplying the mean absolute percentage error of the lowest group by the weighting factor of the lowest group.

10. The laser light reachability learning system according to claim 9.

11. an inference means for inputting a subject three-dimensional model, which is a three-dimensional model of a specific tissue of a subject, into a learned model of a specific tissue of a living body generated by the laser light reachability learning system according to claim 2, thereby calculating an inferred reachability of the laser light at each position of the subject three-dimensional model when the subject three-dimensional model is irradiated with laser light; A laser light arrival degree inference system comprising:

12. a display means for displaying an image representing the degree of inference attainment at each position of the test three-dimensional model; having The laser light arrival degree deduction system according to claim 11.

13. The cross-sectional area of ​​the laser light is larger than the cross-sectional area of ​​a reference laser light given as a condition when the laser light reachability learning system generates the trained model, the inference means regards the laser light as a bundle of a plurality of the reference laser lights, calculates an inference reach of the reference laser light at each position of the test three-dimensional model when each of the reference laser lights is irradiated onto the test three-dimensional model using the learned model, and adopts the sum of the calculated inference reach as an inference result.

13. A laser light arrival degree inference system according to claim 11 or 12.

14. The cross section of the reference laser light is rectangular. The laser light arrival degree deduction system according to claim 13.

15. Acquire, as a data set, a learning three-dimensional model which is a three-dimensional model of a specific tissue of a living body, and a learning reach degree which is a reach degree of the laser light at each position of the learning three-dimensional model when the learning three-dimensional model is irradiated with the laser light; generating a trained model by performing machine learning using the training three-dimensional model and the training achievement level of the data set as input data and correct answer data, respectively; A method for learning a laser light reaching degree, comprising:

16. A subject three-dimensional model, which is a three-dimensional model of a specific tissue of a subject, is input to a learned model of a specific tissue of a living body generated by the laser light reachability learning method according to claim 15, thereby calculating an inferred reachability of the laser light at each position of the subject three-dimensional model when the subject three-dimensional model is irradiated with laser light. A method for estimating the degree of laser light arrival, comprising:

17. causing a computer to execute an acquisition process for acquiring, as a data set, a learning three-dimensional model which is a three-dimensional model of a specific tissue of a living body, and a learning reach degree which is a reach degree of the laser light at each position of the learning three-dimensional model when the learning three-dimensional model is irradiated with the laser light; A generation process is performed on the computer to generate a trained model by performing machine learning using the training three-dimensional model and the training achievement level of the data set as input data and correct answer data, respectively. A computer program comprising:

18. A subject three-dimensional model, which is a three-dimensional model of a specific tissue of a subject, is input to a learned model of a specific tissue of a living body generated by the laser light reachability learning method according to claim 15, and a calculation process is executed by a computer to calculate an inferred reachability of the laser light at each position of the subject three-dimensional model when the subject three-dimensional model is irradiated with laser light. A computer program comprising:

Images