Laser beam reach degree learning and inference system, laser beam reach degree learning system, laser beam reach degree inference system, laser beam reach degree learning method, laser beam reach degree inference method, and computer program

Abstract

To quickly infer by a computer whose specification is not high, a level of intensity of attainment of a laser beam radiated to a tissue of a living body such as a human being, to each portion in the tissue.SOLUTION: A laser beam attainment degree learning inference system is configured to: collect, as a data set 50, a three-dimensional tissue model 51 of a specific tissue of a living body, and a light fluence distribution 52 which is an attainment degree of a laser beam in each position on the three-dimensional tissue model 51 when radiating the laser beam to the three-dimensional tissue model 51; perform machine learning using the three-dimensional tissue model 51 and the light fluence distribution 52 as input data and correct answer data, respectively, for generating a learned neural network 67; input a three-dimensional tissue model 54 of the specific tissue of a subject, to the learned neural network 67, for calculating a light fluence distribution 55 on each position of the three-dimensional tissue model 54 when radiating the laser beam to the three-dimensional tissue model 54.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for estimating the degree to which irradiated laser light reaches each position in tissue of a living body. [Background technology]

[0002] In recent years, photodynamic therapy (PDT) has been attracting attention as a treatment method for removing tumors. PDT, written as "photodynamic therapy" in Japanese, is a tumor-selective, minimally invasive treatment that utilizes the tumor accumulation of photosensitizers (PS), or photosensitizing drugs, and their photochemical reaction to specific wavelengths of light. When administered into the body, PS accumulates inside tumor cells. When irradiated with laser light of a specific wavelength, a photochemical reaction occurs in PS, generating singlet oxygen, which then kills the tumor cells.

[0003] These characteristics enable PDT to induce necrosis of tumor cells while minimizing the effects on surrounding normal cells.

[0004] By optimizing treatment conditions such as the irradiation position or angle of the laser light in PDT, it is expected that the risk of damage to tissues such as organs irradiated with the laser light can be reduced.

[0005] To achieve optimization, it is necessary to understand the light distribution within the tissue, i.e., how the irradiated laser light reaches the tissue. Because it is difficult to understand the light distribution within the tissue experimentally, it is effective to use simulations.

[0006] For example, application software described in Non-Patent Document 1 makes it possible to simulate the intensity of laser light irradiated onto tissue such as the human brain and reaching each part of the tissue. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "Monte Carlo Simulation of Photon Migration in 3D Turbid Media Accelerated by Graphics Processing Units", Qianqian Fang & David Boas, 2009 Summary of the Invention [Problem to be solved by the invention]

[0008] However, existing simulation methods such as those described in the above-mentioned Non-Patent Document 1 require high-spec resources such as a large-scale parallel computer, limiting the environments in which they can be used. While it is technically possible to implement a similar method on a computer with lower specs, the calculation time required is long. Therefore, it is difficult to use them in situations requiring real-time performance, such as during surgery.

[0009] In view of these problems, the present invention aims to infer, faster than ever before, using a computer with low specifications, the intensity at which laser light irradiated onto the tissue of a living body such as a human, will reach each part of the tissue. [Means for solving the problem]

[0010] A laser light reachability learning and inference system according to one embodiment of the present invention comprises an acquisition means for acquiring, as a dataset, a training three-dimensional model, which is a three-dimensional model of specific tissue of a living organism, and a training reachability, which is the reachability of the laser light at each position of the training three-dimensional model when laser light is irradiated onto each of the training three-dimensional models; a learning means for generating a trained model by performing machine learning using the training three-dimensional model and the training reachability of the dataset as input data and ground truth data, respectively; and an inference means for inputting a test three-dimensional model, which is a three-dimensional model of the specific tissue of a subject, into the trained model, and calculating the inferred reachability of the laser light at each position of the test three-dimensional model when the test three-dimensional model is irradiated with the laser light. [Effects of the Invention]

[0011] According to the present invention, it is possible to estimate, faster than ever before, the intensity at which laser light irradiated onto the tissue of a living body such as a human being reaches each part of the tissue, using a computer with low specifications. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a learning inference system. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning inference device. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning inference device. [Figure 4] FIG. 1 is a diagram illustrating an example of a three-dimensional tissue model. [Figure 5] FIG. 10 is a diagram illustrating an example of light fluence. [Figure 6] FIG. 10 is a diagram illustrating an example of a light fluence distribution. [Figure 7] FIG. 1 is a diagram illustrating an example of a neural network. [Figure 8] FIG. 10 is a diagram illustrating an example of a three-dimensional model with double zero padding. [Figure 9]FIG. 10 is a diagram showing an example of how a three-dimensional model is enlarged 2×2×2 times. [Figure 10] 10 is a flowchart illustrating an example of the overall processing flow of a learning inference program. [Figure 11] FIG. 10 is a diagram illustrating an example of a mini-batch. [Figure 12] FIG. 10 is a diagram showing examples of attributes of four regions of a voxel group. [Figure 13] FIG. 10 is a diagram showing examples of attributes of five regions of a voxel group. [Figure 14] FIG. 10 is a diagram illustrating an example of splitting a laser beam. [Figure 15] FIG. 10 is a diagram illustrating an example of splitting a laser beam. [Figure 16] FIG. 10 is a diagram illustrating an example of splitting a laser beam. DETAILED DESCRIPTION OF THE INVENTION

[0013] [1. Overall system configuration] Fig. 1 is a diagram illustrating an example of the overall configuration of a learning inference system 1. Fig. 2 is a diagram illustrating an example of the hardware configuration of a learning inference device 2. Fig. 3 is a diagram illustrating an example of the functional configuration of the learning inference device 2.

[0014] The learning inference system 1 shown in Fig. 1 is a system that uses AI (Artificial Intelligence) technology to infer how laser light emitted from a light source will reach the tissues of a living body, and is composed of a learning inference device 2 and a three-dimensional measurement device 3. Below, we will explain the example of inferring how laser light will reach each position in the human brain.

[0015] The learning and inference device 2 generates a trained model through machine learning and infers how the irradiated laser light reaches the brain based on the trained model. The following describes an example in which a personal computer is used as the learning and inference device 2.

[0016] As shown in FIG. 2, the learning inference device 2 is composed of a main processor 20, a RAM (Random Access Memory) 21, a ROM (Read Only Memory) 22, an auxiliary storage device 23, a network interface 24, a serial interface 25, a wireless communication device 26, a display 27, a keyboard 28, and a pointing device 29.

[0017] In addition to the operating system, computer programs such as a learning and inference program 40 are installed in the ROM 22 or the auxiliary storage device 23. As the auxiliary storage device 23, a hard disk or an SSD (Solid State Drive) is used.

[0018] 3, functions such as a three-dimensional model generation unit 401, a light fluence calculation unit 402, a data set registration unit 403, a data set storage unit 404, a machine learning unit 405, a learned model storage unit 406, a light fluence inference unit 407, and an inference result output unit 408 are realized. The functions of each unit will be explained in turn later.

[0019] The RAM 21 is the main memory of the learning and inference device 2. Computer programs stored in the ROM 22 or the auxiliary storage device 23 are loaded into the RAM 21 as appropriate.

[0020] The main processor 20 executes a computer program loaded into the RAM 21. As the main processor 20, a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit) or the like is used.

[0021] The network interface 24 communicates with other devices using protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol). As the network interface 24, a NIC (Network Interface Card) or a communication device for Wi-Fi is used.

[0022] The serial interface 25 communicates with peripheral devices using a serial communication method, and is implemented as a board conforming to standards such as USB (Universal Serial Bus).

[0023] The wireless communication device 26 communicates with peripheral devices by short-range wireless communication. A board conforming to standards such as Bluetooth is used as the wireless communication device 26.

[0024] The serial interface 25 and the wireless communication device 26 are used particularly for communicating with the three-dimensional measuring device 3. The serial interface 25 and the wireless communication device 26 are used depending on the communication standard of the three-dimensional measuring device 3.

[0025] The display 27 displays a screen for inputting commands or data, a screen showing the results of calculations performed by the main processor 20, and the like.

[0026] The keyboard 28 and the pointing device 29 are input devices that allow the operator to input commands, data, and the like.

[0027] The three-dimensional measurement device 3 measures the three-dimensional shape of a living body. In this embodiment, it particularly measures the three-dimensional shape of the entire brain or a part of it. Then, data indicating the measured three-dimensional shape is transmitted to the learning and inference device 2.

[0028] Below, we will explain the functions of each part shown in Figure 3, broadly dividing them into the dataset preparation phase, machine learning phase, and inference phase.

[0029] [2. Preparing the dataset] Fig. 4 is a diagram showing an example of a three-dimensional tissue model 51. Fig. 5 is a diagram showing an example of light fluence. Fig. 6 is a diagram showing an example of light fluence distribution 52.

[0030] The three-dimensional model generation unit 401, the light fluence calculation unit 402, and the data set registration unit 403 execute the process of preparing a data set for machine learning as follows.

[0031] The three-dimensional model generating unit 401 generates a large number of three-dimensional tissue models 51 that mimic a portion of the surface of the human brain as shown in FIG. 4. For example, 40,000 to 100,000 three-dimensional tissue models 51 are generated. X ×B Y ×B Z The image is made up of voxels and is divided into two layers: a tissue layer 51f and an air layer 51g. The tissue layer 51f represents the brain tissue. The air layer 51g represents the outside of the brain, i.e., the air.

[0032] The boundary surface between the two layers is generally parallel to the XY plane and has small irregularities. The three-dimensional tissue model 51 is generated, for example, by the following method.

[0033] The three-dimensional model generation unit 401 generates a three-dimensional model for each voxel on the XY plane (i.e., B X ×B Y voxels), an integer value D between D1 and D2 inclusive XY is given randomly. Then, the coordinates are calculated from (X,Y,0) to (X,Y,(D0+D XY )) are defined as voxels of the air layer 51g, and (X, Y, (D0+D XY +1)) to (X,Y,B Z ) are defined as air layers 51g. Note that D0, D1, and D2 are all constants.

[0034] For example, for a voxel with X=12 and Y=7, the integer value D XY "1" is given as B Zis "31" and "15" is defined as D0, then the voxels (12,7,0) to (12,7,16) are defined as voxels in the air layer 51g, and the voxels (12,7,17) to (12,7,31) are defined as voxels in the tissue layer 51f.

[0035] Below, B X =B Y =B Z The processing of each unit will be described using an example where D1 = 31, the voxels are cubes with sides of 0.1 mm, D1 = 0, D2 = 1, and D0 = 15. Therefore, the three-dimensional model generation unit 401 generates a three-dimensional tissue model 51 having a boundary surface between a tissue layer 51f having an irregularity of 0.1 mm and an air layer 51g at a depth of approximately 1.5 mm. Furthermore, in this embodiment, an example will be described in which 100,000 mutually different three-dimensional tissue models 51 are generated.

[0036] The light fluence calculation unit 402 randomly extracts 80,000 from the 100,000 generated by the three-dimensional model generation unit 401. For each of the extracted 80,000 three-dimensional tissue models 51, the light fluence of each voxel of each three-dimensional tissue model 51 is calculated when laser light 43 used in PDT (Photodynamic Therapy) is irradiated under specified conditions. The light fluence is the intensity of the laser light 43 at the voxel, and is the amount of laser light 43 that reaches the voxel per unit time. It can also be said to be equivalent to illuminance.

[0037] The optical fluence is calculated by a known method such as the Monte Carlo method. For example, it may be calculated using MCX (Monte Carlo eXtreme). MCX is software developed by Qianqian Fang, and its algorithm is described in the aforementioned Non-Patent Document 1. Furthermore, technical information such as its source code is disclosed at http: / / mcx.space / .

[0038] In this embodiment, the predetermined conditions are set such that the angle of incidence of laser light 43 is 90 degrees with respect to the surface of tissue layer 51f, i.e., the XY plane, that laser light 43 is incident from the center (16,16,0) of the surface of tissue layer 51f, and that laser light 43 has a radius of 0.1 mm. Furthermore, the scattering coefficient, absorption coefficient, and scattering anisotropy when laser light 43 is irradiated onto human brain tissue are determined in advance and input into the learning and inference device 2 by the operator.

[0039] When laser light 43 is incident on tissue layer 51f of three-dimensional tissue model 51 under these conditions, the optical fluence appears as shown in Fig. 5. Hereinafter, the 32 x 32 x 32 optical fluences as shown in Fig. 6 calculated by optical fluence calculation unit 402 will be referred to as "optical fluence distribution 52."

[0040] The dataset registration unit 403 combines the i-th three-dimensional tissue model 51 and the light fluence distribution 52 as one set of dataset 50, and stores the set in the dataset storage unit 404. As a result, 80,000 sets of datasets 50 are stored in the dataset storage unit 404.

[0041] [3. Machine Learning] Fig. 7 is a diagram showing an example of a neural network 60. Fig. 8 is a diagram showing an example of a three-dimensional model that has been double-zero padded. Fig. 9 is a diagram showing an example of a three-dimensional model being enlarged 2x2x2 times.

[0042] The machine learning unit 405 generates a trained model for inferring how the irradiated laser light 43 reaches the brain by performing machine learning based on the dataset 50 stored in the dataset storage unit 404. The machine learning method will be described below with reference to Figs. 7 to 9.

[0043] The machine learning unit 405 is provided with a neural network 60 as shown in FIG.

[0044] The neural network 60 is a U-Net type deep neural network, consisting of an encoder 61 and a decoder 62. U-Net is a fully convolutional network (FCN) and is a semantic segmentation model developed by Olaf et al. for biomedical applications. Details of U-Net are available at https: / / arxiv.org / abs / 1505.04597.

[0045] The machine learning phase mainly includes a training step and a validation step. In the training step, the machine learning unit 405 trains the neural network 60 as follows.

[0046] The encoder 61 is composed of convolutional layers 611 to 618 and pooling layers 61A to 61C, etc. The decoder 62 is composed of convolutional layers 621 to 627 and upward convolutional layers 62A to 62C, etc. The processes performed as appropriate in these layers, such as convolution, pooling, upward convolution, feature map (channel) expansion, and feature map (channel) compression, are performed using known techniques.

[0047] Below, the processing at each layer will be explained using an example in which the neural network 60 is trained using the first data set 50.

[0048] The convolutional layer 611 also functions as an input layer, and receives the three-dimensional tissue model 51 of the first dataset 50 as input.

[0049] Then, the convolution layer 611 performs double zero padding on the input three-dimensional tissue model 51. Double zero padding is a process of adding two layers of "0" voxels to each of the six faces of the three-dimensional tissue model, and adding 2 × 2 × 2 "0" voxels to the outside of the eight vertices, as shown in Figure 8. This process changes the size of the three-dimensional tissue model 51 from 32 × 32 × 32 to 36 × 36 × 36.

[0050] The convolutional layer 611 generates 16 feature maps 5A by convolving the zero-padded three-dimensional tissue model 51 with 16 different filters and activating them with an activation function. The 16 feature maps 5A are then sent to the convolutional layer 612. For example, ReLU is used as the activation function. The following describes an example in which ReLU is used as the activation function in the neural network 60.

[0051] All 16 filters are 3x3x3 filters. Convolution is performed with one stride. Therefore, the size of each of the 16 feature maps 5A is 34x34x34. In the convolution layers 612 to 614, convolution is also performed with one stride using 3x3x3 filters.

[0052] The convolutional layer 612 performs double zero padding on each of the 16 feature maps 5A. This changes the size of each feature map 5A from 34 × 34 × 34 to 38 × 38 × 38. The convolutional layer 612 convolves each of the 16 zero-padded feature maps 5A with two different filters and activates them with an activation function to generate 32 feature maps 5B. The 32 feature maps 5B are then sent to the pooling layer 61A and the convolutional layer 625.

[0053] The pooling layer 61A generates 32 feature maps 5C by performing 3×3×3 max pooling with one stride on each of the 32 feature maps 5B, and sends these to the convolution layer 613.

[0054] Like the convolutional layer 612, the convolutional layers 613 and 614 perform double zero padding on the feature maps sent from the previous layer, convolve the feature maps with two different filters at one stride, and activate them using an activation function to generate new feature maps. The generated feature maps are then sent to the next layer. As a result, 64 feature maps 5D are generated in the convolutional layer 613 and sent to the convolutional layer 614. Furthermore, 128 feature maps 5E are generated in the convolutional layer 614 and sent to the pooling layer 61B and the convolutional layer 623.

[0055] The pooling layer 61B generates 128 feature maps 5F by performing 3×3×3 max pooling with a stride of 2 on each of the 128 feature maps 5E, and sends these to the convolution layer 615.

[0056] Each of the convolutional layers 615 and 616 generates a new feature map by double-padding the feature map sent from the previous layer, convolving it with two different 3x3x3 filters at two strides, and activating it with an activation function. The generated feature map is then sent to the next layer. As a result, 256 feature maps 5G are generated in the convolutional layer 615 and sent to the convolutional layer 616. Also, 512 feature maps 5H are generated in the convolutional layer 616 and sent to the pooling layer 61C and the convolutional layer 621.

[0057] Similar to pooling layer 61B, pooling layer 61C performs 3 × 3 × 3 max pooling with 2 strides on each of the feature maps sent from the previous layer, i.e., the 512 feature maps 5H, and sends them to the next layer, i.e., convolutional layer 617. As a result, 512 feature maps 5J are generated and sent to the convolutional layer 617.

[0058] Like the convolutional layers 615 and 616, the convolutional layer 617 performs double zero padding on the feature maps sent from the previous layer, convolves them with two different 3x3x3 filters at two strides, and activates them using an activation function to generate new feature maps.The generated feature maps are then sent to the next layer.As a result, 1024 feature maps 5K are generated and sent to the convolutional layer 618.

[0059] The convolutional layer 618 generates 1024 feature maps 5L by double-zero padding the 1024 feature maps 5K, convolving them with a 3x3x3 filter with two strides, and activating them with an activation function. The feature maps 5L are then sent to the ascending convolutional layer 62A.

[0060] The ascending convolution layer 62A generates 1024 feature maps 5M by performing upward convolution on the 1024 feature maps 5L as follows, and sends the 1024 feature maps 5M to the convolution layer 621. Here, an example of a method of upward convolution will be described.

[0061] The ascending convolutional layer 62A enlarges one feature map 5L by 2x2x2. That is, as shown in FIG. 9, each of the eight voxels in the feature map 5L is copied 2x2x2 times. The enlarged feature map 5L is then double-padding with zeros. It is then convolved with a 3x3x3 filter at one stride. This generates one feature map 5M. The remaining 1023 feature maps 5L are processed in the same way, generating 1023 feature maps 5M.

[0062] The convolutional layer 621 generates a feature map 5N by combining the k-th feature map 5M and the roundup(k / 2)-th feature map 5H out of the 1,024 feature maps 5M sent from the ascending convolutional layer 62A and the 512 feature maps 5H sent from the convolutional layer 616. Note that roundup(k / 2) is the value obtained by dividing k by 2 and rounding up to the nearest integer. Therefore, when k=1, 2, 3, 4, 5, ..., roundup(k / 2)=1, 1, 2, 2, 3, ....

[0063] The convolutional layer 621 then performs six-fold zero padding on the (2j-1)th feature map 5N and the 2jth feature map 5N, convolves them with a 3x3x3 filter at one stride, and activates them using an activation function. This generates 512 feature maps 5P. These feature maps 5P are sent to the convolutional layer 622.

[0064] Similar to the convolutional layer 621, the convolutional layer 622 performs six-fold zero padding on the (2j-1)th feature map 5P and the 2jth feature map 5P, then performs convolution with a 3x3x3 filter at one stride and activates them using an activation function. This generates 256 feature maps 5Q. These feature maps 5Q are sent to the ascending convolutional layer 62B.

[0065] The ascending convolutional layer 62B scales each of the 256 feature maps 5Q by 2x2x2 and double-padding them with zeros. It then convolves each of these maps with a 5x5x5 filter with a stride of 2 to generate 256 feature maps 5R. These feature maps 5R are then sent to the convolutional layer 623.

[0066] The convolutional layer 623 generates a feature map 5S by combining the kth feature map 5R and the roundup(k / 2)th feature map 5R from the 256 feature maps 5R sent from the ascending convolutional layer 62B and the 128 feature maps 5E sent from the convolutional layer 614. Then, the (2j-1)th feature map 5S and the 2jth feature map 5S are double-padding with zeros, convolved with a 3x3x3 filter at one stride, and activated using an activation function. This generates 128 feature maps 5T. These feature maps 5T are sent to the convolutional layer 624.

[0067] Similar to the convolutional layer 623, the convolutional layer 624 performs double zero padding on the (2j-1)th feature map 5T and the 2jth feature map 5T, then performs convolution with a 3x3x3 filter at one stride and activates them using an activation function. This generates 64 feature maps 5U. These feature maps 5U are sent to the ascending convolutional layer 62C.

[0068] The ascending convolutional layer 62C scales each of the 64 feature maps 5U by 2x2x2 and performs 14-fold zero padding. It then convolves each of these maps with a 5x5x5 filter with a stride of 3 to generate 64 feature maps 5V. These feature maps 5V are then sent to the convolutional layer 625.

[0069] The convolutional layer 625 generates a feature map 5W by combining the kth feature map 5V and the roundup(k / 2)th feature map 5B from the 64 feature maps 5V sent from the ascending convolutional layer 62C and the 32 feature maps 5B sent from the convolutional layer 612. Then, the (2j-1)th feature map 5W and the 2jth feature map 5W are double-padding with zeros, convolved with a 9x9x9 filter at one stride, and activated using an activation function. This generates 32 feature maps 5X. These feature maps 5X are sent to the convolutional layer 626.

[0070] The convolutional layer 626 performs double zero padding on the (2j-1)th feature map 5X and the 2jth feature map 5X, then performs convolution with a 7x7x7 filter at one stride and activates them using an activation function. This generates 16 feature maps 5Y. These feature maps 5Y are sent to the convolutional layer 627.

[0071] The convolutional layer 627 performs double zero padding on the 16 feature maps 5Y, convolves them with a 3x3x3 filter at one stride, and activates them using an activation function. This generates one feature map 5Z, which the convolutional layer 627 outputs as prediction data.

[0072] The machine learning unit 405 calculates the error between the light fluence distribution 52 of the first data set 50, i.e., the correct data, and the feature map 5Z, i.e., the predicted data, using a loss function. Then, the calculated error is input into an optimization function to update the weight coefficients and biases of the neural network 60. This completes training based on the first data set 50. Note that, in order to evaluate the absolute error as the error, for example, MSELoss is used as the loss function. Also, Adam (Adaptive Momentum Estimation) is used as the optimization function.

[0073] Then, the machine learning unit 405 trains the neural network 60 by similarly using the second and subsequent data sets 50. However, the learning rate η t is decayed after each learning. η t =η min +(η max -η min )(1+cos((T cur / T cur )π)) / 2 …(1)

[0074] If it is verified in the verification step that the trained neural network 67 has a certain level of accuracy, the machine learning unit 405 stores the trained neural network 60 as the trained neural network 67 in the trained model storage unit 406.

[0075] However, if a certain level of accuracy is not achieved, the machine learning unit 405 prepares a new dataset 50 and performs training and validation steps until a certain level of accuracy is achieved.

[0076] [4. Reasoning] Once the machine learning is complete, the light fluence distribution when laser light is irradiated onto the brain of a patient (subject) in actual PDT can be inferred using the trained neural network 67. Below, the inference process will be explained using as an example a case where PDT is performed using laser light with the same conditions (radius, intensity, wavelength, etc.) as those used in machine learning, i.e., laser light 43.

[0077] A medical staff member, such as a doctor, measures the three-dimensional shape of the part of the patient's brain that is to be treated using a three-dimensional measurement device 3. This results in a three-dimensional tissue model 54 of that part. The three-dimensional tissue model 54 is then sent from the three-dimensional measurement device 3 to the learning and inference device 2.

[0078] Three-dimensional tissue model 54 has the same size as three-dimensional tissue model 51. Therefore, in this embodiment, it is composed of 32 × 32 × 32 voxels, and each voxel is a cube with each side measuring 0.1 mm. Alternatively, three-dimensional measuring device 3 may acquire a three-dimensional model larger than three-dimensional tissue model 51, and learning and inference device 2 may trim this three-dimensional model to the above size to acquire three-dimensional tissue model 54.

[0079] The optical fluence inference unit 407 infers the optical fluence distribution of the three-dimensional tissue model 54 by inputting the three-dimensional tissue model 54 to the input layer of the trained neural network 67 and obtaining an output value from the output layer of the trained neural network 67. Hereinafter, this output value will be referred to as the "optical fluence distribution 55."

[0080] The inference result output unit 408 colors each voxel of the three-dimensional tissue model 54 with a color or density corresponding to the value (light fluence) of each voxel shown in the light fluence distribution 55. This results in a light distribution model 56. Then, a rendering image 571 obtained by rendering the light distribution model 56 in accordance with the orientation specified by the medical staff is displayed on the display 27. Alternatively, a cross-sectional image 572 obtained by cross-sectioning the light distribution model 56 at an arbitrary position may be displayed.

[0081] Medical staff can perform PDT while viewing the light distribution model 56 displayed on the display 27.

[0082] 5. Overall Processing Flow and Effects of This Embodiment FIG. 10 is a flowchart illustrating an example of the overall processing flow by the learning and inference program 40.

[0083] Next, the overall processing flow of the learning and inference device 2 will be described with reference to a flowchart.

[0084] The learning and inference device 2 executes the process of constructing an AI and the process of inferring how the laser light 43 reaches the brain of a patient based on the learning and inference program 40 in the procedure shown in FIG.

[0085] In the dataset preparation phase, the learning and inference device 2 generates a dataset 50 (#11 in FIG. 10) and stores it (#12). By performing machine learning based on the dataset 50, the learning and inference device 2 generates a trained neural network 67 as a trained model (#13) and stores it (#14).

[0086] Then, in the PDT, when the learning and inference device 2 receives the three-dimensional shape of the patient's brain, i.e., the three-dimensional tissue model 54, from the three-dimensional measurement device 3 (#15), it inputs the three-dimensional tissue model 54 into the trained neural network 67 to calculate the light fluence distribution 55 (#16). This allows the way in which the laser light 43 reaches the patient's brain to be inferred.

[0087] Furthermore, the learning and inference device 2 generates a light distribution model 56 based on the light fluence distribution 55 (#17), and renders or cross-sections the light distribution model 56 according to the specified direction to display a rendered image 571 or a cross-sectional image 572 (#18).

[0088] According to this embodiment, it is possible to infer, faster than conventional methods, the intensity at which laser light irradiated onto the human brain reaches each part of the brain using a computer with low specifications.

[0089] Furthermore, since a learned model is generated by training a U-Net such as the neural network 60, it is computationally efficient and can perform inference at the speed of light.

[0090] [6. Modifications and Applications] Fig. 11 is a diagram showing an example of a mini-batch 58. Fig. 12 is a diagram showing examples of attributes of each region when a voxel group is divided into four regions. Fig. 13 is a diagram showing examples of attributes of each region when a voxel group is divided into five regions.

[0091] In this embodiment, in the machine learning phase, the neural network 60 is trained by updating its parameters, such as weight coefficients and biases, for each set of data sets 50. However, with this method, even if a certain data set 50 contains outliers, noise, or abnormal values, the data set 50 is used as is, which may have a negative impact on the neural network 60. In other words, this method is vulnerable to outliers and the like. Furthermore, since the number of updates is large, training takes a long time.

[0092] To overcome these drawbacks, machine learning may be performed using mini-batch learning. The inventors have trained the neural network 60 by setting the batch size to 4 and 32, respectively, and found that the neural network 60 trains better in the former case than in the latter case.

[0093] Therefore, the learning inference system 1 may perform machine learning using mini-batch learning with a batch size of "4." That is, the learning inference system 1 randomly groups 80,000 data sets 50 into groups of four, thereby generating 20,000 mini-batches 58 as shown in FIG. 11. For each mini-batch 58, the gradient of the three-dimensional tissue model 51 for each of the four data sets 50 that make it up is calculated, and the parameters of the neural network 60 are updated by the average of the four gradients. Therefore, the number of epochs is 20,000, and the neural network 60 is trained 20,000 times.

[0094] Incidentally, in estimating the effect of PDT treatment, the intensity of voxels that are thought to be strongly irradiated with laser light (the intensity of light irradiation) in the three-dimensional tissue model 51 is important. However, since such voxels are limited to a small fraction of the whole, the intensity of such voxels is likely to be evaluated as an outlier.

[0095] To solve this problem, the learning inference system 1 may divide the voxels used to calibrate the three-dimensional tissue model 51 into several groups according to intensity, calculate the error for each group, and calculate the sum of these errors as the error for the entire three-dimensional tissue model 51. For example, a loss function such as Loss_fnc in the following equation (2) may be prepared, and the three-dimensional tissue model 51 may be divided into four groups, a first group, a second group, a third group, and a fourth group, and the errors may be calculated. Loss_fnc=α4×MAPE4+α1×MSE1+α2×MSE2+α3×MSE3…(2)

[0096] As shown in FIG. 12, the first group is a group whose common logarithm of intensity (Fluence) is equal to or greater than "1". The second group is a group whose common logarithm of intensity is equal to or greater than "-1" and less than "1". The third group is a group whose common logarithm of intensity is equal to or greater than "-3" and less than "-1". The fourth group is a group whose common logarithm of intensity is less than "-3". The unit of intensity is "J / mm 3 " α1 to α4 are weighting coefficients for the first to fourth groups, respectively, and α3>α2>α1>α4.

[0097] "MSE1", "MSE2", and "MSE3" are the mean squared errors of the first, second, and third groups, respectively. "MAPE4" is the mean absolute percentage error of the fourth group. Thus, "10 -3 The error can be calculated more precisely by taking into account small intensities below 0.01. Also, the fourth group of errors can be better evaluated by applying MAPE (Mean Absolute Percentage Error) instead of MSE (Mean Squared Error).

[0098] The numbers of voxels in each of the first to fourth groups shown in FIG. 12 are an example of the number of voxels belonging to each group in a certain three-dimensional tissue model 51. As can be seen from this example, the number of voxels in the third group accounts for the majority, at 76% of the total, the number of voxels in the second group is approximately 22% of the total, and the number of voxels in the first group is less than 1% of the total. In consideration of these differences in share, MSE3, MSE2, and MSE1 are weighted in this order. For example, α3=10 8 , α2=10 6 , α1=10 3 It is set as follows.

[0099] The number of voxels in the fourth group is less than 1% of the total. In consideration of the fact that MAPE is used instead of MSE as the error for the fourth group, the share of the number of voxels in the fourth group, and the importance of the fourth group, the weighting coefficient α4 of MAPE4 is set to a coefficient smaller than α1, for example, "1."

[0100] The weighting coefficients α1, α2, α3, and α4 may be adjusted according to the share rates of the first group, the second group, the third group, and the fourth group, respectively.

[0101] Alternatively, the range of the common logarithm of the intensity of each of the first, second, third, and fourth groups is not limited to the above example. For example, the range of the common logarithm of each intensity may be set to a range of 0.8 or greater, a range of -1.2 or greater but less than 0.8, or a range of -3.2 or greater but less than -1.2, or less than -3.2. The range of each group may be set according to the target market share of each group. In the example of FIG. 12, the market shares of the first to fourth groups are approximately 0.07%, approximately 22.9%, approximately 76.3%, and approximately 0.72%, respectively.

[0102] Alternatively, the voxels of the three-dimensional tissue model 51 may be divided into three groups, or five or more groups.

[0103] For example, as shown in FIG. 13, the three-dimensional tissue model 51 may be divided into five groups, namely, a first group to a fifth group, and the error may be calculated based on a loss function such as Loss_fnc in the following equation (3). Loss_fnc=β1×MAPE5+β2×MSE6+β3×MSE7+β4×MSE8 +β5×MSE9…(3)

[0104] As can be seen by comparing this example with the previous example (see FIG. 12) where the images were divided into four groups, the second group in the previous example is further divided into two groups using a threshold value of "0."

[0105] That is, the first group is a group in which the common logarithm of intensity (Fluence) is equal to or greater than "1". The second group is a group in which the common logarithm of intensity is equal to or greater than "0" and less than "1". The third group is a group in which the common logarithm of intensity is equal to or greater than "-1" and less than "0". The fourth group is a group in which the common logarithm of intensity is equal to or greater than "-3" and less than "-1". The fifth group is a group in which the common logarithm of intensity is less than "-3". The unit of intensity is "J / mm 3 β1 to β5 are weighting coefficients for the first to fifth groups, respectively, and the order is β4>β3>β2>β1>β5.

[0106] "MSE5", "MSE6", "MSE7", and "MSE8" are the mean squared errors of the first, second, third, and fourth groups, respectively. "MAPE9" is the mean absolute percentage error of the fifth group.

[0107] 13 is an example of the number of voxels belonging to each group in a certain three-dimensional tissue model 51. In addition, when the weighting coefficient is, for example, β1=10 3 , β2=10 5 , β3=10 6 , β4=10 10 , β5=1. The range of each group can be set according to the target share rate of each group. In the example of Figure 13, the share rates of the first to fifth groups are approximately 0.09%, approximately 1.69%, approximately 24.4%, approximately 73.5%, and approximately 0.36%, respectively.

[0108] Fig. 13 is a diagram showing an example of splitting of laser light 44. Fig. 14 is a diagram showing an example of splitting of laser light 45. Fig. 15 is a diagram showing an example of splitting of laser light 47.

[0109] In this embodiment, the learning and inference system 1 performed each process assuming that the laser beam 43 would be used in both the machine learning phase and the inference phase. However, it is also possible to infer a light distribution model 56 in the inference phase when a laser beam with a radius larger than that of the laser beam 43 is used.

[0110] Specifically, a laser beam with a radius greater than 0.1 mm is divided into multiple laser beams (segments) with a radius of 0.1 mm. An optical fluence distribution 55 is calculated for each segment based on a trained neural network 67. The calculated optical fluence distributions 55 are then summed up and used as the inference result, i.e., the target optical fluence distribution.

[0111] For example, when a laser beam 44 with a radius of 0.3 mm is used, the learning and inference system 1 divides the laser beam 44 into seven segments (laser beams 43) as shown in Fig. 14. In other words, the laser beam 43 is regarded as a bundle of eight laser beams 43. An optical fluence distribution 55 obtained when each of the eight laser beams 43 is irradiated onto a three-dimensional tissue model 54 is calculated based on the trained neural network 67. Then, the sum of the calculated eight optical fluence distributions 55 is adopted as the inference result, i.e., the target optical fluence distribution.

[0112] It can be said that the cross-sectional area of ​​the laser beam 44 is approximately equal to, but not exactly equal to, the sum of the cross-sectional areas of the eight laser beams 43. Therefore, even with the above-described method, it is possible to obtain a light fluence distribution that is sufficiently useful for PDT.

[0113] Note that trained neural networks 67 corresponding to several laser beams with smaller radii (e.g., laser beams with 0.03 mm, 0.05 mm, 0.07 mm, etc.) may be generated in the same manner as above, in addition to the trained neural network 67 corresponding to laser beam 43, i.e., laser beam with a radius of 0.1 mm. The learning and inference system 1 may then divide a laser beam with a large radius such as laser beam 44 into several laser beams with smaller radii and infer the light fluence distribution using each trained neural network 67.

[0114] For example, as shown in Figure 15, a laser beam 45 with a radius of 0.2 mm is considered to be a bundle of one laser beam 43 and nine laser beams 46 with a radius of 0.05 mm. The learning and inference system 1 calculates an optical fluence distribution 55 when one laser beam 43 is irradiated based on a trained neural network 67 for the laser beam 43. Furthermore, the learning and inference system 1 calculates an optical fluence distribution 55 when each of nine laser beams 46 is irradiated based on a trained neural network 67 for the laser beam 46. The sum of the calculated ten optical fluence distributions 55 is then adopted as the inference result, i.e., the target optical fluence distribution.

[0115] Alternatively, the laser beam used in PDT may be divided into laser beams (segments) each having a rectangular cross section. For example, a laser beam 47 as shown in FIG. 16 is divided into laser beams 48 (segments) each having a square cross section. Note that when divided in this manner, there are some parts that are less than a single segment. If each of these parts were replaced with a segment, the sum of the segments would clearly be greater than the laser beam 47. Therefore, it is possible to exclude from among all the segments those whose distance between a predetermined position of the segment itself (e.g., the lower left vertex) and the center of the laser beam 47 exceeds the radius of the laser beam 47. In the example of FIG. 16, the hatched segments are excluded.

[0116] Additionally, a trained neural network 67 for the laser light 48 is generated in advance in the above manner.

[0117] Then, for the remaining segments, i.e., 115 of the 137 segments shown in Figure 16, the light fluence distribution 55 is calculated based on the trained neural network 67, and the sum of the calculated multiple light fluence distributions 55 is adopted as the inference result, i.e., the target light fluence distribution.

[0118] Furthermore, trained neural networks 67 may be generated and used separately for laser light having different conditions such as output intensity and wavelength in addition to radius. In particular, trained neural networks 67 with multiple sets of output intensity and radius may be generated and used separately for each laser light having a wavelength corresponding to each PS (photosensitizer).

[0119] 3 are provided in the learning and inference device 2, they may also be provided in a plurality of devices. For example, the three-dimensional model generation unit 401, the light fluence calculation unit 402, and the dataset registration unit 403 may be provided in a preparation computer, the dataset storage unit 404 and the machine learning unit 405 may be provided in a machine learning computer, and the trained model storage unit 406, the light fluence inference unit 407, and the inference result output unit 408 may be provided in the inference computer. In this case, these computers may be connected via a communication line such as the Internet, a LAN (Local Area Network), or a public line, and the dataset 50, the trained neural network 67, and the like may be exchanged between the computers.

[0120] The preparation computer and the machine learning computer may be implemented using cloud computing, and the inference computer may be implemented using a personal computer. The preparation computer and the machine learning computer require high specifications because the processing is computationally intensive, but the inference computer does not require as much computational effort, so a computer with low specifications may suffice. Therefore, the inference computer may be installed in a medical setting, such as an operating room, and easily used by medical staff during PDT. Alternatively, an inference computer may be installed at each medical setting.

[0121] In this embodiment, the learning inference system 1 generates the trained neural network 67 by training the neural network 60 shown in Fig. 7, but it may also be generated by training a U-Net with another configuration, or by training a neural network other than a U-Net.

[0122] In this embodiment, the learning and inference system 1 generates a trained neural network 67 for the human brain and infers the light fluence distribution 55, but the above-described method can also be used to generate trained neural networks for other human tissues (e.g., gastrointestinal tract, small intestine, large intestine, kidney, liver, etc.) and infer light fluence distributions. Also, trained neural networks can be generated for each tissue of other animals and infer light fluence distributions.

[0123] Additionally, the overall or individual configurations of the learning inference system 1 and the learning inference device 2, the content of the processing, the order of processing, the configuration of the neural network, and the like can be modified as appropriate in accordance with the spirit of the present invention. [Explanation of symbols]

[0124] 1 Learning Inference System 27 Display (display means) 401 Three-dimensional model generation unit (acquisition means) 402 Light fluence calculation unit (acquisition means) 405 Machine Learning Department (Learning Method) 407 Light fluence distribution inference section (inference means) 408 Inference result output section (display means) 43 Laser light (reference laser light) 44 Laser light 45 Laser light 50 datasets 51 3D tissue model (3D model for learning) 52 Light Fluence Distribution (Learning Achievement Level) 54 Three-dimensional tissue model (test three-dimensional model) 55 Light fluence distribution (inference attainment degree) 571 rendering images 572 cross-sectional images 60 Neural Network (U-Net) 67 Trained Neural Networks (Trained Models)

Claims

1. an acquisition means for acquiring, as a data set, a training three-dimensional model which is a three-dimensional model of a specific tissue of a living body, and a training reach degree which is the reach degree of the laser light at each position of the training three-dimensional model when the training three-dimensional model is irradiated with the laser light; a learning means for generating a trained model by performing machine learning using the training three-dimensional model and the training achievement level of the data set as input data and correct answer data, respectively; an inference means for inputting a subject three-dimensional model, which is a three-dimensional model of the specific tissue of a subject, into the trained model, and calculating an inferred reach of the laser light at each position of the subject three-dimensional model when the laser light is irradiated onto the subject three-dimensional model; A laser light arrival degree learning and inference system comprising:

2. an acquisition means for acquiring, as a data set, a training three-dimensional model which is a three-dimensional model of a specific tissue of a living body, and a training reach degree which is the reach degree of the laser light at each position of the training three-dimensional model when the training three-dimensional model is irradiated with the laser light; a learning means for generating a trained model by performing machine learning using the training three-dimensional model and the training achievement level of the data set as input data and correct answer data, respectively; A laser light reachability learning system comprising:

3. The learning means generates the learned model by training a convolutional neural network.

3. The laser beam reach learning system according to claim 2.

4. The learning means generates the learned model by training U-Net as the convolutional neural network.

4. The laser beam reach learning system according to claim 3.

5. the learning means calculates an error between a feature map obtained by inputting the training three-dimensional model into the convolutional neural network and the training achievement level, and trains the convolutional neural network so as to reduce the error; 5. The laser beam reach learning system according to claim 3 or 4.

6. the learning means classifies each position of the training three-dimensional model into one of a plurality of groups according to the magnitude of the degree of achievement of the position indicated in the training achievement degree, calculates an error as an individual error for each group using a different loss function or weighting coefficient, and trains the convolutional neural network so as to reduce the sum of the individual errors; 6. The laser beam reach learning system according to claim 5.

7. the plurality of groups includes a top-level group; the learning means classifies the positions of the training three-dimensional model into the highest group based on a threshold value that is set so that positions of the training three-dimensional model whose reach level is equal to or greater than a predetermined value are classified into the highest group; 7. The laser beam reach learning system according to claim 6.

8. The plurality of groups further includes a lowest-level group, the learning means classifies the positions of the training three-dimensional model into the lowest group based on a second threshold value that is set so that positions of the training three-dimensional model whose degree of reach is less than a predetermined value are classified into the lowest group; 8. The laser beam reach learning system according to claim 7.

9. the plurality of groups further includes a first intermediate group and a second intermediate group; the learning means calculates, as the individual error for each of the highest group, the first intermediate group, and the second intermediate group, a value obtained by multiplying the mean square error by a respective weighting factor, and calculates, as the individual error for the lowest group, a value obtained by multiplying the mean absolute percentage error of the lowest group by the weighting factor of the lowest group.

9. The laser beam reach learning system according to claim 8.

10. the plurality of groups further includes a first intermediate group, a second intermediate group, and a third group; the learning means calculates, as the individual error for each of the top group, the first intermediate group, the second intermediate group, and the third group, a value obtained by multiplying the mean square error by a respective weighting factor, and calculates, as the individual error for the lowest group, a value obtained by multiplying the mean absolute percentage error of the lowest group by the weighting factor of the lowest group.

10. The laser beam reach learning system according to claim 9.

11. an inference means for inputting a test three-dimensional model, which is a three-dimensional model of a specific tissue of a subject, into a trained model of the specific tissue of a living body generated by the laser light reachability learning system according to claim 2, and calculating an inferred reachability of the laser light at each position of the test three-dimensional model when the test three-dimensional model is irradiated with laser light; A laser light arrival degree inference system comprising:

12. a display means for displaying an image representing the degree of inference attainment at each position of the test three-dimensional model; having The laser beam arrival degree inference system according to claim 11.

13. the cross-sectional area of ​​the laser light is larger than the cross-sectional area of ​​a reference laser light given as a condition when the laser light reachability learning system generates the trained model, the inference means regards the laser light as a bundle of a plurality of reference laser lights, calculates an inferred reach of the reference laser light at each position of the test three-dimensional model when each of the reference laser lights is irradiated onto the test three-dimensional model using the trained model, and adopts the sum of the calculated inferred reach as the inference result.

13. The laser beam arrival degree inference system according to claim 11 or 12.

14. The cross section of the reference laser light is rectangular. The laser beam arrival degree deduction system according to claim 13.

15. acquiring, as a data set, a training three-dimensional model that is a three-dimensional model of a specific tissue of a living body, and a training reach degree that is a reach degree of the laser light at each position of the training three-dimensional model when the training three-dimensional model is irradiated with the laser light; generating a trained model by performing machine learning using the training 3D model and the training achievement level of the dataset as input data and correct answer data, respectively; A method for learning the degree of laser light arrival, comprising:

16. a test three-dimensional model, which is a three-dimensional model of a specific tissue of a subject, inputted into a trained model of the specific tissue of a living body generated by the laser light reachability learning method according to claim 15, thereby calculating an inferred reachability of the laser light at each position of the test three-dimensional model when the test three-dimensional model is irradiated with laser light; A method for estimating the degree of laser light arrival, comprising:

17. causing a computer to execute an acquisition process for acquiring, as a data set, a training three-dimensional model which is a three-dimensional model of a specific tissue of a living body, and a training reach degree which is a reach degree of the laser light at each position of the training three-dimensional model when the training three-dimensional model is irradiated with the laser light; causing the computer to execute a generation process of generating a trained model by performing machine learning using the training 3D model and the training achievement level of the data set as input data and correct answer data, respectively; A computer program characterized by:

18. a test three-dimensional model, which is a three-dimensional model of a specific tissue of a subject, is input into a trained model of the specific tissue of a living body generated by the laser light reachability learning method according to claim 15, and the computer is caused to execute a calculation process of calculating an inferred reachability of the laser light at each position of the test three-dimensional model when the test three-dimensional model is irradiated with laser light; A computer program characterized by:

Images