Organ deformation estimation device, treatment device, treatment assistance device, organ deformation estimation method, and program

Abstract

This organ deformation estimation device is provided with: a three-dimensional model acquisition unit which acquires a three-dimensional model that shows the shape of an organ, the three-dimensional model being generated on the basis of a three-dimensional image of the organ which has been captured previously before the implementation of a treatment; a captured image acquisition unit which acquires a captured image that is an image of the inside of the organ which has been captured during the process of the treatment; a partial image extraction unit which extracts a partial image from the three-dimensional image; an alignment unit which performs the alignment of the captured image with the partial image; a target position calculation unit which calculates a target position on the basis of the result of the alignment; and a displacement estimation unit which estimates the deformation of the organ during the process of the treatment by deforming the three-dimensional model on the basis of a three-dimensional simulation in such a manner that the position of a part corresponding to the inside among parts constituting the three-dimensional model is displaced to the target position.

Description

Organ deformation estimation device, treatment device, treatment support device, organ deformation estimation method, and program

[0001] The present invention relates to an organ deformation estimation device, a treatment device, a treatment support device, an organ deformation estimation method, and a program. This application claims priority to Japanese Patent Application No. 2022-119514, filed on July 27, 2022, the contents of which are incorporated herein by reference.

[0002] Radiation therapy is currently widely used in cancer treatment because it can be performed without incising the body. However, the therapeutic results of radiation therapy depend on the intensity of the radiation. Some organs cannot be exposed to high doses of radiation. For example, the pancreas is an organ located mostly behind the stomach and spreads widely from the duodenum to the spleen. In addition, the opening of the pancreatic duct is located in the duodenum, so the pancreas and duodenum are particularly closely related. Therefore, high doses of radiation cannot be applied to the pancreas due to the risk of accidentally irradiating other organs, making radiation therapy for pancreatic cancer more difficult than for cancers of other organs.

[0003] In recent years, a device (MR-Linac) that performs radiation therapy while capturing MR (Magnetic Resonance) images in real time has been put into practical use, and it is expected that high-dose radiation therapy will be possible for the pancreas. However, only a few cross sections can be captured in real time, and it is difficult to change pre-planned labels in accordance with the movement of the organ while taking into account contact with surrounding organs and spontaneous deformation. Therefore, it is necessary to estimate the displacement of the entire model from a subset of images.

[0004] Research into MR image-guided radiation therapy is currently underway. For example, a technique for estimating organ displacement using machine learning is known (see Patent Document 1 or Non-Patent Document 1). This technique for estimating organ displacement using machine learning simultaneously learns data from a large number of patients to represent a variety of movements.

[0005] International Publication No. 2020 / 054503

[0006] "Medical Image Computing and Computer Assisted Intervention - MICCAI 2021, Lecture Notes in Computer Science", September 21, 2021, Vol. 12904, pp. 238-248

[0007] However, the deformation of the pancreas, which is affected by multiple surrounding organs, is aperiodic and varies from patient to patient. This makes it difficult to extract features using machine learning, and it is thought that a huge amount of data set would be required for training. There is a need to be able to estimate organ displacement from a small number of captured images during the treatment process.

[0008] The present invention has been made in consideration of the above points, and provides an organ deformation estimation device, a treatment device, a treatment support device, an organ deformation estimation method, and a program that can estimate organ displacement from a small number of captured images during the treatment process.

[0009] the organ deformation estimation device includes: a three-dimensional model acquisition unit that acquires a three-dimensional model that is generated based on three-dimensional images of the organ taken in advance before treatment, and that shows the shape of the organ before the treatment is performed; a photographed image acquisition unit that acquires photographed images, which are two-dimensional images of the inside of the organ taken during the treatment; a partial image extraction unit that extracts from the three-dimensional image a partial image, which is a two-dimensional image of a portion included in the three-dimensional image that corresponds to a position of the inside of the organ photographed in the photographed image; a registration unit that aligns pixels that make up the photographed image with pixels that make up the partial image so that pixels that show the same part of the inside correspond to each other; a target position calculation unit that calculates a target position for displacement of a part of the three-dimensional model that corresponds to the inside of the organ based on the result of the registration; and a displacement estimation unit that estimates deformation of the organ during the treatment by deforming the three-dimensional model based on a three-dimensional simulation so as to displace the position of the part of the three-dimensional model that corresponds to the inside of the organ to the target position.

[0010] In one aspect of the present invention, in the organ deformation estimation device, the organ comprises an organ that is a target of treatment and another organ adjacent to the target organ.

[0011] In one aspect of the present invention, the organ deformation estimation device further includes a three-dimensional image acquisition unit that acquires the three-dimensional image.

[0012] In one aspect of the present invention, in the organ deformation estimation device, the displacement estimation unit performs the three-dimensional simulation based on a meshfree method.

[0013] In one aspect of the present invention, the organ deformation estimation device further comprises a slice selection unit that selects a slice to be captured as the captured image from within the organ.

[0014] Furthermore, in one aspect of the present invention, in the organ deformation estimation device, the tomography selection unit selects the imaging tomography from among candidate tomography sections for one or more tomography directions and one or more positions in the tomography directions, based on an error in the estimated result of the organ deformation estimated when a two-dimensional image obtained for the candidate tomography section from ground truth data, which is a three-dimensional image obtained by randomly applying deformation to the three-dimensional image, is used instead of the imaging image, relative to the ground truth data.

[0015] Another aspect of the present invention is a treatment device including the organ deformation estimation device.

[0016] Another aspect of the present invention is a medical treatment support device including the organ deformation estimation device.

[0017] and a displacement estimation step of estimating the deformation of the organ during the treatment by deforming the three-dimensional model based on a three-dimensional image of the organ taken in advance at a time before the treatment is performed, the three-dimensional model being generated based on three-dimensional images of the organ taken in advance at a time before the treatment is performed, and the three-dimensional model being indicative of the shape of the organ before the treatment is performed; a photographed image acquisition step of acquiring photographed images, which are two-dimensional images of the interior of the organ taken during the treatment; a partial image extraction step of extracting from the three-dimensional image, partial images, which are two-dimensional images of portions included in the three-dimensional image that correspond to positions of the interior of the organ photographed in the photographed image, from the three-dimensional image; a registration step of aligning pixels constituting the photographed image with pixels constituting the partial image so that pixels indicating the same portion of the interior correspond to each other; a target position calculation step of calculating a target position for displacement of a portion of the three-dimensional model that corresponds to the interior of the organ based on the result of the registration; and a displacement estimation step of estimating the deformation of the organ during the treatment by deforming the three-dimensional model based on a three-dimensional simulation so as to displace the position of the portion of the three-dimensional model that corresponds to the interior of the organ to the target position.

[0018] and a displacement estimation step of estimating deformation of the organ during the treatment by deforming the three-dimensional model based on a three-dimensional image of the organ taken in advance before the treatment is performed. The three-dimensional model acquisition step acquires a three-dimensional model showing the shape of the organ before the treatment is performed, the three-dimensional model being generated based on three-dimensional images of the organ taken in advance before the treatment is performed, the three-dimensional model being generated based on three-dimensional images of the organ taken in advance before the treatment is performed, the three-dimensional model being generated based on three-dimensional images of the organ taken in advance before the treatment is performed, the three-dimensional model being generated based on three-dimensional images of the organ taken in advance before the treatment is performed, the three-dimensional model being generated based on three-dimensional images of the organ taken in advance before the treatment is performed, the three-dimensional model being generated based on three-dimensional images of the organ

[0019] According to the present invention, the displacement of an organ can be estimated from a small number of captured images during the course of treatment.

[0020] FIG. 1 is a diagram illustrating an overview of organ deformation estimation processing according to a first embodiment of the present invention. FIG. 2 is a diagram for explaining an overview of the Material Point Method according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating an example of the functional configuration of an organ deformation estimation system according to the first embodiment of the present invention. FIG. 4 is a diagram illustrating an example of the flow of organ deformation estimation processing according to the first embodiment of the present invention. FIG. 5 is a diagram illustrating an example of an overall algorithm of a 3D simulation according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating parameters according to an example of the first embodiment of the present invention. FIG. 7 is a diagram illustrating a 3D model before being driven according to the example of the first embodiment of the present invention. FIG. 8 is a diagram illustrating a 3D model after being driven according to the example of the first embodiment of the present invention. FIG. 9 is a diagram illustrating an example of high accuracy among results obtained by applying organ deformation estimation processing according to an example of the first embodiment of the present invention. FIG. 10 is a diagram illustrating an example of low accuracy among results obtained by applying organ deformation estimation processing according to an example of the first embodiment of the present invention. FIG. 11 is a box plot illustrating the distribution of errors in the results obtained by organ deformation estimation processing according to an example of the first embodiment of the present invention. FIG. 12 is a diagram illustrating an example of the functional configuration of an organ deformation estimation system according to a second embodiment of the present invention. FIG. 13 is a diagram illustrating an example of the flow of a fault selection process according to the second embodiment of the present invention when the number of faults is finite and full search is possible. 1 is a diagram for explaining an example of a fault selection process when the fault is finite and cannot be fully searched, or when the fault is not finite, according to the second embodiment of the present invention. FIG. 2 is a diagram showing errors and fault numbers for five examples of correct answer data according to an example of the second embodiment of the present invention. FIG. 3 is a diagram showing a graph of error transition when the fault number is changed according to an example of the second embodiment of the present invention. FIG. 4 is a diagram showing a graph of error transition when the fault number is changed according to an example of the second embodiment of the present invention. FIG. 5 is a diagram showing a graph of error transition when the fault number is changed according to an example of the second embodiment of the present invention. FIG. 6 is a diagram showing a graph of error transition when the fault number is changed according to an example of the second embodiment of the present invention. FIG. 7 is a diagram showing a graph of error transition when the fault number is changed according to an example of the second embodiment of the present invention.FIG. 1 is a graph showing the relationship between the transition of error and the number of particles when the fault number is changed according to an example of the second embodiment of the present invention. FIG. 2 is a graph showing the relationship between the transition of error and the number of particles when the fault number is changed according to an example of the second embodiment of the present invention. FIG. 3 is a graph showing the relationship between the transition of error and the number of particles when the fault number is changed according to an example of the second embodiment of the present invention. FIG. 4 is a graph showing the relationship between the transition of error and the number of particles when the fault number is changed according to an example of the second embodiment of the present invention.

[0021] (First Embodiment) Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the following description, estimating the displacement of each part constituting an organ as a three-dimensional shape is referred to as estimating the deformation of the organ's shape, or estimating the deformation of the organ. Also, a cross section of an organ is referred to as an organ slice. In this embodiment, the process of estimating the deformation of the organ's shape is referred to as organ deformation estimation processing. Also, in each embodiment, an example of a case where organ deformation estimation processing is applied to radiation therapy will be described, but organ deformation estimation processing may also be applied to treatments other than radiation therapy.

[0022] [Outline of Organ Deformation Estimation Processing] FIG. 1 is a diagram illustrating an outline of the organ deformation estimation processing according to this embodiment. In the organ deformation estimation processing, a three-dimensional image A1 of an organ is captured in advance before radiation therapy is performed. A three-dimensional model B1 of the organ is generated from the three-dimensional image A1. In the organ deformation estimation processing, the three-dimensional model B1 is driven by a three-dimensional simulation. A captured image C1 is used when driving the three-dimensional model B1 by the three-dimensional simulation. The captured image C1 is a two-dimensional image of a cross section of the organ captured during radiation therapy. A two-dimensional image of the three-dimensional image A1 corresponding to the cross section captured in the captured image C1 is aligned with the captured image C1. Through this alignment, the displacement of each part of the cross section due to the deformation of the organ during treatment from its position before treatment is calculated. In the organ deformation estimation processing, the part of the three-dimensional model B1 corresponding to the cross section captured in the captured image C1 is displaced by the three-dimensional simulation by the displacement calculated through the alignment. Parts of the three-dimensional model B1 other than the part corresponding to the cross section are displaced following the displacement of the part corresponding to the cross section. As a result, the overall displacement of the three-dimensional shape of the organ is estimated from the photographed image C1 in which a cross section of the organ is photographed.

[0023] In this embodiment, the pancreas is used as an example of an organ whose deformation is to be estimated. The pancreas is adjacent to surrounding organs, and it is generally difficult to estimate the deformation of the organ while taking into account spontaneous movement and contact with the surrounding organs. The organ deformation estimation process estimates the deformation of the pancreas' shape with high accuracy by taking into account the fact that the pancreas is adjacent to the surrounding organs. In the organ deformation estimation process, the pancreas and surrounding organs are each modeled as three-dimensional models, and a three-dimensional simulation is performed. In the organ deformation estimation process, the position information of the pancreas and surrounding organs obtained from the tomographic image is used to perform a three-dimensional simulation that takes into account contact between the pancreas and surrounding organs, thereby estimating the deformation of the organ's shape.

[0024] In the organ deformation estimation process, the Material Point Method (MPM) is used to perform a 3D simulation that takes into account contact between the pancreas and surrounding organs. MPM is a mesh-free method in which each part of the target is replaced with particles and discretized, while calculations are performed using a grid prepared separately from the particles. MPM has the advantage of being able to handle contact between different objects in a natural algorithmic flow because it calculates physical quantities after transmitting information about all particles to grid points.

[0025] FIG. 2 is a diagram for explaining an overview of the MPM according to this embodiment. In the MPM, particles are used to track the position, mass, velocity, deformation gradient, etc. of an object. In other words, the particles carry the physical information of the object. On the other hand, in the MPM, lattice points are used to update particle information based on conservation laws and constitutive laws. The constitutive laws depend on the object in question. On the other hand, the conservation laws are common to all objects, and the law of conservation of mass, the law of conservation of momentum, and the law of conservation of angular momentum are used. The law of conservation of mass and the law of conservation of momentum are expressed by the following equations (1) and (2), respectively.

[0026]

[0027]

[0028] Here, ρ is density, t is time, v is velocity, σ is stress, and g is gravitational acceleration. The law of conservation of angular momentum is guaranteed because the stress tensor is symmetric. These conservation laws are solved on the grid by transmitting the physical information carried by the particles to the grid points. Therefore, MPM can be said to be a calculation method that combines Eulerian and Lagrangian perspectives. Furthermore, because the object is discretized into particles, MPM has the advantages of a particle method.

[0029] Figure 2(A) shows the position of a particle before deformation. Figure 2(B) shows the state in which the physical information carried by the particle has been transmitted to the lattice point. Figure 2(C) shows the state in which the position of the lattice point has been updated based on the conservation law and the constitutive law, and the position of the particle has been updated based on the physical information transmitted from the lattice point to the particle. Figure 2(D) shows the position of the particle after deformation.

[0030] 3 is a diagram showing an example of the functional configuration of the organ deformation estimation system 1 according to this embodiment. The organ deformation estimation system 1 includes an organ deformation estimation device 2, a three-dimensional image supply unit 3, and a captured image supply unit 40. The captured image supply unit 40 is provided in the radiation therapy device 4.

[0031] The organ deformation estimation device 2 performs organ deformation estimation processing. For example, the organ deformation estimation device 2 is a computer such as a personal computer (PC), a workstation, or a server. For example, the organ deformation estimation device 2 is a computer separate from the radiation therapy device 4, but it may also be built into the console of the radiation therapy device 4 and provided integrally with the radiation therapy device 4.

[0032] The three-dimensional image supply unit 3 supplies the three-dimensional image A1 to the organ deformation estimation device 2. The three-dimensional image supply unit 3 is a medical imaging device, such as a magnetic resonance imaging (MRI) device.

[0033] The radiotherapy device 4 is an apparatus (MR-Linac) that performs radiotherapy while capturing MR (Magnetic Resonance) images in real time. Therefore, the radiotherapy device 4 also functions as a medical imaging device. In this embodiment, the three-dimensional image supply unit 3 is separate from the radiotherapy device 4, but the three-dimensional image supply unit 3 may be included in the radiotherapy device 4.

[0034] The three-dimensional image supply unit 3 may be a computed tomography (CT) device. The radiotherapy device 4 may be a device that performs radiotherapy while capturing CT images in real time. The radiotherapy device 4 may be a device (Linac) that performs radiotherapy while capturing images of the organ to be treated with X-rays in real time.

[0035] The organ deformation estimation device 2 includes a control unit 20 and a storage unit 21. The control unit 20 includes, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a RAM (Random Access Memory), and the like, and performs various calculations and information transmission and reception. Each functional unit of the control unit 20 is realized by the CPU (Central Processing Unit) reading a program from a ROM (Read Only Memory) and executing the program. The ROM is included in the storage unit 21.

[0036] The control unit 20 includes a three-dimensional image acquisition unit 200, a three-dimensional model generation unit 201, a three-dimensional model acquisition unit 202, a partial image extraction unit 203, a captured image acquisition unit 204, an alignment unit 205, a target position calculation unit 206, a displacement estimation unit 207, and an output unit 208.

[0037] The three-dimensional image acquisition unit 200 acquires a three-dimensional image A1 from the three-dimensional image supply unit 3. The three-dimensional image A1 is a three-dimensional image of an organ captured in advance before radiation is irradiated to the organ in radiation therapy. In this embodiment, the organ consists of the pancreas, which is the organ targeted for radiation therapy, and other organs adjacent to the pancreas (surrounding organs). The three-dimensional image acquisition unit 200 stores the acquired three-dimensional image A1 in the memory unit 21. Note that the other organs adjacent to the organ targeted for radiation therapy (surrounding organs) may or may not be in contact with the organ targeted for radiation therapy.

[0038] The three-dimensional model generating unit 201 generates a three-dimensional model B1 based on the three-dimensional image A1. The three-dimensional model B1 is a three-dimensional model that represents the shape of an organ before radiation therapy is performed. The three-dimensional model generating unit 201 stores the generated three-dimensional model B1 in the storage unit 21.

[0039] The three-dimensional model acquisition unit 202 acquires the three-dimensional model B1 generated by the three-dimensional model generation unit 201. The three-dimensional model acquisition unit 202 supplies the acquired three-dimensional model B1 to the displacement estimation unit 207.

[0040] The three-dimensional model B1 may be generated by a computer separate from the organ deformation estimation device 2. In this case, the three-dimensional model generation unit 201 may be omitted from the configuration of the organ deformation estimation device 2, and the three-dimensional model acquisition unit 202 acquires the three-dimensional model B1 from the computer.

[0041] The partial image extracting unit 203 extracts a partial image D1 from the three-dimensional image A1. The partial image D1 is a two-dimensional image of a portion included in the three-dimensional image A1 that corresponds to the position of the inside of the organ captured in the captured image C1. In this embodiment, the inside of the organ is a cross-section of the organ.

[0042] In this embodiment, the position of the organ slice is determined in advance. Information indicating the position of the slice is stored in advance in the storage unit 21 as slice position information E1. The organ slice is, for example, a cross section in a direction passing through the center of gravity of the organ (pancreas) to be radiotherapy treated. The cross section passing through the center of gravity may be any cross section, for example, a cross section in any of the three axes when a three-dimensional Cartesian coordinate system is set, a cross section along the patient's body axis, or a coronal or sagittal cross section. The organ slice may be determined so as to maximize the cross section area of ​​the organ (pancreas) to be radiotherapy treated. Furthermore, the direction passing through the center of gravity may be a direction including a position where the organ (pancreas) to be radiotherapy treated and a surrounding organ are in contact with each other. Furthermore, the direction passing through the center of gravity may be a direction in which the distance between the closest parts (particles) of the organ (pancreas) to be radiotherapy treated and the parts of the surrounding organ are shortest.

[0043] The positions of the organ slices may be determined so that the slices include a greater number (or types) of organs (or their labels).The positions of the organ slices may be determined so that the slices include each organ as evenly as possible in terms of area.

[0044] The captured image acquisition unit 204 acquires a captured image C1. The captured image C1 is a two-dimensional image of the inside (tomography) of an organ captured during radiation therapy. In this embodiment, the captured image C1 is a tomography image of a tomography of the organ.

[0045] The registration unit 205 aligns the pixels constituting the photographed image C1 with the pixels constituting the partial image D1 so that pixels showing the same part of the inside (slice) of the organ correspond to each other.

[0046] Based on the result of the alignment performed by the alignment unit 205, the target position calculation unit 206 calculates the target position of displacement for the part that corresponds to the inside (slice) of the organ among the parts that make up the three-dimensional model B1.

[0047] The displacement estimation unit 207 estimates the deformation of the organ during the radiation therapy process based on a three-dimensional simulation. Here, the displacement estimation unit 207 estimates the deformation of the organ during the radiation therapy process by deforming the three-dimensional model B1 based on the three-dimensional simulation so as to displace the position of a portion corresponding to the interior (slope) of the three-dimensional model B1 to the target position calculated by the target position calculation unit 206. The displacement estimation unit 207 performs the three-dimensional simulation based on a meshfree method. One example of the meshfree method is MPM.

[0048] The output unit 208 outputs the result of organ deformation estimated by the displacement estimation unit 207 as an estimation result F1 to the radiotherapy device 4. The estimation result F1 includes, for example, one or more of a deformed tomographic image, a deformed three-dimensional model, and a deformed three-dimensional label. The three-dimensional label indicates the target position within the organ to which radiation is emitted. The three-dimensional model includes, for example, pixel value information of the three-dimensional image A1 and three-dimensional label information. Therefore, when the three-dimensional model is output, it is equivalent to outputting the deformed tomographic image and three-dimensional label information together. The radiotherapy device 4 changes the pre-planned three-dimensional label in real time during the course of radiotherapy based on the estimation result F1.

[0049] The storage unit 21 stores various types of information. The information stored in the storage unit 21 includes a three-dimensional image A1, a three-dimensional model B1, and tomographic position information E1. The storage unit 21 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device.

[0050] The organ deformation estimation device 2 and the 3D image supply unit 3 may communicate via a cable connection, or via a wireless network such as a LAN (Local Area Network). The 3D image A1 captured by the 3D image supply unit 3 may be stored in an external storage device and then supplied to the organ deformation estimation device 2 via the external storage device. In this case, the 3D image A1 may be transferred from the external storage device to the organ deformation estimation device 2 by the user.

[0051] The organ deformation estimation device 2 and the radiotherapy device 4 may be connected by a cable to communicate with each other, or may communicate with each other via a wireless network such as a LAN.

[0052] 4 is a diagram showing an example of the flow of organ deformation estimation processing according to this embodiment. The organ deformation estimation processing is made up of pre-treatment processing, calibration, and in-treatment processing.

[0053] Step S10: The organ deformation estimation system 1 executes pre-treatment processing. The pre-treatment processing is executed before radiation therapy is performed. The pre-treatment processing is processing for creating a three-dimensional model B1 from the three-dimensional image A1. The period before radiation therapy is performed is, for example, one day before radiation therapy is performed. It is required that the state of the organ changes as little as possible between the period before radiation therapy and the course of radiation therapy. Therefore, it is preferable that the pre-treatment processing be performed as soon as possible before radiation therapy. Furthermore, the pre-treatment processing may be performed after radiation therapy has started, for example, as long as it is performed before radiation is irradiated onto the organ in radiation therapy. In this case, the three-dimensional image A1 is captured in advance after radiation therapy has started and before calibration. As the pre-treatment processing, the processes from step S110 to step S140 are executed.

[0054] Step S110: The three-dimensional image supply unit 3 captures a three-dimensional image A1. The three-dimensional image supply unit 3 captures a three-dimensional image A1 of an organ by, for example, MRI.

[0055] Step S120 : The three-dimensional image acquisition unit 200 acquires the three-dimensional image A1 from the three-dimensional image supply unit 3 .

[0056] Step S130: The 3D model generation unit 201 extracts organ contours from the 3D image A1 based on one or more processes of segmentation and contouring. In this embodiment, the contours of the pancreas and the stomach as a surrounding organ are extracted. The contours include the external contour and the internal contour of the organ.

[0057] Step S140: The three-dimensional model generation unit 201 generates a three-dimensional model B1 based on the extracted outlines of the organs. In this embodiment, the three-dimensional model B1 is a three-dimensional model of the pancreas and stomach. As an example, the three-dimensional model B1 is a three-dimensional model in which each part of an organ is replaced with a particle to discretize the shape of the organ. If the three-dimensional model B1 includes multiple organs, it is possible to identify which of the multiple organs each particle included in the three-dimensional model B1 belongs to by assigning a label indicating the organ to the particle. In one example of this embodiment, it is possible to identify which of the pancreas and stomach each particle included in the three-dimensional model B1 belongs to. The three-dimensional model generation unit 201 stores the generated three-dimensional model B1 in the storage unit 21.

[0058] Step S20: The organ deformation estimation system 1 performs calibration. The calibration is performed at the beginning of radiation therapy. The beginning of radiation therapy refers to, for example, immediately after radiation therapy is initiated. The 3D image A1 and the captured images C1 taken during the treatment are captured by different devices (in this embodiment, the 3D image supply unit 3 and the radiation therapy device 4). Furthermore, the organ may be captured in different positions between the 3D image A1 and the captured images C1 taken during the treatment. Therefore, in order to perform a 3D simulation of the organ, it is necessary to associate the captured images C1 taken during the treatment with the 3D model B1 in terms of the positions of each part of the organ. In the calibration, a part of the 3D model B1 that corresponds to the cross section captured in the captured image C10 is selected. Calibration involves the execution of the processes from step S210 to step S240.

[0059] Step S210: The radiotherapy device 4 captures a photographed image C10 at the beginning of radiotherapy. The radiotherapy device 4 captures a cross section of an organ as the photographed image C10 by, for example, MRI.

[0060] Step S220: The captured image acquisition unit 204 acquires the captured image C10 from the radiotherapy device 4.

[0061] Step S230: The partial image extraction unit 203 extracts a partial image D1 from the three-dimensional image A1, which is a two-dimensional image of a portion of the three-dimensional image A1 that corresponds to the position of the tomographic slice captured in the captured image C10. Here, the partial image extraction unit 203 reads the three-dimensional image A1 and the tomographic slice position information E1 from the storage unit 21. Based on the read three-dimensional image A1, the tomographic slice position information E1, and the captured image C10 acquired by the captured image acquisition unit 204, the partial image extraction unit 203 selects a portion of the three-dimensional image A1 that corresponds to the internal position of the organ captured in the captured image C10. The portion included in the three-dimensional image A1 is selected as a two-dimensional image included in the three-dimensional image A1. The process of selecting the portion may include, for example, a process of determining a direction in the three-dimensional image A1 that corresponds to the direction of the normal to the plane of the tomographic slice captured in the captured image C10.

[0062] Step S240: The partial image extraction unit 203 selects a portion that corresponds to the cross section captured in the captured image C10 from among the portions that make up the three-dimensional model B1.

[0063] Step S30: The organ deformation estimation system 1 executes in-treatment processing. In the in-treatment processing, a 3D simulation of the 3D model B1 is performed. The in-treatment processing is repeatedly performed during the course of radiation therapy. In this embodiment, the in-treatment processing is performed in real time each time an image C1 of a tomographic image of an organ is captured by the radiation therapy device 4. Note that the in-treatment processing does not have to be performed for each captured image C1. In order to reduce the processing load on the organ deformation estimation device 2, the in-treatment processing may be performed, for example, once every N (N is a natural number equal to or greater than 2) captured images C1. The in-treatment processing may also be performed at least once during the course of treatment. As the in-treatment processing, each of the processes from step S310 to step S350 is executed.

[0064] Step S310: The radiotherapy device 4 captures a photographed image C11 during the course of radiotherapy. The radiotherapy device 4 captures a cross section of an organ as a photographed image C11 by, for example, MRI.

[0065] Step S320: The captured image acquisition unit 204 acquires the captured image C11 from the radiotherapy apparatus 4.

[0066] Step S330: The alignment unit 205 aligns the captured image C11 with the partial image D1. Here, the alignment unit 205 aligns the pixels constituting the captured image C11 with the pixels constituting the partial image D1 so that pixels representing the same part of the cross section of the organ correspond to each other. The partial image D1 is the two-dimensional image selected in the calibration step S230.

[0067] In the registration, a displacement u(x) is calculated between a d-dimensional image IF(x) (where x is an element of a d-dimensional real space) and another image IM(x) (where x is also an element of a d-dimensional real space) so that equation (3) is satisfied.

[0068]

[0069] That is, the alignment is reduced to the problem of finding a transformation function T that satisfies equation (4).

[0070]

[0071] However, in reality, two images rarely match perfectly, so an optimal transformation function T (T with a hat in equation (5)) as expressed by equations (5) and (6) is determined.

[0072]

[0073]

[0074] That is, the registration is formulated as a minimization problem with respect to the deformation function T (T with a hat in equation (5)). In equations (5) and (6), C is a similarity function, P is a penalty function, and γ is a weight for the penalty.

[0075] The degree of freedom of deformation varies depending on the type of function assumed for the deformation function T (T with a hat in equation (5)), and there are rigid deformations that allow only translation and rotation, and non-rigid deformations. There are many models of non-rigid deformations, such as those that assume affine transformations or B-spline transformations. In this embodiment, as an example, it is assumed that the deformation function T (T with a hat in equation (5)) is an affine transformation, but it may also be a rigid transformation, a B-spline transformation, or other transformations.

[0076] A predetermined 3D medical image registration library may be used for the registration. In this embodiment, Elastic is used as an example of the 3D medical image registration library. Elastic is a toolbox that supports multiple medical images and can align two-dimensional images using various types of deformations and evaluation indices, regardless of whether they are rigid or non-rigid. MI (Mutual Information) is used as the evaluation indices. By using Elastic, the affine transformation matrix A and translation vector t shown in Equation (7) can be obtained.

[0077]

[0078] The affine transformation matrix A and the translation vector t are used as inputs to the three-dimensional simulation: where x is the pixel position vector and c is the position vector of the image center.

[0079] Step S340: The target position calculation unit 206 calculates a target position for displacement of a portion of the three-dimensional model B1 that corresponds to a slice of the organ, based on the result of the alignment by the alignment unit 205. The target position calculation unit 206 calculates a target position of a particle present in a portion of the three-dimensional model B1 that corresponds to the slice captured in the captured image C11. As an example, the target position calculation unit 206 uses the above-mentioned affine transformation matrix A and translation vector t to calculate the target position.

[0080] Step S350: The displacement estimation unit 207 estimates the deformation of the pancreas during the radiation therapy process based on the 3D simulation using MPM. In the 3D simulation using MPM, contact between organs can be handled without changing the MPM algorithm from the case where there is no contact between organs.

[0081] In this embodiment, the pancreas and stomach are modeled as linear elastic bodies, and one particle is assigned to each pixel. Note that N particles may be assigned to M pixels (M and N are natural numbers). The three-dimensional model B1 is driven by applying a force expressed by equation (8) to each particle until the difference between the current position and the target position falls below a threshold.

[0082]

[0083] Here, xdiff is given by equation (9).

[0084]

[0085] where xtar is the target position, xcur is the position vector of the current position, Kp is the coefficient for the proportional term, Ki is the coefficient for the integral term, and Kd is the coefficient for the differential term. The overall algorithm for the three-dimensional simulation is shown in FIG.

[0086] The 3D simulation by the displacement estimation unit 207 takes into consideration contact between the organ targeted by radiation therapy (pancreas) and the surrounding organ (stomach). Therefore, the organ deformation estimation process makes it possible to express, as deformation of the organ targeted by radiation therapy, the positional relationship with the surrounding organs and non-periodic deformation due to the movement of the surrounding organs.

[0087] Step S360: The output unit 208 outputs the estimation result F1 to the radiation therapy device 4. The three-dimensional labels included in the estimation result F1 indicate pixels in the three-dimensional image A1 of the organ that correspond to the area to be irradiated with radiation. The three-dimensional labels may be specified using particles that make up the three-dimensional model B1. The three-dimensional labels may include not only the labels of the organs that are the target of radiation therapy, but also the labels of surrounding organs.

[0088] In the present embodiment, an example has been described in which one captured image C11 is used for one 3D simulation, but this is not limiting. Multiple captured images C11 may be used for one 3D simulation. In this case, the multiple captured images C11 may be, for example, multiple tomographic images obtained by capturing slices having the same normal direction to the slices but at different positions in that direction, or multiple tomographic images obtained by capturing slices having normal directions different from each other.

[0089] In this embodiment, an example has been described in which the position of the fault is specified in advance by the fault position information E1, but this is not limiting. The position of the fault may be determined based on the captured image C10 captured in step S210. In this case, the position of the fault captured in the captured image C10 is determined, and information indicating the determined position of the fault is set as the fault position information E1.

[0090] The tomographic position information E1 may also be calculated after pre-treatment processing and before calibration. A case in which the tomographic position information E1 is calculated will be described later in a second embodiment.

[0091] In this embodiment, an example in which the captured image C1 is an MR image has been described, but this is not limiting. The captured image C1 may be a CT image. However, since an MR image has a higher contrast than a CT image, it is preferable to use an MR image as the captured image C1.

[0092] The above-described MPM algorithm is merely an example, and various modifications may be made. Furthermore, in the present embodiment, an example of a case where a 3D simulation is performed based on MPM has been described, but the present invention is not limited to this. The 3D simulation may be performed based on a meshfree method other than MPM. Furthermore, the 3D simulation may be performed based on a method other than the meshfree method. In the present embodiment, the 3D model B1 is a 3D model in which each part of an organ is replaced with particles and discretized, but the 3D model B1 is modified depending on the method used for the 3D simulation.

[0093] In this embodiment, an example has been described in which the organs consist of the organ to be radiotherapy and other organs adjacent to the target organ, and the 3D simulation is performed taking into account contact between the organs accordingly, but this is not limiting. The organs may consist only of the organ to be radiotherapy. For example, the organ may be only the pancreas.

[0094] In this embodiment, an example in which the organ to be radiotherapy is the pancreas has been described, but the present invention is not limited to this. The organ to be radiotherapy is also applicable to organs other than the pancreas, such as the duodenum, spleen, and brain.

[0095] In step S230, an X-ray image of the inside of the organ captured by X-rays (tomography) may be used to select a portion of the three-dimensional image A1 that corresponds to the internal position of the organ captured in the captured image C10. In this case, a two-dimensional image that most closely matches the X-ray image is selected from the three-dimensional image A1 as the partial image D1.

[0096] As described above, the organ deformation estimation device 2 according to this embodiment is an organ deformation estimation device for radiation therapy, and includes a 3D image acquisition unit 200, a 3D model acquisition unit 202, a partial image extraction unit 203, a captured image acquisition unit 204, a registration unit 205, a target position calculation unit 206, and a displacement estimation unit 207. The 3D image acquisition unit 200 acquires a 3D image A1 of an organ (in this embodiment, the pancreas) captured in advance at a time before radiation is irradiated to the organ during radiation therapy. The 3D model acquisition unit 202 acquires a 3D model B1 generated based on the 3D image A1 and representing the shape of the organ (in this embodiment, the pancreas) at a time before radiation therapy is performed. The captured image acquisition unit 204 acquires a captured image C1, which is a 2D image of the interior (in this embodiment, a cross section) of the organ (in this embodiment, the pancreas) captured during radiation therapy. The partial image extraction unit 203 extracts from the three-dimensional image A1 a partial image D1, which is a two-dimensional image of a portion included in the three-dimensional image A1 that corresponds to the position of the interior (in this embodiment, a slice) captured in the captured image C1. The alignment unit 205 aligns the pixels constituting the captured image C1 with the pixels constituting the partial image D1 so that pixels representing the same portion of the interior (in this embodiment, a slice) correspond to each other. The target position calculation unit 206 calculates a target position for displacement of a portion of the three-dimensional model B1 that corresponds to the interior (in this embodiment, a slice) based on the alignment result by the alignment unit 205. The displacement estimation unit 207 deforms the three-dimensional model B1 based on a three-dimensional simulation (in this embodiment, MPM) so as to displace the position of the portion of the three-dimensional model B1 that corresponds to the interior (in this embodiment, a slice) to the target position calculated by the target position calculation unit 206, thereby estimating the deformation of the organ (in this embodiment, the pancreas) during radiation therapy.

[0097] With this configuration, the organ deformation estimation device 2 according to this embodiment can estimate the deformation of an organ (in this embodiment, the pancreas) during radiation therapy by deforming the 3D model B1 based on a 3D simulation (in this embodiment, the MPM) so as to displace the position of a portion of the 3D model B1 corresponding to the interior of the organ (in this embodiment, a slice) to a target position calculated using captured images C1 of the interior of the organ (in this embodiment, the pancreas) during radiation therapy. Therefore, the organ deformation estimation device 2 according to this embodiment can estimate the displacement of an organ from a small number of captured images during radiation therapy. The small number refers to the number of images that can be captured during radiation therapy, such as one to several (e.g., two to three). Radiation therapy is an example of a treatment. The period before radiation is irradiated onto the organ during radiation therapy is an example of a period before treatment is performed. The 3D image acquisition unit 200 may be omitted from the configuration of the organ deformation estimation device 2.

[0098] Conventionally, a device known as MR-Linac, which performs radiation therapy while capturing MR images of organ sections in real time, can capture only a few tomographic images in real time during treatment. The organ deformation estimation device 2 according to this embodiment is suitable for use in estimating organ displacement during radiation therapy and changing the 3D label for emitting radiation in real time when only a few tomographic images can be captured.

[0099] (First Example) A first example, which is an example according to the first embodiment, will be described. In this example, a single tomographic image of a central slice of a volume in which an organ is present was used as the captured image C1 used to align two-dimensional images. The Multi-Atlas Labeling Beyond the Cranial Vaultsegmentation challenge dataset was used for the organ image. MATLAB (registered trademark) was used for data processing to perform the three-dimensional simulation. The parameters used in this example are shown in FIG. 6.

[0100] 7 and 8 show how the three-dimensional model B1 is driven by the three-dimensional simulation. Each of Fig. 7 and Fig. 8 shows a three-dimensional Cartesian coordinate system (XYZ coordinate system). In the three-dimensional Cartesian coordinate system, the Z-axis direction is perpendicular to the fault, and the X-axis and Y-axis directions are parallel to the plane of the fault.

[0101] Fig. 7 shows the three-dimensional model before being driven. Three-dimensional model B21 is a three-dimensional model of the pancreas before being driven. Three-dimensional model B31 is a three-dimensional model of the stomach before being driven. Fig. 8 shows the three-dimensional model B12 after being driven. Three-dimensional model B22 is a three-dimensional model of the pancreas after being driven. Three-dimensional model B32 is a three-dimensional model of the stomach after being driven.

[0102] The target position T1 indicates a target position for displacing a portion of the three-dimensional model B11 corresponding to the fault. When the portion corresponding to the fault is displaced to the target position T1 by a three-dimensional simulation using the MPM, portions of the three-dimensional model B11 other than the portion corresponding to the fault also displace accordingly.

[0103] In this example, the ground truth (GT) was created by applying random in-plane forces to the particles constituting the three-dimensional model B1 to deform it. The error ε used in this example is expressed by equation (10).

[0104]

[0105] Here, n is the total number of particles in the physical model, xgt,i is the position vector of the i-th particle in GT, and xest,i is the position vector of the i-th particle estimated by the organ deformation estimation process.

[0106] By comparing the error caused by the organ deformation estimation process with the results of aligning 3D images using Elastic, we verified the accuracy of 3D displacement estimation using only one tomographic image. We also performed a 3D simulation using the organ deformation estimation process without discretizing the stomach, a neighboring organ, to verify whether taking the neighboring organs into account improves accuracy.

[0107] In this example, the organ deformation estimation process was applied to 20 examples of data. Results for one example with high accuracy and one example with low accuracy among the 20 examples of data are shown in Figures 9 and 10, respectively. In Figures 9 and 10, "data from estimation process only" refers to data indicating positions not included in the GT among positions estimated by the organ deformation estimation process. "GT only data" refers to data indicating positions not estimated by the organ deformation estimation process among the GT data. "data from estimation process and GT" refers to data indicating positions included in the GT among positions estimated by the organ deformation estimation process.

[0108] Figure 9 shows that, although slight errors are observed in the surface area, the displacement can be estimated with high accuracy. The errors present in the surface area are thought to be due to the boundary condition being defined only for contact with other organs. It is believed that errors can be reduced by increasing the number of surrounding organs considered or by changing the contact algorithm to one with higher accuracy. Figure 10 shows that errors are particularly prevalent at both ends of the organ. In this example, a cross-sectional image of the center of the pancreas region was used to standardize the experimental conditions. The axial direction of the cross-section to be used and the selection of which cross-section in each axial direction are thought to be important.

[0109] Figure 11 shows the error results of the organ deformation estimation process for 20 data cases. Figure 11 is a boxplot showing the distribution of errors for the 20 cases. The pancreas position error resulting from the organ deformation estimation process was 12.7 ± 6.93 (pixels). When contact with surrounding organs was not considered in the organ deformation estimation process, the pancreas position error was 21.3 ± 9.30 (pixels). The pancreas position error resulting from the alignment of 3D images was 1.97 ± 0.981 (pixels). Looking at the average results, we were unable to achieve accuracy approaching that of the alignment of 3D images. However, Figure 11 shows that in approximately 40% of the cases, the organ deformation estimation process achieved accuracy similar to that achieved by the alignment of 3D images. If an appropriate cross-section can be selected, the organ deformation estimation process can achieve accuracy similar to that achieved by the alignment of 3D images. A method for selecting an appropriate cross-section will be described later.

[0110] Compared to not considering contact with surrounding organs, taking contact into account produced better results in terms of mean and standard deviation, and a statistically significant difference was obtained between the medians of both cases at a significance level of 0.001. This suggests that taking contact with surrounding organs into account improves accuracy.

[0111] Second Embodiment A second embodiment of the present invention will be described in detail below with reference to the drawings. In the first embodiment, the case where the position and direction of the slice to be imaged are predetermined is described. In this embodiment, the case where the position and direction of the slice to be imaged are selected is described. The organ deformation estimation system according to this embodiment is referred to as organ deformation estimation system 1a. Note that the same components as those in the first embodiment described above are denoted by the same reference numerals, and descriptions of the same components and operations may be omitted.

[0112] 12 is a diagram showing an example of the functional configuration of the organ deformation estimation system 1a according to this embodiment. The organ deformation estimation system 1a includes an organ deformation estimation device 2a, a three-dimensional image supply unit 3, and a captured image supply unit 40.

[0113] The organ deformation estimation device 2 includes a control unit 20a and a storage unit 21a. The control unit 20a includes a 3D image acquisition unit 200, a 3D model generation unit 201, a 3D model acquisition unit 202, a partial image extraction unit 203, a photographed image acquisition unit 204, a registration unit 205, a target position calculation unit 206, a displacement estimation unit 207, an output unit 208, and a slice selection unit 209a. Comparing the control unit 20a ( FIG. 12 ) according to this embodiment with the control unit 20 ( FIG. 3 ) according to the first embodiment, the slice selection unit 209a is different. The functions of the other components (the 3D image acquisition unit 200, the 3D model generation unit 201, the 3D model acquisition unit 202, the partial image extraction unit 203, the photographed image acquisition unit 204, the registration unit 205, the target position calculation unit 206, the displacement estimation unit 207, and the output unit 208) are the same as those of the first embodiment.

[0114] The tomography selection unit 209a selects a tomography to be captured as the captured image C1 from the inside (tomography) of the organ. The tomography is a tomography to be captured as the captured image C1 from the inside (tomography) of the organ. The tomography is selected by specifying the direction of the tomography and a position in that direction. The tomography selection unit 209a stores information indicating the selected tomography as tomography position information E1a in the storage unit 21a.

[0115] [Layer Selection Process] The process by which the layer selection unit 209a selects a layer is called the layer selection process. The layer selection unit 209a executes the layer selection process after pre-treatment processing and before calibration. The layer selection process will now be described with reference to Figures 13 and 14. There are cases where the number of imaging layer candidates that can be acquired is finite and a full search is possible, where the number of imaging layer candidates is finite and a full search is not possible, or where the number of imaging layer candidates is not finite. The layer selection process differs in each case.

[0116] FIG. 13 is a diagram showing an example of the flow of the tomographic section selection process in the case where the number of imaging tomographic section candidates according to this embodiment is finite and a full search is possible.

[0117] Step S410: The slice selection unit 209a randomly applies deformation to the 3D image A1 to generate correct answer data. The correct answer data is a 3D image generated by applying random deformation pixel by pixel to each part of the organ captured in the 3D image A1. The random deformation is applied using random numbers in a predetermined number of patterns. The slice selection unit 209a generates correct answer data for each of the predetermined number of random deformation patterns. The predetermined number is, for example, five. Note that for simplicity, only two correct answer data are shown in FIG. 13, and the other correct answer data are omitted.

[0118] Step S420: The slice selection unit 209a measures the error of the results of the organ deformation estimation process for all combinations of slice directions and slice numbers. The slice numbers are numbers assigned to slices at one or more positions in a certain slice direction.

[0119] The slice selection unit 209a extracts a two-dimensional image from the correct answer data for a slice specified by a certain slice direction and a certain slice number. The control unit 20a executes organ deformation estimation processing using the extracted two-dimensional image instead of the captured image C1. The slice selection unit 209a measures the error of the organ deformation estimation result from the organ deformation estimation processing relative to the correct answer data.

[0120] For example, if there are two cross-sectional directions and 30 cross-sections, there are 900 possible combinations of cross-sectional directions and cross-section numbers. If the number of combinations is enormous, the number of combinations may be reduced by using other parameter tuning methods, rather than performing organ deformation estimation processing for all combinations. Examples of other parameter tuning methods include random search and Bayesian optimization.

[0121] Step S430: The fault selection unit 209a selects the top few faults (for example, the top five faults) with the smallest measured errors.

[0122] The above processes of step S420 and step S430 are executed for each correct answer data. Step S440: The tomography selection unit 209a selects the tomography with the smallest error as the optimal solution (photographed tomography) for the group of tomography selected in step S430. The tomography selection unit 209a, for example, selects the tomography with the smallest error by taking a weighted average by rank. The tomography selection unit 209a stores, for example, the tomography number of the photographed tomography in the storage unit 21a as tomography position information E1.

[0123] 14 is a diagram illustrating an example of a tomography selection process according to the present embodiment when the number of imaging tomography candidates is finite and exhaustive search is impossible, or when the number of imaging tomography candidates is not finite. When the number of imaging tomography candidates is finite and exhaustive search is impossible, this includes, for example, a case where the number of imaging tomography candidates is enormous. Furthermore, when the number of imaging tomography candidates is not finite, this also includes, for example, a case where a tomography is virtually defined using image interpolation or the like and used as a candidate imaging tomography, even for a position where the candidate imaging tomography position is not included in the three-dimensional image A1. Therefore, the imaging tomography candidates may include not only tomography corresponding to a position included in the three-dimensional image A1 captured in advance, but also tomography corresponding to a position not included in the three-dimensional image A1.

[0124] FIG. 14 shows the error of the estimation result relative to the correct data for two slice directions, "slice direction 1" and "slice direction 2." The estimation result is obtained by the same process as in step S420 described above. Among the errors shown in FIG. 14, the two-dimensional image used instead of the photographed image C1 to perform the organ deformation estimation process is, for example, a two-dimensional image generated from the three-dimensional image A1 using image interpolation or the like. When the number of candidate photographic slices is finite and a full search is not possible, or when the number of candidate photographic slices is not finite, the slice selection unit 209a searches for the optimal cross section based on an optimization method. The optimization method is, for example, stochastic gradient descent.

[0125] As described above, the organ deformation estimation device 2a according to this embodiment includes a tomography selection unit 209a. The tomography selection unit 209a selects an imaging tomography from among candidate imaging tomography sections for one or more tomography directions and one or more positions in the tomography directions, based on the error, relative to the correct answer data, of the organ deformation estimation result estimated when a two-dimensional image obtained for the candidate tomography section from correct answer data, which is a three-dimensional image obtained by randomly applying deformation to the three-dimensional image A1, is used instead of the imaging image C1. With this configuration, the organ deformation estimation device 2a according to this embodiment can select a tomography section that will reduce the error in the organ deformation estimation result, thereby reducing the error in the estimation result.

[0126] The method for selecting an appropriate fault plane is not limited to the fault plane selection process described in this embodiment. Another method for selecting an appropriate fault plane may involve creating a regression model using the fault plane position as an explanatory variable and the evaluation index, the registration error, or the value and ranking of the Dice coefficient as the objective variable, to obtain the optimal fault plane position. When the above regression model was actually created, a highly accurate regression model with a coefficient of determination of 0.903 to 0.951 was created. The execution time was 1.57±0.213 ms, confirming that real-time fault plane selection during treatment is possible.

[0127] Second Example A second example according to the second embodiment will be described. In this example, five examples generated by randomly transforming a single three-dimensional image using random numbers were used as correct data, and three-dimensional simulations were performed on all cross sections. In the three-dimensional simulations, the pancreas, stomach, and duodenum were discretized, and combinations that reduced the estimation error were also considered.

[0128] For each of the five examples of correct data, the errors and tomography numbers for the top five tomography sections, in order of smallest error, are shown in Figure 15. The tomography numbers range from 79 to 113, and the median tomography number is 96. In this example, the following four types of organ combinations (referred to as "types") were performed. The first type of combination is when the pancreas, stomach, and duodenum are all discretized (referred to as "all"). The second type of combination is when the pancreas and stomach are discretized (referred to as "with duo"). The third type of combination is when the pancreas and duodenum are discretized (referred to as "with stomach"). The fourth type of combination is when only the pancreas is discretized (referred to as "panc").

[0129] 15, the columns "pancreas," "stomach," and "duodenum" indicate the proportion of each organ's area on the cross section. There are many blank spaces in the duodenum area, which indicates that the duodenum is smaller in area than the pancreas and stomach, and therefore is discretized as a 3D model B1 but does not exist on the cross section.

[0130] For each example, graphs showing the transition of error when the tomographic layer number is changed are shown in Figures 16 to 20. Graphs showing the relationship between the transition of error and the number of particles when the tomographic layer number is changed are shown in Figures 21 to 25. The results of this example confirmed that the accuracy of organ deformation estimation increases when more organs are used in the 3D simulation.

[0131] In the above-described embodiment, an example in which the organ deformation estimation device 2, 2a is used in radiation therapy has been described, but the present invention is not limited to this. The organ deformation estimation device 2, 2a may also be used for treatments other than radiation therapy or for treatment support. Treatments other than radiation therapy include, for example, heavy ion beam therapy or surgical operations. Treatment support includes, for example, image-guided surgery support and robot-assisted surgery. The organ deformation estimation device 2, 2a may also be used as a treatment device or treatment system including the organ deformation estimation device 2, 2a for various treatments. The organ deformation estimation device 2, 2a may also be used as a treatment support device or treatment support system including the organ deformation estimation device 2, 2a for various treatment support. An example of a treatment system is a radiation treatment system including the organ deformation estimation device 2, an imaging device, and a radiation irradiation device.

[0132] Note that parts of the organ deformation estimation device 2, 2a in the above-described embodiment, such as the 3D image acquisition unit 200, 3D model generation unit 201, 3D model acquisition unit 202, partial image extraction unit 203, captured image acquisition unit 204, alignment unit 205, target position calculation unit 206, displacement estimation unit 207, output unit 208, and slice selection unit 209a, may be implemented by a computer. In this case, a program for implementing these control functions may be recorded on a computer-readable recording medium, and the program may be read and executed by a computer system. Note that the term "computer system" used here refers to a computer system built into the organ deformation estimation device 2, 2a, including hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into the computer system. Furthermore, the term "computer-readable recording medium" may include a medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or a medium that stores a program for a fixed period of time, such as a volatile memory within a computer system that serves as a server or client in such a case. The program may be for implementing part of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system. Furthermore, part or all of the organ deformation estimation devices 2 and 2a in the above-described embodiments may be implemented as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the organ deformation estimation devices 2 and 2a may be implemented as a processor individually, or part or all of them may be integrated into a processor. The integrated circuit implementation method is not limited to LSI, and may also be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0133] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0134] 2, 2a... Organ deformation estimation device, 200... 3D image acquisition unit, 202... 3D model acquisition unit, 203... partial image extraction unit, 203... partial image extraction unit, 204... photographed image acquisition unit, 205... alignment unit, 206... target position calculation unit, 207... displacement estimation unit, A1... 3D image, B1, B11, B12, B21, B22, B31, B32... 3D model, C1, C10, C11... photographed image, D1... partial image

Claims

1. a three-dimensional model acquisition unit that acquires a three-dimensional model that is generated based on a three-dimensional image of the organ captured in advance before the treatment is performed and indicates a shape of the organ before the treatment is performed; an image acquisition unit for acquiring a two-dimensional image of the inside of the organ during the course of the treatment; a partial image extraction unit that extracts from the three-dimensional image a partial image that is a two-dimensional image of a portion that corresponds to the internal position captured in the captured image among portions included in the three-dimensional image; a position matching unit that performs position matching between pixels constituting the captured image and pixels constituting the partial image so that pixels indicating the same internal portion correspond to each other; a target position calculation unit that calculates a target position of a portion corresponding to the interior among portions constituting the three-dimensional model based on a result of the alignment; and a displacement estimation unit that estimates deformation of the organ during the course of the treatment by deforming the three-dimensional model based on a three-dimensional simulation based on a meshfree method so as to displace a position of a portion corresponding to the interior among portions constituting the three-dimensional model to the target position; Equipped with The organ comprises a target organ of the treatment and a surrounding organ in contact with the target organ, The deformation of the organ to be treated estimated by the displacement estimation unit includes a deformation that has no periodicity due to the movement of the surrounding organ. Organ deformation estimation device.

2. (delete)

3. The three-dimensional image acquisition unit further acquires the three-dimensional image. The organ deformation estimation device according to claim 1 .

4. (delete)

5. a slice selection unit for selecting a slice to be captured as the captured image from within the internal organ The organ deformation estimation device according to claim 1 .

6. The tomography selection unit selects the imaging tomography from among the candidate tomography for one or more tomography directions and one or more positions in the tomography directions, based on an error of an estimated result of deformation of the organ estimated when a two-dimensional image obtained from ground-answer data, which is a three-dimensional image obtained by randomly deforming the three-dimensional image, is used instead of the imaging image, relative to the ground-answer data. The organ deformation estimation device according to claim 5.

7. A treatment apparatus comprising the organ deformation estimation device according to claim 1.

8. A medical treatment support device comprising the organ deformation estimation device according to claim 1.

9. a three-dimensional model acquisition step of acquiring a three-dimensional model that is generated based on a three-dimensional image of the organ captured in advance before the treatment is performed and that indicates the shape of the organ before the treatment is performed; an image acquiring step of acquiring an image, which is a two-dimensional image of the inside of the organ, during the course of the treatment; a partial image extraction step of extracting from the three-dimensional image a partial image which is a two-dimensional image of a portion included in the three-dimensional image and corresponds to the internal position captured in the captured image; a registration step of performing registration between pixels constituting the photographed image and pixels constituting the partial image such that pixels indicating the same internal portion correspond to each other; a target position calculation step of calculating a target position of a portion corresponding to the interior among portions constituting the three-dimensional model based on a result of the alignment; a displacement estimation step of estimating deformation of the organ during the course of the treatment by deforming the three-dimensional model based on a three-dimensional simulation based on a meshfree method so as to displace a position of a portion corresponding to the inside among portions constituting the three-dimensional model to the target position; having The organ comprises a target organ of the treatment and a surrounding organ in contact with the target organ, The deformation of the organ to be treated estimated by the displacement estimation unit includes a deformation that has no periodicity due to the movement of the surrounding organ. Method for estimating organ deformation.

10. On the computer, a three-dimensional model acquisition step of acquiring a three-dimensional model that is generated based on a three-dimensional image of the organ captured in advance before the treatment is performed and that indicates the shape of the organ before the treatment is performed; an image acquiring step of acquiring an image, which is a two-dimensional image of the inside of the organ, during the course of the treatment; a partial image extraction step of extracting from the three-dimensional image a partial image which is a two-dimensional image of a portion included in the three-dimensional image and corresponds to the internal position captured in the captured image; a registration step of performing registration between pixels constituting the photographed image and pixels constituting the partial image such that pixels indicating the same internal portion correspond to each other; a target position calculation step of calculating a target position of a portion corresponding to the interior among portions constituting the three-dimensional model based on a result of the alignment; a displacement estimation step of estimating deformation of the organ during the course of the treatment by deforming the three-dimensional model based on a three-dimensional simulation based on a meshfree method so as to displace a position of a portion corresponding to the inside among portions constituting the three-dimensional model to the target position; A program for executing The organ comprises a target organ of the treatment and a surrounding organ in contact with the target organ, The deformation of the organ to be treated estimated by the displacement estimation unit includes a deformation that has no periodicity due to the movement of the surrounding organ. program.