Target shape estimation device and treatment device

Abstract

By using this therapeutic device (1) which is characterized in that a shape of a target (30) is estimated using an identifier (61) trained on the basis of a first image (23) in which subject in vivo information is captured, a second image (30) showing the target (63) in vivo, and a third image (20) capturing the same subject using a different device from the first image (23), it is possible to accurately estimate the shape of a target treatment area when carrying out training on the shape of a target treatment area using a DRR image processed from a CT image.

Description

[Technical Field]

[0001] The present invention relates to a target shape estimation device and a therapeutic device. [Background technology]

[0002] When treating tumors using radiation such as X-rays or positron beams, it is necessary to identify the external shape of the tumor to ensure that radiation is not irradiated to areas other than the tumor itself. Furthermore, because the position of the treatment target (tumor, etc.) shifts due to the patient's breathing and heartbeat, it is necessary to track the target using real-time fluoroscopic images to identify its position and irradiate only when the target has moved to the irradiation position, in order to avoid irradiating areas other than the target (normal tissue). Furthermore, the technique of using images to identify and track the position of a target and guide the irradiation position of radiation, known as image-guided target tracking, generally uses X-ray fluoroscopy images taken with kV-order X-rays, but is not limited to this. For example, it is possible to use not only X-ray fluoroscopy images, but also X-ray images taken with MV-order X-rays, ultrasound images, MRI (Magnetic Resonance Imaging) images, CT (Computed Tomography) images, and PET (Positron Emission Tomography) images. It is also possible to use X-ray backscatter images, which utilize backscattering, instead of fluoroscopy images. The following techniques are known for identifying and tracking the location of a target.

[0003] Patent Document 1 (International Publication No. 2018 / 159775) describes a technique for tracking targets such as tumors that move with the patient's breathing. In the technique described in Patent Document 1, the bone structure DRR (Digitally Reconstructed Radiography) image and the soft tissue DRR image of the tumor are separated from the image containing the target (tumor, etc.). Multiple superimposed images are created by randomly overlaying (randomly transforming) the bone structure DRR image onto the soft tissue DRR image. Then, deep learning is used to train the system using these multiple superimposed images. During treatment, the area of ​​the target to be tracked is identified from the fluoroscopic image taken during treatment and the training results, and therapeutic X-rays are irradiated. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] International Publication No. 2018 / 159775: WO2018 / 159775A1 ("0031" to "0053", Figures 1 to 6) [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] (Problems with conventional technology) The technology described in Patent Document 1 uses DRR images created from CT images for training, but during treatment, real-time performance is prioritized, and tumor tracking is performed based on fluoroscopic images. Therefore, the difference in image quality between the DRR images processed from CT images and the fluoroscopic images becomes a problem. For example, if the image quality of the DRR images is lower than that of the fluoroscopic images, problems may occur such as the tumor's outline being distorted, a single tumor being recognized as having two or more divided regions, or a tumor without holes being recognized as having holes inside. Therefore, if training is performed when the tumor is not accurately recognized, the accuracy of tumor tracking will decrease.

[0006] The present invention aims to accurately estimate the shape of a target area in a configuration that learns the shape of a target area using images processed from CT images. [Means for solving the problem]

[0007] To solve the aforementioned technical problems, the target shape estimation device of the invention described in claim 1 is: The first image shows internal body information of the subject, the second image shows a target inside the body, and the third image shows the same subject taken with a different device than the first image. and the subject's internal information including the target Based on that, The first image and the third image A classifier that has been trained to take either of the following as input and output the outline of a target, wherein the outline of the target output with the first image as input and the outline of the target in the second image Using a loss function that identifies the geometric similarity of the target's outline Please provide feedback for further improvement Using the trained classifier, It is characterized by estimating the external shape of the target.

[0008] The invention described in claim 2 is a target shape estimation device according to claim 1, Use a third image taken before or immediately before the treatment date. It is characterized by the following:

[0011] Claim 3 The invention described herein relates to the target shape estimation device described in claim 1, In the shape of the target derived using the third image and the classifier, the loss function derived based on the number of regions and the number of holes inside the target, It is characterized by having the following features.

[0012] Claim 4 The invention described herein relates to the target shape estimation device described in claim 1, In the shape of the target derived using the first image and the classifier, the loss function derived based on the number of regions and the number of holes inside the target, It is characterized by having the following features.

[0013] Claim 5The invention described herein relates to the target shape estimation device described in claim 1, A similar image to the third image is obtained from a database in which the first image is stored, and the loss function is based on the similarity between the target outline derived using the classifier and the similar image, and a similar target image in which the target outline in the similar image is identified. It is characterized by having the following features.

[0014] Claim 6 The invention described herein relates to the target shape estimation device described in claim 1, A classifier comprising: a generator that generates information about the outline of a target from the first image or the third image; a discriminator that discriminates based on the outline of a target generated by the generator based on the first image, and also discriminates between the outline of a pre-identified target and the outline of a target generated by the generator based on the third image, wherein the classifier learns based on the discrimination results of the discriminator, It is characterized by having the following features.

[0015] Claim 7 The invention described herein relates to the target shape estimation device described in claim 1, The classifier into which the first or third image is directly input, It is characterized by having the following features.

[0016] Claim 8 The invention described is Claim 6 or 7 In the target shape estimation device described above, If the area of ​​the target region differs from a predetermined value, the loss function will have a larger loss. It is characterized by having the following features.

[0017] Claim 9 The invention described is Claim 6 or 7 In the target shape estimation device described above, If the aspect ratio of the target region differs from a predetermined value, the loss function will have a larger loss. It is characterized by having the following features.

[0018] Claim 10 The invention described is Claim 6 or 7 In the target shape estimation device described above, If the length of the diagonal of the target region differs from a predetermined value, the loss function will have a larger loss. It is characterized by having the following features.

[0019] Claim 11 The invention described herein relates to the target shape estimation device described in claim 1, Each of the aforementioned images is one or a combination of the following: X-ray image, nuclear magnetic resonance image, ultrasound image, positron emission tomography image, body surface topography image, and photoacoustic imaging image. It is characterized by the following:

[0020] In order to solve the aforementioned technical problems, Claim 12 The therapeutic device of the invention described above is A target shape estimation device according to claim 1, Based on the target shape estimated by the target shape estimation device, an irradiation means for irradiating the target with therapeutic radiation, It is characterized by having the following features. [Effects of the Invention]

[0021] Claim 1, 12 According to the invention described above, in a configuration that learns the shape of a target area using an image processed from a CT image, the shape of the target area can be estimated with high accuracy. Furthermore, according to the invention described in claim 1, the similarity of the target's external shape can be identified and learned using a loss function. Furthermore, according to the invention described in claim 1, the geometric similarity of the target's outline can be learned using a loss function, and the shape of the target area can be estimated with high accuracy. According to the invention described in claim 2, learning can be deepened using images from before the treatment day or immediately before treatment, and the shape of the target area can be estimated with high accuracy. 。

[0022] Claim 3 According to the invention described above, the shape of the target area can be estimated by learning from a loss function that corresponds to the number of target regions and the number of internal holes in the target shape derived from the third image. Claim 4 According to the invention described above, the shape of the target area can be estimated by learning from a loss function that corresponds to the number of target regions and the number of internal holes in the target shape derived from the first image.

[0023] Claim 5 According to the invention described above, the shape of the target area can be estimated by learning from a loss function derived using a similar image that is similar to the third image. Claim 6 According to the invention described above, a classifier having a generator and a discriminator can be used to learn and estimate the shape of a target area. Claim 7 According to the invention described above, a classifier that receives a first image or a third image as direct input can be used to learn and estimate the shape of a target area.

[0024] Claim 8 According to the invention described above, it is expected that the shape of the target area can be estimated with greater accuracy compared to cases where a loss function corresponding to the area of ​​the target region is not used. Claim 9 According to the invention described above, it is expected that the shape of the target area can be estimated with greater accuracy compared to cases where a loss function corresponding to the aspect ratio of the target region is not used. Claim 10 According to the invention described above, it is expected that the shape of the target area can be estimated with greater accuracy compared to the case where a loss function corresponding to the diagonal length of the target area is not used. Claim 11 According to the invention described herein, images can be acquired using one or a combination of X-ray images, nuclear magnetic resonance images, ultrasound images, positron emission tomography images, body surface shape images, and photoacoustic imaging images, and these images can be used for treatment with X-rays or particle beams. [Brief explanation of the drawing]

[0025] [Figure 1] Figure 1 is an explanatory diagram of a radiotherapy machine to which the tumor shape estimation device of Embodiment 1 of the present invention is applied. [Figure 2]Figure 2 is a block diagram showing the various functions of the control unit of the radiotherapy machine in Example 1. [Figure 3] Figure 3 is an explanatory diagram of an example of processing in the control unit of Embodiment 1. [Figure 4] Figure 4 is an explanatory diagram illustrating the relationship between the number of target regions and the number of pores. Figure 4A is an explanatory diagram for the case where both the number of regions and the number of pores are zero. Figure 4B is an explanatory diagram for the case where the number of regions is 1 and the number of pores is zero. Figure 4C is an explanatory diagram for the case where the number of regions is 3 and the number of pores is zero. Figure 4D is an explanatory diagram for the case where the number of regions is 1 and the number of pores is 1. [Figure 5] Figure 5 is an explanatory diagram of the effects of Example 1, with Figure 5A being an explanatory diagram when topology coefficients are not used, and Figure 5B being an explanatory diagram when topology coefficients are used. [Figure 6] Figure 6 is an explanatory diagram of Example 2, and corresponds to Figure 3 of Example 1. [Figure 7] Figure 7 is an explanatory diagram of Example 4, and corresponds to Figure 3 of Example 1. [Modes for carrying out the invention]

[0026] Next, with reference to the drawings, specific examples of embodiments of the present invention will be described, but the present invention is not limited to the following embodiments. In the following explanation using diagrams, diagrams of components other than those necessary for the explanation have been omitted as appropriate for ease of understanding. [Examples]

[0027] Figure 1 is an explanatory diagram of a radiotherapy machine to which the tumor shape estimation device of Embodiment 1 of the present invention is applied. In Figure 1, the radiotherapy machine (an example of a treatment device) 1 to which the subject tracking device of Embodiment 1 of the present invention is applied has a bed 3 on which the patient 2, who is the subject of treatment, lies. Below the bed 3, an X-ray irradiation device 4 for fluoroscopy is positioned. On the opposite side of each X-ray irradiation device 4, with patient 2 in between, is an imaging device (imaging means) 6. The imaging device 6 receives the X-rays that have passed through the patient and captures an X-ray fluoroscopic image. The image captured by the imaging device 6 is converted into an electrical signal by an image generator 7 and input to the control system 8.

[0028] Furthermore, a therapeutic radiation irradiator 11 is positioned above bed 3. The therapeutic radiation irradiator 11 is configured to receive control signals from the control system 8. The therapeutic radiation irradiator 11 is configured to irradiate X-rays, as an example of therapeutic radiation, towards a preset position (the affected area of ​​patient 2) in response to the input of control signals. In Embodiment 1, a Multi-Leaf Collimator (MLC, not shown) is installed outside the X-ray source of the therapeutic radiation irradiator 11 as an example of a diaphragm that can adjust the X-ray passage area (the shape of the opening through which the X-rays pass) to match the outline of the target being tracked (tumor, etc.). While MLCs are conventionally known and commercially available ones can be used, the invention is not limited to MLCs, and any configuration that can adjust the X-ray passage area can be adopted. In Example 1, X-rays with an acceleration voltage on the order of kV (kilovolts) were irradiated for fluoroscopy, and X-rays with an acceleration voltage on the order of MV (megavolts) were irradiated for therapeutic purposes.

[0029] (Description of the control system (control unit) in Example 1) Figure 2 is a block diagram showing the various functions of the control unit of the radiotherapy machine in Example 1. In Figure 2, the control unit C of the control system 8 has an input / output interface (I / O) for inputting and outputting signals to and from the outside. The control unit C also has a ROM (read-only memory) where programs and information for performing necessary processing are stored. Furthermore, the control unit C has a RAM (random-access memory) for temporarily storing necessary data. Finally, the control unit C has a CPU (central processing unit) that performs processing according to the programs stored in the ROM, etc. Therefore, the control unit C in Embodiment 1 is composed of a small information processing device, a so-called microcomputer. Thus, the control unit C can realize various functions by executing programs stored in the ROM, etc.

[0030] (Signal output element connected to control unit C) The control unit C receives output signals from signal output elements such as the operation unit UI, the image generator 7, and sensors (not shown). The user interface (UI) includes a touch panel UI0, which is an example of a display unit and an example of an input unit. The user interface also includes various input components such as a button UI1 for starting the learning process, a button UI2 for inputting training data, and a button UI3 for starting fluoroscopic imaging. The image generator 7 inputs the image captured by the imaging device 6 to the control unit C.

[0031] (Controlled element connected to control unit C) The control unit C is connected to the fluoroscopy X-ray irradiation device 4, the therapeutic radiation irradiator 11, and other control elements (not shown). The control unit C outputs control signals to the fluoroscopy X-ray irradiation device 4, the therapeutic radiation irradiator 11, etc. The fluoroscopic X-ray irradiation device 4 irradiates the patient 2 with X-rays to take X-ray fluoroscopic images during learning or treatment. The therapeutic radiation irradiator 11 irradiates the patient 2 with therapeutic radiation (X-rays) during treatment.

[0032] (Functions of Control Unit C) The control unit C has the function of performing processing according to the input signal from the signal output element and outputting control signals to each of the control elements. In other words, the control unit C has the following functions:

[0033] Figure 3 is an explanatory diagram of an example of processing in the control unit of Embodiment 1. C1: Learning image reading means (first shooting means) The learning image reading means C1 reads (receives) the image input from the image generator 7. In Example 1, the learning image reading means C1 reads the image input from the image generator 7 when the button UI1 for starting the learning process is pressed. In Example 1, the X-ray CT image (first fluoroscopic image) is read for a predetermined learning period starting from the time the button UI1 for starting the learning process is pressed. In Example 1, real-time learning processing is not performed for the acquisition of X-ray CT images, but if the processing speed improves due to CPU speed increases, etc., and real-time processing becomes possible, it will be possible to perform it in real time.

[0034] C2: Image separation means The image separation means C2 separates and extracts an X-ray CT image into a tracked area image 23 (a soft tissue DRR (Digitally Reconstructed Radiography) image as an example) that includes the tracked area 21, and a separated background image (a non-tracked image, a separated non-tracked image) 24 (a bone structure DRR image (bone image) as an example) that does not include the tracked area 21, based on the source image 22 which includes three-dimensional learning information including the tracked area image region (target region) 21. In Example 1, the image separation means C2 separates the tracked area image 23 (first image, soft tissue DRR image) and the separated background image 24 (bone structure DRR image) based on the CT value, which is the contrast information of the CT image. In Example 1, as an example, the separated background image 24 is configured as a bone structure DRR image in the area with a CT value of 200 or more, and the tracked area image 23 is configured as a soft tissue DRR image in the area with a CT value of less than 200. In Example 1, as an example, a tumor (target of tracking) occurring in the lung, i.e., the target of treatment, is shown as a soft tissue DRR image in the target area image 23. However, if the target of tracking is, for example, an abnormal part of the bone, then a bone structure DRR image is selected as the target area image 23, and a soft tissue DRR image is selected as the isolated background image 24. Thus, the selection of the target area image and the background image (non-target of tracking) is appropriately selected according to the target of tracking and the background image including obstacles.

[0035] C3: Perspective image reading means (second shooting means) The fluoroscopic image reading means C3 reads (acquires) the image input from the image generator 7. In Example 1, the fluoroscopic image reading means C3 reads (acquires) the image input from the image generator 7 when the button UI3 for starting fluoroscopy is pressed. In Example 1, the X-ray fluoroscopy image 20 (third image), which is the image taken by each imaging device 6 after the X-rays irradiated from the fluoroscopic X-ray irradiation device 4 pass through the patient 2, is read.

[0036] C4: Input receiving means for training images (target image storage means) The teacher image input receiving means C4 receives input of a teacher image (second image) 30, which includes a teacher image region 27 as an example of an image to be used to teach the target to be tracked, in response to input to the touch panel UI0 or to the button UI2 for inputting teacher data. In Embodiment 1, the source image (CT image) 22 for learning is displayed on the touch panel UI0, and the system is configured so that a physician can determine the teacher image region 27 by inputting on the screen such that the image region of the target to be tracked, which is the target of treatment, is enclosed.

[0037] C5: Learning means (classifier creation means) The learning means C5 learns at least one of the region information and position information of the target image region 21 in the images based on multiple X-ray fluoroscopic images 20 and the target area image (soft tissue DRR image) 23 to create a classifier 61. In Example 1, both the region and position of the target image region 21 are learned. In Example 1, the X-ray fluoroscopic images 20 and DRR images 23 are directly input to the classifier 61.

[0038] Furthermore, in Example 1, the centroid of the image region 21 is used as the position of the image region 21, but it can be changed to any position such as the top, bottom, right, or left edge of the region, depending on the design and specifications. The learning means C5 can adopt any conventionally known configuration, but it is preferable to use so-called deep learning (a multi-layered neural network), and in particular, it is preferable to use CNN (Convolutional Neural Network). In Example 1, Caffe was used as an example of deep learning, but it is not limited to this, and any learning means (framework, algorithm, software) can be adopted.

[0039] C6: Method for deriving the loss coefficient The loss coefficient derivation means C6 derives (calculates) the loss coefficient from the classifier (CNN) 61 derived by the learning means C5. The loss coefficient derivation means C6 in Example 1 includes a first derivation means C6A, a third derivation means C6B, and a second derivation means C6C. C6A: First derivation means The first derivation means C6A uses the soft tissue DRR image (first image) 23 and the classifier 61 derived once by the learning means C5 to derive the target outline (target estimated image) 62, and the Jackard coefficient (first loss function) L, which is the similarity between the derived target outline 62 and the training image region 27 (target outline) of the training image 30 (target image). jacc The following is derived. Note that the Jaccard coefficient L jaccSince it is a well-known coefficient and function indicating the similarity between two sets, detailed description thereof will be omitted. In Example 1, the case of using the Jaccard coefficient as an example of the first loss function indicating similarity is illustrated, but it is not limited thereto. For example, as another function indicating similarity, it is also possible to use the Dice coefficient, the Simpson coefficient (overlap coefficient), etc.

[0040] FIG. 4 is an explanatory diagram of the relationship between the number of target regions and the number of holes. FIG. 4A is an explanatory diagram when both the number of regions and the number of holes are zero. FIG. 4B is an explanatory diagram when the number of regions is 1 and the number of holes is zero. FIG. 4C is an explanatory diagram when the number of regions is 3 and the number of holes is zero. FIG. 4D is an explanatory diagram when the number of regions is 1 and the number of holes is 1. C6B: Third derivation means The third derivation means C6B uses the soft tissue DRR image (first fluoroscopic image) 23 and the discriminator 61 derived once by the learning means C5, and in the derived target shape (target outer shape 62), a third topology coefficient (third loss function) L based on the number β0 of target regions and the number β1 of internal holes is derived. topo3 When the derived target outer shape 62 is as shown in FIG. 4A, β0 = 0 and β1 = 0. When the derived target outer shape 62 is as shown in FIG. 4B, β0 = 1 and β1 = 0. When the derived target outer shape 62 is as shown in FIG. 4C, β0 = 3 and β1 = 0. When the derived target outer shape 62 is as shown in FIG. 4D, β0 = 1 and β1 = 1. As a practical problem, in normal radiotherapy targets, the number of regions is one, and the region should not be divided or have holes. That is, as a practical problem, β0 = 1 and β1 = 0 should be the case. Therefore, a penalty other than β0 = 1 and β1 = 0, that is, the topology coefficient L which is a loss coefficient regarded as incorrect is defined. topo3 In Example 1, as an example, the topology coefficient L is defined by the following formula (1). topo3 L topo3 = (1 / nb) Σ nb i=1 (1 - β 0,i - β 1,i ) 2 …Formula (1)​ In equation (1), nb is the batch size, which refers to the number of images processed at once. That is, even if the number of images processed increases or decreases, multiplying by "(1 / nb)" in equation (1) will result in the topology coefficient L topo3 It will be standardized (normalized).

[0041] In Example 1, equation (1) is correct when β0=1, β1=0, and the topology coefficient L topo3 The function (loss function) minimizes the value, and increases the value if the answer is incorrect.

[0042] Furthermore, in Example 1, the topology coefficient L topo3 The function defined by equation (1) is shown as an example, but it is not limited to this. For example, equation (1) is in a form where squaring makes it a positive number, but it is also possible to use equation (2), which is a function that uses the absolute value instead of squaring. L topo3 =(1 / nb)Σ nb i=1 |1-β 0,i -β 1,i | …Formula (2) Additionally, if the tumor is divided into two parts, it is possible to define the topology coefficients such that equations (3) and (4) below simultaneously satisfy the following loss function, which rejects any answer other than the correct answer β0=2, β1=0 as incorrect. L topo3 =(1 / nb)Σ nb i=1 (2-β 0,i -β 1,i ) 2 ...Formula (3) ,and, Σ nb i=1 β 1,i =0…Formula (4) Similarly, if there are three or more regions, the topology coefficients will be set according to the correct answer. Furthermore, while Example 1 uses two-dimensional topology coefficients, for three-dimensional structures, it is also possible to use topology coefficients with additional parameters such as β2.

[0043] C6C: Second derivation method The second derivation means C6C uses the X-ray fluoroscopic image 20 and the classifier 61 derived once by the learning means C5 to determine the shape of the target (target outline (target estimated image) 63), and then uses a second topology coefficient (second loss function) L based on the number of target regions β0 and the number of internal holes β1. topo2 The second topology coefficient L is derived. topo2 The only difference is the shape of the target; the definition of the coefficient itself is the third topology coefficient L topo3 The same applies. Therefore, the second topology coefficient L topo2 A detailed explanation will be omitted.

[0044] C7: Feedback coefficient storage means The feedback coefficient storage means C7 stores topology coefficients L topo2 ,L topo3 The coefficient λ used for feedback is stored. In Example 1, the feedback coefficient λ is set to 0.01 as an example. Note that the feedback coefficient λ can be arbitrarily changed depending on the design, specifications, required learning accuracy, learning time, etc. The learning means C5 of Example 1 uses the first loss function L jacc , the second loss function L topo2 and the third loss function L topo3 Based on this, the classifier 61 is further trained.

[0045] C8: Learning result storage method The learning result storage means C8 stores the learning results of the learning means C5. That is, it stores the CNN optimized through retraining as the final classifier 61.

[0046] C9: Tumor identification means (target identification means) The tumor identification means C9 identifies the external shape of the target tumor based on the image captured during treatment. The identification method can employ conventionally known image analysis techniques, learn from images of multiple tumors to perform discrimination, or use the technology described in Patent Document 1. The "external shape" of the target is not limited to the external shape of the tumor itself (= the boundary between the normal area and the tumor), but also includes cases where it is set to be inside or outside the tumor at the discretion of the physician. In other words, the "external shape" can also be a region defined by the user. Therefore, the "target" is not limited to a tumor, but can also be a region that the user wants to track.

[0047] C10: External shape estimation means The external shape estimation means C10 estimates the external shape of the tumor based on the X-ray fluoroscopic image 20 taken immediately before the treatment and the classifier 61, and outputs the result. C11: Radiation irradiation means The radiation irradiation means C11 controls the therapeutic radiation irradiator 11 and irradiates the tumor with therapeutic X-rays when the area and location of the tumor estimated by the shape estimation means C10 are included in the irradiation range of the therapeutic X-rays. In Example 1, the radiation irradiation means C11 controls the MLC according to the area (shape) of the tumor estimated by the shape estimation means C10 to adjust the irradiation area (irradiation field) of the therapeutic X-rays to match the shape of the tumor. When therapeutic X-rays are irradiated, the radiation irradiation means C11 controls the MLC in real time in response to the output of the estimated shape of the tumor, which changes over time. The target outline estimation device of Example 1 is comprised of a fluoroscopic X-ray irradiation device 4, an imaging device 6, an image generator 7, a control system 8, and the aforementioned means C1 to C11.

[0048] (Effect of Example 1) In the radiotherapy machine 1 of Embodiment 1, which has the above configuration, learning is performed using X-ray fluoroscopic images 20 and soft tissue DRR images 23. That is, learning is performed with the DRR images 23 and X-ray fluoroscopic images 20 mixed together. Therefore, even if there is a difference in image quality between the DRR images 23 and the X-ray fluoroscopic images 20, learning is performed with both mixed together. Consequently, image quality can be improved compared to conventional techniques that perform learning using only DRR images. Therefore, it is possible to accurately estimate the shape of the tumor's external form. Consequently, it is possible to accurately irradiate the tumor and affected area with X-rays during treatment. In particular, while techniques for training using only supervised DRR images 23 and techniques for training using only unsupervised X-ray fluoroscopy images 20 existed separately, training by mixing supervised and unsupervised images using topology, as in Example 1, had not been done conventionally.

[0049] In contrast, in Example 1, accuracy is improved by mixing two images with different modalities (classification, style) (DRR image 23 and X-ray fluoroscopy image 20) using topology for training. In particular, in Example 1, the tracking target image 23 and the training image 30 are paired images, making supervised learning possible, but the X-ray fluoroscopy image 20 has no training data (unpaired image). Even in this case, it is possible to train by including the X-ray fluoroscopy image 20. Furthermore, X-ray fluoroscopic images 20 have low contrast between malignant tumors and normal areas, making it difficult to provide a complete ground truth image (training image). Additionally, the shape and size of the tumor change with the progression of symptoms over time, making supervised learning difficult.

[0050] Figure 5 is an explanatory diagram of the effects of Example 1, with Figure 5A being an explanatory diagram when topology coefficients are not used, and Figure 5B being an explanatory diagram when topology coefficients are used. In particular, in Example 1, the similarity L of the shape of the target outline 62 derived from the DRR image 23 and the classifier 61 is A ( =L jacc +λ·L topo3) is derived. In addition, the similarity L between the shape of the target outline 63 derived from the X-ray fluoroscopy image 20 and the classifier 61 is derived. B (=λ·L topo2 ) is derived. And the sum of the two similarities L total This is fed back and retrained. When using only conventionally known Jackard coefficients, problems can occur, such as having two or more regions (when the training time is too short, half the optimal time) or having holes within the regions (when the training time becomes four times longer, resulting in overfitting), as shown in Figure 5A. In contrast, in Example 1, as shown in Figure 5B, the model stabilizes after a certain amount of time has elapsed.

[0051] In particular, X-ray CT images are taken on the day of diagnosis or when the treatment plan is formulated, and are not taken frequently thereafter until the treatment day due to the burden on the patient such as radiation exposure. However, X-ray fluoroscopy images are often taken multiple times until the treatment day. Therefore, the DRR image 23 based on the X-ray CT image uses the first image taken, and subsequent changes in the shape of the tumor are shown using the X-ray fluoroscopy image 20, and the second topology coefficient L topo2 It is also possible to repeat the retraining process using the same data. In other words, since the X-ray fluoroscopy images 20 are unpaired images and do not incur the cost of creating training data, they can be used for training immediately after acquisition. In this way, it is possible to improve accuracy by retraining using the most recent X-ray fluoroscopy images possible. Furthermore, in Example 1, since the model is trained using images of patient 2 who is receiving treatment, the accuracy is improved compared to using images taken by a third party. [Examples]

[0052] Next, we will describe Embodiment 2 of the present invention. In this description of Embodiment 2, the same reference numerals are used for components corresponding to the components of Embodiment 1, and their detailed descriptions are omitted. This embodiment differs from Embodiment 1 in the following respects, but is otherwise configured in the same manner as Embodiment 1.

[0053] Figure 6 is an explanatory diagram of Example 2, and corresponds to Figure 3 of Example 1. In Example 1, the second topology coefficient L is used for the X-ray fluoroscopic image 20. topo2 Although only the first method was calculated, in Example 2, as shown in Figure 6, the Jackard coefficient is also derived. In Example 2, a DRR image similar to the X-ray fluoroscopy image 20 (similar DRR image 71) is searched for and obtained from the patient's DRR image database. Known image analysis techniques can be used to search for similar images. Then, the second Jackard coefficient (fourth loss function) L is the similarity between the shape of the target's outline (similar target image) 72, which is associated with the searched similar DRR image 71, and the outline of the target 63. jacc2 Derive the following.

[0054] The learning means C5 of Example 2 uses the first loss function L jacc , the second loss function L topo2 , the third loss function L topo3 , the fourth loss function L jacc2 Based on this, the classifier 61 is further trained. Specifically, the feedback value L total ( =L jacc +λ·L topo3 +λj·L jacc2 +λ·L topo2 Retrain using ). Furthermore, the similar target image 72 in Example 2 is not strictly accurate training data, but rather, so to speak, "somewhat inaccurate training data." Therefore, in Example 2, the second Jackard coefficient L jacc2 Instead of directly feeding back the result, the feedback coefficient λj (<1) is multiplied before the feedback is applied. In Example 2, the feedback coefficient λj is set to λj = 0.3 as an example, but the specific value can be changed as appropriate depending on the design and specifications.

[0055] (Effect of Example 2) In the radiotherapy machine 1 of Embodiment 2 having the above configuration, the second Jackard coefficient L is also applied to the X-ray fluoroscopic image 20. jacc2Feedback is provided using this method. In other words, in Example 2, for an X-ray fluoroscopic image 20 without a training image, a similar DRR image 71 is searched for, and a similar target image 72 associated with it is used. This allows the system to learn in a way that is somewhat close to having a training image (having a ground truth image) (with slightly inaccurate training data). Therefore, it is expected that the shape of the tumor's external form can be estimated with even greater accuracy compared to Example 1. [Examples]

[0056] Next, we will describe Embodiment 3 of the present invention. In this description of Embodiment 3, the same reference numerals are used for components corresponding to the components of Embodiment 1, and their detailed descriptions are omitted. This embodiment differs from Embodiment 1 in the following respects, but is otherwise configured in the same manner as Embodiment 1.

[0057] In Example 1, the loss function is the topology coefficient L topo2 , L topo3 While the number of regions and holes were taken into account using the previous method, in Example 3, a loss function based on the target area, aspect ratio, and diagonal length is also used. Area coefficient L is the loss function based on area. area The target area S area And a predetermined assumed value S true1 Based on this, the target area S area The expected value is S true1 The loss is calculated based on the following formula (5), such that the greater the difference from the given value, the greater the loss. L area =(1 / nb)Σ nb i=1 (S area -S true1 ) 2 …Equation (5)

[0058] Similarly, the aspect ratio coefficient L is a loss function based on the aspect ratio. asp The aspect ratio S of the target asp And a predetermined assumed value S true2 Based on this, the aspect ratio S of the target asp The expected value is S true2It is calculated based on the following formula (6) such that the greater the difference from L asp =(1 / nb)Σ nb i=1 (S asp -S true2 ) 2 … Formula (6) Also, the diagonal coefficient L, which is a loss function based on the diagonal length, diag is calculated based on the following formula (7) such that the greater the difference between the target diagonal length S diag and a predetermined assumed value S true3 , the greater the loss. diag L true3 =(1 / nb)Σ L diag =(1 / nb)Σ nb i=1 (S diag -S true3 ) 2 … Formula (7)

[0059] In Example 3, for the shape of the target outer contour 62 based on the DRR image 23, the topology coefficients L topo2 , L area1 , L asp1 [[ID=~]] diag1 , L diag1 are derived, and for the shape of the target outer contour 63 based on the fluoroscopic X-ray image 20, the topology coefficients L topo3 , L area2 , L asp2 , L diag2 are derived. Then, the similarity L A (=L jacc +λ·L topo2 +λ ar ·L area1 +λ as ·L asp1 +λ d ·L diag1 ) and the similarity L B (=λ·L topo3 +λ ar ·L area2 +λ as ·L asp2 +λ d ·L diag2 ) are derived. Here, λ ar is the area coefficient L areaThis is the feedback coefficient for λ. as The aspect ratio coefficient L asp This is the feedback coefficient for λ. d L is the diagonal length coefficient. diag This is the feedback coefficient for the similarity L. B Regarding the second Jackard coefficient L of Example 2, jacc2 It is also possible to apply this.

[0060] (Effect of Example 3) In the radiotherapy machine 1 of Embodiment 3 having the above configuration, the area S of the shape of the tumor target 62,63 is estimated using the classifier 61. area However, the assumed area S true1 If it is extremely large or extremely small, the area coefficient L area A penalty is applied to the calculation, making it more likely that the estimated result will be incorrect. Furthermore, the aspect ratio S of the shape of the tumor target outline 62,63 estimated using the classifier 61 asp However, the assumed aspect ratio S true2 If it is extremely large or extremely small, the aspect ratio coefficient L asp A penalty is applied to the calculation, making it more likely that the estimated result will be incorrect. Furthermore, the diagonal length S of the shape of the tumor target outline 62,63 estimated using the discriminator 61 diag However, the assumed aspect ratio S true3 If it is extremely large or extremely small, the diagonal coefficient L diag A penalty is applied to the calculation, making it more likely that the estimated result will be incorrect. Therefore, in Example 3, it is expected that the shape of the tumor's external form can be estimated with even greater accuracy compared to Example 1. [Examples]

[0061] Next, we will describe Embodiment 4 of the present invention. In this description of Embodiment 4, the same reference numerals are used for components corresponding to the components of Embodiment 1, and their detailed descriptions are omitted. This embodiment differs from Embodiment 1 in the following respects, but is otherwise configured in the same manner as Embodiment 1.

[0062] Figure 7 is an explanatory diagram of Example 4, and corresponds to Figure 3 of Example 1. In Figure 7, the radiotherapy machine 1 of Example 4 has a generator 81 that generates the outline 62′ of the target from a DRR image 23 (an example of the first image in Example 4) and generates the outline 63′ of the target from an X-ray fluoroscopy image 20 (an example of the third image in Example 4), instead of the classifier 61 of Example 1. The generator 81 of Example 4 generates images 62′ and 63′ by adding random numbers and noise to the input images 20 and 23. Therefore, the images 62′ and 63′ generated by the generator 81 are so-called false images (images that are presumed to be false).

[0063] Furthermore, the radiotherapy machine 1 of Example 4 has a discriminator 82. The discriminator 82 of Example 4 outputs a discrimination result (true or false identification result) from an image (fake image) 62′ generated from the DRR image 23 by the generator 81 and a training image (true image) 30, or from an image (fake image) 63′ generated from the X-ray fluoroscopy image 20 by the generator 81. A loss function L derived based on the training image 30 and image 62′, and image 63′. jacc , L D From this, the generator 81 is trained. From the similarity of the target outline between the training image 30 and image 62', L jacc The L value is calculated from the outline of the target in images 62' and 63'. D This is calculated.

[0064] Therefore, in Example 1, a Generative Adversarial Network (GAN) using a generator 81 and a discriminator 82 is used as the discriminator 81+82. Note that although two discriminators 82 are shown in Figure 7, this is only to illustrate the flow of information processing; in reality, there is only one discriminator 82. In the radiotherapy machine 1 of Example 4, similar shapes need to be output during both training and inference, so the discriminator 82 is used to determine whether the images are similar during learning.

[0065] (Effect of Example 4) In the radiotherapy machine 1 of Embodiment 4, which has the above configuration, a generator 81 and a discriminator 82 are used, and learning can be deepened by repeatedly generating a large number of false images with the generator 81 and discriminating their truthfulness with the discriminator 82. Therefore, accuracy is also improved in Embodiment 4. Furthermore, the loss function is not limited to one based on topology calculations; it can be represented as a classifier using a neural network (CNN61, generator81, discriminator82) by taking advantage of the fact that the target outline of the same patient is almost the same regardless of the imaging device.

[0066] (Example of change) Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and various modifications can be made within the scope of the gist of the present invention as described in the claims. Examples of modifications to the present invention (H01) to (H03) are shown below. (H01) In the above embodiment, a configuration using X-ray fluoroscopy images with kV-order X-rays as the captured image was illustrated, but the invention is not limited thereto. For example, it is also possible to use X-ray images with MV-order X-rays (images detected after therapeutic MV-X-rays have passed through the patient), ultrasound images (so-called ultrasound echo images), MRI images (Magnetic Resonance Imaging), PET images (Positron Emission tomography), photoacoustic imaging (PAI) images, X-ray backscatter images, etc. It is also possible to make modifications such as combining X-ray images (kV-X-rays or MV-X-rays) with MRI images, etc. Additionally, by taking advantage of the strong correlation between movements such as respiration and tumor movement, it is possible to capture the patient's body surface shape, which changes with respiration, using imaging means such as 3D cameras and distance sensors, and then estimate the location and shape of the tumor using the body surface shape images.

[0067] (H02) In the above embodiment, a configuration for tracking without using markers was illustrated, but the invention is not limited thereto. It is also possible to use X-ray images with embedded markers, etc. (H03) In the above embodiment, the similarity L of the shape of the target outline 62 derived from the DRR image 23 and the classifier 61 A In deriving the third loss function L topo3 The configurations shown as examples are not limited to those, but depending on the required precision, a third loss function L may be used. topo3 It is also possible to configure the system without using it. [Explanation of Symbols]

[0068] 1...Treatment device, 2... Subject, 20...Third image, 23...First image, 30...Second image, 61, 81 + 82… Discriminator, 62, 63, 62′, 63′… Target outline, estimated target image, 71... Similar images, 72... Similar target images, 81...generator, 82... Discriminant, C1... First method of filming, C3... Second method of filming, C4...Target image storage means, C5...Learning methods, C6A... First derivation means, C6B... Third derivation means, C6C... Second derivation means, C10...External shape estimation means, C11...irradiation means, L jacc ...the first loss function, L jacc ,Ljacc2 ,L topo2 ,L topo3 ,L area1 ,L area2 ,L asp1 ,L asp2 ,L diag1 ,L diag2 ...loss function, L jacc2 ...the fourth loss function, L topo2 ,L area1 ,L asp1 ,L diag1 ...the second loss function, L topo3 ,L area2 ,L asp2 ,L diag2 ...the third loss function, S area ...the area of ​​the target region, S asp ...the aspect ratio of the target area, S diag ...the diagonal length of the target area, S true1 ,S true2 ,S true3 ...a predetermined value, β0…Number of target regions, β1…Number of pores inside.

Claims

1. Based on a first image in which the subject's internal information is captured, a second image showing a target inside the body, and a third image taken of the same subject using a different device than the first image, which includes the target and the subject's internal information, A classifier that has been trained to take either the first image or the third image as input and output the outline of a target, and has been further trained by feeding back a loss function that identifies the geometric similarity of the outline of the target from the outline of the target output with the first image as input and the outline of the target in the second image, A target shape estimation device characterized by estimating the external shape of a target.

2. A third image taken before or immediately before the treatment date is used. The target shape estimation device according to feature 1.

3. In the shape of the target derived using the third image and the classifier, the loss function derived based on the number of regions and the number of holes inside the target, The target shape estimation device according to claim 1, characterized by comprising the above.

4. In the shape of the target derived using the first image and the classifier, the loss function derived based on the number of regions and the number of holes inside the target, The target shape estimation device according to claim 1, characterized by comprising the above.

5. A similar image to the third image is obtained from a database in which the first image is stored, and the loss function is based on the similarity between the target outline derived using the classifier and the similar image, and a similar target image in which the target outline in the similar image is identified. The target shape estimation device according to claim 1, characterized by comprising the above.

6. A classifier comprising: a generator that generates information about the outline of a target from the first image or the third image; a discriminator that discriminates based on the outline of a target generated by the generator based on the first image, and also discriminates between the outline of a pre-identified target and the outline of a target generated by the generator based on the third image, wherein the classifier learns based on the discrimination results of the discriminator, The target shape estimation device according to claim 1, characterized by comprising the above.

7. The classifier into which the first image or the third image is directly input, The target shape estimation device according to claim 1, characterized by comprising the above.

8. If the area of ​​the target region differs from a predetermined value, the loss function will have a larger loss. A target shape estimation device according to claim 6 or 7, characterized by comprising the above.

9. If the aspect ratio of the target region differs from a predetermined value, the loss function will have a larger loss. A target shape estimation device according to claim 6 or 7, characterized by comprising the above.

10. If the length of the diagonal of the target region differs from a predetermined value, the loss function will have a larger loss. A target shape estimation device according to claim 6 or 7, characterized by comprising the above.

11. Each of the aforementioned images is one or a combination of the following: X-ray image, nuclear magnetic resonance image, ultrasound image, positron emission tomography image, body surface topography image, and photoacoustic imaging image. The target shape estimation device according to feature 1.

12. A target shape estimation device according to claim 1, Based on the target shape estimated by the target shape estimation device, an irradiation means for irradiating the target with therapeutic radiation, A therapeutic device characterized by being equipped with [a specific feature].

Images